2024-09-11T00:18:59.771 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T00:18:59.796 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T00:18:59.892 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843 branch: wip-hemanth-testing-2024-09-10-0723 description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} email: hyelloji@redhat.com first_in_suite: false job_id: '7899843' kernel: flavor: default kdb: 1 sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-hemanth-testing-2024-09-10-0723 ceph: cephfs: fs: - name: a - name: b session_timeout: 300 cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: client: client_shutdown_timeout: 120 mds: mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 mgr: client mount timeout: 30 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Replacing daemon mds - SLOW_OPS - slow request sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: cephfs_name: a client.1: cephfs_name: b install: ceph: flavor: default sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 kclient: client.0: cephfs_name: a client.1: cephfs_name: b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-hemanth-testing-2024-09-10-0723 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 owner: scheduled_hyelloji@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 6190 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 sleep_before_teardown: 0 subset: 479/1024 suite: fs suite_branch: wip-hemanth-testing-2024-09-10-0723 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBprUoQScclASkFC6iEkUBG1EHV0lJpvsJlZKHhC+hkgTQsfg023d0MAvkMHcYPRBgZwQuqknu1gJtqMES3xYK0= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIu3hLTpBMeScGb/BDZb55wXviStnce68tS81q0/caQp/E9i13Q3CTmPYTI8tniSmGlJiTTfiHDg9trG2roRmDk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_16:59:47 tube: smithi user: hyelloji verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T00:18:59.892 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa; will attempt to use it 2024-09-11T00:18:59.893 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks 2024-09-11T00:18:59.893 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T00:18:59.895 INFO:teuthology.task.internal:Checking packages... 2024-09-11T00:18:59.916 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8f219c476cb4f26fc5719c661c0323719a45cb30' 2024-09-11T00:18:59.916 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T00:18:59.916 INFO:teuthology.packaging:ref: None 2024-09-11T00:18:59.916 INFO:teuthology.packaging:tag: None 2024-09-11T00:18:59.916 INFO:teuthology.packaging:branch: wip-hemanth-testing-2024-09-10-0723 2024-09-11T00:18:59.917 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T00:18:59.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-hemanth-testing-2024-09-10-0723 2024-09-11T00:19:00.071 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4854.g8f219c47 2024-09-11T00:19:00.073 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T00:19:00.078 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T00:19:00.078 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T00:19:00.097 INFO:teuthology.task.internal:Saving configuration 2024-09-11T00:19:00.115 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T00:19:00.121 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T00:19:00.151 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 00:13:39.885220', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBprUoQScclASkFC6iEkUBG1EHV0lJpvsJlZKHhC+hkgTQsfg023d0MAvkMHcYPRBgZwQuqknu1gJtqMES3xYK0='} 2024-09-11T00:19:00.180 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 00:13:39.886547', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIu3hLTpBMeScGb/BDZb55wXviStnce68tS81q0/caQp/E9i13Q3CTmPYTI8tniSmGlJiTTfiHDg9trG2roRmDk='} 2024-09-11T00:19:00.180 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T00:19:00.186 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-11T00:19:00.186 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-09-11T00:19:00.186 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T00:19:00.293 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efc94db7be0>, signals=[15]) 2024-09-11T00:19:00.293 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T00:19:00.298 INFO:teuthology.task.internal:Opening connections... 2024-09-11T00:19:00.298 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:19:00.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:19:00.406 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:19:00.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:19:00.492 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T00:19:00.498 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-09-11T00:19:00.516 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-09-11T00:19:00.517 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-09-11T00:19:00.574 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-09-11T00:19:00.574 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-09-11T00:19:00.574 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-09-11T00:19:00.574 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-09-11T00:19:00.574 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-09-11T00:19:00.574 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-09-11T00:19:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T00:19:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-09-11T00:19:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-09-11T00:19:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T00:19:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-09-11T00:19:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T00:19:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T00:19:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T00:19:00.576 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-09-11T00:19:00.603 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-09-11T00:19:00.622 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-09-11T00:19:00.622 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-09-11T00:19:00.681 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-09-11T00:19:00.681 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-09-11T00:19:00.681 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T00:19:00.682 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T00:19:00.683 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-09-11T00:19:00.711 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T00:19:00.720 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T00:19:00.724 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T00:19:00.725 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-09-11T00:19:00.727 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-09-11T00:19:00.744 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T00:19:00.750 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T00:19:00.750 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-09-11T00:19:00.784 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-09-11T00:19:00.881 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T00:19:00.952 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T00:19:00.967 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': 1, 'sha1': '2ca44a2f35ca8f36983aaaa46d49312239ef519a'} 2024-09-11T00:19:00.967 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': 1, 'sha1': '2ca44a2f35ca8f36983aaaa46d49312239ef519a'}, 'mon.b': {'flavor': 'default', 'kdb': 1, 'sha1': '2ca44a2f35ca8f36983aaaa46d49312239ef519a'}}, timeout 300 2024-09-11T00:19:00.968 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T00:19:00.968 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T00:19:00.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T00:19:00.985 DEBUG:teuthology.task.kernel:sha1 for mon.a is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-11T00:19:00.985 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-11T00:19:00.986 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-09-11T00:19:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T00:19:00.986 DEBUG:teuthology.task.kernel:sha1 for mon.b is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-11T00:19:00.987 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-11T00:19:00.987 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-09-11T00:19:01.043 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-503.el9.x86_64 2024-09-11T00:19:01.043 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-11T00:19:01.043 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-11T00:19:01.043 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-11T00:19:01.055 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-503.el9.x86_64 2024-09-11T00:19:01.055 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-11T00:19:01.055 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-11T00:19:01.055 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-11T00:19:01.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum upgrade -y linux-firmware 2024-09-11T00:19:01.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum upgrade -y linux-firmware 2024-09-11T00:19:03.441 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-11T00:19:04.850 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-11T00:19:05.592 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-09-11T00:19:11.184 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.5 MB 00:01 2024-09-11T00:19:11.292 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 20 MB 00:05 2024-09-11T00:19:13.101 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 19 kB 00:00 2024-09-11T00:19:14.022 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-11T00:19:15.659 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-11T00:19:17.135 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-11T00:19:17.989 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-11T00:19:19.635 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-11T00:19:21.861 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T00:19:21.862 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-09-11T00:19:21.862 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T00:19:21.977 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on mon.a... 2024-09-11T00:19:21.977 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-11T00:19:22.136 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-11T00:19:22.136 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-11T00:19:22.139 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi031.front.sepia.ceph.com to complete... 2024-09-11T00:19:23.428 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-11T00:19:25.506 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-11T00:19:25.507 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-11T00:19:25.507 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-11T00:19:25.612 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on mon.b... 2024-09-11T00:19:25.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-11T00:19:25.768 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-11T00:19:25.769 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-11T00:19:25.771 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi204.front.sepia.ceph.com to complete... 2024-09-11T00:19:29.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11 00:19:29 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-11T00:19:29.914 INFO:teuthology.orchestra.run.smithi031.stderr:FINISHED --2024-09-11 00:19:29-- 2024-09-11T00:19:29.914 INFO:teuthology.orchestra.run.smithi031.stderr:Total wall clock time: 7.7s 2024-09-11T00:19:29.914 INFO:teuthology.orchestra.run.smithi031.stderr:Downloaded: 1 files, 343M in 7.5s (45.9 MB/s) 2024-09-11T00:19:29.917 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on mon.a... 2024-09-11T00:19:29.917 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-11T00:19:32.111 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying... ######################################## 2024-09-11T00:19:32.251 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing... ######################################## 2024-09-11T00:19:32.351 INFO:teuthology.orchestra.run.smithi031.stdout:Updating / installing... 2024-09-11T00:19:35.143 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11 00:19:35 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-11T00:19:35.143 INFO:teuthology.orchestra.run.smithi204.stderr:FINISHED --2024-09-11 00:19:35-- 2024-09-11T00:19:35.143 INFO:teuthology.orchestra.run.smithi204.stderr:Total wall clock time: 9.3s 2024-09-11T00:19:35.143 INFO:teuthology.orchestra.run.smithi204.stderr:Downloaded: 1 files, 343M in 8.9s (38.4 MB/s) 2024-09-11T00:19:35.146 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on mon.b... 2024-09-11T00:19:35.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-11T00:19:37.309 INFO:teuthology.orchestra.run.smithi204.stdout:Verifying... ######################################## 2024-09-11T00:19:37.462 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing... ######################################## 2024-09-11T00:19:37.579 INFO:teuthology.orchestra.run.smithi204.stdout:Updating / installing... 2024-09-11T00:19:40.647 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-11T00:19:45.640 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-11T00:20:20.534 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-11T00:20:20.534 DEBUG:teuthology.orchestra.run.smithi031:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-11T00:20:20.610 INFO:teuthology.orchestra.run.smithi031.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:20.610 INFO:teuthology.orchestra.run.smithi031.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:20.610 INFO:teuthology.orchestra.run.smithi031.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:20.610 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:20.610 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-11T00:20:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-11T00:20:20.612 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-11T00:20:20.613 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-11T00:20:20.614 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-11T00:20:20.615 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-11T00:20:20.616 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-11T00:20:20.617 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-11T00:20:20.618 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-11T00:20:20.619 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-11T00:20:20.620 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-11T00:20:20.621 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-11T00:20:20.622 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-11T00:20:20.623 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-11T00:20:20.624 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-11T00:20:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-11T00:20:20.626 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-11T00:20:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-11T00:20:20.628 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-11T00:20:20.629 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-11T00:20:20.630 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-11T00:20:20.631 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-11T00:20:20.632 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-11T00:20:20.633 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-11T00:20:20.634 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-11T00:20:20.635 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-11T00:20:20.636 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-11T00:20:20.637 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-11T00:20:20.638 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-11T00:20:20.639 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-11T00:20:20.640 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-11T00:20:20.641 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-11T00:20:20.642 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-11T00:20:20.643 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-11T00:20:20.644 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-11T00:20:20.645 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-11T00:20:20.646 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-11T00:20:20.647 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-11T00:20:20.648 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-11T00:20:20.649 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-11T00:20:20.650 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-11T00:20:20.651 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-11T00:20:20.652 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-11T00:20:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-11T00:20:20.654 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-11T00:20:20.655 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-11T00:20:20.656 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-11T00:20:20.657 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-11T00:20:20.658 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-11T00:20:20.659 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-11T00:20:20.660 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-11T00:20:20.661 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-11T00:20:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-11T00:20:20.663 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-11T00:20:20.664 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-11T00:20:20.665 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-11T00:20:20.666 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-11T00:20:20.667 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-11T00:20:20.668 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-11T00:20:20.669 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-11T00:20:20.670 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-11T00:20:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-11T00:20:20.672 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-11T00:20:20.673 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-11T00:20:20.674 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-11T00:20:20.675 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-11T00:20:20.676 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-11T00:20:20.677 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-11T00:20:20.678 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-11T00:20:20.679 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-11T00:20:20.680 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-11T00:20:20.681 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-11T00:20:20.682 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-11T00:20:20.683 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-11T00:20:20.684 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-11T00:20:20.685 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-11T00:20:20.686 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-11T00:20:20.687 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-11T00:20:20.688 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-11T00:20:20.689 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-11T00:20:20.690 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-11T00:20:20.691 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-11T00:20:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-11T00:20:20.693 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-11T00:20:20.694 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-11T00:20:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-11T00:20:20.696 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-11T00:20:20.697 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-11T00:20:20.698 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-11T00:20:20.699 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-11T00:20:20.700 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-11T00:20:20.701 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-11T00:20:20.702 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-11T00:20:20.703 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-11T00:20:20.704 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-11T00:20:20.705 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-11T00:20:20.706 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-11T00:20:20.707 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-11T00:20:20.708 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-11T00:20:20.709 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-11T00:20:20.710 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-11T00:20:20.711 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-11T00:20:20.712 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-11T00:20:20.713 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-11T00:20:20.714 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-11T00:20:20.715 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-11T00:20:20.716 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-11T00:20:20.717 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-11T00:20:20.718 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-11T00:20:20.719 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-11T00:20:20.720 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-11T00:20:20.721 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-11T00:20:20.722 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-11T00:20:20.723 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-11T00:20:20.724 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-11T00:20:20.725 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-11T00:20:20.726 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-11T00:20:20.727 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-11T00:20:20.728 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-11T00:20:20.729 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-11T00:20:20.730 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-11T00:20:20.731 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-11T00:20:20.732 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-11T00:20:20.733 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-11T00:20:20.734 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-11T00:20:20.735 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-11T00:20:20.736 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-11T00:20:20.737 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-11T00:20:20.738 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-11T00:20:20.739 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-11T00:20:20.740 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-11T00:20:20.741 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-11T00:20:20.742 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-11T00:20:20.743 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-11T00:20:20.744 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-11T00:20:20.745 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-11T00:20:20.746 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-11T00:20:20.747 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-11T00:20:20.748 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-11T00:20:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-11T00:20:20.750 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-11T00:20:20.751 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-11T00:20:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-11T00:20:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-11T00:20:20.754 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-11T00:20:20.755 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-11T00:20:20.756 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-11T00:20:20.756 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-11T00:20:20.756 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-11T00:20:20.756 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-11T00:20:20.760 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:20.760 DEBUG:teuthology.orchestra.run.smithi031:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-11T00:20:20.783 INFO:teuthology.orchestra.run.smithi031.stdout:postinstall scriptlet (using /bin/sh): 2024-09-11T00:20:20.783 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-11T00:20:20.783 INFO:teuthology.orchestra.run.smithi031.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout:for file in vmlinuz System.map config; do 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout: fi 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout:done 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-11T00:20:20.784 INFO:teuthology.orchestra.run.smithi031.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:20.785 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-09-11T00:20:20.785 INFO:teuthology.orchestra.run.smithi031.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-11T00:20:20.785 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-11T00:20:20.785 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:20.785 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2024-09-11T00:20:20.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2024-09-11T00:20:20.864 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2024-09-11T00:20:20.864 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2024-09-11T00:20:20.864 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2024-09-11T00:20:20.864 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 82.el9 2024-09-11T00:20:20.864 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2024-09-11T00:20:20.864 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T00:20:20.864 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2024-09-11T00:20:20.864 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8274670 2024-09-11T00:20:20.864 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2024-09-11T00:20:20.864 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T00:20:20.865 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2024-09-11T00:20:20.866 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:20:20.866 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T00:20:20.866 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T00:20:20.867 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:20.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T00:20:21.519 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2024-09-11T00:20:22.936 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T00:20:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:done 2024-09-11T00:20:22.960 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-11T00:20:23.015 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.HICEMY7CC6 2024-09-11T00:20:23.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HICEMY7CC6 2024-09-11T00:20:23.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.HICEMY7CC6 2024-09-11T00:20:23.227 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.HICEMY7CC6 is 6KB 2024-09-11T00:20:23.276 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.HICEMY7CC6 2024-09-11T00:20:23.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T00:20:23.353 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T00:20:23.353 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T00:20:23.353 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T00:20:23.353 INFO:teuthology.orchestra.run.smithi031.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-11T00:20:23.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:23.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2024-09-11T00:20:25.768 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-11T00:20:25.769 DEBUG:teuthology.orchestra.run.smithi204:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-11T00:20:25.861 INFO:teuthology.orchestra.run.smithi204.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:25.861 INFO:teuthology.orchestra.run.smithi204.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:25.861 INFO:teuthology.orchestra.run.smithi204.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:25.861 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:25.861 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-11T00:20:25.861 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-11T00:20:25.861 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-11T00:20:25.862 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-11T00:20:25.863 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-11T00:20:25.864 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-11T00:20:25.864 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-11T00:20:25.864 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-11T00:20:25.864 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-11T00:20:25.864 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-11T00:20:25.864 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-11T00:20:25.864 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-11T00:20:25.864 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-11T00:20:25.865 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-11T00:20:25.866 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-11T00:20:25.867 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-11T00:20:25.868 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-11T00:20:25.869 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-11T00:20:25.870 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-11T00:20:25.871 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-11T00:20:25.872 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-11T00:20:25.873 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-11T00:20:25.874 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-11T00:20:25.875 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-11T00:20:25.876 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-11T00:20:25.877 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-11T00:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-11T00:20:25.879 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-11T00:20:25.880 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-11T00:20:25.881 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-11T00:20:25.882 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-11T00:20:25.883 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-11T00:20:25.884 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-11T00:20:25.885 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-11T00:20:25.886 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-11T00:20:25.887 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-11T00:20:25.888 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-11T00:20:25.889 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-11T00:20:25.890 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-11T00:20:25.891 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-11T00:20:25.892 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-11T00:20:25.893 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-11T00:20:25.894 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-11T00:20:25.895 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-11T00:20:25.896 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-11T00:20:25.897 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-11T00:20:25.898 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-11T00:20:25.899 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-11T00:20:25.900 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-11T00:20:25.901 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-11T00:20:25.902 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-11T00:20:25.903 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-11T00:20:25.904 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-11T00:20:25.905 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-11T00:20:25.906 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-11T00:20:25.907 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-11T00:20:25.908 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-11T00:20:25.909 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-11T00:20:25.910 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-11T00:20:25.911 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-11T00:20:25.912 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-11T00:20:25.913 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-11T00:20:25.914 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-11T00:20:25.915 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-11T00:20:25.916 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-11T00:20:25.917 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-11T00:20:25.918 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-11T00:20:25.919 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-11T00:20:25.920 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-11T00:20:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-11T00:20:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-11T00:20:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-11T00:20:25.924 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-11T00:20:25.925 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-11T00:20:25.926 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-11T00:20:25.927 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-11T00:20:25.928 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-11T00:20:25.929 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-11T00:20:25.930 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-11T00:20:25.931 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-11T00:20:25.932 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-11T00:20:25.933 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-11T00:20:25.934 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-11T00:20:25.935 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-11T00:20:25.936 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-11T00:20:25.937 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-11T00:20:25.938 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-11T00:20:25.939 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-11T00:20:25.940 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-11T00:20:25.941 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-11T00:20:25.942 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-11T00:20:25.943 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-11T00:20:25.944 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-11T00:20:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-11T00:20:25.946 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-11T00:20:25.947 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-11T00:20:25.948 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-11T00:20:25.949 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-11T00:20:25.950 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-11T00:20:25.951 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-11T00:20:25.952 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-11T00:20:25.953 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-11T00:20:25.954 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-11T00:20:25.955 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-11T00:20:25.956 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-11T00:20:25.957 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-11T00:20:25.958 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-11T00:20:25.959 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-11T00:20:25.960 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-11T00:20:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-11T00:20:25.962 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-11T00:20:25.963 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-11T00:20:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-11T00:20:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-11T00:20:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-11T00:20:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-11T00:20:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-11T00:20:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-11T00:20:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-11T00:20:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-11T00:20:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-11T00:20:25.965 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-11T00:20:25.966 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-11T00:20:25.967 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-11T00:20:25.968 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-11T00:20:25.969 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-11T00:20:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-11T00:20:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-11T00:20:25.972 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-11T00:20:25.973 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-11T00:20:25.974 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-11T00:20:25.975 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-11T00:20:25.976 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-11T00:20:25.977 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-11T00:20:25.978 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-11T00:20:25.979 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-11T00:20:25.980 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-11T00:20:25.981 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-11T00:20:25.982 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-11T00:20:25.983 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-11T00:20:25.984 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-11T00:20:25.985 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-11T00:20:25.986 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-11T00:20:25.987 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-11T00:20:25.988 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-11T00:20:25.988 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-11T00:20:25.988 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-11T00:20:25.988 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-11T00:20:25.988 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-11T00:20:25.988 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-11T00:20:25.988 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-11T00:20:25.988 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-11T00:20:25.988 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-11T00:20:25.988 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-11T00:20:25.989 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-11T00:20:25.990 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-11T00:20:25.991 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-11T00:20:25.992 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-11T00:20:25.993 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-11T00:20:25.994 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-11T00:20:25.995 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-11T00:20:25.996 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-11T00:20:25.997 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-11T00:20:25.998 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-11T00:20:25.999 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-11T00:20:26.000 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-11T00:20:26.001 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-11T00:20:26.002 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-11T00:20:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-11T00:20:26.004 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-11T00:20:26.005 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-11T00:20:26.006 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-11T00:20:26.007 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-11T00:20:26.008 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-11T00:20:26.009 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-11T00:20:26.010 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-11T00:20:26.011 INFO:teuthology.orchestra.run.smithi204.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-11T00:20:26.015 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:26.015 DEBUG:teuthology.orchestra.run.smithi204:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-11T00:20:26.038 INFO:teuthology.orchestra.run.smithi204.stdout:postinstall scriptlet (using /bin/sh): 2024-09-11T00:20:26.038 INFO:teuthology.orchestra.run.smithi204.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-11T00:20:26.038 INFO:teuthology.orchestra.run.smithi204.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-11T00:20:26.038 INFO:teuthology.orchestra.run.smithi204.stdout:fi 2024-09-11T00:20:26.038 INFO:teuthology.orchestra.run.smithi204.stdout:for file in vmlinuz System.map config; do 2024-09-11T00:20:26.038 INFO:teuthology.orchestra.run.smithi204.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-11T00:20:26.038 INFO:teuthology.orchestra.run.smithi204.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-11T00:20:26.038 INFO:teuthology.orchestra.run.smithi204.stdout: fi 2024-09-11T00:20:26.038 INFO:teuthology.orchestra.run.smithi204.stdout:done 2024-09-11T00:20:26.038 INFO:teuthology.orchestra.run.smithi204.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout:fi 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout:fi 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:26.039 INFO:teuthology.orchestra.run.smithi204.stdout:fi 2024-09-11T00:20:26.040 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-09-11T00:20:26.113 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-09-11T00:20:26.113 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-09-11T00:20:26.113 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.06 2024-09-11T00:20:26.113 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 82.el9 2024-09-11T00:20:26.113 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-09-11T00:20:26.113 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T00:20:26.113 INFO:teuthology.orchestra.run.smithi204.stdout:Group : Unspecified 2024-09-11T00:20:26.113 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 8274670 2024-09-11T00:20:26.113 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : builder@centos.org 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T00:20:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-09-11T00:20:26.115 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:20:26.115 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T00:20:26.116 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T00:20:26.116 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:26.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T00:20:26.664 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-09-11T00:20:27.965 INFO:teuthology.orchestra.run.smithi204.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T00:20:28.004 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-09-11T00:20:28.006 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-09-11T00:20:28.021 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.zBR2frO6Rk 2024-09-11T00:20:28.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zBR2frO6Rk 2024-09-11T00:20:28.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.zBR2frO6Rk 2024-09-11T00:20:28.239 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.zBR2frO6Rk is 6KB 2024-09-11T00:20:28.286 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.zBR2frO6Rk 2024-09-11T00:20:28.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T00:20:28.363 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T00:20:28.364 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T00:20:28.364 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T00:20:28.364 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-11T00:20:28.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:20:28.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-09-11T00:20:53.528 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T00:20:53.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T00:20:53.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:20:58.521 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-11T00:20:58.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-11T00:20:58.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:21:11.869 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-09-11T00:21:16.989 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-11T00:21:20.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T00:21:20.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:21:25.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-11T00:21:25.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:21:29.085 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-11T00:21:39.261 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-09-11T00:21:41.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-11T00:21:41.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:21:51.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T00:21:51.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:21:54.369 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2024-09-11T00:22:09.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T00:22:09.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:22:09.828 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-09-11T00:22:10.594 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T00:22:10.594 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-11T00:22:10.594 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-09-11T00:22:10.679 INFO:teuthology.orchestra.run.smithi031.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:22:10.679 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:22:10.679 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T00:22:10.680 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T00:22:11.680 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T00:22:11.681 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T00:22:11.787 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2024-09-11T00:22:11.827 DEBUG:teuthology.parallel:result is None 2024-09-11T00:22:41.099 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T00:22:56.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-11T00:22:56.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:22:56.592 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-09-11T00:22:57.118 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-11T00:22:57.118 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-11T00:22:57.118 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-09-11T00:22:57.152 INFO:teuthology.orchestra.run.smithi204.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:22:57.153 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-11T00:22:57.153 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T00:22:57.153 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T00:22:58.153 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-11T00:22:58.154 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T00:22:58.264 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-09-11T00:22:58.300 DEBUG:teuthology.parallel:result is None 2024-09-11T00:22:58.300 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T00:22:58.307 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T00:22:58.307 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T00:22:58.312 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T00:22:58.348 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T00:22:58.354 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T00:22:58.364 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T00:22:58.364 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T00:22:58.391 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T00:22:58.436 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T00:22:58.443 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T00:22:58.443 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T00:22:58.474 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T00:22:58.534 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T00:22:58.538 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T00:22:58.556 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T00:22:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T00:22:58.564 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T00:22:58.571 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T00:22:58.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T00:22:58.575 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T00:22:58.629 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T00:22:58.638 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T00:22:58.639 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T00:22:58.642 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T00:22:58.676 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T00:22:58.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T00:22:58.872 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T00:22:58.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T00:22:59.004 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:22:59.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T00:22:59.059 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T00:22:59.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T00:22:59.220 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T00:22:59.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T00:22:59.354 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:22:59.354 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T00:22:59.409 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-09-11T00:22:59.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-09-11T00:22:59.499 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T00:22:59.538 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T00:23:00.047 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T00:23:00.054 INFO:teuthology.task.internal:Starting timer... 2024-09-11T00:23:00.055 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T00:23:00.064 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T00:23:00.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-09-11T00:23:00.188 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-09-11T00:23:00.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-09-11T00:23:00.286 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-09-11T00:23:00.289 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T00:23:00.290 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-09-11T00:23:00.331 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-09-11T00:23:00.331 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-09-11T00:23:00.377 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-09-11T00:23:00.377 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-09-11T00:23:00.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T00:23:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T00:23:00.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T00:23:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T00:23:00.494 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T00:23:00.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-09-11T00:23:00.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-09-11T00:23:00.621 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T00:23:00.631 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T00:23:00.634 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T00:23:00.802 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T00:23:00.813 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T00:23:00.814 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T00:27:40.927 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-09-11T00:27:40.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T00:27:40.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:27:41.026 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-09-11T00:27:41.139 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-09-11T00:27:41.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-11T00:27:41.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T00:27:41.237 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-09-11T00:27:41.354 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-11T00:27:41.354 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T00:27:41.364 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T00:27:41.365 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T00:27:41.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T00:27:41.369 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T00:27:41.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T00:27:41.427 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T00:27:41.455 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T00:27:41.457 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T00:27:41.485 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T00:27:41.508 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-09-11T00:27:41.532 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-09-11T00:27:41.536 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-09-11T00:27:41.561 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-09-11T00:27:41.561 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T00:27:41.590 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T00:27:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T00:27:41.620 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T00:27:41.686 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-09-11T00:27:41.693 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T00:27:41.693 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-09-11T00:27:41.693 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T00:27:41.693 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T00:27:41.693 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T00:27:41.693 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T00:27:41.723 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-09-11T00:27:41.730 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T00:27:41.730 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-09-11T00:27:41.730 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T00:27:41.730 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T00:27:41.730 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T00:27:41.730 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T00:27:41.731 INFO:teuthology.run_tasks:Running task install... 2024-09-11T00:27:41.739 DEBUG:teuthology.task.install:project ceph 2024-09-11T00:27:41.739 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-11T00:27:41.739 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'} 2024-09-11T00:27:41.740 INFO:teuthology.task.install:Using flavor: default 2024-09-11T00:27:41.745 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T00:27:41.745 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-11T00:27:41.745 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'tag': None, 'wait_for_package': False} 2024-09-11T00:27:41.746 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T00:27:41.748 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'tag': None, 'wait_for_package': False} 2024-09-11T00:27:41.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T00:27:41.903 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/ 2024-09-11T00:27:41.903 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4854.g8f219c47 2024-09-11T00:27:41.918 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/ 2024-09-11T00:27:41.918 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4854.g8f219c47 2024-09-11T00:27:42.028 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T00:27:42.028 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:27:42.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T00:27:42.039 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T00:27:42.039 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:27:42.039 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T00:27:42.085 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-11T00:27:42.085 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8f219c476cb4f26fc5719c661c0323719a45cb30/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T00:27:42.096 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-11T00:27:42.097 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8f219c476cb4f26fc5719c661c0323719a45cb30/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T00:27:42.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T00:27:42.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T00:27:42.264 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T00:27:42.272 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T00:27:42.320 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-09-11T00:27:42.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-09-11T00:27:42.328 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-09-11T00:27:42.331 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-09-11T00:27:42.596 INFO:teuthology.orchestra.run.smithi204.stdout:45 files removed 2024-09-11T00:27:42.597 INFO:teuthology.orchestra.run.smithi031.stdout:45 files removed 2024-09-11T00:27:42.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-11T00:27:42.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-11T00:27:43.317 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 286 kB/s | 84 kB 00:00 2024-09-11T00:27:43.355 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 274 kB/s | 84 kB 00:00 2024-09-11T00:27:43.626 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 91 kB/s | 24 kB 00:00 2024-09-11T00:27:43.672 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 89 kB/s | 24 kB 00:00 2024-09-11T00:27:43.935 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 7.1 kB/s | 1.9 kB 00:00 2024-09-11T00:27:43.965 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 7.6 kB/s | 1.9 kB 00:00 2024-09-11T00:27:44.631 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-11T00:27:45.763 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.2 MB 00:01 2024-09-11T00:27:47.882 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 20 MB 00:02 2024-09-11T00:27:51.353 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-11T00:27:52.611 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-11T00:27:54.334 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 19 kB 00:00 2024-09-11T00:27:55.305 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-11T00:27:56.653 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.5 MB 00:01 2024-09-11T00:27:58.571 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 19 kB 00:00 2024-09-11T00:27:59.650 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-11T00:28:01.927 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-09-11T00:28:04.509 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-11T00:28:04.510 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-11T00:28:04.524 INFO:teuthology.orchestra.run.smithi031.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-11T00:28:04.525 INFO:teuthology.orchestra.run.smithi031.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-11T00:28:04.526 INFO:teuthology.orchestra.run.smithi031.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-11T00:28:04.527 INFO:teuthology.orchestra.run.smithi031.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-11T00:28:04.529 INFO:teuthology.orchestra.run.smithi031.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-11T00:28:04.531 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-11T00:28:04.532 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-11T00:28:04.603 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-09-11T00:28:04.625 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-09-11T00:28:04.625 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-09-11T00:28:04.625 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-09-11T00:28:04.625 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-09-11T00:28:04.625 INFO:teuthology.orchestra.run.smithi031.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-11T00:28:04.625 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 6.6 k 2024-09-11T00:28:04.625 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-11T00:28:04.625 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 900 k 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 145 k 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 1.5 M 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 154 k 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 14 M 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 7.4 M 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 49 k 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 11 M 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 50 M 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 285 k 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 783 k 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 225 k 2024-09-11T00:28:04.626 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 27 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 18 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 32 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 755 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 127 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-11T00:28:04.627 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 161 k 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 321 k 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 306 k 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 99 k 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 85 k 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 171 k 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-11T00:28:04.628 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.5 M 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 22 M 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 34 k 2024-09-11T00:28:04.629 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 2.4 M 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 291 k 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 4.8 M 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 17 M 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 16 k 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 25 k 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T00:28:04.630 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 162 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 507 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-11T00:28:04.631 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-11T00:28:04.632 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 45 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 143 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T00:28:04.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T00:28:04.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T00:28:04.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T00:28:04.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T00:28:04.637 INFO:teuthology.orchestra.run.smithi031.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-11T00:28:04.638 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T00:28:04.638 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-11T00:28:04.638 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2024-09-11T00:28:04.638 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T00:28:04.638 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:28:04.638 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-09-11T00:28:04.638 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-09-11T00:28:04.638 INFO:teuthology.orchestra.run.smithi031.stdout:Install 149 Packages 2024-09-11T00:28:04.638 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 8 Packages 2024-09-11T00:28:04.638 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:28:04.639 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 240 M 2024-09-11T00:28:04.639 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-09-11T00:28:05.512 INFO:teuthology.orchestra.run.smithi031.stdout:(1/157): ceph-19.3.0-4854.g8f219c47.el9.x86_64. 47 kB/s | 6.6 kB 00:00 2024-09-11T00:28:05.695 INFO:teuthology.orchestra.run.smithi031.stdout:(2/157): ceph-fuse-19.3.0-4854.g8f219c47.el9.x8 4.8 MB/s | 900 kB 00:00 2024-09-11T00:28:05.754 INFO:teuthology.orchestra.run.smithi031.stdout:(3/157): ceph-immutable-object-cache-19.3.0-485 2.4 MB/s | 145 kB 00:00 2024-09-11T00:28:05.929 INFO:teuthology.orchestra.run.smithi031.stdout:(4/157): ceph-base-19.3.0-4854.g8f219c47.el9.x8 9.9 MB/s | 5.5 MB 00:00 2024-09-11T00:28:05.987 INFO:teuthology.orchestra.run.smithi031.stdout:(5/157): ceph-mds-19.3.0-4854.g8f219c47.el9.x86 10 MB/s | 2.4 MB 00:00 2024-09-11T00:28:05.989 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-09-11T00:28:06.089 INFO:teuthology.orchestra.run.smithi031.stdout:(6/157): ceph-mgr-19.3.0-4854.g8f219c47.el9.x86 9.5 MB/s | 1.5 MB 00:00 2024-09-11T00:28:06.363 INFO:teuthology.orchestra.run.smithi031.stdout:(7/157): ceph-mon-19.3.0-4854.g8f219c47.el9.x86 13 MB/s | 4.8 MB 00:00 2024-09-11T00:28:06.772 INFO:teuthology.orchestra.run.smithi031.stdout:(8/157): ceph-common-19.3.0-4854.g8f219c47.el9. 16 MB/s | 22 MB 00:01 2024-09-11T00:28:06.822 INFO:teuthology.orchestra.run.smithi031.stdout:(9/157): ceph-selinux-19.3.0-4854.g8f219c47.el9 505 kB/s | 25 kB 00:00 2024-09-11T00:28:07.097 INFO:teuthology.orchestra.run.smithi031.stdout:(10/157): ceph-radosgw-19.3.0-4854.g8f219c47.el 15 MB/s | 11 MB 00:00 2024-09-11T00:28:07.281 INFO:teuthology.orchestra.run.smithi031.stdout:(11/157): ceph-osd-19.3.0-4854.g8f219c47.el9.x8 14 MB/s | 17 MB 00:01 2024-09-11T00:28:07.306 INFO:teuthology.orchestra.run.smithi031.stdout:(12/157): cephfs-mirror-19.3.0-4854.g8f219c47.e 1.1 MB/s | 225 kB 00:00 2024-09-11T00:28:07.331 INFO:teuthology.orchestra.run.smithi031.stdout:(13/157): cephfs-shell-19.3.0-4854.g8f219c47.el 539 kB/s | 27 kB 00:00 2024-09-11T00:28:07.356 INFO:teuthology.orchestra.run.smithi031.stdout:(14/157): libcephfs-devel-19.3.0-4854.g8f219c47 638 kB/s | 32 kB 00:00 2024-09-11T00:28:07.390 INFO:teuthology.orchestra.run.smithi031.stdout:(15/157): libcephfs2-19.3.0-4854.g8f219c47.el9. 13 MB/s | 755 kB 00:00 2024-09-11T00:28:07.424 INFO:teuthology.orchestra.run.smithi031.stdout:(16/157): libcephsqlite-19.3.0-4854.g8f219c47.e 2.4 MB/s | 162 kB 00:00 2024-09-11T00:28:07.449 INFO:teuthology.orchestra.run.smithi031.stdout:(17/157): librados-devel-19.3.0-4854.g8f219c47. 2.1 MB/s | 127 kB 00:00 2024-09-11T00:28:07.491 INFO:teuthology.orchestra.run.smithi031.stdout:(18/157): libradosstriper1-19.3.0-4854.g8f219c4 7.4 MB/s | 507 kB 00:00 2024-09-11T00:28:07.558 INFO:teuthology.orchestra.run.smithi031.stdout:(19/157): python3-ceph-argparse-19.3.0-4854.g8f 681 kB/s | 45 kB 00:00 2024-09-11T00:28:07.641 INFO:teuthology.orchestra.run.smithi031.stdout:(20/157): python3-ceph-common-19.3.0-4854.g8f21 1.7 MB/s | 143 kB 00:00 2024-09-11T00:28:07.725 INFO:teuthology.orchestra.run.smithi031.stdout:(21/157): librgw2-19.3.0-4854.g8f219c47.el9.x86 20 MB/s | 5.5 MB 00:00 2024-09-11T00:28:07.750 INFO:teuthology.orchestra.run.smithi031.stdout:(22/157): python3-cephfs-19.3.0-4854.g8f219c47. 1.5 MB/s | 161 kB 00:00 2024-09-11T00:28:07.776 INFO:teuthology.orchestra.run.smithi031.stdout:(23/157): python3-rados-19.3.0-4854.g8f219c47.e 6.3 MB/s | 321 kB 00:00 2024-09-11T00:28:07.801 INFO:teuthology.orchestra.run.smithi031.stdout:(24/157): python3-rbd-19.3.0-4854.g8f219c47.el9 6.0 MB/s | 306 kB 00:00 2024-09-11T00:28:07.826 INFO:teuthology.orchestra.run.smithi031.stdout:(25/157): python3-rgw-19.3.0-4854.g8f219c47.el9 1.9 MB/s | 99 kB 00:00 2024-09-11T00:28:07.851 INFO:teuthology.orchestra.run.smithi031.stdout:(26/157): rbd-fuse-19.3.0-4854.g8f219c47.el9.x8 1.7 MB/s | 85 kB 00:00 2024-09-11T00:28:07.910 INFO:teuthology.orchestra.run.smithi031.stdout:(27/157): rbd-nbd-19.3.0-4854.g8f219c47.el9.x86 2.9 MB/s | 171 kB 00:00 2024-09-11T00:28:07.969 INFO:teuthology.orchestra.run.smithi031.stdout:(28/157): rbd-mirror-19.3.0-4854.g8f219c47.el9. 22 MB/s | 3.2 MB 00:00 2024-09-11T00:28:08.025 INFO:teuthology.orchestra.run.smithi031.stdout:(29/157): ceph-grafana-dashboards-19.3.0-4854.g 299 kB/s | 34 kB 00:00 2024-09-11T00:28:08.053 INFO:teuthology.orchestra.run.smithi031.stdout:(30/157): ceph-mgr-cephadm-19.3.0-4854.g8f219c4 1.8 MB/s | 154 kB 00:00 2024-09-11T00:28:08.545 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-11T00:28:08.545 INFO:teuthology.orchestra.run.smithi031.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 15 MB/s | 7.4 MB 00:00 2024-09-11T00:28:08.546 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-11T00:28:08.559 INFO:teuthology.orchestra.run.smithi204.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-11T00:28:08.560 INFO:teuthology.orchestra.run.smithi204.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-11T00:28:08.561 INFO:teuthology.orchestra.run.smithi204.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-11T00:28:08.562 INFO:teuthology.orchestra.run.smithi204.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-11T00:28:08.564 INFO:teuthology.orchestra.run.smithi204.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-11T00:28:08.567 INFO:teuthology.orchestra.run.smithi204.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-11T00:28:08.568 INFO:teuthology.orchestra.run.smithi204.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-11T00:28:08.612 INFO:teuthology.orchestra.run.smithi031.stdout:(32/157): ceph-mgr-modules-core-19.3.0-4854.g8f 4.3 MB/s | 291 kB 00:00 2024-09-11T00:28:08.634 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-11T00:28:08.658 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-11T00:28:08.658 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-11T00:28:08.658 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-11T00:28:08.658 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-11T00:28:08.658 INFO:teuthology.orchestra.run.smithi204.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-11T00:28:08.658 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 6.6 k 2024-09-11T00:28:08.658 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-11T00:28:08.658 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 900 k 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 145 k 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 1.5 M 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 154 k 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 14 M 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 7.4 M 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 49 k 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 11 M 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 50 M 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 285 k 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 783 k 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 225 k 2024-09-11T00:28:08.659 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-shell x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 27 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-top noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 18 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 32 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 755 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 127 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-11T00:28:08.660 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 161 k 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 321 k 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 306 k 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 99 k 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 85 k 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 171 k 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-11T00:28:08.661 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.5 M 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 3.2 M 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 22 M 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 34 k 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 2.4 M 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 291 k 2024-09-11T00:28:08.662 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 4.8 M 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 17 M 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4854.g8f219c47.el9 ceph-noarch 16 k 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 25 k 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 162 k 2024-09-11T00:28:08.663 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 507 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 5.5 M 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T00:28:08.664 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T00:28:08.665 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 45 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:19.3.0-4854.g8f219c47.el9 ceph 143 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T00:28:08.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T00:28:08.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T00:28:08.668 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T00:28:08.669 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-11T00:28:08.670 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T00:28:08.671 INFO:teuthology.orchestra.run.smithi204.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-11T00:28:08.671 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-09-11T00:28:08.671 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T00:28:08.671 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:28:08.671 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-11T00:28:08.671 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-11T00:28:08.671 INFO:teuthology.orchestra.run.smithi204.stdout:Install 149 Packages 2024-09-11T00:28:08.671 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 8 Packages 2024-09-11T00:28:08.671 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:28:08.672 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 240 M 2024-09-11T00:28:08.672 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-11T00:28:08.787 INFO:teuthology.orchestra.run.smithi031.stdout:(33/157): ceph-mgr-dashboard-19.3.0-4854.g8f219 19 MB/s | 14 MB 00:00 2024-09-11T00:28:08.812 INFO:teuthology.orchestra.run.smithi031.stdout:(34/157): ceph-mgr-rook-19.3.0-4854.g8f219c47.e 246 kB/s | 49 kB 00:00 2024-09-11T00:28:08.837 INFO:teuthology.orchestra.run.smithi031.stdout:(35/157): ceph-prometheus-alerts-19.3.0-4854.g8 330 kB/s | 16 kB 00:00 2024-09-11T00:28:08.871 INFO:teuthology.orchestra.run.smithi031.stdout:(36/157): ceph-volume-19.3.0-4854.g8f219c47.el9 4.8 MB/s | 285 kB 00:00 2024-09-11T00:28:08.905 INFO:teuthology.orchestra.run.smithi031.stdout:(37/157): cephadm-19.3.0-4854.g8f219c47.el9.noa 11 MB/s | 783 kB 00:00 2024-09-11T00:28:08.948 INFO:teuthology.orchestra.run.smithi031.stdout:(38/157): cephfs-top-19.3.0-4854.g8f219c47.el9. 238 kB/s | 18 kB 00:00 2024-09-11T00:28:09.064 INFO:teuthology.orchestra.run.smithi031.stdout:(39/157): fuse-2.9.9-16.el9.x86_64.rpm 504 kB/s | 80 kB 00:00 2024-09-11T00:28:09.106 INFO:teuthology.orchestra.run.smithi031.stdout:(40/157): libgfortran-11.5.0-2.el9.x86_64.rpm 5.0 MB/s | 801 kB 00:00 2024-09-11T00:28:09.139 INFO:teuthology.orchestra.run.smithi031.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-11T00:28:09.141 INFO:teuthology.orchestra.run.smithi204.stdout:(1/157): ceph-19.3.0-4854.g8f219c47.el9.x86_64. 44 kB/s | 6.6 kB 00:00 2024-09-11T00:28:09.181 INFO:teuthology.orchestra.run.smithi031.stdout:(42/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-09-11T00:28:09.240 INFO:teuthology.orchestra.run.smithi031.stdout:(43/157): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-09-11T00:28:09.265 INFO:teuthology.orchestra.run.smithi031.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-09-11T00:28:09.316 INFO:teuthology.orchestra.run.smithi031.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-09-11T00:28:09.324 INFO:teuthology.orchestra.run.smithi204.stdout:(2/157): ceph-fuse-19.3.0-4854.g8f219c47.el9.x8 4.8 MB/s | 900 kB 00:00 2024-09-11T00:28:09.374 INFO:teuthology.orchestra.run.smithi031.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-09-11T00:28:09.391 INFO:teuthology.orchestra.run.smithi204.stdout:(3/157): ceph-immutable-object-cache-19.3.0-485 2.1 MB/s | 145 kB 00:00 2024-09-11T00:28:09.424 INFO:teuthology.orchestra.run.smithi031.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-09-11T00:28:09.567 INFO:teuthology.orchestra.run.smithi204.stdout:(4/157): ceph-base-19.3.0-4854.g8f219c47.el9.x8 9.6 MB/s | 5.5 MB 00:00 2024-09-11T00:28:09.634 INFO:teuthology.orchestra.run.smithi204.stdout:(5/157): ceph-mds-19.3.0-4854.g8f219c47.el9.x86 9.8 MB/s | 2.4 MB 00:00 2024-09-11T00:28:09.721 INFO:teuthology.orchestra.run.smithi204.stdout:(6/157): ceph-mgr-19.3.0-4854.g8f219c47.el9.x86 9.9 MB/s | 1.5 MB 00:00 2024-09-11T00:28:09.942 INFO:teuthology.orchestra.run.smithi031.stdout:(48/157): ceph-test-19.3.0-4854.g8f219c47.el9.x 16 MB/s | 50 MB 00:03 2024-09-11T00:28:10.001 INFO:teuthology.orchestra.run.smithi204.stdout:(7/157): ceph-mon-19.3.0-4854.g8f219c47.el9.x86 13 MB/s | 4.8 MB 00:00 2024-09-11T00:28:10.417 INFO:teuthology.orchestra.run.smithi031.stdout:(49/157): automake-1.16.2-8.el9.noarch.rpm 661 kB/s | 689 kB 00:01 2024-09-11T00:28:10.442 INFO:teuthology.orchestra.run.smithi031.stdout:(50/157): boost-program-options-1.75.0-8.el9.x8 214 kB/s | 107 kB 00:00 2024-09-11T00:28:10.526 INFO:teuthology.orchestra.run.smithi031.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 200 kB/s | 22 kB 00:00 2024-09-11T00:28:10.634 INFO:teuthology.orchestra.run.smithi031.stdout:(52/157): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-09-11T00:28:10.651 INFO:teuthology.orchestra.run.smithi204.stdout:(8/157): ceph-common-19.3.0-4854.g8f219c47.el9. 13 MB/s | 22 MB 00:01 2024-09-11T00:28:10.684 INFO:teuthology.orchestra.run.smithi031.stdout:(53/157): bison-3.7.4-5.el9.x86_64.rpm 749 kB/s | 943 kB 00:01 2024-09-11T00:28:10.735 INFO:teuthology.orchestra.run.smithi031.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 1.0 MB/s | 313 kB 00:00 2024-09-11T00:28:10.785 INFO:teuthology.orchestra.run.smithi031.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 182 kB/s | 18 kB 00:00 2024-09-11T00:28:10.868 INFO:teuthology.orchestra.run.smithi031.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 83 kB/s | 11 kB 00:00 2024-09-11T00:28:10.910 INFO:teuthology.orchestra.run.smithi031.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 1.3 MB/s | 171 kB 00:00 2024-09-11T00:28:10.986 INFO:teuthology.orchestra.run.smithi031.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-09-11T00:28:10.993 INFO:teuthology.orchestra.run.smithi204.stdout:(9/157): ceph-osd-19.3.0-4854.g8f219c47.el9.x86 13 MB/s | 17 MB 00:01 2024-09-11T00:28:11.018 INFO:teuthology.orchestra.run.smithi204.stdout:(10/157): ceph-selinux-19.3.0-4854.g8f219c47.el 68 kB/s | 25 kB 00:00 2024-09-11T00:28:11.061 INFO:teuthology.orchestra.run.smithi031.stdout:(59/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.0 MB/s | 3.0 MB 00:00 2024-09-11T00:28:11.086 INFO:teuthology.orchestra.run.smithi031.stdout:(60/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 258 kB/s | 45 kB 00:00 2024-09-11T00:28:11.201 INFO:teuthology.orchestra.run.smithi204.stdout:(11/157): ceph-radosgw-19.3.0-4854.g8f219c47.el 9.1 MB/s | 11 MB 00:01 2024-09-11T00:28:11.211 INFO:teuthology.orchestra.run.smithi031.stdout:(61/157): libtool-2.4.6-46.el9.x86_64.rpm 4.6 MB/s | 581 kB 00:00 2024-09-11T00:28:11.227 INFO:teuthology.orchestra.run.smithi204.stdout:(12/157): cephfs-mirror-19.3.0-4854.g8f219c47.e 1.1 MB/s | 225 kB 00:00 2024-09-11T00:28:11.236 INFO:teuthology.orchestra.run.smithi031.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 163 kB/s | 24 kB 00:00 2024-09-11T00:28:11.244 INFO:teuthology.orchestra.run.smithi204.stdout:(13/157): cephfs-shell-19.3.0-4854.g8f219c47.el 627 kB/s | 27 kB 00:00 2024-09-11T00:28:11.261 INFO:teuthology.orchestra.run.smithi204.stdout:(14/157): libcephfs-devel-19.3.0-4854.g8f219c47 948 kB/s | 32 kB 00:00 2024-09-11T00:28:11.287 INFO:teuthology.orchestra.run.smithi031.stdout:(63/157): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-09-11T00:28:11.311 INFO:teuthology.orchestra.run.smithi204.stdout:(15/157): libcephsqlite-19.3.0-4854.g8f219c47.e 3.2 MB/s | 162 kB 00:00 2024-09-11T00:28:11.345 INFO:teuthology.orchestra.run.smithi204.stdout:(16/157): libcephfs2-19.3.0-4854.g8f219c47.el9. 7.3 MB/s | 755 kB 00:00 2024-09-11T00:28:11.346 INFO:teuthology.orchestra.run.smithi031.stdout:(64/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 355 kB/s | 47 kB 00:00 2024-09-11T00:28:11.379 INFO:teuthology.orchestra.run.smithi204.stdout:(17/157): librados-devel-19.3.0-4854.g8f219c47. 1.9 MB/s | 127 kB 00:00 2024-09-11T00:28:11.379 INFO:teuthology.orchestra.run.smithi031.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-11T00:28:11.404 INFO:teuthology.orchestra.run.smithi031.stdout:(66/157): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-11T00:28:11.412 INFO:teuthology.orchestra.run.smithi204.stdout:(18/157): libradosstriper1-19.3.0-4854.g8f219c4 7.4 MB/s | 507 kB 00:00 2024-09-11T00:28:11.455 INFO:teuthology.orchestra.run.smithi031.stdout:(67/157): openblas-0.3.26-2.el9.x86_64.rpm 353 kB/s | 38 kB 00:00 2024-09-11T00:28:11.479 INFO:teuthology.orchestra.run.smithi204.stdout:(19/157): python3-ceph-argparse-19.3.0-4854.g8f 678 kB/s | 45 kB 00:00 2024-09-11T00:28:11.588 INFO:teuthology.orchestra.run.smithi204.stdout:(20/157): python3-ceph-common-19.3.0-4854.g8f21 1.3 MB/s | 143 kB 00:00 2024-09-11T00:28:11.597 INFO:teuthology.orchestra.run.smithi031.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-11T00:28:11.672 INFO:teuthology.orchestra.run.smithi204.stdout:(21/157): python3-cephfs-19.3.0-4854.g8f219c47. 1.9 MB/s | 161 kB 00:00 2024-09-11T00:28:11.772 INFO:teuthology.orchestra.run.smithi204.stdout:(22/157): librgw2-19.3.0-4854.g8f219c47.el9.x86 14 MB/s | 5.5 MB 00:00 2024-09-11T00:28:11.806 INFO:teuthology.orchestra.run.smithi204.stdout:(23/157): python3-rados-19.3.0-4854.g8f219c47.e 2.3 MB/s | 321 kB 00:00 2024-09-11T00:28:11.831 INFO:teuthology.orchestra.run.smithi204.stdout:(24/157): python3-rbd-19.3.0-4854.g8f219c47.el9 5.1 MB/s | 306 kB 00:00 2024-09-11T00:28:11.856 INFO:teuthology.orchestra.run.smithi204.stdout:(25/157): python3-rgw-19.3.0-4854.g8f219c47.el9 1.9 MB/s | 99 kB 00:00 2024-09-11T00:28:11.881 INFO:teuthology.orchestra.run.smithi204.stdout:(26/157): rbd-fuse-19.3.0-4854.g8f219c47.el9.x8 1.7 MB/s | 85 kB 00:00 2024-09-11T00:28:11.957 INFO:teuthology.orchestra.run.smithi204.stdout:(27/157): rbd-nbd-19.3.0-4854.g8f219c47.el9.x86 2.2 MB/s | 171 kB 00:00 2024-09-11T00:28:12.057 INFO:teuthology.orchestra.run.smithi204.stdout:(28/157): rbd-mirror-19.3.0-4854.g8f219c47.el9. 16 MB/s | 3.2 MB 00:00 2024-09-11T00:28:12.082 INFO:teuthology.orchestra.run.smithi204.stdout:(29/157): ceph-grafana-dashboards-19.3.0-4854.g 274 kB/s | 34 kB 00:00 2024-09-11T00:28:12.107 INFO:teuthology.orchestra.run.smithi204.stdout:(30/157): ceph-mgr-cephadm-19.3.0-4854.g8f219c4 3.0 MB/s | 154 kB 00:00 2024-09-11T00:28:12.456 INFO:teuthology.orchestra.run.smithi031.stdout:(69/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-09-11T00:28:12.678 INFO:teuthology.orchestra.run.smithi204.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 13 MB/s | 7.4 MB 00:00 2024-09-11T00:28:12.735 INFO:teuthology.orchestra.run.smithi204.stdout:(32/157): ceph-mgr-modules-core-19.3.0-4854.g8f 4.9 MB/s | 291 kB 00:00 2024-09-11T00:28:12.756 INFO:teuthology.orchestra.run.smithi031.stdout:(70/157): postgresql-private-libs-13.16-1.el9.x 471 kB/s | 137 kB 00:00 2024-09-11T00:28:12.925 INFO:teuthology.orchestra.run.smithi204.stdout:(33/157): ceph-mgr-dashboard-19.3.0-4854.g8f219 17 MB/s | 14 MB 00:00 2024-09-11T00:28:12.942 INFO:teuthology.orchestra.run.smithi204.stdout:(34/157): ceph-mgr-rook-19.3.0-4854.g8f219c47.e 236 kB/s | 49 kB 00:00 2024-09-11T00:28:12.956 INFO:teuthology.orchestra.run.smithi031.stdout:(71/157): postgresql-contrib-13.16-1.el9.x86_64 669 kB/s | 910 kB 00:01 2024-09-11T00:28:12.959 INFO:teuthology.orchestra.run.smithi204.stdout:(35/157): ceph-prometheus-alerts-19.3.0-4854.g8 491 kB/s | 16 kB 00:00 2024-09-11T00:28:13.223 INFO:teuthology.orchestra.run.smithi031.stdout:(72/157): openblas-openmp-0.3.26-2.el9.x86_64.r 2.7 MB/s | 4.9 MB 00:01 2024-09-11T00:28:13.423 INFO:teuthology.orchestra.run.smithi031.stdout:(73/157): python3-attrs-20.3.0-7.el9.noarch.rpm 438 kB/s | 87 kB 00:00 2024-09-11T00:28:13.463 INFO:teuthology.orchestra.run.smithi204.stdout:(36/157): ceph-test-19.3.0-4854.g8f219c47.el9.x 20 MB/s | 50 MB 00:02 2024-09-11T00:28:13.502 INFO:teuthology.orchestra.run.smithi204.stdout:(37/157): ceph-volume-19.3.0-4854.g8f219c47.el9 510 kB/s | 285 kB 00:00 2024-09-11T00:28:13.527 INFO:teuthology.orchestra.run.smithi204.stdout:(38/157): cephfs-top-19.3.0-4854.g8f219c47.el9. 291 kB/s | 18 kB 00:00 2024-09-11T00:28:13.560 INFO:teuthology.orchestra.run.smithi204.stdout:(39/157): cephadm-19.3.0-4854.g8f219c47.el9.noa 1.3 MB/s | 783 kB 00:00 2024-09-11T00:28:13.686 INFO:teuthology.orchestra.run.smithi204.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 434 kB/s | 80 kB 00:00 2024-09-11T00:28:13.778 INFO:teuthology.orchestra.run.smithi204.stdout:(41/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-09-11T00:28:13.833 INFO:teuthology.orchestra.run.smithi204.stdout:(42/157): libquadmath-11.5.0-2.el9.x86_64.rpm 703 kB/s | 191 kB 00:00 2024-09-11T00:28:13.870 INFO:teuthology.orchestra.run.smithi204.stdout:(43/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-11T00:28:13.895 INFO:teuthology.orchestra.run.smithi204.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 106 kB 00:00 2024-09-11T00:28:13.929 INFO:teuthology.orchestra.run.smithi204.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-09-11T00:28:13.954 INFO:teuthology.orchestra.run.smithi204.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-09-11T00:28:13.996 INFO:teuthology.orchestra.run.smithi204.stdout:(47/157): python3-cryptography-36.0.1-4.el9.x86 5.6 MB/s | 1.2 MB 00:00 2024-09-11T00:28:14.021 INFO:teuthology.orchestra.run.smithi204.stdout:(48/157): python3-urllib3-1.26.5-6.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-09-11T00:28:14.130 INFO:teuthology.orchestra.run.smithi204.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 989 kB/s | 107 kB 00:00 2024-09-11T00:28:14.164 INFO:teuthology.orchestra.run.smithi204.stdout:(50/157): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-09-11T00:28:14.214 INFO:teuthology.orchestra.run.smithi204.stdout:(51/157): elfutils-libelf-devel-0.191-4.el9.x86 259 kB/s | 22 kB 00:00 2024-09-11T00:28:14.264 INFO:teuthology.orchestra.run.smithi204.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 943 kB 00:00 2024-09-11T00:28:14.290 INFO:teuthology.orchestra.run.smithi204.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 435 kB/s | 33 kB 00:00 2024-09-11T00:28:14.315 INFO:teuthology.orchestra.run.smithi204.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 2.0 MB/s | 313 kB 00:00 2024-09-11T00:28:14.340 INFO:teuthology.orchestra.run.smithi204.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-09-11T00:28:14.365 INFO:teuthology.orchestra.run.smithi204.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-09-11T00:28:14.390 INFO:teuthology.orchestra.run.smithi204.stdout:(57/157): libnbd-1.20.2-2.el9.x86_64.rpm 3.3 MB/s | 171 kB 00:00 2024-09-11T00:28:14.433 INFO:teuthology.orchestra.run.smithi204.stdout:(58/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-09-11T00:28:14.457 INFO:teuthology.orchestra.run.smithi204.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 679 kB/s | 45 kB 00:00 2024-09-11T00:28:14.473 INFO:teuthology.orchestra.run.smithi031.stdout:(74/157): protobuf-3.14.0-13.el9.x86_64.rpm 681 kB/s | 1.0 MB 00:01 2024-09-11T00:28:14.516 INFO:teuthology.orchestra.run.smithi204.stdout:(60/157): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 662 kB 00:00 2024-09-11T00:28:14.574 INFO:teuthology.orchestra.run.smithi204.stdout:(61/157): libtool-2.4.6-46.el9.x86_64.rpm 4.9 MB/s | 581 kB 00:00 2024-09-11T00:28:14.590 INFO:teuthology.orchestra.run.smithi031.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 525 kB/s | 61 kB 00:00 2024-09-11T00:28:14.600 INFO:teuthology.orchestra.run.smithi204.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 293 kB/s | 24 kB 00:00 2024-09-11T00:28:14.625 INFO:teuthology.orchestra.run.smithi204.stdout:(63/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 928 kB/s | 47 kB 00:00 2024-09-11T00:28:14.701 INFO:teuthology.orchestra.run.smithi204.stdout:(64/157): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-09-11T00:28:14.726 INFO:teuthology.orchestra.run.smithi204.stdout:(65/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-11T00:28:14.751 INFO:teuthology.orchestra.run.smithi204.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 768 kB/s | 38 kB 00:00 2024-09-11T00:28:14.965 INFO:teuthology.orchestra.run.smithi031.stdout:(76/157): python3-devel-3.9.19-8.el9.x86_64.rpm 653 kB/s | 245 kB 00:00 2024-09-11T00:28:15.043 INFO:teuthology.orchestra.run.smithi204.stdout:(67/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.8 MB/s | 3.0 MB 00:00 2024-09-11T00:28:15.349 INFO:teuthology.orchestra.run.smithi031.stdout:(77/157): python3-jinja2-2.11.3-6.el9.noarch.rp 648 kB/s | 248 kB 00:00 2024-09-11T00:28:15.457 INFO:teuthology.orchestra.run.smithi031.stdout:(78/157): python3-jsonpatch-1.21-16.el9.noarch. 237 kB/s | 26 kB 00:00 2024-09-11T00:28:15.485 INFO:teuthology.orchestra.run.smithi204.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 3.6 MB/s | 1.6 MB 00:00 2024-09-11T00:28:15.574 INFO:teuthology.orchestra.run.smithi031.stdout:(79/157): python3-jsonpointer-2.0-4.el9.noarch. 164 kB/s | 19 kB 00:00 2024-09-11T00:28:15.753 INFO:teuthology.orchestra.run.smithi204.stdout:(69/157): postgresql-contrib-13.16-1.el9.x86_64 3.3 MB/s | 910 kB 00:00 2024-09-11T00:28:15.828 INFO:teuthology.orchestra.run.smithi204.stdout:(70/157): postgresql-private-libs-13.16-1.el9.x 1.8 MB/s | 137 kB 00:00 2024-09-11T00:28:15.858 INFO:teuthology.orchestra.run.smithi031.stdout:(80/157): python3-mako-1.1.4-6.el9.noarch.rpm 609 kB/s | 172 kB 00:00 2024-09-11T00:28:15.983 INFO:teuthology.orchestra.run.smithi031.stdout:(81/157): python3-markupsafe-1.1.1-12.el9.x86_6 278 kB/s | 35 kB 00:00 2024-09-11T00:28:16.077 INFO:teuthology.orchestra.run.smithi204.stdout:(71/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-09-11T00:28:16.170 INFO:teuthology.orchestra.run.smithi204.stdout:(72/157): openblas-openmp-0.3.26-2.el9.x86_64.r 3.4 MB/s | 4.9 MB 00:01 2024-09-11T00:28:16.220 INFO:teuthology.orchestra.run.smithi204.stdout:(73/157): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-09-11T00:28:16.467 INFO:teuthology.orchestra.run.smithi031.stdout:(82/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-09-11T00:28:16.470 INFO:teuthology.orchestra.run.smithi204.stdout:(74/157): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-09-11T00:28:16.512 INFO:teuthology.orchestra.run.smithi204.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-09-11T00:28:16.592 INFO:teuthology.orchestra.run.smithi031.stdout:(83/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 250 kB 00:00 2024-09-11T00:28:16.604 INFO:teuthology.orchestra.run.smithi204.stdout:(76/157): python3-devel-3.9.19-8.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-09-11T00:28:16.705 INFO:teuthology.orchestra.run.smithi204.stdout:(77/157): python3-jinja2-2.11.3-6.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-09-11T00:28:16.709 INFO:teuthology.orchestra.run.smithi031.stdout:(84/157): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-09-11T00:28:16.746 INFO:teuthology.orchestra.run.smithi204.stdout:(78/157): python3-jsonpatch-1.21-16.el9.noarch. 622 kB/s | 26 kB 00:00 2024-09-11T00:28:16.788 INFO:teuthology.orchestra.run.smithi204.stdout:(79/157): python3-jsonpointer-2.0-4.el9.noarch. 461 kB/s | 19 kB 00:00 2024-09-11T00:28:16.826 INFO:teuthology.orchestra.run.smithi031.stdout:(85/157): python3-packaging-20.9-5.el9.noarch.r 661 kB/s | 77 kB 00:00 2024-09-11T00:28:16.864 INFO:teuthology.orchestra.run.smithi204.stdout:(80/157): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-11T00:28:16.906 INFO:teuthology.orchestra.run.smithi204.stdout:(81/157): python3-markupsafe-1.1.1-12.el9.x86_6 841 kB/s | 35 kB 00:00 2024-09-11T00:28:16.934 INFO:teuthology.orchestra.run.smithi031.stdout:(86/157): python3-prettytable-0.7.2-27.el9.noar 390 kB/s | 42 kB 00:00 2024-09-11T00:28:17.051 INFO:teuthology.orchestra.run.smithi031.stdout:(87/157): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-09-11T00:28:17.168 INFO:teuthology.orchestra.run.smithi031.stdout:(88/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-11T00:28:17.293 INFO:teuthology.orchestra.run.smithi031.stdout:(89/157): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 279 kB 00:00 2024-09-11T00:28:17.402 INFO:teuthology.orchestra.run.smithi031.stdout:(90/157): python3-pytz-2021.1-5.el9.noarch.rpm 471 kB/s | 51 kB 00:00 2024-09-11T00:28:17.505 INFO:teuthology.orchestra.run.smithi031.stdout:(91/157): python3-requests-oauthlib-1.3.0-12.el 538 kB/s | 54 kB 00:00 2024-09-11T00:28:17.656 INFO:teuthology.orchestra.run.smithi204.stdout:(82/157): postgresql-server-13.16-1.el9.x86_64. 3.2 MB/s | 5.8 MB 00:01 2024-09-11T00:28:17.756 INFO:teuthology.orchestra.run.smithi204.stdout:(83/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.5 MB/s | 250 kB 00:00 2024-09-11T00:28:17.881 INFO:teuthology.orchestra.run.smithi204.stdout:(84/157): python3-babel-2.9.1-2.el9.noarch.rpm 3.6 MB/s | 6.0 MB 00:01 2024-09-11T00:28:17.982 INFO:teuthology.orchestra.run.smithi204.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 986 kB/s | 222 kB 00:00 2024-09-11T00:28:17.999 INFO:teuthology.orchestra.run.smithi204.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 660 kB/s | 77 kB 00:00 2024-09-11T00:28:18.024 INFO:teuthology.orchestra.run.smithi204.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-09-11T00:28:18.091 INFO:teuthology.orchestra.run.smithi204.stdout:(88/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-09-11T00:28:18.116 INFO:teuthology.orchestra.run.smithi204.stdout:(89/157): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-09-11T00:28:18.158 INFO:teuthology.orchestra.run.smithi204.stdout:(90/157): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-11T00:28:18.169 INFO:teuthology.orchestra.run.smithi031.stdout:(92/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-09-11T00:28:18.200 INFO:teuthology.orchestra.run.smithi204.stdout:(91/157): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-09-11T00:28:18.225 INFO:teuthology.orchestra.run.smithi204.stdout:(92/157): python3-pyasn1-modules-0.4.8-6.el9.no 2.0 MB/s | 279 kB 00:00 2024-09-11T00:28:18.267 INFO:teuthology.orchestra.run.smithi204.stdout:(93/157): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-09-11T00:28:18.303 INFO:teuthology.orchestra.run.smithi031.stdout:(93/157): python3-toml-0.10.2-6.el9.noarch.rpm 313 kB/s | 42 kB 00:00 2024-09-11T00:28:18.335 INFO:teuthology.orchestra.run.smithi204.stdout:(94/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 663 kB/s | 44 kB 00:00 2024-09-11T00:28:18.403 INFO:teuthology.orchestra.run.smithi031.stdout:(94/157): postgresql-server-13.16-1.el9.x86_64. 1.0 MB/s | 5.8 MB 00:05 2024-09-11T00:28:18.418 INFO:teuthology.orchestra.run.smithi204.stdout:(95/157): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-11T00:28:18.429 INFO:teuthology.orchestra.run.smithi031.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 352 kB/s | 44 kB 00:00 2024-09-11T00:28:18.460 INFO:teuthology.orchestra.run.smithi204.stdout:(96/157): qatlib-service-24.02.0-1.el9.x86_64.r 871 kB/s | 36 kB 00:00 2024-09-11T00:28:18.502 INFO:teuthology.orchestra.run.smithi204.stdout:(97/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-09-11T00:28:18.512 INFO:teuthology.orchestra.run.smithi031.stdout:(96/157): python3-babel-2.9.1-2.el9.noarch.rpm 1.2 MB/s | 6.0 MB 00:05 2024-09-11T00:28:18.537 INFO:teuthology.orchestra.run.smithi031.stdout:(97/157): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-11T00:28:18.563 INFO:teuthology.orchestra.run.smithi031.stdout:(98/157): qatlib-service-24.02.0-1.el9.x86_64.r 268 kB/s | 36 kB 00:00 2024-09-11T00:28:18.619 INFO:teuthology.orchestra.run.smithi204.stdout:(98/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.0 MB/s | 5.1 MB 00:01 2024-09-11T00:28:18.622 INFO:teuthology.orchestra.run.smithi031.stdout:(99/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 436 kB/s | 47 kB 00:00 2024-09-11T00:28:18.663 INFO:teuthology.orchestra.run.smithi031.stdout:(100/157): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-11T00:28:18.686 INFO:teuthology.orchestra.run.smithi204.stdout:(99/157): uuid-1.6.2-55.el9.x86_64.rpm 872 kB/s | 58 kB 00:00 2024-09-11T00:28:18.688 INFO:teuthology.orchestra.run.smithi031.stdout:(101/157): uuid-1.6.2-55.el9.x86_64.rpm 461 kB/s | 58 kB 00:00 2024-09-11T00:28:18.711 INFO:teuthology.orchestra.run.smithi204.stdout:(100/157): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-11T00:28:18.745 INFO:teuthology.orchestra.run.smithi204.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.6 MB/s | 96 kB 00:00 2024-09-11T00:28:18.755 INFO:teuthology.orchestra.run.smithi031.stdout:(102/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 717 kB/s | 96 kB 00:00 2024-09-11T00:28:18.770 INFO:teuthology.orchestra.run.smithi204.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-11T00:28:18.781 INFO:teuthology.orchestra.run.smithi031.stdout:(103/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 544 kB/s | 64 kB 00:00 2024-09-11T00:28:18.795 INFO:teuthology.orchestra.run.smithi204.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 897 kB/s | 45 kB 00:00 2024-09-11T00:28:18.806 INFO:teuthology.orchestra.run.smithi031.stdout:(104/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 382 kB/s | 45 kB 00:00 2024-09-11T00:28:18.865 INFO:teuthology.orchestra.run.smithi031.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-09-11T00:28:18.898 INFO:teuthology.orchestra.run.smithi031.stdout:(106/157): gperftools-libs-2.9.1-2.el9.x86_64.r 9.1 MB/s | 309 kB 00:00 2024-09-11T00:28:18.923 INFO:teuthology.orchestra.run.smithi031.stdout:(107/157): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-09-11T00:28:18.949 INFO:teuthology.orchestra.run.smithi031.stdout:(108/157): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-09-11T00:28:18.982 INFO:teuthology.orchestra.run.smithi031.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-11T00:28:19.008 INFO:teuthology.orchestra.run.smithi031.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-11T00:28:19.033 INFO:teuthology.orchestra.run.smithi031.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-09-11T00:28:19.058 INFO:teuthology.orchestra.run.smithi031.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-09-11T00:28:19.071 INFO:teuthology.orchestra.run.smithi204.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-09-11T00:28:19.092 INFO:teuthology.orchestra.run.smithi031.stdout:(113/157): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-09-11T00:28:19.130 INFO:teuthology.orchestra.run.smithi204.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-09-11T00:28:19.159 INFO:teuthology.orchestra.run.smithi031.stdout:(114/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-11T00:28:19.163 INFO:teuthology.orchestra.run.smithi204.stdout:(106/157): gperftools-libs-2.9.1-2.el9.x86_64.r 9.2 MB/s | 309 kB 00:00 2024-09-11T00:28:19.188 INFO:teuthology.orchestra.run.smithi204.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 785 kB/s | 19 kB 00:00 2024-09-11T00:28:19.209 INFO:teuthology.orchestra.run.smithi031.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 876 kB/s | 43 kB 00:00 2024-09-11T00:28:19.234 INFO:teuthology.orchestra.run.smithi031.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-11T00:28:19.259 INFO:teuthology.orchestra.run.smithi031.stdout:(117/157): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-09-11T00:28:19.285 INFO:teuthology.orchestra.run.smithi031.stdout:(118/157): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-09-11T00:28:19.305 INFO:teuthology.orchestra.run.smithi204.stdout:(108/157): libarrow-9.0.0-11.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2024-09-11T00:28:19.310 INFO:teuthology.orchestra.run.smithi031.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-11T00:28:19.330 INFO:teuthology.orchestra.run.smithi204.stdout:(109/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-11T00:28:19.344 INFO:teuthology.orchestra.run.smithi031.stdout:(120/157): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-09-11T00:28:19.356 INFO:teuthology.orchestra.run.smithi204.stdout:(110/157): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-11T00:28:19.377 INFO:teuthology.orchestra.run.smithi031.stdout:(121/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-09-11T00:28:19.403 INFO:teuthology.orchestra.run.smithi031.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-11T00:28:19.414 INFO:teuthology.orchestra.run.smithi204.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-11T00:28:19.436 INFO:teuthology.orchestra.run.smithi031.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-09-11T00:28:19.440 INFO:teuthology.orchestra.run.smithi204.stdout:(112/157): luarocks-3.9.2-5.el9.noarch.rpm 5.8 MB/s | 151 kB 00:00 2024-09-11T00:28:19.461 INFO:teuthology.orchestra.run.smithi031.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 435 kB/s | 11 kB 00:00 2024-09-11T00:28:19.473 INFO:teuthology.orchestra.run.smithi204.stdout:(113/157): protobuf-compiler-3.14.0-13.el9.x86_ 1.2 MB/s | 863 kB 00:00 2024-09-11T00:28:19.487 INFO:teuthology.orchestra.run.smithi031.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 712 kB/s | 18 kB 00:00 2024-09-11T00:28:19.512 INFO:teuthology.orchestra.run.smithi031.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 913 kB/s | 23 kB 00:00 2024-09-11T00:28:19.532 INFO:teuthology.orchestra.run.smithi204.stdout:(114/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-11T00:28:19.538 INFO:teuthology.orchestra.run.smithi031.stdout:(127/157): python3-jaraco-functools-3.5.0-2.el9 785 kB/s | 19 kB 00:00 2024-09-11T00:28:19.557 INFO:teuthology.orchestra.run.smithi204.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.9 MB/s | 43 kB 00:00 2024-09-11T00:28:19.563 INFO:teuthology.orchestra.run.smithi031.stdout:(128/157): python3-jaraco-text-3.2.0-6.el9.noar 794 kB/s | 20 kB 00:00 2024-09-11T00:28:19.574 INFO:teuthology.orchestra.run.smithi204.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 1.9 MB/s | 32 kB 00:00 2024-09-11T00:28:19.599 INFO:teuthology.orchestra.run.smithi204.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 573 kB/s | 14 kB 00:00 2024-09-11T00:28:19.622 INFO:teuthology.orchestra.run.smithi031.stdout:(129/157): python3-kubernetes-26.1.0-2.el9.noar 18 MB/s | 1.0 MB 00:00 2024-09-11T00:28:19.633 INFO:teuthology.orchestra.run.smithi204.stdout:(118/157): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-09-11T00:28:19.658 INFO:teuthology.orchestra.run.smithi204.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-09-11T00:28:19.668 INFO:teuthology.orchestra.run.smithi031.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.0 MB/s | 46 kB 00:00 2024-09-11T00:28:19.692 INFO:teuthology.orchestra.run.smithi204.stdout:(120/157): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-09-11T00:28:19.713 INFO:teuthology.orchestra.run.smithi031.stdout:(131/157): python3-grpcio-1.46.7-10.el9.x86_64. 6.1 MB/s | 2.0 MB 00:00 2024-09-11T00:28:19.717 INFO:teuthology.orchestra.run.smithi204.stdout:(121/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-09-11T00:28:19.738 INFO:teuthology.orchestra.run.smithi031.stdout:(132/157): python3-more-itertools-8.12.0-2.el9. 1.1 MB/s | 79 kB 00:00 2024-09-11T00:28:19.764 INFO:teuthology.orchestra.run.smithi031.stdout:(133/157): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-09-11T00:28:19.784 INFO:teuthology.orchestra.run.smithi204.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 2.4 MB/s | 220 kB 00:00 2024-09-11T00:28:19.789 INFO:teuthology.orchestra.run.smithi031.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 662 kB/s | 16 kB 00:00 2024-09-11T00:28:19.814 INFO:teuthology.orchestra.run.smithi031.stdout:(135/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-11T00:28:19.818 INFO:teuthology.orchestra.run.smithi204.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-09-11T00:28:19.843 INFO:teuthology.orchestra.run.smithi204.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-09-11T00:28:19.869 INFO:teuthology.orchestra.run.smithi204.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 704 kB/s | 18 kB 00:00 2024-09-11T00:28:19.881 INFO:teuthology.orchestra.run.smithi031.stdout:(136/157): libarrow-9.0.0-11.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-09-11T00:28:19.894 INFO:teuthology.orchestra.run.smithi204.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 932 kB/s | 23 kB 00:00 2024-09-11T00:28:19.898 INFO:teuthology.orchestra.run.smithi031.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 339 kB/s | 28 kB 00:00 2024-09-11T00:28:19.919 INFO:teuthology.orchestra.run.smithi204.stdout:(127/157): python3-jaraco-functools-3.5.0-2.el9 782 kB/s | 19 kB 00:00 2024-09-11T00:28:19.923 INFO:teuthology.orchestra.run.smithi031.stdout:(138/157): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-09-11T00:28:19.940 INFO:teuthology.orchestra.run.smithi031.stdout:(139/157): python3-repoze-lru-0.7-16.el9.noarch 525 kB/s | 31 kB 00:00 2024-09-11T00:28:19.944 INFO:teuthology.orchestra.run.smithi204.stdout:(128/157): python3-jaraco-text-3.2.0-6.el9.noar 791 kB/s | 20 kB 00:00 2024-09-11T00:28:19.965 INFO:teuthology.orchestra.run.smithi031.stdout:(140/157): python3-rsa-4.9-2.el9.noarch.rpm 2.4 MB/s | 59 kB 00:00 2024-09-11T00:28:19.982 INFO:teuthology.orchestra.run.smithi031.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 863 kB/s | 36 kB 00:00 2024-09-11T00:28:20.003 INFO:teuthology.orchestra.run.smithi204.stdout:(129/157): python3-grpcio-1.46.7-10.el9.x86_64. 7.1 MB/s | 2.0 MB 00:00 2024-09-11T00:28:20.007 INFO:teuthology.orchestra.run.smithi031.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.7 MB/s | 74 kB 00:00 2024-09-11T00:28:20.033 INFO:teuthology.orchestra.run.smithi031.stdout:(143/157): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-09-11T00:28:20.037 INFO:teuthology.orchestra.run.smithi204.stdout:(130/157): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-09-11T00:28:20.058 INFO:teuthology.orchestra.run.smithi031.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-11T00:28:20.062 INFO:teuthology.orchestra.run.smithi204.stdout:(131/157): python3-logutils-0.3.5-21.el9.noarch 794 kB/s | 46 kB 00:00 2024-09-11T00:28:20.083 INFO:teuthology.orchestra.run.smithi031.stdout:(145/157): python3-zc-lockfile-2.0-10.el9.noarc 812 kB/s | 20 kB 00:00 2024-09-11T00:28:20.087 INFO:teuthology.orchestra.run.smithi204.stdout:(132/157): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-09-11T00:28:20.109 INFO:teuthology.orchestra.run.smithi031.stdout:(146/157): python3-webob-1.8.8-2.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-09-11T00:28:20.113 INFO:teuthology.orchestra.run.smithi204.stdout:(133/157): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-09-11T00:28:20.163 INFO:teuthology.orchestra.run.smithi204.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-09-11T00:28:20.167 INFO:teuthology.orchestra.run.smithi031.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 2.2 MB/s | 191 kB 00:00 2024-09-11T00:28:20.188 INFO:teuthology.orchestra.run.smithi204.stdout:(135/157): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-09-11T00:28:20.192 INFO:teuthology.orchestra.run.smithi031.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-09-11T00:28:20.213 INFO:teuthology.orchestra.run.smithi204.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-09-11T00:28:20.239 INFO:teuthology.orchestra.run.smithi204.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-09-11T00:28:20.251 INFO:teuthology.orchestra.run.smithi031.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-11T00:28:20.264 INFO:teuthology.orchestra.run.smithi204.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 610 kB/s | 31 kB 00:00 2024-09-11T00:28:20.289 INFO:teuthology.orchestra.run.smithi204.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.4 MB/s | 59 kB 00:00 2024-09-11T00:28:20.315 INFO:teuthology.orchestra.run.smithi204.stdout:(140/157): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-09-11T00:28:20.340 INFO:teuthology.orchestra.run.smithi204.stdout:(141/157): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-09-11T00:28:20.365 INFO:teuthology.orchestra.run.smithi204.stdout:(142/157): python3-typing-extensions-4.12.2-2.e 1.5 MB/s | 74 kB 00:00 2024-09-11T00:28:20.368 INFO:teuthology.orchestra.run.smithi031.stdout:(150/157): NetworkManager-1.49.90-1.el9.x86_64. 20 MB/s | 2.3 MB 00:00 2024-09-11T00:28:20.391 INFO:teuthology.orchestra.run.smithi204.stdout:(143/157): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-11T00:28:20.424 INFO:teuthology.orchestra.run.smithi204.stdout:(144/157): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-11T00:28:20.450 INFO:teuthology.orchestra.run.smithi204.stdout:(145/157): python3-zc-lockfile-2.0-10.el9.noarc 803 kB/s | 20 kB 00:00 2024-09-11T00:28:20.460 INFO:teuthology.orchestra.run.smithi031.stdout:(151/157): NetworkManager-libnm-1.49.90-1.el9.x 20 MB/s | 1.8 MB 00:00 2024-09-11T00:28:20.483 INFO:teuthology.orchestra.run.smithi204.stdout:(146/157): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-11T00:28:20.519 INFO:teuthology.orchestra.run.smithi031.stdout:(152/157): librbd1-19.3.0-4854.g8f219c47.el9.x8 10 MB/s | 3.2 MB 00:00 2024-09-11T00:28:20.542 INFO:teuthology.orchestra.run.smithi204.stdout:(147/157): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-09-11T00:28:20.544 INFO:teuthology.orchestra.run.smithi031.stdout:(153/157): NetworkManager-team-1.49.90-1.el9.x8 489 kB/s | 41 kB 00:00 2024-09-11T00:28:20.601 INFO:teuthology.orchestra.run.smithi204.stdout:(148/157): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-11T00:28:20.627 INFO:teuthology.orchestra.run.smithi031.stdout:(154/157): librados2-19.3.0-4854.g8f219c47.el9. 7.5 MB/s | 3.5 MB 00:00 2024-09-11T00:28:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:(155/157): NetworkManager-tui-1.49.90-1.el9.x86 1.8 MB/s | 249 kB 00:00 2024-09-11T00:28:20.695 INFO:teuthology.orchestra.run.smithi031.stdout:(156/157): openssl-3.2.2-6.el9.x86_64.rpm 9.0 MB/s | 1.4 MB 00:00 2024-09-11T00:28:20.793 INFO:teuthology.orchestra.run.smithi204.stdout:(149/157): librados2-19.3.0-4854.g8f219c47.el9. 14 MB/s | 3.5 MB 00:00 2024-09-11T00:28:20.820 INFO:teuthology.orchestra.run.smithi031.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-09-11T00:28:20.826 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-09-11T00:28:20.826 INFO:teuthology.orchestra.run.smithi031.stdout:Total 15 MB/s | 240 MB 00:16 2024-09-11T00:28:20.910 INFO:teuthology.orchestra.run.smithi204.stdout:(150/157): librbd1-19.3.0-4854.g8f219c47.el9.x8 11 MB/s | 3.2 MB 00:00 2024-09-11T00:28:21.085 INFO:teuthology.orchestra.run.smithi204.stdout:(151/157): NetworkManager-1.49.90-1.el9.x86_64. 7.9 MB/s | 2.3 MB 00:00 2024-09-11T00:28:21.143 INFO:teuthology.orchestra.run.smithi204.stdout:(152/157): NetworkManager-team-1.49.90-1.el9.x8 703 kB/s | 41 kB 00:00 2024-09-11T00:28:21.203 INFO:teuthology.orchestra.run.smithi204.stdout:(153/157): NetworkManager-tui-1.49.90-1.el9.x86 4.2 MB/s | 249 kB 00:00 2024-09-11T00:28:21.280 INFO:teuthology.orchestra.run.smithi204.stdout:(154/157): NetworkManager-libnm-1.49.90-1.el9.x 4.9 MB/s | 1.8 MB 00:00 2024-09-11T00:28:21.339 INFO:teuthology.orchestra.run.smithi204.stdout:(155/157): openssl-3.2.2-6.el9.x86_64.rpm 9.9 MB/s | 1.4 MB 00:00 2024-09-11T00:28:21.423 INFO:teuthology.orchestra.run.smithi204.stdout:(156/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2024-09-11T00:28:22.934 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-09-11T00:28:23.137 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-09-11T00:28:23.137 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-09-11T00:28:23.698 INFO:teuthology.orchestra.run.smithi204.stdout:(157/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.0 MB/s | 16 MB 00:05 2024-09-11T00:28:23.704 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-11T00:28:23.704 INFO:teuthology.orchestra.run.smithi204.stdout:Total 16 MB/s | 240 MB 00:15 2024-09-11T00:28:25.708 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-09-11T00:28:25.708 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-09-11T00:28:25.781 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-11T00:28:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-11T00:28:25.964 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-11T00:28:28.579 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-11T00:28:28.579 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-11T00:28:32.985 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-09-11T00:28:33.125 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-11T00:28:33.185 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-11T00:28:33.276 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-11T00:28:33.319 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-11T00:28:33.351 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-11T00:28:33.424 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-11T00:28:33.834 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-11T00:28:33.942 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-11T00:28:34.091 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-11T00:28:34.135 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-11T00:28:34.249 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-11T00:28:34.285 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 10/165 2024-09-11T00:28:34.323 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-11T00:28:34.563 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-11T00:28:34.631 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-11T00:28:34.656 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-11T00:28:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-11T00:28:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 15/165 2024-09-11T00:28:34.898 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 15/165 2024-09-11T00:28:34.925 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-11T00:28:34.971 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-11T00:28:35.193 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-11T00:28:35.365 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-11T00:28:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-11T00:28:35.550 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 18/165 2024-09-11T00:28:35.577 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-11T00:28:35.634 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-11T00:28:35.686 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-11T00:28:35.727 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-11T00:28:35.783 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-11T00:28:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-11T00:28:35.978 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-11T00:28:36.070 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-11T00:28:36.111 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-11T00:28:36.263 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-11T00:28:36.297 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-11T00:28:36.342 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-11T00:28:36.426 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-11T00:28:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-11T00:28:36.504 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-11T00:28:36.550 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-11T00:28:36.571 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-11T00:28:36.605 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-11T00:28:36.643 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-11T00:28:36.676 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-11T00:28:36.749 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-11T00:28:36.769 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-11T00:28:36.780 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-11T00:28:36.847 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-11T00:28:36.881 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-11T00:28:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-11T00:28:36.927 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-11T00:28:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 10/165 2024-09-11T00:28:37.010 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-11T00:28:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-11T00:28:37.067 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9. 34/165 2024-09-11T00:28:37.111 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el 35/165 2024-09-11T00:28:37.139 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_6 36/165 2024-09-11T00:28:37.236 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-11T00:28:37.251 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-11T00:28:37.301 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-11T00:28:37.318 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-11T00:28:37.344 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-11T00:28:37.404 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-11T00:28:37.505 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-11T00:28:37.537 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 15/165 2024-09-11T00:28:37.592 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-11T00:28:37.644 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 15/165 2024-09-11T00:28:37.655 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-11T00:28:37.671 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-11T00:28:37.710 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-11T00:28:37.948 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-11T00:28:38.111 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-11T00:28:38.317 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 18/165 2024-09-11T00:28:38.333 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-11T00:28:38.439 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-11T00:28:38.442 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-11T00:28:38.472 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-11T00:28:38.517 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-11T00:28:38.555 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_6 46/165 2024-09-11T00:28:38.573 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-11T00:28:38.751 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-11T00:28:38.852 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 19/165 2024-09-11T00:28:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-11T00:28:39.000 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-11T00:28:39.058 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-11T00:28:39.070 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-11T00:28:39.114 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-11T00:28:39.179 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-11T00:28:39.197 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-11T00:28:39.250 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-11T00:28:39.253 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-11T00:28:39.297 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-11T00:28:39.351 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-11T00:28:39.358 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-11T00:28:39.390 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-11T00:28:39.401 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-11T00:28:39.438 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-11T00:28:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-11T00:28:39.497 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-11T00:28:39.537 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-11T00:28:39.538 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-11T00:28:39.583 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-11T00:28:39.610 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-11T00:28:39.637 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-11T00:28:39.644 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-11T00:28:39.691 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-11T00:28:39.780 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-11T00:28:39.819 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-11T00:28:39.824 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9. 34/165 2024-09-11T00:28:39.866 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el 35/165 2024-09-11T00:28:39.868 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-11T00:28:39.894 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_6 36/165 2024-09-11T00:28:39.912 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-11T00:28:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-11T00:28:40.038 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-11T00:28:40.061 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-11T00:28:40.110 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-11T00:28:40.129 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-11T00:28:40.152 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-11T00:28:40.207 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/165 2024-09-11T00:28:40.231 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-11T00:28:40.312 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-11T00:28:40.365 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-11T00:28:40.375 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-11T00:28:40.535 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-11T00:28:40.581 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-11T00:28:40.618 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-11T00:28:40.703 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-11T00:28:40.751 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-11T00:28:40.799 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-11T00:28:40.836 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-11T00:28:40.870 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-11T00:28:40.903 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-11T00:28:40.948 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-11T00:28:40.980 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-11T00:28:41.024 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-11T00:28:41.084 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-11T00:28:41.142 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-11T00:28:41.165 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-11T00:28:41.195 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-11T00:28:41.218 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-11T00:28:41.239 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-11T00:28:41.303 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_6 46/165 2024-09-11T00:28:41.353 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-11T00:28:41.355 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-11T00:28:41.431 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-11T00:28:41.544 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-11T00:28:41.614 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-11T00:28:41.745 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-11T00:28:41.817 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-11T00:28:41.901 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-11T00:28:41.949 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-11T00:28:41.985 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-11T00:28:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-11T00:28:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-11T00:28:42.156 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-11T00:28:42.205 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-11T00:28:42.373 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-11T00:28:42.378 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-11T00:28:42.432 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-11T00:28:42.432 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-11T00:28:42.466 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-11T00:28:42.473 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-11T00:28:42.587 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-11T00:28:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-11T00:28:42.631 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-11T00:28:42.658 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-11T00:28:42.687 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-11T00:28:42.711 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-11T00:28:42.719 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/165 2024-09-11T00:28:42.769 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-11T00:28:42.812 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-11T00:28:42.860 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-11T00:28:42.888 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-11T00:28:43.057 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-11T00:28:43.098 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-11T00:28:43.120 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-11T00:28:43.151 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-11T00:28:43.156 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-11T00:28:43.193 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-11T00:28:43.234 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-11T00:28:43.254 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-11T00:28:43.274 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-11T00:28:43.311 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-11T00:28:43.346 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-11T00:28:43.379 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-11T00:28:43.422 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-11T00:28:43.455 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-11T00:28:43.578 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-11T00:28:43.647 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-11T00:28:43.684 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-11T00:28:43.804 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-11T00:28:43.872 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-11T00:28:44.116 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-11T00:28:44.268 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-11T00:28:44.756 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-11T00:28:44.815 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-11T00:28:44.849 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-11T00:28:44.964 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-11T00:28:45.035 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-11T00:28:45.089 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-11T00:28:45.184 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-11T00:28:45.230 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-11T00:28:45.465 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-11T00:28:45.502 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-11T00:28:45.598 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-11T00:28:45.962 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-11T00:28:46.060 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-11T00:28:46.096 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-11T00:28:46.129 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-11T00:28:46.400 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-11T00:28:46.509 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-11T00:28:46.568 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-11T00:28:46.599 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 102/165 2024-09-11T00:28:46.712 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 102/165 2024-09-11T00:28:46.747 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 103/165 2024-09-11T00:28:46.804 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-11T00:28:47.386 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-11T00:28:47.418 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 106/165 2024-09-11T00:28:47.510 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 106/165 2024-09-11T00:28:47.545 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 107/165 2024-09-11T00:28:48.141 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-11T00:28:48.242 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-11T00:28:48.310 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-11T00:28:48.345 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-11T00:28:48.695 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-11T00:28:48.756 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 102/165 2024-09-11T00:28:48.869 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 102/165 2024-09-11T00:28:48.904 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 103/165 2024-09-11T00:28:48.961 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-11T00:28:49.497 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-11T00:28:49.534 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 106/165 2024-09-11T00:28:49.608 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 106/165 2024-09-11T00:28:49.637 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 107/165 2024-09-11T00:28:50.265 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-11T00:28:50.308 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-11T00:28:50.981 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-11T00:28:51.036 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 109/165 2024-09-11T00:28:51.184 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 109/165 2024-09-11T00:28:51.184 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T00:28:51.184 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T00:28:51.184 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:28:51.237 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 110/165 2024-09-11T00:28:52.205 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-11T00:28:52.242 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-11T00:28:52.885 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 108/165 2024-09-11T00:28:52.928 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 109/165 2024-09-11T00:28:53.057 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 109/165 2024-09-11T00:28:53.057 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T00:28:53.058 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T00:28:53.058 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:28:53.121 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 110/165 2024-09-11T00:29:04.445 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 110/165 2024-09-11T00:29:04.445 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-09-11T00:29:04.446 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-09-11T00:29:04.446 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-09-11T00:29:04.446 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-09-11T00:29:04.446 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-09-11T00:29:04.446 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-09-11T00:29:04.446 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-09-11T00:29:04.446 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:04.655 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 111/165 2024-09-11T00:29:04.692 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 111/165 2024-09-11T00:29:04.692 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:04.692 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T00:29:04.692 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:05.713 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 110/165 2024-09-11T00:29:05.713 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-09-11T00:29:05.713 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-09-11T00:29:05.713 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-09-11T00:29:05.713 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-09-11T00:29:05.713 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-09-11T00:29:05.713 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-09-11T00:29:05.714 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-09-11T00:29:05.714 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 111/165 2024-09-11T00:29:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 111/165 2024-09-11T00:29:05.966 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:05.966 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T00:29:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 112/165 2024-09-11T00:29:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 112/165 2024-09-11T00:29:06.443 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:06.443 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T00:29:06.443 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T00:29:06.443 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T00:29:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:06.740 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 113/165 2024-09-11T00:29:06.792 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 113/165 2024-09-11T00:29:06.792 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:06.792 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T00:29:06.792 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T00:29:06.792 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T00:29:06.792 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:07.317 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 114/165 2024-09-11T00:29:07.369 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 114/165 2024-09-11T00:29:07.369 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:07.369 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T00:29:07.369 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T00:29:07.369 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T00:29:07.369 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:07.510 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-11T00:29:07.571 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-11T00:29:07.613 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 112/165 2024-09-11T00:29:07.644 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-11T00:29:07.671 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 112/165 2024-09-11T00:29:07.671 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:07.671 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T00:29:07.671 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T00:29:07.672 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T00:29:07.672 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:07.898 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-11T00:29:07.914 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 113/165 2024-09-11T00:29:07.963 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-11T00:29:07.965 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 113/165 2024-09-11T00:29:07.965 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:07.965 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T00:29:07.965 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T00:29:07.966 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T00:29:07.966 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:08.052 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-11T00:29:08.163 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-11T00:29:08.350 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-11T00:29:08.412 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-11T00:29:08.452 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-11T00:29:08.473 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 114/165 2024-09-11T00:29:08.477 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-11T00:29:08.524 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 114/165 2024-09-11T00:29:08.524 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:08.524 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T00:29:08.524 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T00:29:08.524 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T00:29:08.524 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:08.649 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-11T00:29:08.742 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-11T00:29:08.806 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-11T00:29:09.034 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-11T00:29:09.054 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 126/165 2024-09-11T00:29:09.094 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 126/165 2024-09-11T00:29:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-11T00:29:09.128 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.e 127/165 2024-09-11T00:29:09.189 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-11T00:29:09.212 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47. 128/165 2024-09-11T00:29:09.241 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 129/165 2024-09-11T00:29:09.314 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-11T00:29:09.473 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-11T00:29:09.533 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-11T00:29:09.575 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-11T00:29:09.601 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-11T00:29:10.143 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 126/165 2024-09-11T00:29:10.209 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 126/165 2024-09-11T00:29:10.251 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.e 127/165 2024-09-11T00:29:10.331 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47. 128/165 2024-09-11T00:29:10.365 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 129/165 2024-09-11T00:29:13.115 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 129/165 2024-09-11T00:29:13.180 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 130/165 2024-09-11T00:29:14.018 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 129/165 2024-09-11T00:29:14.087 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 130/165 2024-09-11T00:29:14.337 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 130/165 2024-09-11T00:29:14.363 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 131/165 2024-09-11T00:29:14.522 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 131/165 2024-09-11T00:29:14.704 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el 132/165 2024-09-11T00:29:14.740 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 133/165 2024-09-11T00:29:14.793 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 133/165 2024-09-11T00:29:14.793 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:14.794 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T00:29:14.794 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T00:29:14.794 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T00:29:14.794 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:14.849 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 134/165 2024-09-11T00:29:14.876 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 134/165 2024-09-11T00:29:14.997 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 135/165 2024-09-11T00:29:15.069 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 136/165 2024-09-11T00:29:15.180 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 130/165 2024-09-11T00:29:15.204 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 131/165 2024-09-11T00:29:15.358 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 131/165 2024-09-11T00:29:15.536 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el 132/165 2024-09-11T00:29:15.573 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 133/165 2024-09-11T00:29:15.625 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 133/165 2024-09-11T00:29:15.626 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:15.626 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T00:29:15.626 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T00:29:15.626 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T00:29:15.626 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:15.704 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 134/165 2024-09-11T00:29:15.730 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 134/165 2024-09-11T00:29:15.851 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 135/165 2024-09-11T00:29:15.916 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 136/165 2024-09-11T00:29:16.196 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-11T00:29:16.227 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 138/165 2024-09-11T00:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 138/165 2024-09-11T00:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T00:29:16.279 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T00:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T00:29:16.280 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:16.327 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4854.g8f219 139/165 2024-09-11T00:29:16.378 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4854.g8f219 139/165 2024-09-11T00:29:16.379 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:16.379 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T00:29:16.379 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:16.428 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 140/165 2024-09-11T00:29:16.481 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 140/165 2024-09-11T00:29:16.481 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:16.481 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-11T00:29:16.481 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-11T00:29:16.481 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-11T00:29:16.482 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:16.837 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 141/165 2024-09-11T00:29:16.889 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 141/165 2024-09-11T00:29:16.889 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:16.890 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T00:29:16.890 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T00:29:16.890 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T00:29:16.890 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:16.990 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-11T00:29:17.027 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 138/165 2024-09-11T00:29:17.078 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 138/165 2024-09-11T00:29:17.078 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:17.078 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T00:29:17.078 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T00:29:17.078 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T00:29:17.078 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:17.127 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4854.g8f219 139/165 2024-09-11T00:29:17.177 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4854.g8f219 139/165 2024-09-11T00:29:17.178 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:17.178 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T00:29:17.178 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:17.237 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 140/165 2024-09-11T00:29:17.287 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 140/165 2024-09-11T00:29:17.287 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:17.287 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-11T00:29:17.287 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-11T00:29:17.287 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-11T00:29:17.287 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:17.629 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 141/165 2024-09-11T00:29:17.680 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 141/165 2024-09-11T00:29:17.680 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T00:29:17.680 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T00:29:17.680 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T00:29:17.680 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T00:29:17.680 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:22.327 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 142/165 2024-09-11T00:29:22.376 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 143/165 2024-09-11T00:29:22.429 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 144/165 2024-09-11T00:29:22.465 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-11T00:29:22.522 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-11T00:29:22.782 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 147/165 2024-09-11T00:29:22.811 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 142/165 2024-09-11T00:29:22.839 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-11T00:29:22.851 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 143/165 2024-09-11T00:29:22.869 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_ 149/165 2024-09-11T00:29:22.895 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 144/165 2024-09-11T00:29:22.909 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 150/165 2024-09-11T00:29:22.923 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-11T00:29:22.959 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-11T00:29:22.977 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 151/165 2024-09-11T00:29:23.199 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 147/165 2024-09-11T00:29:23.256 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-11T00:29:23.286 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_ 149/165 2024-09-11T00:29:23.317 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 150/165 2024-09-11T00:29:23.362 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 151/165 2024-09-11T00:29:23.748 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-11T00:29:23.808 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-11T00:29:23.850 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-11T00:29:24.335 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-11T00:29:24.377 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-11T00:29:24.419 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-11T00:29:25.232 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 153/165 2024-09-11T00:29:25.371 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-11T00:29:25.434 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-11T00:29:25.547 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-11T00:29:25.582 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-11T00:29:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-11T00:29:25.661 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-11T00:29:25.661 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-11T00:29:25.693 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 153/165 2024-09-11T00:29:25.791 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-11T00:29:25.801 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-11T00:29:25.802 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-11T00:29:25.843 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-11T00:29:25.911 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-11T00:29:25.935 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-11T00:29:25.951 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-11T00:29:25.952 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-11T00:29:25.959 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-11T00:29:25.994 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-11T00:29:26.052 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-11T00:29:26.089 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-11T00:29:26.089 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-11T00:29:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-11T00:29:26.204 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-11T00:29:26.205 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-11T00:29:26.205 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-11T00:29:26.313 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-11T00:29:26.330 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-11T00:29:26.337 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-11T00:29:26.353 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-11T00:29:26.353 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-11T00:29:26.407 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-11T00:29:26.599 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-11T00:29:26.599 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-11T00:29:26.616 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-11T00:29:26.617 INFO:teuthology.orchestra.run.smithi031.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-11T00:29:26.617 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:26.783 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-11T00:29:26.857 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-11T00:29:27.100 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-11T00:29:27.100 INFO:teuthology.orchestra.run.smithi204.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-11T00:29:27.104 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:32.797 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 1/165 2024-09-11T00:29:32.797 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2/165 2024-09-11T00:29:32.797 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 3/165 2024-09-11T00:29:32.797 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 4/165 2024-09-11T00:29:32.797 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4854.g8f219 5/165 2024-09-11T00:29:32.797 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 6/165 2024-09-11T00:29:32.797 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 7/165 2024-09-11T00:29:32.797 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 10/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 11/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 12/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 13/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_ 15/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 16/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 17/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_6 18/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 19/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 20/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el 21/165 2024-09-11T00:29:32.798 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9. 22/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_6 23/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 24/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 25/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 26/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 27/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 28/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 29/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47. 30/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 31/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 32/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 33/165 2024-09-11T00:29:32.799 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el 34/165 2024-09-11T00:29:32.800 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 35/165 2024-09-11T00:29:32.800 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.e 36/165 2024-09-11T00:29:32.800 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 37/165 2024-09-11T00:29:32.800 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 38/165 2024-09-11T00:29:32.800 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 39/165 2024-09-11T00:29:32.800 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-11T00:29:32.800 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-11T00:29:32.800 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-11T00:29:32.800 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-11T00:29:32.801 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-11T00:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-11T00:29:32.803 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-11T00:29:32.804 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-11T00:29:32.805 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-11T00:29:32.806 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-11T00:29:32.807 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-11T00:29:32.808 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 150/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 152/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 154/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 156/165 2024-09-11T00:29:32.809 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-11T00:29:32.810 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 158/165 2024-09-11T00:29:32.810 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-11T00:29:32.810 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 160/165 2024-09-11T00:29:32.810 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-11T00:29:32.810 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-11T00:29:32.810 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-11T00:29:32.810 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-11T00:29:33.575 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 1/165 2024-09-11T00:29:33.575 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2/165 2024-09-11T00:29:33.575 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 3/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 4/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4854.g8f219 5/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 6/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 7/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 8/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 9/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 10/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 11/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 12/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 13/165 2024-09-11T00:29:33.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 14/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_ 15/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 16/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 17/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_6 18/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86 19/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 20/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el 21/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:19.3.0-4854.g8f219c47.el9. 22/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_6 23/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 24/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 25/165 2024-09-11T00:29:33.577 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 26/165 2024-09-11T00:29:33.578 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 27/165 2024-09-11T00:29:33.589 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 28/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 29/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47. 30/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noa 31/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.n 32/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f2 33/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el 34/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 35/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.e 36/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 37/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 38/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 39/165 2024-09-11T00:29:33.590 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-11T00:29:33.591 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-11T00:29:33.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-11T00:29:33.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-11T00:29:33.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-11T00:29:33.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-11T00:29:33.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-11T00:29:33.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-11T00:29:33.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-11T00:29:33.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-11T00:29:33.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-11T00:29:33.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-11T00:29:33.593 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-11T00:29:33.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-11T00:29:33.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-11T00:29:33.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-11T00:29:33.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-11T00:29:33.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-11T00:29:33.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 150/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 152/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 154/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 156/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 158/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 160/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-11T00:29:33.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-11T00:29:33.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-11T00:29:35.045 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-11T00:29:35.045 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:35.045 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-09-11T00:29:35.045 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-11T00:29:35.045 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-11T00:29:35.045 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: automake-1.16.2-8.el9.noarch 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: bison-3.7.4-5.el9.x86_64 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.046 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.047 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-11T00:29:35.048 INFO:teuthology.orchestra.run.smithi031.stdout: flex-2.6.4-9.el9.x86_64 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.049 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T00:29:35.050 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-13.16-1.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-11T00:29:35.051 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.052 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T00:29:35.053 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T00:29:35.054 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T00:29:35.055 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T00:29:35.056 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T00:29:35.057 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T00:29:35.058 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T00:29:35.058 INFO:teuthology.orchestra.run.smithi031.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-11T00:29:35.058 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-11T00:29:35.058 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T00:29:35.058 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-11T00:29:35.058 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:35.058 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-09-11T00:29:35.748 DEBUG:teuthology.parallel:result is None 2024-09-11T00:29:35.848 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-11T00:29:35.848 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T00:29:35.849 INFO:teuthology.orchestra.run.smithi204.stdout: automake-1.16.2-8.el9.noarch 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: bison-3.7.4-5.el9.x86_64 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.850 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-shell-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.851 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-top-2:19.3.0-4854.g8f219c47.el9.noarch 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: flex-2.6.4-9.el9.x86_64 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.852 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T00:29:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-11T00:29:35.854 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-13.16-1.el9.x86_64 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T00:29:35.855 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T00:29:35.856 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T00:29:35.857 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T00:29:35.858 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T00:29:35.859 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.860 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.861 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:19.3.0-4854.g8f219c47.el9.x86_64 2024-09-11T00:29:35.861 INFO:teuthology.orchestra.run.smithi204.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T00:29:35.861 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T00:29:35.861 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T00:29:35.861 INFO:teuthology.orchestra.run.smithi204.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-11T00:29:35.861 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-11T00:29:35.861 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T00:29:35.861 INFO:teuthology.orchestra.run.smithi204.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-11T00:29:35.861 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:35.862 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-11T00:29:36.531 DEBUG:teuthology.parallel:result is None 2024-09-11T00:29:36.531 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T00:29:36.687 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T00:29:36.733 INFO:teuthology.orchestra.run.smithi031.stdout:19.3.0-4854.g8f219c47.el9 2024-09-11T00:29:36.733 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854.g8f219c47.el9 2024-09-11T00:29:36.733 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854.g8f219c47 is installed. 2024-09-11T00:29:36.735 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T00:29:36.891 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T00:29:36.937 INFO:teuthology.orchestra.run.smithi204.stdout:19.3.0-4854.g8f219c47.el9 2024-09-11T00:29:36.937 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854.g8f219c47.el9 2024-09-11T00:29:36.937 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854.g8f219c47 is installed. 2024-09-11T00:29:36.939 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T00:29:36.939 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:36.939 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T00:29:37.005 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:29:37.005 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T00:29:37.066 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T00:29:37.066 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:37.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T00:29:37.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T00:29:37.220 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:29:37.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T00:29:37.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T00:29:37.374 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T00:29:37.374 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:37.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T00:29:37.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T00:29:37.528 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:29:37.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T00:29:37.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T00:29:37.681 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T00:29:37.681 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:37.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T00:29:37.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T00:29:37.835 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:29:37.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T00:29:37.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T00:29:37.991 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T00:29:38.071 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T00:29:38.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2024-09-11T00:29:38.075 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2024-09-11T00:29:38.137 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T00:29:38.138 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T00:29:38.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T00:29:38.199 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T00:29:38.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T00:29:38.203 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T00:29:38.269 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-11T00:29:38.270 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-11T00:29:38.270 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-11T00:29:38.270 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:38.270 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-11T00:29:38.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-11T00:29:38.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-11T00:29:38.535 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:29:38.535 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-11T00:29:38.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-11T00:29:38.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-11T00:29:38.812 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T00:29:38.812 INFO:tasks.ceph:config {'conf': {'client': {'client_shutdown_timeout': 120}, 'mds': {'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T00:29:38.812 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843', 'branch': 'wip-hemanth-testing-2024-09-10-0723', 'description': 'fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}', 'email': 'hyelloji@redhat.com', 'first_in_suite': False, 'job_id': '7899843', 'kernel': {'flavor': 'default', 'kdb': 1, 'sha1': '2ca44a2f35ca8f36983aaaa46d49312239ef519a'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-hemanth-testing-2024-09-10-0723'}, 'ceph': {'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'client_shutdown_timeout': 120}, 'mds': {'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'kclient': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-hemanth-testing-2024-09-10-0723', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'owner': 'scheduled_hyelloji@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 6190, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'sleep_before_teardown': 0, 'subset': '479/1024', 'suite': 'fs', 'suite_branch': 'wip-hemanth-testing-2024-09-10-0723', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBprUoQScclASkFC6iEkUBG1EHV0lJpvsJlZKHhC+hkgTQsfg023d0MAvkMHcYPRBgZwQuqknu1gJtqMES3xYK0=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIu3hLTpBMeScGb/BDZb55wXviStnce68tS81q0/caQp/E9i13Q3CTmPYTI8tniSmGlJiTTfiHDg9trG2roRmDk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': 1, 'sha1': '2ca44a2f35ca8f36983aaaa46d49312239ef519a'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'client_shutdown_timeout': 120}, 'mds': {'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_16:59:47', 'tube': 'smithi', 'user': 'hyelloji', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T00:29:38.813 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T00:29:38.817 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T00:29:38.852 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T00:29:38.893 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T00:29:38.952 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:38.952 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T00:29:38.987 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T00:29:38.987 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-09-11T00:29:39.063 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T00:29:39.063 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T00:29:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-09-11T00:29:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T00:29:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T00:29:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-11 00:29:26.987693024 +0000 2024-09-11T00:29:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-11 00:27:17.673729674 +0000 2024-09-11T00:29:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-11 00:27:17.673729674 +0000 2024-09-11T00:29:39.064 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-11 00:27:17.673729674 +0000 2024-09-11T00:29:39.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T00:29:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-11T00:29:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-11T00:29:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000275448 s, 1.9 MB/s 2024-09-11T00:29:39.158 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T00:29:39.195 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-09-11T00:29:39.271 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T00:29:39.271 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T00:29:39.272 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-09-11T00:29:39.272 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T00:29:39.272 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T00:29:39.272 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-11 00:29:26.987693024 +0000 2024-09-11T00:29:39.272 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-11 00:27:18.139721967 +0000 2024-09-11T00:29:39.272 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-11 00:27:18.139721967 +0000 2024-09-11T00:29:39.272 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-11 00:27:18.139721967 +0000 2024-09-11T00:29:39.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T00:29:39.364 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-11T00:29:39.364 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-11T00:29:39.364 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000260193 s, 2.0 MB/s 2024-09-11T00:29:39.367 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T00:29:39.403 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-09-11T00:29:39.478 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T00:29:39.479 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T00:29:39.479 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-09-11T00:29:39.479 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T00:29:39.479 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T00:29:39.479 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-11 00:29:26.988693008 +0000 2024-09-11T00:29:39.479 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-11 00:27:18.608714210 +0000 2024-09-11T00:29:39.479 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-11 00:27:18.608714210 +0000 2024-09-11T00:29:39.479 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-11 00:27:18.608714210 +0000 2024-09-11T00:29:39.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T00:29:39.571 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-11T00:29:39.572 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-11T00:29:39.572 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000258586 s, 2.0 MB/s 2024-09-11T00:29:39.574 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T00:29:39.611 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-09-11T00:29:39.687 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T00:29:39.687 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T00:29:39.687 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-09-11T00:29:39.687 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T00:29:39.687 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T00:29:39.687 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-09-11 00:29:26.989692992 +0000 2024-09-11T00:29:39.688 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-11 00:27:19.076706469 +0000 2024-09-11T00:29:39.688 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-11 00:27:19.076706469 +0000 2024-09-11T00:29:39.688 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-09-11 00:27:19.076706469 +0000 2024-09-11T00:29:39.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T00:29:39.781 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-09-11T00:29:39.781 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-09-11T00:29:39.782 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000263438 s, 1.9 MB/s 2024-09-11T00:29:39.784 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T00:29:39.820 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T00:29:39.820 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:29:39.821 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T00:29:39.856 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T00:29:39.856 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-09-11T00:29:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T00:29:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T00:29:39.932 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 638 Links: 1 2024-09-11T00:29:39.932 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T00:29:39.932 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T00:29:39.932 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-11 00:29:27.462252999 +0000 2024-09-11T00:29:39.932 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-11 00:27:15.800572005 +0000 2024-09-11T00:29:39.932 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-11 00:27:15.800572005 +0000 2024-09-11T00:29:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-11 00:27:15.800572005 +0000 2024-09-11T00:29:39.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T00:29:40.024 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-11T00:29:40.024 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-11T00:29:40.025 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.00025704 s, 2.0 MB/s 2024-09-11T00:29:40.027 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T00:29:40.064 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-09-11T00:29:40.140 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T00:29:40.140 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T00:29:40.140 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 657 Links: 1 2024-09-11T00:29:40.140 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T00:29:40.140 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T00:29:40.140 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-11 00:29:27.463252984 +0000 2024-09-11T00:29:40.140 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-11 00:27:16.272565160 +0000 2024-09-11T00:29:40.140 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-11 00:27:16.272565160 +0000 2024-09-11T00:29:40.140 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-11 00:27:16.272565160 +0000 2024-09-11T00:29:40.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T00:29:40.232 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-11T00:29:40.233 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-11T00:29:40.233 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000275132 s, 1.9 MB/s 2024-09-11T00:29:40.235 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T00:29:40.271 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-09-11T00:29:40.346 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T00:29:40.347 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T00:29:40.347 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 676 Links: 1 2024-09-11T00:29:40.347 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T00:29:40.347 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T00:29:40.347 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-11 00:29:27.464252969 +0000 2024-09-11T00:29:40.347 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-11 00:27:16.730558517 +0000 2024-09-11T00:29:40.347 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-11 00:27:16.730558517 +0000 2024-09-11T00:29:40.347 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-11 00:27:16.730558517 +0000 2024-09-11T00:29:40.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T00:29:40.439 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-11T00:29:40.439 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-11T00:29:40.439 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000296797 s, 1.7 MB/s 2024-09-11T00:29:40.441 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T00:29:40.478 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-09-11T00:29:40.553 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T00:29:40.553 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T00:29:40.553 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 696 Links: 1 2024-09-11T00:29:40.553 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T00:29:40.554 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T00:29:40.554 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-11 00:29:27.465252954 +0000 2024-09-11T00:29:40.554 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-11 00:27:17.206551613 +0000 2024-09-11T00:29:40.554 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-11 00:27:17.207551599 +0000 2024-09-11T00:29:40.554 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-11 00:27:17.206551613 +0000 2024-09-11T00:29:40.554 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T00:29:40.645 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-11T00:29:40.646 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-11T00:29:40.646 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000302051 s, 1.7 MB/s 2024-09-11T00:29:40.648 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T00:29:40.684 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-11T00:29:40.684 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-11T00:29:40.684 INFO:tasks.ceph:Generating config... 2024-09-11T00:29:40.686 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-09-11T00:29:40.686 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2024-09-11T00:29:40.686 INFO:tasks.ceph:[mds] ms inject delay max = 1 2024-09-11T00:29:40.686 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2024-09-11T00:29:40.686 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2024-09-11T00:29:40.686 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2024-09-11T00:29:40.686 INFO:tasks.ceph:[mgr] client mount timeout = 30 2024-09-11T00:29:40.686 INFO:tasks.ceph:[mgr] debug client = 20 2024-09-11T00:29:40.687 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T00:29:40.687 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T00:29:40.687 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T00:29:40.687 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T00:29:40.687 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T00:29:40.687 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-11T00:29:40.687 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-11T00:29:40.687 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-11T00:29:40.687 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-11T00:29:40.687 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T00:29:40.687 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T00:29:40.687 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-11T00:29:40.687 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] filestore flush min = 0 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] ms inject delay max = 1 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2024-09-11T00:29:40.688 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2024-09-11T00:29:40.689 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-11T00:29:40.689 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T00:29:40.689 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-11T00:29:40.689 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T00:29:40.689 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T00:29:40.689 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T00:29:40.783 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T00:29:40.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T00:29:40.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T00:29:40.931 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31'), ('mon.b', '172.21.15.204'), ('mon.c', '[v2:172.21.15.204:3301,v1:172.21.15.204:6790]')] 2024-09-11T00:29:40.932 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31,172.21.15.204,[v2:172.21.15.204:3301,v1:172.21.15.204:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'client mount timeout': 30, 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client_shutdown_timeout': 120}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500}} 2024-09-11T00:29:40.933 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:40.933 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T00:29:40.969 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --add b 172.21.15.204 --addv c '[v2:172.21.15.204:3301,v1:172.21.15.204:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:29:41.087 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:29:41.087 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid 4a2c9b4f-c931-4bcb-847e-5521a1bafcf6 2024-09-11T00:29:41.087 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-09-11T00:29:41.087 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2024-09-11T00:29:41.087 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 4a2c9b4f-c931-4bcb-847e-5521a1bafcf6 2024-09-11T00:29:41.087 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2024-09-11T00:29:41.086238+0000 2024-09-11T00:29:41.087 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-09-11T00:29:41.086238+0000 2024-09-11T00:29:41.088 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 17 (quincy) 2024-09-11T00:29:41.088 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2024-09-11T00:29:41.088 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-09-11T00:29:41.088 INFO:teuthology.orchestra.run.smithi031.stdout:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2024-09-11T00:29:41.088 INFO:teuthology.orchestra.run.smithi031.stdout:2: [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] mon.c 2024-09-11T00:29:41.088 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-11T00:29:41.090 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T00:29:41.124 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4a2c9b4f-c931-4bcb-847e-5521a1bafcf6... 2024-09-11T00:29:41.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T00:29:41.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T00:29:41.262 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-09-11T00:29:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2024-09-11T00:29:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T00:29:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2024-09-11T00:29:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2024-09-11T00:29:41.262 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T00:29:41.263 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T00:29:41.264 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = "172.21.15.31,172.21.15.204,[v2:172.21.15.204:3301,v1:172.21.15.204:6790]" 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 4a2c9b4f-c931-4bcb-847e-5521a1bafcf6 2024-09-11T00:29:41.265 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2024-09-11T00:29:41.266 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug trim objects = true 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.267 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: bdev async discard = True 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: bdev enable discard = True 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore allocator = bitmap 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 1/20 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 1/20 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 4/10 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: filestore flush min = 0 2024-09-11T00:29:41.268 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay max = 1 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay probability = 0.005 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay type = client mds 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 2500 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: osd blocked scrub grace period = 3600 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: osd max markdown count = 1000 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2024-09-11T00:29:41.269 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: client mount timeout = 30 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: debug client = 20 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2024-09-11T00:29:41.270 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T00:29:41.271 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: mon down mkfs grace = 300 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2024-09-11T00:29:41.272 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: client_shutdown_timeout = 120 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.b] 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.c] 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a] 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c] 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b] 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d] 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout:[mds] 2024-09-11T00:29:41.273 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2024-09-11T00:29:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay max = 1 2024-09-11T00:29:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay probability = 0.005 2024-09-11T00:29:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject delay type = client mds osd 2024-09-11T00:29:41.274 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 2500 2024-09-11T00:29:41.274 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2024-09-11T00:29:41.275 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.276 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T00:29:41.276 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2024-09-11T00:29:41.276 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.276 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T00:29:41.276 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T00:29:41.276 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.276 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2024-09-11T00:29:41.276 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.276 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T00:29:41.277 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = "172.21.15.31,172.21.15.204,[v2:172.21.15.204:3301,v1:172.21.15.204:6790]" 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = 4a2c9b4f-c931-4bcb-847e-5521a1bafcf6 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.278 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2024-09-11T00:29:41.279 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2024-09-11T00:29:41.280 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug trim objects = true 2024-09-11T00:29:41.280 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.280 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2024-09-11T00:29:41.280 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2024-09-11T00:29:41.280 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.280 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2024-09-11T00:29:41.280 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.280 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2024-09-11T00:29:41.280 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.280 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: bdev async discard = True 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: bdev enable discard = True 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore allocator = bitmap 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore block size = 96636764160 2024-09-11T00:29:41.281 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore fsck on mount = True 2024-09-11T00:29:41.282 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluefs = 1/20 2024-09-11T00:29:41.282 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluestore = 1/20 2024-09-11T00:29:41.282 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-11T00:29:41.282 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2024-09-11T00:29:41.282 INFO:teuthology.orchestra.run.smithi204.stdout: debug rocksdb = 4/10 2024-09-11T00:29:41.282 INFO:teuthology.orchestra.run.smithi204.stdout: filestore flush min = 0 2024-09-11T00:29:41.282 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2024-09-11T00:29:41.282 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T00:29:41.282 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd full ratio = 0.9 2024-09-11T00:29:41.282 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd nearfull ratio = 0.8 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject delay max = 1 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject delay probability = 0.005 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject delay type = client mds 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 2500 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout: osd blocked scrub grace period = 3600 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout: osd failsafe full ratio = 0.95 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout: osd max markdown count = 1000 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = bluestore 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2024-09-11T00:29:41.283 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-11T00:29:41.284 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2024-09-11T00:29:41.284 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-09-11T00:29:41.284 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-09-11T00:29:41.284 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-09-11T00:29:41.284 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-09-11T00:29:41.284 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2024-09-11T00:29:41.284 INFO:teuthology.orchestra.run.smithi204.stdout: client mount timeout = 30 2024-09-11T00:29:41.284 INFO:teuthology.orchestra.run.smithi204.stdout: debug client = 20 2024-09-11T00:29:41.284 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.285 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2024-09-11T00:29:41.285 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-11T00:29:41.285 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-09-11T00:29:41.285 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2024-09-11T00:29:41.285 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-09-11T00:29:41.285 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2024-09-11T00:29:41.285 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2024-09-11T00:29:41.285 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-09-11T00:29:41.285 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2024-09-11T00:29:41.285 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2024-09-11T00:29:41.286 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-09-11T00:29:41.286 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.286 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T00:29:41.286 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T00:29:41.286 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T00:29:41.286 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.286 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T00:29:41.286 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T00:29:41.286 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T00:29:41.286 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: mon down mkfs grace = 300 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T00:29:41.287 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout: client_shutdown_timeout = 120 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.b] 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.c] 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.a] 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.c] 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.b] 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.d] 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout:[mds] 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject delay max = 1 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject delay probability = 0.005 2024-09-11T00:29:41.288 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject delay type = client mds osd 2024-09-11T00:29:41.289 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 2500 2024-09-11T00:29:41.292 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T00:29:41.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T00:29:41.393 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T00:29:41.393 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:41.393 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T00:29:41.428 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:41.428 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T00:29:41.503 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:29:41.503 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:41.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T00:29:41.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T00:29:41.621 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:29:41.621 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:29:41.696 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:29:41.696 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:29:41.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T00:29:41.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T00:29:41.775 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:29:41.775 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:29:41.851 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T00:29:41.851 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T00:29:41.851 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-11T00:29:41.956 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-11T00:29:41.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T00:29:42.078 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T00:29:42.084 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T00:29:42.085 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-11T00:29:42.189 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-11T00:29:42.194 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-11T00:29:42.251 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-11T00:29:42.355 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-11T00:29:42.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-11T00:29:42.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-11T00:29:42.521 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-11T00:29:42.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-11T00:29:42.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-11T00:29:42.686 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-11T00:29:42.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-11T00:29:42.747 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T00:29:42.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T00:29:42.829 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T00:29:42.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-11T00:29:42.939 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-11T00:29:42.967 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T00:29:42.967 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-11T00:29:42.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T00:29:43.023 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T00:29:43.023 INFO:tasks.ceph:role: osd.0 2024-09-11T00:29:43.023 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:29:43.023 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T00:29:43.086 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T00:29:43.086 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T00:29:43.086 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T00:29:43.086 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T00:29:43.086 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T00:29:43.087 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-11T00:29:43.087 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T00:29:43.087 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T00:29:43.087 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T00:29:43.087 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T00:29:43.495 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-11T00:29:43.499 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-11T00:29:43.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T00:29:43.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T00:29:43.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T00:29:43.762 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T00:29:43.762 INFO:tasks.ceph:role: osd.1 2024-09-11T00:29:43.762 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:29:43.762 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T00:29:43.825 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T00:29:43.825 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T00:29:43.825 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T00:29:43.825 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T00:29:43.825 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T00:29:43.825 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-11T00:29:43.825 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T00:29:43.825 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T00:29:43.825 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T00:29:43.825 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T00:29:44.181 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-11T00:29:44.186 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-11T00:29:44.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T00:29:44.274 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T00:29:44.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T00:29:44.391 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T00:29:44.391 INFO:tasks.ceph:role: osd.2 2024-09-11T00:29:44.391 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:29:44.391 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T00:29:44.454 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T00:29:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T00:29:44.454 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T00:29:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T00:29:44.455 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T00:29:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-11T00:29:44.455 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T00:29:44.455 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T00:29:44.455 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T00:29:44.455 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T00:29:44.849 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-11T00:29:44.853 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-11T00:29:44.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T00:29:44.949 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T00:29:45.025 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-11T00:29:45.081 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-11T00:29:45.081 INFO:tasks.ceph:role: osd.3 2024-09-11T00:29:45.081 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:29:45.081 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-11T00:29:45.144 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T00:29:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T00:29:45.144 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T00:29:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T00:29:45.145 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T00:29:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2024-09-11T00:29:45.145 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T00:29:45.145 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T00:29:45.145 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T00:29:45.145 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T00:29:45.552 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2024-09-11T00:29:45.555 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2024-09-11T00:29:45.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-11T00:29:45.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-11T00:29:45.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:29:45.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:45.858+0000 7fabb5123640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-11T00:29:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:45.862+0000 7fabb5123640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-11T00:29:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:45.862+0000 7fabb5123640 -1 bdev(0x55f031149800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-11T00:29:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:45.863+0000 7fabb5123640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-11T00:29:48.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T00:29:48.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:29:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:49.039+0000 7fcde3bc5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-11T00:29:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:49.039+0000 7fcde3bc5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-11T00:29:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:49.039+0000 7fcde3bc5640 -1 bdev(0x555ff321f800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-11T00:29:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:49.040+0000 7fcde3bc5640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-11T00:29:52.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T00:29:52.105 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:29:52.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:52.196+0000 7fae4000c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-11T00:29:52.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:52.196+0000 7fae4000c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-11T00:29:52.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:52.196+0000 7fae4000c640 -1 bdev(0x564fd45a5800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-11T00:29:52.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:52.197+0000 7fae4000c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-11T00:29:55.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T00:29:56.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:29:56.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:56.107+0000 7fa7a5212640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-11T00:29:56.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:56.107+0000 7fa7a5212640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-11T00:29:56.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:56.107+0000 7fa7a5212640 -1 bdev(0x561379f2d800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-11T00:29:56.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:29:56.108+0000 7fa7a5212640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-11T00:29:59.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-11T00:29:59.461 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-11T00:29:59.523 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-11T00:29:59.523 INFO:tasks.ceph:role: osd.4 2024-09-11T00:29:59.523 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:29:59.523 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T00:29:59.587 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T00:29:59.588 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T00:29:59.588 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T00:29:59.588 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T00:29:59.588 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T00:29:59.588 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-11T00:29:59.588 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T00:29:59.588 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T00:29:59.588 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T00:29:59.588 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T00:30:00.006 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-11T00:30:00.009 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-11T00:30:00.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-11T00:30:00.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-11T00:30:00.208 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-11T00:30:00.265 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-11T00:30:00.265 INFO:tasks.ceph:role: osd.5 2024-09-11T00:30:00.265 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:30:00.266 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T00:30:00.328 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T00:30:00.328 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T00:30:00.328 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T00:30:00.328 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T00:30:00.328 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T00:30:00.328 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-11T00:30:00.328 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T00:30:00.328 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T00:30:00.328 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T00:30:00.328 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T00:30:00.683 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-11T00:30:00.687 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-11T00:30:00.687 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-11T00:30:00.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-11T00:30:00.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-11T00:30:00.906 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-11T00:30:00.906 INFO:tasks.ceph:role: osd.6 2024-09-11T00:30:00.907 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:30:00.907 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T00:30:00.972 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T00:30:00.972 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T00:30:00.972 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T00:30:00.972 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T00:30:00.972 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T00:30:00.972 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-11T00:30:00.972 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T00:30:00.972 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T00:30:00.972 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T00:30:00.972 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T00:30:01.355 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-11T00:30:01.358 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-11T00:30:01.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-11T00:30:01.453 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-11T00:30:01.517 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-11T00:30:01.574 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-11T00:30:01.574 INFO:tasks.ceph:role: osd.7 2024-09-11T00:30:01.574 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:30:01.574 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-11T00:30:01.637 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T00:30:01.637 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T00:30:01.637 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T00:30:01.638 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T00:30:01.638 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T00:30:01.638 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-11T00:30:01.638 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T00:30:01.638 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T00:30:01.638 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T00:30:01.638 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T00:30:02.047 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-11T00:30:02.051 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-11T00:30:02.051 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-11T00:30:02.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-11T00:30:02.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:30:02.356 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:02.354+0000 7f63cf9b2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-11T00:30:02.358 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:02.357+0000 7f63cf9b2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-11T00:30:02.358 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:02.357+0000 7f63cf9b2640 -1 bdev(0x5579d8cc3800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-11T00:30:02.358 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:02.357+0000 7f63cf9b2640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-11T00:30:05.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-11T00:30:05.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:30:05.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:05.798+0000 7f51ba824640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-11T00:30:05.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:05.798+0000 7f51ba824640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-11T00:30:05.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:05.798+0000 7f51ba824640 -1 bdev(0x5602ea985800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-11T00:30:05.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:05.798+0000 7f51ba824640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-11T00:30:08.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:30:08.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:30:08.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T00:30:08.937 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:30:09.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:09.027+0000 7f552fc13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-11T00:30:09.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:09.028+0000 7f552fc13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-11T00:30:09.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:09.028+0000 7f552fc13640 -1 bdev(0x557715bd7800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-11T00:30:09.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:09.028+0000 7f552fc13640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-11T00:30:12.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-11T00:30:12.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:30:12.192 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:12.190+0000 7f4b31c0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-11T00:30:12.193 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:12.191+0000 7f4b31c0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-11T00:30:12.193 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:12.191+0000 7f4b31c0f640 -1 bdev(0x562f11405800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-11T00:30:12.193 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:12.191+0000 7f4b31c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-11T00:30:15.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-11T00:30:15.291 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T00:30:15.291 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:15.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-11T00:30:15.354 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:15.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-11T00:30:15.452 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:15.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-11T00:30:15.550 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:15.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T00:30:15.646 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:15.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T00:30:15.742 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:15.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T00:30:15.838 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:15.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-11T00:30:15.934 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:15.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T00:30:15.996 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:15.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-11T00:30:16.095 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:16.096 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-11T00:30:16.191 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:16.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-11T00:30:16.288 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:16.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-11T00:30:16.383 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:16.383 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-11T00:30:16.479 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:16.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-11T00:30:16.577 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:16.577 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T00:30:16.615 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:16.616 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-11T00:30:16.655 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T00:30:16.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T00:30:16.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T00:30:16.711 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.y] 2024-09-11T00:30:16.712 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB15OBmR0D2OBAA2LQiIesU51YcYEGi6cta1g== 2024-09-11T00:30:16.712 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a] 2024-09-11T00:30:16.712 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB25OBmYpY7CxAANfoa8HShyEnI9dD2P1rm5g== 2024-09-11T00:30:16.712 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c] 2024-09-11T00:30:16.712 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB25OBm/OIpFRAA4o5YBVAlBCiipV3KBBKXQQ== 2024-09-11T00:30:16.712 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2024-09-11T00:30:16.712 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB55OBmpKw7MxAAyrrMBsf2cqwi7RSGzjQoyg== 2024-09-11T00:30:16.712 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2024-09-11T00:30:16.712 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB95OBmsFZlAhAA38gP6TnRUk3yoWY9SfPeNA== 2024-09-11T00:30:16.712 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCA5OBmLJHACxAAZAxiXoLBOD7eGcslS9vfeQ== 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.3] 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCE5OBm2E5wBhAA5c5YRRdDpDT9bqZJbb5PvA== 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB25OBmKOOcBBAA6Y3siq75kt2B/qs+AJwI3g== 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b] 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB25OBmXToGHxAAJNalOXejysTGAHu9SfxlMg== 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d] 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB25OBm0JHlKBAAzPNHYrFjM4yxEQJLITB/rQ== 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.4] 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCK5OBmHbAwFRAAsYku92WnAAsg20OMfTZ8SQ== 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.5] 2024-09-11T00:30:16.713 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCN5OBme7yeLxAARpl19Rpf+CZttXZBmIq+Pg== 2024-09-11T00:30:16.714 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.6] 2024-09-11T00:30:16.714 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCR5OBmyu2yARAAvh1wW1ootKwwPIVlhpc5gA== 2024-09-11T00:30:16.714 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.7] 2024-09-11T00:30:16.714 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCU5OBmt3drCxAAS/Hq4x25GmDRiU6ZfE8ZiA== 2024-09-11T00:30:16.714 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-09-11T00:30:16.714 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB25OBmt6RpMRAAaKGEr69XAmeXm5hP57fXHw== 2024-09-11T00:30:16.714 INFO:teuthology.orchestra.run.smithi031.stdout:[client.1] 2024-09-11T00:30:16.714 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQB25OBmilL5NxAAJpsAcqc2f/FsqHFShJaccg== 2024-09-11T00:30:16.749 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.y] 2024-09-11T00:30:16.749 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB15OBmR0D2OBAA2LQiIesU51YcYEGi6cta1g== 2024-09-11T00:30:16.749 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.a] 2024-09-11T00:30:16.749 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB25OBmYpY7CxAANfoa8HShyEnI9dD2P1rm5g== 2024-09-11T00:30:16.749 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.c] 2024-09-11T00:30:16.749 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB25OBm/OIpFRAA4o5YBVAlBCiipV3KBBKXQQ== 2024-09-11T00:30:16.749 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2024-09-11T00:30:16.749 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB55OBmpKw7MxAAyrrMBsf2cqwi7RSGzjQoyg== 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB95OBmsFZlAhAA38gP6TnRUk3yoWY9SfPeNA== 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQCA5OBmLJHACxAAZAxiXoLBOD7eGcslS9vfeQ== 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.3] 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQCE5OBm2E5wBhAA5c5YRRdDpDT9bqZJbb5PvA== 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB25OBmKOOcBBAA6Y3siq75kt2B/qs+AJwI3g== 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.b] 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB25OBmXToGHxAAJNalOXejysTGAHu9SfxlMg== 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout:[mds.d] 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB25OBm0JHlKBAAzPNHYrFjM4yxEQJLITB/rQ== 2024-09-11T00:30:16.750 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.4] 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQCK5OBmHbAwFRAAsYku92WnAAsg20OMfTZ8SQ== 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.5] 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQCN5OBme7yeLxAARpl19Rpf+CZttXZBmIq+Pg== 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.6] 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQCR5OBmyu2yARAAvh1wW1ootKwwPIVlhpc5gA== 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.7] 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQCU5OBmt3drCxAAS/Hq4x25GmDRiU6ZfE8ZiA== 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB25OBmt6RpMRAAaKGEr69XAmeXm5hP57fXHw== 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout:[client.1] 2024-09-11T00:30:16.751 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQB25OBmilL5NxAAJpsAcqc2f/FsqHFShJaccg== 2024-09-11T00:30:16.753 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T00:30:16.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T00:30:16.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-11T00:30:16.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-11T00:30:16.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-11T00:30:16.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-11T00:30:17.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.388 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T00:30:17.485 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T00:30:17.579 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-11T00:30:17.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-11T00:30:17.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-11T00:30:17.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-11T00:30:17.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:17.981 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:18.076 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:18.080 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T00:30:18.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T00:30:18.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T00:30:18.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T00:30:18.277 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T00:30:18.374 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T00:30:18.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T00:30:18.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T00:30:18.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T00:30:18.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-11T00:30:18.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T00:30:18.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-11T00:30:18.814 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-11T00:30:18.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T00:30:18.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-11T00:30:19.034 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:30:19.038 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T00:30:19.073 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T00:30:19.073 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T00:30:19.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T00:30:19.114 INFO:tasks.ceph.mon.a:Started 2024-09-11T00:30:19.115 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T00:30:19.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T00:30:19.119 INFO:tasks.ceph.mon.b:Started 2024-09-11T00:30:19.119 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T00:30:19.119 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T00:30:19.122 INFO:tasks.ceph.mon.c:Started 2024-09-11T00:30:19.122 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T00:30:19.123 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-11T00:30:19.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-11T00:30:19.126 INFO:tasks.ceph.mgr.y:Started 2024-09-11T00:30:19.126 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T00:30:19.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T00:30:19.134 INFO:tasks.ceph.mgr.x:Started 2024-09-11T00:30:19.134 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-09-11T00:30:19.134 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-09-11T00:30:19.138 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-09-11T00:30:19.139 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set client debug_client 20 2024-09-11T00:30:19.184 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-09-11T00:30:19.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set client debug_ms 1 2024-09-11T00:30:19.231 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-09-11T00:30:19.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-09-11T00:30:19.277 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-09-11T00:30:19.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-09-11T00:30:19.324 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-09-11T00:30:19.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-09-11T00:30:19.370 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-09-11T00:30:19.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-09-11T00:30:19.418 INFO:tasks.ceph:setting config [mds] debug_ms = 2 2024-09-11T00:30:19.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds debug_ms 2 2024-09-11T00:30:19.470 INFO:tasks.ceph:setting config [mds] mds_bal_fragment_size_max = 10000 2024-09-11T00:30:19.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_bal_fragment_size_max 10000 2024-09-11T00:30:19.524 INFO:tasks.ceph:setting config [mds] mds_bal_merge_size = 5 2024-09-11T00:30:19.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_bal_merge_size 5 2024-09-11T00:30:19.551 INFO:tasks.ceph:setting config [mds] mds_bal_split_bits = 3 2024-09-11T00:30:19.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_bal_split_bits 3 2024-09-11T00:30:19.608 INFO:tasks.ceph:setting config [mds] mds_bal_split_size = 100 2024-09-11T00:30:19.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_bal_split_size 100 2024-09-11T00:30:19.633 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-09-11T00:30:19.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-09-11T00:30:19.684 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-09-11T00:30:19.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-09-11T00:30:19.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.703+0000 7fe81a000640 1 -- 172.21.15.31:0/4153579368 <== mon.1 v2:172.21.15.204:3300/0 10 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v4)=0 v4) ==== 109+0+0 (secure 0 0 0) 0x7fe81403bc20 con 0x7fe82414ea20 2024-09-11T00:30:19.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.705+0000 7f38ba200640 1 -- 172.21.15.31:0/431942750 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b41e84e0 con 0x7f38b41e7c10 2024-09-11T00:30:19.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.705+0000 7f38ba200640 1 -- 172.21.15.31:0/431942750 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b4072230 con 0x7f38b41e7c10 2024-09-11T00:30:19.716 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-09-11T00:30:19.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-09-11T00:30:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.710+0000 7fe81a000640 1 -- 172.21.15.31:0/4153579368 <== mon.1 v2:172.21.15.204:3300/0 11 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fe814044950 con 0x7fe82414ea20 2024-09-11T00:30:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.711+0000 7fe828c00640 1 -- 172.21.15.31:0/4153579368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe82414ea20 msgr2=0x7fe8241f3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.711+0000 7fe828c00640 1 --2- 172.21.15.31:0/4153579368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe82414ea20 0x7fe8241f3460 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe814005fd0 tx=0x7fe8140046c0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.711+0000 7fe828c00640 1 -- 172.21.15.31:0/4153579368 shutdown_connections 2024-09-11T00:30:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.710+0000 7fc227e00640 1 -- 172.21.15.31:0/1687796957 <== mon.1 v2:172.21.15.204:3300/0 9 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fc21c031950 con 0x7fc2300c99e0 2024-09-11T00:30:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.711+0000 7fc227e00640 1 -- 172.21.15.31:0/1687796957 <== mon.1 v2:172.21.15.204:3300/0 10 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7fc230076890 con 0x7fc2300c99e0 2024-09-11T00:30:19.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.707+0000 7f38ab400640 1 -- 172.21.15.31:0/431942750 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f38ac00b070 con 0x7f38b41e7c10 2024-09-11T00:30:19.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.707+0000 7f38ab400640 1 -- 172.21.15.31:0/431942750 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f38ac019950 con 0x7f38b41e7c10 2024-09-11T00:30:19.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.707+0000 7f38ba200640 1 -- 172.21.15.31:0/431942750 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b4074810 con 0x7f38b41e7c10 2024-09-11T00:30:19.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.714+0000 7f38ab400640 1 -- 172.21.15.31:0/431942750 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f38b4074810 con 0x7f38b41e7c10 2024-09-11T00:30:19.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.714+0000 7f38ab400640 1 -- 172.21.15.31:0/431942750 <== mon.2 v2:172.21.15.204:3301/0 7 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f38ac019420 con 0x7f38b41e7c10 2024-09-11T00:30:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7fe828c00640 1 --2- 172.21.15.31:0/4153579368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe82414ea20 0x7fe8241f3460 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7fe828c00640 1 --2- 172.21.15.31:0/4153579368 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8240cc5b0 0x7fe8241579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7fe828c00640 1 --2- 172.21.15.31:0/4153579368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8240bf2c0 0x7fe82414dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7fe828c00640 1 -- 172.21.15.31:0/4153579368 >> 172.21.15.31:0/4153579368 conn(0x7fe8240addf0 msgr2=0x7fe8240c9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7fe828c00640 1 -- 172.21.15.31:0/4153579368 shutdown_connections 2024-09-11T00:30:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7fe828c00640 1 -- 172.21.15.31:0/4153579368 wait complete. 2024-09-11T00:30:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.711+0000 7f9a5e000640 1 -- 172.21.15.31:0/1506452563 <== mon.1 v2:172.21.15.204:3300/0 8 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f9a5801e400 con 0x7f9a60049a30 2024-09-11T00:30:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7eff00a00640 1 -- 172.21.15.31:0/3034710016 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efefc0c1310 msgr2=0x7efefc1608d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7eff00a00640 1 --2- 172.21.15.31:0/3034710016 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efefc0c1310 0x7efefc1608d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efee8005fd0 tx=0x7efee802a4a0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7eff00a00640 1 -- 172.21.15.31:0/3034710016 shutdown_connections 2024-09-11T00:30:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7eff00a00640 1 --2- 172.21.15.31:0/3034710016 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efefc0c1310 0x7efefc1608d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7eff00a00640 1 --2- 172.21.15.31:0/3034710016 >> v2:172.21.15.204:3300/0 conn(0x7efefc070050 0x7efefc0c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.716+0000 7eff00a00640 1 -- 172.21.15.31:0/3034710016 >> 172.21.15.31:0/3034710016 conn(0x7efefc0a9750 msgr2=0x7efefc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:19.730 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-09-11T00:30:19.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-09-11T00:30:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.722+0000 7fc237000640 1 -- 172.21.15.31:0/1687796957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2300c99e0 msgr2=0x7fc2301f3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.722+0000 7fc237000640 1 --2- 172.21.15.31:0/1687796957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2300c99e0 0x7fc2301f3270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009980 tx=0x7fc21c0046c0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.722+0000 7fc237000640 1 -- 172.21.15.31:0/1687796957 shutdown_connections 2024-09-11T00:30:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.722+0000 7fc237000640 1 --2- 172.21.15.31:0/1687796957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2300c99e0 0x7fc2301f3270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.722+0000 7fc237000640 1 --2- 172.21.15.31:0/1687796957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2300c8ca0 0x7fc23014fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.722+0000 7fc237000640 1 --2- 172.21.15.31:0/1687796957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc23006f5b0 0x7fc230151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.722+0000 7fc237000640 1 -- 172.21.15.31:0/1687796957 >> 172.21.15.31:0/1687796957 conn(0x7fc2300ada10 msgr2=0x7fc2300ac730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.722+0000 7fc237000640 1 -- 172.21.15.31:0/1687796957 shutdown_connections 2024-09-11T00:30:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.722+0000 7fc237000640 1 -- 172.21.15.31:0/1687796957 wait complete. 2024-09-11T00:30:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.718+0000 7eff00a00640 1 -- 172.21.15.31:0/3034710016 shutdown_connections 2024-09-11T00:30:19.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.720+0000 7eff00a00640 1 -- 172.21.15.31:0/3034710016 wait complete. 2024-09-11T00:30:19.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.730+0000 7eff00a00640 1 Processor -- start 2024-09-11T00:30:19.765 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-09-11T00:30:19.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-09-11T00:30:19.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.739+0000 7eff00a00640 1 -- start start 2024-09-11T00:30:19.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.743+0000 7eff00a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efefc070050 0x7efefc0c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:19.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.743+0000 7eff00a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efefc0c1310 0x7efefc150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:19.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.743+0000 7eff00a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efefc156a40 0x7efefc150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:19.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.745+0000 7efefb400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efefc070050 0x7efefc0c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.745+0000 7efefb400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efefc070050 0x7efefc0c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33666/0 (socket says 172.21.15.31:33666) 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.745+0000 7eff00a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efefc0c32c0 con 0x7efefc156a40 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.745+0000 7eff00a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efefc0c1fd0 con 0x7efefc070050 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.745+0000 7eff00a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efefc0c37e0 con 0x7efefc0c1310 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.745+0000 7efefaa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efefc0c1310 0x7efefc150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.745+0000 7efefb400640 1 -- 172.21.15.31:0/159116959 learned_addr learned my addr 172.21.15.31:0/159116959 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.747+0000 7efefaa00640 1 -- 172.21.15.31:0/159116959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efefc070050 msgr2=0x7efefc0c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.747+0000 7efefaa00640 1 --2- 172.21.15.31:0/159116959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efefc070050 0x7efefc0c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.747+0000 7efefaa00640 1 -- 172.21.15.31:0/159116959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efefc156a40 msgr2=0x7efefc150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.747+0000 7efefaa00640 1 --2- 172.21.15.31:0/159116959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efefc156a40 0x7efefc150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.747+0000 7efefaa00640 1 -- 172.21.15.31:0/159116959 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efefc1efba0 con 0x7efefc0c1310 2024-09-11T00:30:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.750+0000 7efefaa00640 1 --2- 172.21.15.31:0/159116959 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efefc0c1310 0x7efefc150a00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efeec00e9f0 tx=0x7efeec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.759+0000 7efef3400640 1 -- 172.21.15.31:0/159116959 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec00ccc0 con 0x7efefc0c1310 2024-09-11T00:30:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.759+0000 7efef3400640 1 -- 172.21.15.31:0/159116959 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7efeec010430 con 0x7efefc0c1310 2024-09-11T00:30:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.759+0000 7efef3400640 1 -- 172.21.15.31:0/159116959 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec010840 con 0x7efefc0c1310 2024-09-11T00:30:19.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.762+0000 7eff00a00640 1 -- 172.21.15.31:0/159116959 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efefc1efe40 con 0x7efefc0c1310 2024-09-11T00:30:19.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.762+0000 7eff00a00640 1 -- 172.21.15.31:0/159116959 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efefc1e1ad0 con 0x7efefc0c1310 2024-09-11T00:30:19.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.768+0000 7efef3400640 1 -- 172.21.15.31:0/159116959 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7efeec0109e0 con 0x7efefc0c1310 2024-09-11T00:30:19.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.768+0000 7efef3400640 1 -- 172.21.15.31:0/159116959 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efeec0213f0 con 0x7efefc0c1310 2024-09-11T00:30:19.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.768+0000 7efef0c00640 1 -- 172.21.15.31:0/159116959 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed0005290 con 0x7efefc0c1310 2024-09-11T00:30:19.779 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-09-11T00:30:19.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-09-11T00:30:19.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.771+0000 7efef3400640 1 -- 172.21.15.31:0/159116959 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7efeec018e00 con 0x7efefc0c1310 2024-09-11T00:30:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.816+0000 7fe644a00640 1 -- 172.21.15.31:0/235719666 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe63804dbd0 msgr2=0x7fe638005d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.816+0000 7fe644a00640 1 --2- 172.21.15.31:0/235719666 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe63804dbd0 0x7fe638005d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe62c009830 tx=0x7fe62c02a410 comp rx=0 tx=0).stop 2024-09-11T00:30:19.827 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-09-11T00:30:19.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-09-11T00:30:19.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.831+0000 7fe644a00640 1 -- 172.21.15.31:0/235719666 shutdown_connections 2024-09-11T00:30:19.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.831+0000 7fe644a00640 1 --2- 172.21.15.31:0/235719666 >> v2:172.21.15.204:3300/0 conn(0x7fe63804d530 0x7fe638049fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.831+0000 7fe644a00640 1 --2- 172.21.15.31:0/235719666 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe63804dbd0 0x7fe638005d50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.831+0000 7fe644a00640 1 -- 172.21.15.31:0/235719666 >> 172.21.15.31:0/235719666 conn(0x7fe63804c0b0 msgr2=0x7fe63804c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:19.843 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-09-11T00:30:19.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-09-11T00:30:19.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.831+0000 7fe644a00640 1 -- 172.21.15.31:0/235719666 shutdown_connections 2024-09-11T00:30:19.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.835+0000 7fe644a00640 1 -- 172.21.15.31:0/235719666 wait complete. 2024-09-11T00:30:19.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.845+0000 7fe644a00640 1 Processor -- start 2024-09-11T00:30:19.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.845+0000 7fe644a00640 1 -- start start 2024-09-11T00:30:19.864 INFO:tasks.ceph:setting config [mon] mon_warn_on_pool_no_app = False 2024-09-11T00:30:19.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set mon mon_warn_on_pool_no_app False 2024-09-11T00:30:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.851+0000 7fe644a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638006be0 0x7fe6380dc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.851+0000 7fe644a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe63804d530 0x7fe6381655e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.851+0000 7fe644a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe63804dbd0 0x7fe6380dd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.852+0000 7fe644a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe6380daaa0 con 0x7fe638006be0 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.852+0000 7fe644a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe6380d9e70 con 0x7fe63804d530 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.852+0000 7fe644a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe6380d9b40 con 0x7fe63804dbd0 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.852+0000 7fe63de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638006be0 0x7fe6380dc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.852+0000 7fe63d400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe63804d530 0x7fe6381655e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.852+0000 7fe63e800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe63804dbd0 0x7fe6380dd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.853+0000 7fe63e800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe63804dbd0 0x7fe6380dd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:49246/0 (socket says 172.21.15.31:49246) 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.853+0000 7fe63e800640 1 -- 172.21.15.31:0/3478026721 learned_addr learned my addr 172.21.15.31:0/3478026721 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.858+0000 7fe63e800640 1 -- 172.21.15.31:0/3478026721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe63804d530 msgr2=0x7fe6381655e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.858+0000 7fe63e800640 1 --2- 172.21.15.31:0/3478026721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe63804d530 0x7fe6381655e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.858+0000 7fe63e800640 1 -- 172.21.15.31:0/3478026721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638006be0 msgr2=0x7fe6380dc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.858+0000 7fe63e800640 1 --2- 172.21.15.31:0/3478026721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638006be0 0x7fe6380dc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.858+0000 7fe63e800640 1 -- 172.21.15.31:0/3478026721 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe638031580 con 0x7fe63804dbd0 2024-09-11T00:30:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.858+0000 7fe63de00640 1 --2- 172.21.15.31:0/3478026721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638006be0 0x7fe6380dc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:30:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.859+0000 7fe63e800640 1 --2- 172.21.15.31:0/3478026721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe63804dbd0 0x7fe6380dd710 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe640066670 tx=0x7fe6400772e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.859+0000 7fe636000640 1 -- 172.21.15.31:0/3478026721 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640073aa0 con 0x7fe63804dbd0 2024-09-11T00:30:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.859+0000 7fe636000640 1 -- 172.21.15.31:0/3478026721 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fe640073c60 con 0x7fe63804dbd0 2024-09-11T00:30:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.859+0000 7fe636000640 1 -- 172.21.15.31:0/3478026721 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6400956e0 con 0x7fe63804dbd0 2024-09-11T00:30:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.863+0000 7fe644a00640 1 -- 172.21.15.31:0/3478026721 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe638031860 con 0x7fe63804dbd0 2024-09-11T00:30:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.864+0000 7fe644a00640 1 -- 172.21.15.31:0/3478026721 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe638031e20 con 0x7fe63804dbd0 2024-09-11T00:30:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.865+0000 7fe636000640 1 -- 172.21.15.31:0/3478026721 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fe640095c10 con 0x7fe63804dbd0 2024-09-11T00:30:19.890 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-09-11T00:30:19.890 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-09-11T00:30:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.869+0000 7fe636000640 1 -- 172.21.15.31:0/3478026721 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe640094db0 con 0x7fe63804dbd0 2024-09-11T00:30:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.877+0000 7fe644a00640 1 -- 172.21.15.31:0/3478026721 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe638042d50 con 0x7fe63804dbd0 2024-09-11T00:30:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:19.880+0000 7fe636000640 1 -- 172.21.15.31:0/3478026721 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fe6400a3020 con 0x7fe63804dbd0 2024-09-11T00:30:19.950 DEBUG:tasks.ceph:set 22 configs 2024-09-11T00:30:19.950 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi031') 2024-09-11T00:30:19.951 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi031') 2024-09-11T00:30:19.951 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi031') 2024-09-11T00:30:19.951 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi031') 2024-09-11T00:30:19.951 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.001+0000 7fa45ce00640 1 -- 172.21.15.31:0/1835170221 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa43c004d40 msgr2=0x7fa43c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.003+0000 7efef0c00640 1 -- 172.21.15.31:0/159116959 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7efed0005580 con 0x7efefc0c1310 2024-09-11T00:30:20.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.001+0000 7fa45ce00640 1 --2- 172.21.15.31:0/1835170221 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa43c004d40 0x7fa43c0051c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa450001e90 tx=0x7fa43c005830 comp rx=0 tx=0).stop 2024-09-11T00:30:20.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.003+0000 7fa456800640 1 -- 172.21.15.31:0/1835170221 reap_dead start 2024-09-11T00:30:20.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.004+0000 7fa45ce00640 1 -- 172.21.15.31:0/1835170221 shutdown_connections 2024-09-11T00:30:20.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.004+0000 7fa45ce00640 1 -- 172.21.15.31:0/1835170221 >> 172.21.15.31:0/1835170221 conn(0x7fa4580addf0 msgr2=0x7fa4580cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.006+0000 7fa45ce00640 1 -- 172.21.15.31:0/1835170221 shutdown_connections 2024-09-11T00:30:20.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.008+0000 7fa45ce00640 1 -- 172.21.15.31:0/1835170221 wait complete. 2024-09-11T00:30:20.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.014+0000 7f1040c00640 1 -- 172.21.15.31:0/2307326190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f103c0cc5b0 msgr2=0x7f103c06deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.014+0000 7f1040c00640 1 --2- 172.21.15.31:0/2307326190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f103c0cc5b0 0x7f103c06deb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1034005fd0 tx=0x7f103402a6e0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.014+0000 7f9a6b200640 1 -- 172.21.15.31:0/1506452563 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f9a6000a210 con 0x7f9a60049a30 2024-09-11T00:30:20.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.015+0000 7fa45ce00640 1 Processor -- start 2024-09-11T00:30:20.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.020+0000 7fa45ce00640 1 -- start start 2024-09-11T00:30:20.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.022+0000 7efef3400640 1 -- 172.21.15.31:0/159116959 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v6)=0 v6) ==== 117+0+0 (secure 0 0 0) 0x7efeec018420 con 0x7efefc0c1310 2024-09-11T00:30:20.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.023+0000 7efef3400640 1 -- 172.21.15.31:0/159116959 <== mon.2 v2:172.21.15.204:3301/0 8 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efeec0186c0 con 0x7efefc0c1310 2024-09-11T00:30:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.024+0000 7f38ab400640 1 -- 172.21.15.31:0/431942750 <== mon.2 v2:172.21.15.204:3301/0 8 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38ac019ba0 con 0x7f38b41e7c10 2024-09-11T00:30:20.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.024+0000 7f9a5e000640 1 -- 172.21.15.31:0/1506452563 <== mon.1 v2:172.21.15.204:3300/0 9 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a5801e650 con 0x7f9a60049a30 2024-09-11T00:30:20.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.025+0000 7fa45ce00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa43c004d40 0x7fa4580c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.025+0000 7fa45ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4581f8eb0 0x7fa4580cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.026+0000 7fa45ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4581fc310 0x7fa4581f56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.020+0000 7f1040c00640 1 -- 172.21.15.31:0/2307326190 shutdown_connections 2024-09-11T00:30:20.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.020+0000 7f1040c00640 1 --2- 172.21.15.31:0/2307326190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f103c0cc5b0 0x7f103c06deb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.020+0000 7f1040c00640 1 -- 172.21.15.31:0/2307326190 >> 172.21.15.31:0/2307326190 conn(0x7f103c0addf0 msgr2=0x7f103c0cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.026+0000 7f9a5e000640 1 -- 172.21.15.31:0/1506452563 <== mon.1 v2:172.21.15.204:3300/0 10 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v7)=0 v7) ==== 123+0+0 (secure 0 0 0) 0x7f9a58023020 con 0x7f9a60049a30 2024-09-11T00:30:20.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.026+0000 7fa455400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4581f8eb0 0x7fa4580cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.027+0000 7fa455e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa43c004d40 0x7fa4580c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.027+0000 7fa455e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa43c004d40 0x7fa4580c4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:49258/0 (socket says 172.21.15.31:49258) 2024-09-11T00:30:20.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.027+0000 7fa456800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4581fc310 0x7fa4581f56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.027+0000 7fa455400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4581f8eb0 0x7fa4580cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33722/0 (socket says 172.21.15.31:33722) 2024-09-11T00:30:20.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.027+0000 7fa455400640 1 -- 172.21.15.31:0/3409781850 learned_addr learned my addr 172.21.15.31:0/3409781850 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.027+0000 7fa45ce00640 1 -- 172.21.15.31:0/3409781850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa45806f480 con 0x7fa4581fc310 2024-09-11T00:30:20.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.027+0000 7fa45ce00640 1 -- 172.21.15.31:0/3409781850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa45806f300 con 0x7fa4581f8eb0 2024-09-11T00:30:20.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.027+0000 7fa45ce00640 1 -- 172.21.15.31:0/3409781850 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa4580b1a00 con 0x7fa43c004d40 2024-09-11T00:30:20.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.028+0000 7fa455400640 1 -- 172.21.15.31:0/3409781850 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa43c004d40 msgr2=0x7fa4580c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.028+0000 7fa455400640 1 --2- 172.21.15.31:0/3409781850 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa43c004d40 0x7fa4580c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.028+0000 7fa455400640 1 -- 172.21.15.31:0/3409781850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4581fc310 msgr2=0x7fa4581f56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.028+0000 7fa455400640 1 --2- 172.21.15.31:0/3409781850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4581fc310 0x7fa4581f56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.028+0000 7fa455400640 1 -- 172.21.15.31:0/3409781850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4580c55e0 con 0x7fa4581f8eb0 2024-09-11T00:30:20.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.028+0000 7fa456800640 1 --2- 172.21.15.31:0/3409781850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4581fc310 0x7fa4581f56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:20.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.026+0000 7f1040c00640 1 -- 172.21.15.31:0/2307326190 shutdown_connections 2024-09-11T00:30:20.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.026+0000 7f1040c00640 1 -- 172.21.15.31:0/2307326190 wait complete. 2024-09-11T00:30:20.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.029+0000 7fa455400640 1 --2- 172.21.15.31:0/3409781850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4581f8eb0 0x7fa4580cb190 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa44400e790 tx=0x7fa44400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.033+0000 7f1040c00640 1 Processor -- start 2024-09-11T00:30:20.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.034+0000 7f1040c00640 1 -- start start 2024-09-11T00:30:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.032+0000 7f38ba200640 1 -- 172.21.15.31:0/431942750 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f38880052f0 con 0x7f38b41e7c10 2024-09-11T00:30:20.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.036+0000 7fe636000640 1 -- 172.21.15.31:0/3478026721 <== mon.2 v2:172.21.15.204:3301/0 7 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6400751c0 con 0x7fe63804dbd0 2024-09-11T00:30:20.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.029+0000 7fa44e000640 1 -- 172.21.15.31:0/3409781850 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44400eea0 con 0x7fa4581f8eb0 2024-09-11T00:30:20.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.036+0000 7fa45ce00640 1 -- 172.21.15.31:0/3409781850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4580c58c0 con 0x7fa4581f8eb0 2024-09-11T00:30:20.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.036+0000 7fa45ce00640 1 -- 172.21.15.31:0/3409781850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4581fbdf0 con 0x7fa4581f8eb0 2024-09-11T00:30:20.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.037+0000 7fa44e000640 1 -- 172.21.15.31:0/3409781850 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa44400ccc0 con 0x7fa4581f8eb0 2024-09-11T00:30:20.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.037+0000 7f9a6b200640 1 -- 172.21.15.31:0/1506452563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a60049a30 msgr2=0x7f9a60042460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.037+0000 7f9a6b200640 1 --2- 172.21.15.31:0/1506452563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a60049a30 0x7f9a60042460 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a58007890 tx=0x7f9a58007c90 comp rx=0 tx=0).stop 2024-09-11T00:30:20.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7f9a6b200640 1 -- 172.21.15.31:0/1506452563 shutdown_connections 2024-09-11T00:30:20.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7f9a6b200640 1 --2- 172.21.15.31:0/1506452563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a600d3490 0x7f9a60042e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7f9a6b200640 1 --2- 172.21.15.31:0/1506452563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a60049a30 0x7f9a60042460 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7f9a6b200640 1 --2- 172.21.15.31:0/1506452563 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9a60046540 0x7f9a60044bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7f9a6b200640 1 -- 172.21.15.31:0/1506452563 >> 172.21.15.31:0/1506452563 conn(0x7f9a60047c30 msgr2=0x7f9a60006360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.040+0000 7f9a6b200640 1 -- 172.21.15.31:0/1506452563 shutdown_connections 2024-09-11T00:30:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.040+0000 7f9a6b200640 1 -- 172.21.15.31:0/1506452563 wait complete. 2024-09-11T00:30:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7efef0c00640 1 -- 172.21.15.31:0/159116959 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efefc0c1310 msgr2=0x7efefc150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7efef0c00640 1 --2- 172.21.15.31:0/159116959 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efefc0c1310 0x7efefc150a00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efeec00e9f0 tx=0x7efeec00eeb0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7efef0c00640 1 -- 172.21.15.31:0/159116959 shutdown_connections 2024-09-11T00:30:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7efef0c00640 1 --2- 172.21.15.31:0/159116959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efefc156a40 0x7efefc150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7efef0c00640 1 --2- 172.21.15.31:0/159116959 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efefc0c1310 0x7efefc150a00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7efef0c00640 1 --2- 172.21.15.31:0/159116959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efefc070050 0x7efefc0c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.039+0000 7efef0c00640 1 -- 172.21.15.31:0/159116959 >> 172.21.15.31:0/159116959 conn(0x7efefc0a9750 msgr2=0x7efefc152e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.037+0000 7fa44e000640 1 -- 172.21.15.31:0/3409781850 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44400eea0 con 0x7fa4581f8eb0 2024-09-11T00:30:20.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.041+0000 7f1040c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f103c0bf2c0 0x7f103c0c2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.043+0000 7f1039e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f103c0bf2c0 0x7f103c0c2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.043+0000 7f1039e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f103c0bf2c0 0x7f103c0c2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34302/0 (socket says 172.21.15.31:34302) 2024-09-11T00:30:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.043+0000 7f1039e00640 1 -- 172.21.15.31:0/2565397410 learned_addr learned my addr 172.21.15.31:0/2565397410 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.043+0000 7f1040c00640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f103c0cc5b0 0x7f103c14c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.044+0000 7f1040c00640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f103c14ea20 0x7f103c0bfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.044+0000 7f103a800640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f103c14ea20 0x7f103c0bfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.044+0000 7f1039400640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f103c0cc5b0 0x7f103c14c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.045+0000 7efef0c00640 1 -- 172.21.15.31:0/159116959 shutdown_connections 2024-09-11T00:30:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.045+0000 7efef0c00640 1 -- 172.21.15.31:0/159116959 wait complete. 2024-09-11T00:30:20.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.047+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f103c0b1a00 con 0x7f103c0bf2c0 2024-09-11T00:30:20.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.047+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f103c0c7480 con 0x7f103c0cc5b0 2024-09-11T00:30:20.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.047+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f103c0b1b80 con 0x7f103c14ea20 2024-09-11T00:30:20.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.047+0000 7f103a800640 1 -- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f103c0cc5b0 msgr2=0x7f103c14c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.048+0000 7f103a800640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f103c0cc5b0 0x7f103c14c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.048+0000 7f103a800640 1 -- 172.21.15.31:0/2565397410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f103c0bf2c0 msgr2=0x7f103c0c2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.048+0000 7f103a800640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f103c0bf2c0 0x7f103c0c2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.053+0000 7f38ab400640 1 -- 172.21.15.31:0/431942750 <== mon.2 v2:172.21.15.204:3301/0 9 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v8)=0 v8) ==== 108+0+0 (secure 0 0 0) 0x7f38ac027080 con 0x7f38b41e7c10 2024-09-11T00:30:20.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.048+0000 7f103a800640 1 -- 172.21.15.31:0/2565397410 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103c14c990 con 0x7f103c14ea20 2024-09-11T00:30:20.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.053+0000 7f103a800640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f103c14ea20 0x7f103c0bfae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f102800c400 tx=0x7f102800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.053+0000 7fa443400640 1 -- 172.21.15.31:0/3409781850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4580cc990 con 0x7fa4581f8eb0 2024-09-11T00:30:20.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.053+0000 7f1032000640 1 -- 172.21.15.31:0/2565397410 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028017070 con 0x7f103c14ea20 2024-09-11T00:30:20.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.054+0000 7f1032000640 1 -- 172.21.15.31:0/2565397410 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10280035f0 con 0x7f103c14ea20 2024-09-11T00:30:20.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.057+0000 7fa44e000640 1 -- 172.21.15.31:0/3409781850 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fa4440053b0 con 0x7fa4581f8eb0 2024-09-11T00:30:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.054+0000 7f1032000640 1 -- 172.21.15.31:0/2565397410 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1028003a40 con 0x7f103c14ea20 2024-09-11T00:30:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.054+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103c1eb870 con 0x7f103c14ea20 2024-09-11T00:30:20.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.054+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f103c158190 con 0x7f103c14ea20 2024-09-11T00:30:20.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.058+0000 7fa44e000640 1 -- 172.21.15.31:0/3409781850 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa444018b70 con 0x7fa4581f8eb0 2024-09-11T00:30:20.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.061+0000 7fa44e000640 1 -- 172.21.15.31:0/3409781850 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fa444026080 con 0x7fa4581f8eb0 2024-09-11T00:30:20.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.058+0000 7f38ba200640 1 -- 172.21.15.31:0/431942750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f38b41e7c10 msgr2=0x7f38b41562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.058+0000 7f38ba200640 1 --2- 172.21.15.31:0/431942750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f38b41e7c10 0x7f38b41562f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00f9f0 tx=0x7f38ac00feb0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.058+0000 7f38ba200640 1 -- 172.21.15.31:0/431942750 shutdown_connections 2024-09-11T00:30:20.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.058+0000 7f38ba200640 1 --2- 172.21.15.31:0/431942750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4148910 0x7f38b4144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.058+0000 7f38ba200640 1 --2- 172.21.15.31:0/431942750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38b41e7ff0 0x7f38b4074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.058+0000 7f38ba200640 1 --2- 172.21.15.31:0/431942750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f38b41e7c10 0x7f38b41562f0 secure :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00f9f0 tx=0x7f38ac00feb0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.058+0000 7f38ba200640 1 -- 172.21.15.31:0/431942750 >> 172.21.15.31:0/431942750 conn(0x7f38b40c7b10 msgr2=0x7f38b4151d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.063+0000 7f38ba200640 1 -- 172.21.15.31:0/431942750 shutdown_connections 2024-09-11T00:30:20.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.063+0000 7f38ba200640 1 -- 172.21.15.31:0/431942750 wait complete. 2024-09-11T00:30:20.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.067+0000 7f1032000640 1 -- 172.21.15.31:0/2565397410 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f1028003be0 con 0x7f103c14ea20 2024-09-11T00:30:20.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.067+0000 7f1032000640 1 -- 172.21.15.31:0/2565397410 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1028012740 con 0x7f103c14ea20 2024-09-11T00:30:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.082+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1014005290 con 0x7f103c14ea20 2024-09-11T00:30:20.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.091+0000 7f1032000640 1 -- 172.21.15.31:0/2565397410 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f1028021020 con 0x7f103c14ea20 2024-09-11T00:30:20.117 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.133 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.133 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.146+0000 7fbc78e00640 1 -- 172.21.15.31:0/635948761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc6c00a230 msgr2=0x7fbc6c00a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.146+0000 7fbc78e00640 1 --2- 172.21.15.31:0/635948761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc6c00a230 0x7fbc6c00a610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbc74052430 tx=0x7fbc740a6d80 comp rx=0 tx=0).stop 2024-09-11T00:30:20.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.154+0000 7f0cf9c00640 1 -- 172.21.15.31:0/1387145819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0cec04d530 msgr2=0x7f0cec049fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.154+0000 7f0cf9c00640 1 --2- 172.21.15.31:0/1387145819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0cec04d530 0x7f0cec049fb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009830 tx=0x7f0ce002a410 comp rx=0 tx=0).stop 2024-09-11T00:30:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.157+0000 7f0cf9c00640 1 -- 172.21.15.31:0/1387145819 shutdown_connections 2024-09-11T00:30:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.157+0000 7f0cf9c00640 1 --2- 172.21.15.31:0/1387145819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0cec04d530 0x7f0cec049fb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.157+0000 7f0cf9c00640 1 --2- 172.21.15.31:0/1387145819 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0cec04dbd0 0x7f0cec005d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.154+0000 7fbc78e00640 1 -- 172.21.15.31:0/635948761 shutdown_connections 2024-09-11T00:30:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.154+0000 7fbc78e00640 1 --2- 172.21.15.31:0/635948761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc6c00a230 0x7fbc6c00a610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.154+0000 7fbc78e00640 1 -- 172.21.15.31:0/635948761 >> 172.21.15.31:0/635948761 conn(0x7fbc6c047c50 msgr2=0x7fbc6c048030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.157+0000 7f0cf9c00640 1 -- 172.21.15.31:0/1387145819 >> 172.21.15.31:0/1387145819 conn(0x7f0cec04c0b0 msgr2=0x7f0cec04c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.160+0000 7f0cf9c00640 1 -- 172.21.15.31:0/1387145819 shutdown_connections 2024-09-11T00:30:20.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.160+0000 7f0cf9c00640 1 -- 172.21.15.31:0/1387145819 wait complete. 2024-09-11T00:30:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.154+0000 7fbc78e00640 1 -- 172.21.15.31:0/635948761 shutdown_connections 2024-09-11T00:30:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.154+0000 7fbc78e00640 1 -- 172.21.15.31:0/635948761 wait complete. 2024-09-11T00:30:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.157+0000 7fbc78e00640 1 Processor -- start 2024-09-11T00:30:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.165+0000 7fbc78e00640 1 -- start start 2024-09-11T00:30:20.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.162+0000 7f0cf9c00640 1 Processor -- start 2024-09-11T00:30:20.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.170+0000 7f0cf9c00640 1 -- start start 2024-09-11T00:30:20.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.171+0000 7fbc78e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc6c006c40 0x7fbc6c02d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.171+0000 7fbc78e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc6c00a230 0x7fbc6c0441f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.171+0000 7fbc78e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc6c0495e0 0x7fbc6c044730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.172+0000 7fbc71e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc6c006c40 0x7fbc6c02d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.172+0000 7fbc71e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc6c006c40 0x7fbc6c02d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33754/0 (socket says 172.21.15.31:33754) 2024-09-11T00:30:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.172+0000 7fbc71400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc6c00a230 0x7fbc6c0441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.172+0000 7fbc71400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc6c00a230 0x7fbc6c0441f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:49280/0 (socket says 172.21.15.31:49280) 2024-09-11T00:30:20.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.172+0000 7f0cf9c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0cec006be0 0x7f0cec044300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.172+0000 7f0cf9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cec04d530 0x7f0cec17bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.172+0000 7f0cf9c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0cec04dbd0 0x7f0cec0472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.174+0000 7f0cf2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0cec006be0 0x7f0cec044300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.174+0000 7f0cf2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0cec006be0 0x7f0cec044300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:49282/0 (socket says 172.21.15.31:49282) 2024-09-11T00:30:20.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.176+0000 7fbc72800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc6c0495e0 0x7fbc6c044730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.176+0000 7fbc72800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc6c0495e0 0x7fbc6c044730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34328/0 (socket says 172.21.15.31:34328) 2024-09-11T00:30:20.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.176+0000 7fbc71e00640 1 -- 172.21.15.31:0/1978487960 learned_addr learned my addr 172.21.15.31:0/1978487960 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.176+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c007050 con 0x7fbc6c0495e0 2024-09-11T00:30:20.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.176+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c030ce0 con 0x7fbc6c006c40 2024-09-11T00:30:20.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.176+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbc6c0d7740 con 0x7fbc6c00a230 2024-09-11T00:30:20.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7f0cf2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cec04d530 0x7f0cec17bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7f0cf2a00640 1 -- 172.21.15.31:0/494291203 learned_addr learned my addr 172.21.15.31:0/494291203 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec0daaa0 con 0x7f0cec04d530 2024-09-11T00:30:20.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec0d9e70 con 0x7f0cec04dbd0 2024-09-11T00:30:20.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0cec0d9b40 con 0x7f0cec006be0 2024-09-11T00:30:20.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7fbc72800640 1 -- 172.21.15.31:0/1978487960 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc6c00a230 msgr2=0x7fbc6c0441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7fbc72800640 1 --2- 172.21.15.31:0/1978487960 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc6c00a230 0x7fbc6c0441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7fbc72800640 1 -- 172.21.15.31:0/1978487960 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc6c006c40 msgr2=0x7fbc6c02d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7fbc72800640 1 --2- 172.21.15.31:0/1978487960 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc6c006c40 0x7fbc6c02d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7fbc72800640 1 -- 172.21.15.31:0/1978487960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6c0d7ab0 con 0x7fbc6c0495e0 2024-09-11T00:30:20.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.177+0000 7fbc71400640 1 --2- 172.21.15.31:0/1978487960 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc6c00a230 0x7fbc6c0441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:20.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.181+0000 7f0cf2a00640 1 -- 172.21.15.31:0/494291203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0cec04dbd0 msgr2=0x7f0cec0472e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:30:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.181+0000 7f0cf2a00640 1 --2- 172.21.15.31:0/494291203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0cec04dbd0 0x7f0cec0472e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.182+0000 7fbc72800640 1 --2- 172.21.15.31:0/1978487960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc6c0495e0 0x7fbc6c044730 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400b410 tx=0x7fbc6400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.182+0000 7fbc71e00640 1 --2- 172.21.15.31:0/1978487960 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc6c006c40 0x7fbc6c02d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.183+0000 7fbc6a000640 1 -- 172.21.15.31:0/1978487960 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64015020 con 0x7fbc6c0495e0 2024-09-11T00:30:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.183+0000 7fbc6a000640 1 -- 172.21.15.31:0/1978487960 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc64011920 con 0x7fbc6c0495e0 2024-09-11T00:30:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.183+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc6c180890 con 0x7fbc6c0495e0 2024-09-11T00:30:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.183+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc6c180b70 con 0x7fbc6c0495e0 2024-09-11T00:30:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.184+0000 7f0cf2a00640 1 -- 172.21.15.31:0/494291203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cec04d530 msgr2=0x7f0cec17bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.186+0000 7f0cf2a00640 1 --2- 172.21.15.31:0/494291203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cec04d530 0x7f0cec17bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.196+0000 7fbc6a000640 1 -- 172.21.15.31:0/1978487960 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64011d70 con 0x7fbc6c0495e0 2024-09-11T00:30:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.197+0000 7fbc6a000640 1 -- 172.21.15.31:0/1978487960 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fbc6401a790 con 0x7fbc6c0495e0 2024-09-11T00:30:20.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.187+0000 7f0cf2a00640 1 -- 172.21.15.31:0/494291203 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cec047040 con 0x7f0cec006be0 2024-09-11T00:30:20.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.198+0000 7fbc6a000640 1 -- 172.21.15.31:0/1978487960 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbc64019c00 con 0x7fbc6c0495e0 2024-09-11T00:30:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.198+0000 7f0cf2000640 1 --2- 172.21.15.31:0/494291203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cec04d530 0x7f0cec17bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.198+0000 7f0cf2a00640 1 --2- 172.21.15.31:0/494291203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0cec006be0 0x7f0cec044300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0cf406a110 tx=0x7f0cf40772e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.199+0000 7f0ceaa00640 1 -- 172.21.15.31:0/494291203 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4073890 con 0x7f0cec006be0 2024-09-11T00:30:20.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.199+0000 7f0ceaa00640 1 -- 172.21.15.31:0/494291203 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cf4073b20 con 0x7f0cec006be0 2024-09-11T00:30:20.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.202+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cec17c980 con 0x7f0cec006be0 2024-09-11T00:30:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.203+0000 7f0ceaa00640 1 -- 172.21.15.31:0/494291203 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4095480 con 0x7f0cec006be0 2024-09-11T00:30:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.202+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc4c005290 con 0x7fbc6c0495e0 2024-09-11T00:30:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.203+0000 7f0ceaa00640 1 -- 172.21.15.31:0/494291203 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f0cf4095a50 con 0x7f0cec006be0 2024-09-11T00:30:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.204+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cec031d20 con 0x7f0cec006be0 2024-09-11T00:30:20.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.209+0000 7f0ceaa00640 1 -- 172.21.15.31:0/494291203 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0cf409e3f0 con 0x7f0cec006be0 2024-09-11T00:30:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.210+0000 7fe644a00640 1 -- 172.21.15.31:0/3478026721 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fe6380d9750 con 0x7fe63804dbd0 2024-09-11T00:30:20.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.211+0000 7fbc6a000640 1 -- 172.21.15.31:0/1978487960 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fbc6401f020 con 0x7fbc6c0495e0 2024-09-11T00:30:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.214+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cec04eb60 con 0x7f0cec006be0 2024-09-11T00:30:20.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.228+0000 7f0ceaa00640 1 -- 172.21.15.31:0/494291203 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f0cf4095d70 con 0x7f0cec006be0 2024-09-11T00:30:20.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.235+0000 7fe636000640 1 -- 172.21.15.31:0/3478026721 <== mon.2 v2:172.21.15.204:3301/0 8 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v9)=0 v9) ==== 106+0+0 (secure 0 0 0) 0x7fe6400943e0 con 0x7fe63804dbd0 2024-09-11T00:30:20.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.240+0000 7fe644a00640 1 -- 172.21.15.31:0/3478026721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe63804dbd0 msgr2=0x7fe6380dd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.240+0000 7fe644a00640 1 --2- 172.21.15.31:0/3478026721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe63804dbd0 0x7fe6380dd710 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe640066670 tx=0x7fe6400772e0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.244+0000 7fe644a00640 1 -- 172.21.15.31:0/3478026721 shutdown_connections 2024-09-11T00:30:20.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.245+0000 7fe644a00640 1 --2- 172.21.15.31:0/3478026721 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe63804dbd0 0x7fe6380dd710 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.245+0000 7fe644a00640 1 --2- 172.21.15.31:0/3478026721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe63804d530 0x7fe6381655e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.245+0000 7fe644a00640 1 --2- 172.21.15.31:0/3478026721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe638006be0 0x7fe6380dc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.245+0000 7fe644a00640 1 -- 172.21.15.31:0/3478026721 >> 172.21.15.31:0/3478026721 conn(0x7fe63804c0b0 msgr2=0x7fe63804e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.249+0000 7fe644a00640 1 -- 172.21.15.31:0/3478026721 shutdown_connections 2024-09-11T00:30:20.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.249+0000 7fe644a00640 1 -- 172.21.15.31:0/3478026721 wait complete. 2024-09-11T00:30:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.288+0000 7f5cf2000640 1 -- 172.21.15.31:0/692588464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cec06e4e0 msgr2=0x7f5cec14f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.288+0000 7f5cf2000640 1 --2- 172.21.15.31:0/692588464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cec06e4e0 0x7f5cec14f8a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc005fd0 tx=0x7f5cdc02a4d0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.290+0000 7f5cf2000640 1 -- 172.21.15.31:0/692588464 shutdown_connections 2024-09-11T00:30:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.290+0000 7f5cf2000640 1 --2- 172.21.15.31:0/692588464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cec06e4e0 0x7f5cec14f8a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.290+0000 7f5cf2000640 1 --2- 172.21.15.31:0/692588464 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5cec156970 0x7f5cec06dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.290+0000 7f5cf2000640 1 -- 172.21.15.31:0/692588464 >> 172.21.15.31:0/692588464 conn(0x7f5cec0cbaa0 msgr2=0x7f5cec0cbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.299+0000 7f5cf2000640 1 -- 172.21.15.31:0/692588464 shutdown_connections 2024-09-11T00:30:20.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.299+0000 7f5cf2000640 1 -- 172.21.15.31:0/692588464 wait complete. 2024-09-11T00:30:20.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.302+0000 7f5cf2000640 1 Processor -- start 2024-09-11T00:30:20.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.302+0000 7f5cf2000640 1 -- start start 2024-09-11T00:30:20.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.303+0000 7f5cf2000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5cec06e4e0 0x7f5cec0ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.303+0000 7f5cf2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cec1597b0 0x7f5cec0adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.309+0000 7f5cebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cec1597b0 0x7f5cec0adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.309+0000 7f5cebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cec1597b0 0x7f5cec0adfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34346/0 (socket says 172.21.15.31:34346) 2024-09-11T00:30:20.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.310+0000 7f5cf2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cec1e05a0 0x7f5cec1587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.310+0000 7f5cf0c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5cec06e4e0 0x7f5cec0ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.310+0000 7f5cf0c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5cec06e4e0 0x7f5cec0ad020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:49298/0 (socket says 172.21.15.31:49298) 2024-09-11T00:30:20.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.310+0000 7f5cebe00640 1 -- 172.21.15.31:0/3578596779 learned_addr learned my addr 172.21.15.31:0/3578596779 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.313+0000 7f1bfb800640 1 -- 172.21.15.31:0/1603345401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bf40a6fb0 msgr2=0x7f1bf40c3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.313+0000 7f1bfb800640 1 --2- 172.21.15.31:0/1603345401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bf40a6fb0 0x7f1bf40c3920 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00afd0 tx=0x7f1bec02b570 comp rx=0 tx=0).stop 2024-09-11T00:30:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.313+0000 7f1bfb800640 1 -- 172.21.15.31:0/1603345401 shutdown_connections 2024-09-11T00:30:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.313+0000 7f1bfb800640 1 --2- 172.21.15.31:0/1603345401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bf40a6fb0 0x7f1bf40c3920 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.313+0000 7f1bfb800640 1 --2- 172.21.15.31:0/1603345401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bf406e170 0x7f1bf40c22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.313+0000 7f1bfb800640 1 -- 172.21.15.31:0/1603345401 >> 172.21.15.31:0/1603345401 conn(0x7f1bf40a9670 msgr2=0x7f1bf40a9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.313+0000 7f1bfb800640 1 -- 172.21.15.31:0/1603345401 shutdown_connections 2024-09-11T00:30:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.314+0000 7f5cf2000640 1 -- 172.21.15.31:0/3578596779 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec0c8190 con 0x7f5cec1597b0 2024-09-11T00:30:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.314+0000 7f5cf2000640 1 -- 172.21.15.31:0/3578596779 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5cec0c6b10 con 0x7f5cec1e05a0 2024-09-11T00:30:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.314+0000 7f5cf2000640 1 -- 172.21.15.31:0/3578596779 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5cec06e8f0 con 0x7f5cec06e4e0 2024-09-11T00:30:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.312+0000 7f5cf1600640 1 --2- 172.21.15.31:0/3578596779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cec1e05a0 0x7f5cec1587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.314+0000 7f1bfb800640 1 -- 172.21.15.31:0/1603345401 wait complete. 2024-09-11T00:30:20.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.320+0000 7f1bfb800640 1 Processor -- start 2024-09-11T00:30:20.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.321+0000 7f1bfb800640 1 -- start start 2024-09-11T00:30:20.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.325+0000 7f1bfb800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bf406e170 0x7f1bf4073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.325+0000 7f1bfb800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bf40a6fb0 0x7f1bf4154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.326+0000 7f1bfb800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bf414f990 0x7f1bf4154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.329 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f5cf0c00640 1 -- 172.21.15.31:0/3578596779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cec1e05a0 msgr2=0x7f5cec1587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f1bf8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bf414f990 0x7f1bf4154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f1bf8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bf414f990 0x7f1bf4154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34362/0 (socket says 172.21.15.31:34362) 2024-09-11T00:30:20.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f5cf0c00640 1 --2- 172.21.15.31:0/3578596779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cec1e05a0 0x7f5cec1587c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f5cf0c00640 1 -- 172.21.15.31:0/3578596779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cec1597b0 msgr2=0x7f5cec0adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f5cf0c00640 1 --2- 172.21.15.31:0/3578596779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cec1597b0 0x7f5cec0adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f5cf0c00640 1 -- 172.21.15.31:0/3578596779 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cec14f6a0 con 0x7f5cec06e4e0 2024-09-11T00:30:20.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f5cf1600640 1 --2- 172.21.15.31:0/3578596779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cec1e05a0 0x7f5cec1587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:30:20.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f1bf8e00640 1 -- 172.21.15.31:0/1195986104 learned_addr learned my addr 172.21.15.31:0/1195986104 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.328+0000 7f5cf0c00640 1 --2- 172.21.15.31:0/3578596779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5cec06e4e0 0x7f5cec0ad020 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400be20 tx=0x7f5ce4009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f1bfa200640 1 --2- 172.21.15.31:0/1195986104 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bf406e170 0x7f1bf4073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f1bfb800640 1 -- 172.21.15.31:0/1195986104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf4152040 con 0x7f1bf414f990 2024-09-11T00:30:20.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.327+0000 7f1bfb800640 1 -- 172.21.15.31:0/1195986104 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf41523b0 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.328+0000 7f1bfb800640 1 -- 172.21.15.31:0/1195986104 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf4070140 con 0x7f1bf406e170 2024-09-11T00:30:20.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.329+0000 7f1bf3e00640 1 --2- 172.21.15.31:0/1195986104 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bf40a6fb0 0x7f1bf4154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.329+0000 7f5ce8c00640 1 -- 172.21.15.31:0/3578596779 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4019070 con 0x7f5cec06e4e0 2024-09-11T00:30:20.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.329+0000 7f5ce8c00640 1 -- 172.21.15.31:0/3578596779 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ce4014440 con 0x7f5cec06e4e0 2024-09-11T00:30:20.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.329+0000 7f5ce8c00640 1 -- 172.21.15.31:0/3578596779 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce40148b0 con 0x7f5cec06e4e0 2024-09-11T00:30:20.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.330+0000 7f1bf3e00640 1 -- 172.21.15.31:0/1195986104 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bf406e170 msgr2=0x7f1bf4073f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.330+0000 7f1bf3e00640 1 --2- 172.21.15.31:0/1195986104 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bf406e170 0x7f1bf4073f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.330+0000 7f1bf3e00640 1 -- 172.21.15.31:0/1195986104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bf414f990 msgr2=0x7f1bf4154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.330+0000 7f1bf3e00640 1 --2- 172.21.15.31:0/1195986104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bf414f990 0x7f1bf4154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.330+0000 7f1bf3e00640 1 -- 172.21.15.31:0/1195986104 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf40bca60 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.331+0000 7f1bf8e00640 1 --2- 172.21.15.31:0/1195986104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bf414f990 0x7f1bf4154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:30:20.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.332+0000 7f1bf3e00640 1 --2- 172.21.15.31:0/1195986104 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bf40a6fb0 0x7f1bf4154560 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1be000e9c0 tx=0x7f1be000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.332+0000 7f1bf0c00640 1 -- 172.21.15.31:0/1195986104 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be000ccc0 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.334+0000 7f5cf2000640 1 -- 172.21.15.31:0/3578596779 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cec0ab800 con 0x7f5cec06e4e0 2024-09-11T00:30:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.334+0000 7f5cf2000640 1 -- 172.21.15.31:0/3578596779 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cec0acb00 con 0x7f5cec06e4e0 2024-09-11T00:30:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.333+0000 7f1bf0c00640 1 -- 172.21.15.31:0/1195986104 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1be0004020 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.335+0000 7f1bf0c00640 1 -- 172.21.15.31:0/1195986104 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be000ce60 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.335+0000 7f1bfb800640 1 -- 172.21.15.31:0/1195986104 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf41f0260 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.336+0000 7f1bfb800640 1 -- 172.21.15.31:0/1195986104 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf41f0800 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.339+0000 7f5ce8c00640 1 -- 172.21.15.31:0/3578596779 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f5ce4014a50 con 0x7f5cec06e4e0 2024-09-11T00:30:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.339+0000 7f5ce8c00640 1 -- 172.21.15.31:0/3578596779 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5ce401d4d0 con 0x7f5cec06e4e0 2024-09-11T00:30:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.339+0000 7f1bf0c00640 1 -- 172.21.15.31:0/1195986104 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f1be0003980 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.339+0000 7f5cf2000640 1 -- 172.21.15.31:0/3578596779 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cec0b19a0 con 0x7f5cec06e4e0 2024-09-11T00:30:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.340+0000 7f1be6000640 1 -- 172.21.15.31:0/1195986104 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bc8005290 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.341+0000 7f1bf0c00640 1 -- 172.21.15.31:0/1195986104 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1be0004410 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.347+0000 7f5ce8c00640 1 -- 172.21.15.31:0/3578596779 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f5cec0b19a0 con 0x7f5cec06e4e0 2024-09-11T00:30:20.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.348+0000 7f82aa200640 1 -- 172.21.15.31:0/3645532108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82a40c22c0 msgr2=0x7f82a40c3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.348+0000 7f82aa200640 1 --2- 172.21.15.31:0/3645532108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82a40c22c0 0x7f82a40c3920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f829c00cc10 tx=0x7f829c02c5d0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.348+0000 7f82aa200640 1 -- 172.21.15.31:0/3645532108 shutdown_connections 2024-09-11T00:30:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.348+0000 7f82aa200640 1 --2- 172.21.15.31:0/3645532108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82a40c22c0 0x7f82a40c3920 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.348+0000 7f82aa200640 1 --2- 172.21.15.31:0/3645532108 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82a414f720 0x7f82a40a6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.348+0000 7f82aa200640 1 -- 172.21.15.31:0/3645532108 >> 172.21.15.31:0/3645532108 conn(0x7f82a40a9670 msgr2=0x7f82a40a9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.349+0000 7f82aa200640 1 -- 172.21.15.31:0/3645532108 shutdown_connections 2024-09-11T00:30:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.350+0000 7f1a00800640 1 -- 172.21.15.31:0/2627400137 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19f4004830 msgr2=0x7f19f4004cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.350+0000 7f1a00800640 1 --2- 172.21.15.31:0/2627400137 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19f4004830 0x7f19f4004cb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f19e0006f70 tx=0x7f19f4005860 comp rx=0 tx=0).stop 2024-09-11T00:30:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.350+0000 7f82aa200640 1 -- 172.21.15.31:0/3645532108 wait complete. 2024-09-11T00:30:20.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.355+0000 7f19fde00640 1 -- 172.21.15.31:0/2627400137 reap_dead start 2024-09-11T00:30:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.355+0000 7f19fde00640 1 -- 172.21.15.31:0/2627400137 reap_dead start 2024-09-11T00:30:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.355+0000 7f1a00800640 1 -- 172.21.15.31:0/2627400137 shutdown_connections 2024-09-11T00:30:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.355+0000 7f1a00800640 1 -- 172.21.15.31:0/2627400137 >> 172.21.15.31:0/2627400137 conn(0x7f19f80a9670 msgr2=0x7f19f80a9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.356+0000 7f1a00800640 1 -- 172.21.15.31:0/2627400137 shutdown_connections 2024-09-11T00:30:20.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.355+0000 7f82aa200640 1 Processor -- start 2024-09-11T00:30:20.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.359+0000 7f82aa200640 1 -- start start 2024-09-11T00:30:20.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.358+0000 7f1a00800640 1 -- 172.21.15.31:0/2627400137 wait complete. 2024-09-11T00:30:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.365+0000 7f1a00800640 1 Processor -- start 2024-09-11T00:30:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.365+0000 7f1a00800640 1 -- start start 2024-09-11T00:30:20.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.366+0000 7f82aa200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82a406e170 0x7f82a40c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.366+0000 7f82aa200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82a40c22c0 0x7f82a40739b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.369+0000 7f82a2a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82a406e170 0x7f82a40c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.369+0000 7f82a2a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82a406e170 0x7f82a40c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33836/0 (socket says 172.21.15.31:33836) 2024-09-11T00:30:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.369+0000 7f82aa200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82a414f720 0x7f82a4073ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.370+0000 7f1a00800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80bf440 0x7f19f80bf820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.370+0000 7f82a3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82a414f720 0x7f82a4073ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.370+0000 7f82a3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82a414f720 0x7f82a4073ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:49314/0 (socket says 172.21.15.31:49314) 2024-09-11T00:30:20.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.370+0000 7f1a00800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19f81f9880 0x7f19f80bcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.370+0000 7f82a2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82a40c22c0 0x7f82a40739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.370+0000 7f1a00800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19f80bd030 0x7f19f81f5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.372+0000 7f19fca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19f81f9880 0x7f19f80bcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.372+0000 7f19fca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19f81f9880 0x7f19f80bcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33848/0 (socket says 172.21.15.31:33848) 2024-09-11T00:30:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.372+0000 7f19fca00640 1 -- 172.21.15.31:0/1688659756 learned_addr learned my addr 172.21.15.31:0/1688659756 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.373+0000 7f1a00800640 1 -- 172.21.15.31:0/1688659756 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f19f8152040 con 0x7f19f80bf440 2024-09-11T00:30:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.374+0000 7f82a2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82a40c22c0 0x7f82a40739b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34390/0 (socket says 172.21.15.31:34390) 2024-09-11T00:30:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.374+0000 7f82a3400640 1 -- 172.21.15.31:0/3754749349 learned_addr learned my addr 172.21.15.31:0/3754749349 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.374+0000 7f1a00800640 1 -- 172.21.15.31:0/1688659756 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f19f80c2730 con 0x7f19f81f9880 2024-09-11T00:30:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.374+0000 7f82aa200640 1 -- 172.21.15.31:0/3754749349 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f82a4152320 con 0x7f82a40c22c0 2024-09-11T00:30:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.374+0000 7f82aa200640 1 -- 172.21.15.31:0/3754749349 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82a4152040 con 0x7f82a406e170 2024-09-11T00:30:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.374+0000 7f82aa200640 1 -- 172.21.15.31:0/3754749349 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f82a414fb00 con 0x7f82a414f720 2024-09-11T00:30:20.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.374+0000 7f19fde00640 1 --2- 172.21.15.31:0/1688659756 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19f80bd030 0x7f19f81f5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.374+0000 7f19fd400640 1 --2- 172.21.15.31:0/1688659756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80bf440 0x7f19f80bf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.374+0000 7f82a3400640 1 -- 172.21.15.31:0/3754749349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82a406e170 msgr2=0x7f82a40c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.374+0000 7f82a3400640 1 --2- 172.21.15.31:0/3754749349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82a406e170 0x7f82a40c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.375+0000 7f82a3400640 1 -- 172.21.15.31:0/3754749349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82a40c22c0 msgr2=0x7f82a40739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.375+0000 7f1a00800640 1 -- 172.21.15.31:0/1688659756 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f19f8152320 con 0x7f19f80bd030 2024-09-11T00:30:20.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.377+0000 7f1bf0c00640 1 -- 172.21.15.31:0/1195986104 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f1be0018430 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.377+0000 7f19fd400640 1 -- 172.21.15.31:0/1688659756 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19f80bd030 msgr2=0x7f19f81f5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.377+0000 7f19fd400640 1 --2- 172.21.15.31:0/1688659756 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19f80bd030 0x7f19f81f5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.377+0000 7f19fd400640 1 -- 172.21.15.31:0/1688659756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19f81f9880 msgr2=0x7f19f80bcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.377+0000 7f19fd400640 1 --2- 172.21.15.31:0/1688659756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19f81f9880 0x7f19f80bcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.377+0000 7f19fd400640 1 -- 172.21.15.31:0/1688659756 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f80c0900 con 0x7f19f80bf440 2024-09-11T00:30:20.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.375+0000 7f82a3400640 1 --2- 172.21.15.31:0/3754749349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82a40c22c0 0x7f82a40739b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.375+0000 7f82a3400640 1 -- 172.21.15.31:0/3754749349 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a4074ec0 con 0x7f82a414f720 2024-09-11T00:30:20.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.379+0000 7f82a3400640 1 --2- 172.21.15.31:0/3754749349 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82a414f720 0x7f82a4073ef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f829c02cbc0 tx=0x7f829c00c230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.379+0000 7f19fd400640 1 --2- 172.21.15.31:0/1688659756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80bf440 0x7f19f80bf820 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f19e8011bb0 tx=0x7f19e8007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.380+0000 7f8297400640 1 -- 172.21.15.31:0/3754749349 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c041030 con 0x7f82a414f720 2024-09-11T00:30:20.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.380+0000 7f8297400640 1 -- 172.21.15.31:0/3754749349 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f829c00a920 con 0x7f82a414f720 2024-09-11T00:30:20.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.380+0000 7f19f2000640 1 -- 172.21.15.31:0/1688659756 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e8019070 con 0x7f19f80bf440 2024-09-11T00:30:20.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.380+0000 7f8297400640 1 -- 172.21.15.31:0/3754749349 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c00ab50 con 0x7f82a414f720 2024-09-11T00:30:20.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.382+0000 7f19f2000640 1 -- 172.21.15.31:0/1688659756 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19e8007ca0 con 0x7f19f80bf440 2024-09-11T00:30:20.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.382+0000 7f1a00800640 1 -- 172.21.15.31:0/1688659756 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19f80c0460 con 0x7f19f80bf440 2024-09-11T00:30:20.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.382+0000 7f19f2000640 1 -- 172.21.15.31:0/1688659756 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e80042b0 con 0x7f19f80bf440 2024-09-11T00:30:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.386+0000 7f1a00800640 1 -- 172.21.15.31:0/1688659756 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19f80ac770 con 0x7f19f80bf440 2024-09-11T00:30:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.385+0000 7f82aa200640 1 -- 172.21.15.31:0/3754749349 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a4074be0 con 0x7f82a414f720 2024-09-11T00:30:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.385+0000 7f82aa200640 1 -- 172.21.15.31:0/3754749349 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a41f94b0 con 0x7f82a414f720 2024-09-11T00:30:20.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.390+0000 7f8297400640 1 -- 172.21.15.31:0/3754749349 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f829c04e060 con 0x7f82a414f720 2024-09-11T00:30:20.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.391+0000 7fa443400640 1 -- 172.21.15.31:0/3409781850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7fa4581fbfc0 con 0x7fa4581f8eb0 2024-09-11T00:30:20.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.390+0000 7f8297400640 1 -- 172.21.15.31:0/3754749349 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f829c046750 con 0x7f82a414f720 2024-09-11T00:30:20.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.391+0000 7f8294c00640 1 -- 172.21.15.31:0/3754749349 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8278005290 con 0x7f82a414f720 2024-09-11T00:30:20.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.388+0000 7f19e7400640 1 -- 172.21.15.31:0/1688659756 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d0005290 con 0x7f19f80bf440 2024-09-11T00:30:20.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.392+0000 7f19f2000640 1 -- 172.21.15.31:0/1688659756 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f19e80035f0 con 0x7f19f80bf440 2024-09-11T00:30:20.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.392+0000 7f19f2000640 1 -- 172.21.15.31:0/1688659756 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f19e801e920 con 0x7f19f80bf440 2024-09-11T00:30:20.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.394+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f1014005580 con 0x7f103c14ea20 2024-09-11T00:30:20.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.395+0000 7f19f2000640 1 -- 172.21.15.31:0/1688659756 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f19e8003910 con 0x7f19f80bf440 2024-09-11T00:30:20.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.395+0000 7f8297400640 1 -- 172.21.15.31:0/3754749349 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f829c00a520 con 0x7f82a414f720 2024-09-11T00:30:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.401+0000 7f26ca800640 1 -- 172.21.15.31:0/2310905307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26ac003a00 msgr2=0x7f26ac003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.401+0000 7f26ca800640 1 --2- 172.21.15.31:0/2310905307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26ac003a00 0x7f26ac003e20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f26b000e7f0 tx=0x7f26ac005830 comp rx=0 tx=0).stop 2024-09-11T00:30:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.402+0000 7f26ca800640 1 -- 172.21.15.31:0/2310905307 shutdown_connections 2024-09-11T00:30:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.402+0000 7f26ca800640 1 --2- 172.21.15.31:0/2310905307 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26ac003a00 0x7f26ac003e20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.403+0000 7f26ca800640 1 -- 172.21.15.31:0/2310905307 >> 172.21.15.31:0/2310905307 conn(0x7f26c40c78c0 msgr2=0x7f26c40c7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.403+0000 7f26ca800640 1 -- 172.21.15.31:0/2310905307 shutdown_connections 2024-09-11T00:30:20.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.403+0000 7f26ca800640 1 -- 172.21.15.31:0/2310905307 wait complete. 2024-09-11T00:30:20.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.409+0000 7f26ca800640 1 Processor -- start 2024-09-11T00:30:20.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.409+0000 7f26ca800640 1 -- start start 2024-09-11T00:30:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.410+0000 7fa44e000640 1 -- 172.21.15.31:0/3409781850 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v10)=0 v10) ==== 127+0+0 (secure 0 0 0) 0x7fa4440047a0 con 0x7fa4581f8eb0 2024-09-11T00:30:20.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.411+0000 7f5575e00640 1 -- 172.21.15.31:0/2422985776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55700c22c0 msgr2=0x7f557014cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.411+0000 7f5575e00640 1 --2- 172.21.15.31:0/2422985776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55700c22c0 0x7f557014cfe0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5568005fd0 tx=0x7f556802aaa0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.412+0000 7fa443400640 1 -- 172.21.15.31:0/3409781850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4581f8eb0 msgr2=0x7fa4580cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.412+0000 7fa443400640 1 --2- 172.21.15.31:0/3409781850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4581f8eb0 0x7fa4580cb190 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa44400e790 tx=0x7fa44400ec50 comp rx=0 tx=0).stop 2024-09-11T00:30:20.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7f5575e00640 1 -- 172.21.15.31:0/2422985776 shutdown_connections 2024-09-11T00:30:20.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7f26ca800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26ac003a00 0x7f26c41ea280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7fa443400640 1 -- 172.21.15.31:0/3409781850 shutdown_connections 2024-09-11T00:30:20.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7fa443400640 1 --2- 172.21.15.31:0/3409781850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4581fc310 0x7fa4581f56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7fa443400640 1 --2- 172.21.15.31:0/3409781850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4581f8eb0 0x7fa4580cb190 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7fa443400640 1 --2- 172.21.15.31:0/3409781850 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa43c004d40 0x7fa4580c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7fa443400640 1 -- 172.21.15.31:0/3409781850 >> 172.21.15.31:0/3409781850 conn(0x7fa4580addf0 msgr2=0x7fa4581560a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7f5575e00640 1 --2- 172.21.15.31:0/2422985776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55700c22c0 0x7f557014cfe0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7fa443400640 1 -- 172.21.15.31:0/3409781850 shutdown_connections 2024-09-11T00:30:20.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7fa443400640 1 -- 172.21.15.31:0/3409781850 wait complete. 2024-09-11T00:30:20.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7f26ca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26c4073eb0 0x7f26c40a6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.413+0000 7f5575e00640 1 -- 172.21.15.31:0/2422985776 >> 172.21.15.31:0/2422985776 conn(0x7f55700c78c0 msgr2=0x7f55700c7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.416+0000 7f26ca800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26c41df150 0x7f26c4150bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.416+0000 7f26c9400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26ac003a00 0x7f26c41ea280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.417+0000 7f26c9400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26ac003a00 0x7f26c41ea280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33892/0 (socket says 172.21.15.31:33892) 2024-09-11T00:30:20.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.417+0000 7f26c9400640 1 -- 172.21.15.31:0/3750126278 learned_addr learned my addr 172.21.15.31:0/3750126278 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.417+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f26c40c5e90 con 0x7f26c4073eb0 2024-09-11T00:30:20.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.417+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f26c40c5d10 con 0x7f26ac003a00 2024-09-11T00:30:20.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.418+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f26c40c83b0 con 0x7f26c41df150 2024-09-11T00:30:20.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.418+0000 7f5575e00640 1 -- 172.21.15.31:0/2422985776 shutdown_connections 2024-09-11T00:30:20.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.418+0000 7f5575e00640 1 -- 172.21.15.31:0/2422985776 wait complete. 2024-09-11T00:30:20.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.418+0000 7f26c9e00640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26c41df150 0x7f26c4150bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.419+0000 7f26c8a00640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26c4073eb0 0x7f26c40a6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.419+0000 7f1032000640 1 -- 172.21.15.31:0/2565397410 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v11)=0 v11) ==== 116+0+0 (secure 0 0 0) 0x7f103c14c990 con 0x7f103c14ea20 2024-09-11T00:30:20.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.419+0000 7f26c9400640 1 -- 172.21.15.31:0/3750126278 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26c41df150 msgr2=0x7f26c4150bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.420+0000 7f26c9400640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26c41df150 0x7f26c4150bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.420+0000 7f26c9400640 1 -- 172.21.15.31:0/3750126278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26c4073eb0 msgr2=0x7f26c40a6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.420+0000 7f26c9400640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26c4073eb0 0x7f26c40a6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.420+0000 7f26c9400640 1 -- 172.21.15.31:0/3750126278 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c414c650 con 0x7f26ac003a00 2024-09-11T00:30:20.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.420+0000 7f26c8a00640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26c4073eb0 0x7f26c40a6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:20.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.421+0000 7f26c9e00640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26c41df150 0x7f26c4150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:30:20.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.422+0000 7f26c9400640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26ac003a00 0x7f26c41ea280 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f26b000e740 tx=0x7f26b0005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.426+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f103c14ea20 msgr2=0x7f103c0bfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.426+0000 7f1040c00640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f103c14ea20 0x7f103c0bfae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f102800c400 tx=0x7f102800c8c0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.426+0000 7f26c1600640 1 -- 172.21.15.31:0/3750126278 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b0042020 con 0x7f26ac003a00 2024-09-11T00:30:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.426+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c41de2d0 con 0x7f26ac003a00 2024-09-11T00:30:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.427+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c41df600 con 0x7f26ac003a00 2024-09-11T00:30:20.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.427+0000 7f26c1600640 1 -- 172.21.15.31:0/3750126278 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26b0009810 con 0x7f26ac003a00 2024-09-11T00:30:20.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.428+0000 7f26c1600640 1 -- 172.21.15.31:0/3750126278 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b0037030 con 0x7f26ac003a00 2024-09-11T00:30:20.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.431+0000 7f5575e00640 1 Processor -- start 2024-09-11T00:30:20.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.432+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 shutdown_connections 2024-09-11T00:30:20.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.432+0000 7f1040c00640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f103c14ea20 0x7f103c0bfae0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.432+0000 7f1040c00640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f103c0cc5b0 0x7f103c14c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.432+0000 7f1040c00640 1 --2- 172.21.15.31:0/2565397410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f103c0bf2c0 0x7f103c0c2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.432+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 >> 172.21.15.31:0/2565397410 conn(0x7f103c0addf0 msgr2=0x7f103c1560a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.432+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 shutdown_connections 2024-09-11T00:30:20.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.433+0000 7f1040c00640 1 -- 172.21.15.31:0/2565397410 wait complete. 2024-09-11T00:30:20.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.435+0000 7f5575e00640 1 -- start start 2024-09-11T00:30:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.436+0000 7f5575e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55700c22c0 0x7f55700bfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.436+0000 7f5575e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55700c4320 0x7f55700c1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.436+0000 7f5575e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55700c4700 0x7f55700c0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.436+0000 7f5575e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5570157130 con 0x7f55700c22c0 2024-09-11T00:30:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.436+0000 7f5575e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f55700c4b70 con 0x7f55700c4320 2024-09-11T00:30:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.436+0000 7f5575e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f557015a230 con 0x7f55700c4700 2024-09-11T00:30:20.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.435+0000 7f26c1600640 1 -- 172.21.15.31:0/3750126278 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f26b003d070 con 0x7f26ac003a00 2024-09-11T00:30:20.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.437+0000 7f5574a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55700c22c0 0x7f55700bfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.437+0000 7f5574a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55700c22c0 0x7f55700bfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34422/0 (socket says 172.21.15.31:34422) 2024-09-11T00:30:20.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.437+0000 7f5574a00640 1 -- 172.21.15.31:0/1844210541 learned_addr learned my addr 172.21.15.31:0/1844210541 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.435+0000 7f26c1600640 1 -- 172.21.15.31:0/3750126278 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f26b003eb10 con 0x7f26ac003a00 2024-09-11T00:30:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.438+0000 7f5574a00640 1 -- 172.21.15.31:0/1844210541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55700c4700 msgr2=0x7f55700c0440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T00:30:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.438+0000 7f5574a00640 1 --2- 172.21.15.31:0/1844210541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55700c4700 0x7f55700c0440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.438+0000 7f5574a00640 1 -- 172.21.15.31:0/1844210541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55700c4320 msgr2=0x7f55700c1600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:30:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.438+0000 7f5574a00640 1 --2- 172.21.15.31:0/1844210541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55700c4320 0x7f55700c1600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.438+0000 7f5574a00640 1 -- 172.21.15.31:0/1844210541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55700bf680 con 0x7f55700c22c0 2024-09-11T00:30:20.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.439+0000 7f5574a00640 1 --2- 172.21.15.31:0/1844210541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55700c22c0 0x7f55700bfb60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f555c00c480 tx=0x7f555c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.439+0000 7f556cc00640 1 -- 172.21.15.31:0/1844210541 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c0052d0 con 0x7f55700c22c0 2024-09-11T00:30:20.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.440+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7fbc4c005580 con 0x7fbc6c0495e0 2024-09-11T00:30:20.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.439+0000 7f556cc00640 1 -- 172.21.15.31:0/1844210541 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f555c005470 con 0x7f55700c22c0 2024-09-11T00:30:20.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.439+0000 7f556cc00640 1 -- 172.21.15.31:0/1844210541 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c012890 con 0x7f55700c22c0 2024-09-11T00:30:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.443+0000 7f2f96e00640 1 -- 172.21.15.31:0/356080031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f880329d0 msgr2=0x7f2f8804b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.443+0000 7f2f96e00640 1 --2- 172.21.15.31:0/356080031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f880329d0 0x7f2f8804b4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2f9002eb50 tx=0x7f2f900a6950 comp rx=0 tx=0).stop 2024-09-11T00:30:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.444+0000 7f2f96e00640 1 -- 172.21.15.31:0/356080031 shutdown_connections 2024-09-11T00:30:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.445+0000 7f2f96e00640 1 --2- 172.21.15.31:0/356080031 >> v2:172.21.15.204:3300/0 conn(0x7f2f88049aa0 0x7f2f880daea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.445+0000 7f2f96e00640 1 --2- 172.21.15.31:0/356080031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f880329d0 0x7f2f8804b4c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.445+0000 7f2f96e00640 1 -- 172.21.15.31:0/356080031 >> 172.21.15.31:0/356080031 conn(0x7f2f88009aa0 msgr2=0x7f2f88009eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.440+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a0005290 con 0x7f26ac003a00 2024-09-11T00:30:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.444+0000 7f26c1600640 1 -- 172.21.15.31:0/3750126278 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f26b004c030 con 0x7f26ac003a00 2024-09-11T00:30:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.445+0000 7f2f96e00640 1 -- 172.21.15.31:0/356080031 shutdown_connections 2024-09-11T00:30:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.445+0000 7f2f96e00640 1 -- 172.21.15.31:0/356080031 wait complete. 2024-09-11T00:30:20.453 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.448+0000 7f5575e00640 1 -- 172.21.15.31:0/1844210541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55700be090 con 0x7f55700c22c0 2024-09-11T00:30:20.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.448+0000 7f5575e00640 1 -- 172.21.15.31:0/1844210541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5570070a30 con 0x7f55700c22c0 2024-09-11T00:30:20.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.449+0000 7f2f96e00640 1 Processor -- start 2024-09-11T00:30:20.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.449+0000 7f2f96e00640 1 -- start start 2024-09-11T00:30:20.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.451+0000 7f2f96e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f880329d0 0x7f2f880dacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.451+0000 7f2f96e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f88049aa0 0x7f2f880db200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.450+0000 7f5575e00640 1 -- 172.21.15.31:0/1844210541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55700ad640 con 0x7f55700c22c0 2024-09-11T00:30:20.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.452+0000 7f556cc00640 1 -- 172.21.15.31:0/1844210541 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f555c0035f0 con 0x7f55700c22c0 2024-09-11T00:30:20.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.452+0000 7f556cc00640 1 -- 172.21.15.31:0/1844210541 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f555c01b890 con 0x7f55700c22c0 2024-09-11T00:30:20.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.453+0000 7fbc6a000640 1 -- 172.21.15.31:0/1978487960 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v12)=0 v12) ==== 115+0+0 (secure 0 0 0) 0x7fbc640193b0 con 0x7fbc6c0495e0 2024-09-11T00:30:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.456+0000 7f2f95800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f88049aa0 0x7f2f880db200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.456+0000 7f2f8f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f880329d0 0x7f2f880dacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.456+0000 7f2f96e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f8804ba00 0x7f2f8817fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.456+0000 7f2f8f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f880329d0 0x7f2f880dacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33938/0 (socket says 172.21.15.31:33938) 2024-09-11T00:30:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.456+0000 7f2f8f400640 1 -- 172.21.15.31:0/2184722061 learned_addr learned my addr 172.21.15.31:0/2184722061 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.456+0000 7f2f96e00640 1 -- 172.21.15.31:0/2184722061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2f880d5460 con 0x7f2f88049aa0 2024-09-11T00:30:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.456+0000 7f2f96e00640 1 -- 172.21.15.31:0/2184722061 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8800d550 con 0x7f2f880329d0 2024-09-11T00:30:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.457+0000 7f2f8fe00640 1 --2- 172.21.15.31:0/2184722061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f8804ba00 0x7f2f8817fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.457+0000 7f2f96e00640 1 -- 172.21.15.31:0/2184722061 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2f88048440 con 0x7f2f8804ba00 2024-09-11T00:30:20.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.458+0000 7f556cc00640 1 -- 172.21.15.31:0/1844210541 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f555c020060 con 0x7f55700c22c0 2024-09-11T00:30:20.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.458+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc6c0495e0 msgr2=0x7fbc6c044730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.458+0000 7fbc78e00640 1 --2- 172.21.15.31:0/1978487960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc6c0495e0 0x7fbc6c044730 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400b410 tx=0x7fbc6400b8d0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.458+0000 7fa301c00640 1 -- 172.21.15.31:0/4278854230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2e0004800 msgr2=0x7fa2e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.460+0000 7f2f95800640 1 -- 172.21.15.31:0/2184722061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f8804ba00 msgr2=0x7f2f8817fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.460+0000 7f2f95800640 1 --2- 172.21.15.31:0/2184722061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f8804ba00 0x7f2f8817fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.460+0000 7f2f95800640 1 -- 172.21.15.31:0/2184722061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f880329d0 msgr2=0x7f2f880dacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.460+0000 7f2f95800640 1 --2- 172.21.15.31:0/2184722061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f880329d0 0x7f2f880dacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.460+0000 7f2f95800640 1 -- 172.21.15.31:0/2184722061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f880d6740 con 0x7f2f88049aa0 2024-09-11T00:30:20.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.460+0000 7f2f8f400640 1 --2- 172.21.15.31:0/2184722061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f880329d0 0x7f2f880dacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:30:20.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.459+0000 7f99ff600640 1 -- 172.21.15.31:0/2803770397 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99f814de40 msgr2=0x7f99f80c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.459+0000 7f99ff600640 1 --2- 172.21.15.31:0/2803770397 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99f814de40 0x7f99f80c43c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f99e8005fd0 tx=0x7f99e802a580 comp rx=0 tx=0).stop 2024-09-11T00:30:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.461+0000 7f99ff600640 1 -- 172.21.15.31:0/2803770397 shutdown_connections 2024-09-11T00:30:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.461+0000 7f99ff600640 1 --2- 172.21.15.31:0/2803770397 >> v2:172.21.15.31:3300/0 conn(0x7f99f80c2330 0x7f99f80c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.461+0000 7f99ff600640 1 --2- 172.21.15.31:0/2803770397 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99f814de40 0x7f99f80c43c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.461+0000 7f99ff600640 1 -- 172.21.15.31:0/2803770397 >> 172.21.15.31:0/2803770397 conn(0x7f99f806d560 msgr2=0x7f99f80c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.461+0000 7f99ff600640 1 -- 172.21.15.31:0/2803770397 shutdown_connections 2024-09-11T00:30:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.461+0000 7f99ff600640 1 -- 172.21.15.31:0/2803770397 wait complete. 2024-09-11T00:30:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.458+0000 7fa301c00640 1 --2- 172.21.15.31:0/4278854230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2e0004800 0x7fa2e0004c80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c390 tx=0x7fa2e0005830 comp rx=0 tx=0).stop 2024-09-11T00:30:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.458+0000 7fa301c00640 1 -- 172.21.15.31:0/4278854230 shutdown_connections 2024-09-11T00:30:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.458+0000 7fa301c00640 1 --2- 172.21.15.31:0/4278854230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2e0004800 0x7fa2e0004c80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.459+0000 7fa301c00640 1 -- 172.21.15.31:0/4278854230 >> 172.21.15.31:0/4278854230 conn(0x7fa2fc0c78c0 msgr2=0x7fa2fc0c7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.460+0000 7fa301c00640 1 -- 172.21.15.31:0/4278854230 shutdown_connections 2024-09-11T00:30:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.460+0000 7fa301c00640 1 -- 172.21.15.31:0/4278854230 wait complete. 2024-09-11T00:30:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.462+0000 7f2f95800640 1 --2- 172.21.15.31:0/2184722061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f88049aa0 0x7f2f880db200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2f8804bde0 tx=0x7f2f7c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.464+0000 7f2f83e00640 1 -- 172.21.15.31:0/2184722061 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c017070 con 0x7f2f88049aa0 2024-09-11T00:30:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.465+0000 7f2f83e00640 1 -- 172.21.15.31:0/2184722061 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f7c0035f0 con 0x7f2f88049aa0 2024-09-11T00:30:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.465+0000 7f2f96e00640 1 -- 172.21.15.31:0/2184722061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f8802f790 con 0x7f2f88049aa0 2024-09-11T00:30:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.465+0000 7f2f96e00640 1 -- 172.21.15.31:0/2184722061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f8802fd50 con 0x7f2f88049aa0 2024-09-11T00:30:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.466+0000 7f2f83e00640 1 -- 172.21.15.31:0/2184722061 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c003a40 con 0x7f2f88049aa0 2024-09-11T00:30:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.464+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 shutdown_connections 2024-09-11T00:30:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.464+0000 7fbc78e00640 1 --2- 172.21.15.31:0/1978487960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc6c0495e0 0x7fbc6c044730 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.464+0000 7fbc78e00640 1 --2- 172.21.15.31:0/1978487960 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc6c00a230 0x7fbc6c0441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.464+0000 7fbc78e00640 1 --2- 172.21.15.31:0/1978487960 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc6c006c40 0x7fbc6c02d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.464+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 >> 172.21.15.31:0/1978487960 conn(0x7fbc6c047c50 msgr2=0x7fbc6c0d66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.468+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 shutdown_connections 2024-09-11T00:30:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.468+0000 7fbc78e00640 1 -- 172.21.15.31:0/1978487960 wait complete. 2024-09-11T00:30:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.468+0000 7f99ff600640 1 Processor -- start 2024-09-11T00:30:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.468+0000 7f99ff600640 1 -- start start 2024-09-11T00:30:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.471+0000 7f2f83e00640 1 -- 172.21.15.31:0/2184722061 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f2f7c003ca0 con 0x7f2f88049aa0 2024-09-11T00:30:20.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.471+0000 7f2f83e00640 1 -- 172.21.15.31:0/2184722061 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2f7c012830 con 0x7f2f88049aa0 2024-09-11T00:30:20.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.470+0000 7f99ff600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99f80c2330 0x7f99f81f1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.470+0000 7f99ff600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f814de40 0x7f99f8151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.470+0000 7f99ff600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99f8153e90 0x7f99f8147240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.470+0000 7f99ff600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f99f814e220 con 0x7f99f814de40 2024-09-11T00:30:20.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.470+0000 7f99ff600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f99f80c7030 con 0x7f99f8153e90 2024-09-11T00:30:20.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.470+0000 7f99ff600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f99f814e3a0 con 0x7f99f80c2330 2024-09-11T00:30:20.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.463+0000 7fa301c00640 1 Processor -- start 2024-09-11T00:30:20.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.476+0000 7f99fcc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99f80c2330 0x7f99f81f1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.479 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.476+0000 7f2f81600640 1 -- 172.21.15.31:0/2184722061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f64005290 con 0x7f2f88049aa0 2024-09-11T00:30:20.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.476+0000 7f99fcc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99f80c2330 0x7f99f81f1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:49382/0 (socket says 172.21.15.31:49382) 2024-09-11T00:30:20.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.476+0000 7f99fd600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99f8153e90 0x7f99f8147240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.477+0000 7f99fd600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99f8153e90 0x7f99f8147240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33944/0 (socket says 172.21.15.31:33944) 2024-09-11T00:30:20.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.477+0000 7f99fd600640 1 -- 172.21.15.31:0/2186869119 learned_addr learned my addr 172.21.15.31:0/2186869119 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.476+0000 7f99f7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f814de40 0x7f99f8151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.478+0000 7f99fd600640 1 -- 172.21.15.31:0/2186869119 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99f80c2330 msgr2=0x7f99f81f1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.478+0000 7f99fd600640 1 --2- 172.21.15.31:0/2186869119 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99f80c2330 0x7f99f81f1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.479+0000 7f99fd600640 1 -- 172.21.15.31:0/2186869119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f814de40 msgr2=0x7f99f8151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.483+0000 7f99fd600640 1 --2- 172.21.15.31:0/2186869119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f814de40 0x7f99f8151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.483+0000 7f2f83e00640 1 -- 172.21.15.31:0/2184722061 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f2f7c021070 con 0x7f2f88049aa0 2024-09-11T00:30:20.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.483+0000 7f99fd600640 1 -- 172.21.15.31:0/2186869119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f81f1690 con 0x7f99f8153e90 2024-09-11T00:30:20.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.483+0000 7f99f7e00640 1 --2- 172.21.15.31:0/2186869119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f814de40 0x7f99f8151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:30:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.486+0000 7fa301c00640 1 -- start start 2024-09-11T00:30:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.488+0000 7fa301c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0004800 0x7fa2fc1e8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.488+0000 7fa301c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2fc0ab7d0 0x7fa2fc0bd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.488+0000 7fa301c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2fc1d9450 0x7fa2fc1eac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.485+0000 7f99fd600640 1 --2- 172.21.15.31:0/2186869119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99f8153e90 0x7f99f8147240 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f99f000c6b0 tx=0x7f99f000cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.489+0000 7fa2fb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2fc0ab7d0 0x7fa2fc0bd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.489+0000 7fa2fb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2fc0ab7d0 0x7fa2fc0bd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34446/0 (socket says 172.21.15.31:34446) 2024-09-11T00:30:20.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.489+0000 7f99f4c00640 1 -- 172.21.15.31:0/2186869119 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0003b70 con 0x7f99f8153e90 2024-09-11T00:30:20.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7f99f4c00640 1 -- 172.21.15.31:0/2186869119 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99f0003d30 con 0x7f99f8153e90 2024-09-11T00:30:20.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7f99ff600640 1 -- 172.21.15.31:0/2186869119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f80a4260 con 0x7f99f8153e90 2024-09-11T00:30:20.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7fa2fbe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0004800 0x7fa2fc1e8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7fa2fbe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0004800 0x7fa2fc1e8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33952/0 (socket says 172.21.15.31:33952) 2024-09-11T00:30:20.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7f99ff600640 1 -- 172.21.15.31:0/2186869119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f80a5fa0 con 0x7f99f8153e90 2024-09-11T00:30:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7fa2fb400640 1 -- 172.21.15.31:0/2281292338 learned_addr learned my addr 172.21.15.31:0/2281292338 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7f99f4c00640 1 -- 172.21.15.31:0/2186869119 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0005330 con 0x7f99f8153e90 2024-09-11T00:30:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc0c5e90 con 0x7fa2fc0ab7d0 2024-09-11T00:30:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc0c5d10 con 0x7fa2e0004800 2024-09-11T00:30:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa2fc0c83b0 con 0x7fa2fc1d9450 2024-09-11T00:30:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.491+0000 7fa2fbe00640 1 -- 172.21.15.31:0/2281292338 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2fc1d9450 msgr2=0x7fa2fc1eac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.490+0000 7fa301200640 1 --2- 172.21.15.31:0/2281292338 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2fc1d9450 0x7fa2fc1eac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.492+0000 7fa2fbe00640 1 --2- 172.21.15.31:0/2281292338 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2fc1d9450 0x7fa2fc1eac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.492+0000 7fa2fbe00640 1 -- 172.21.15.31:0/2281292338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2fc0ab7d0 msgr2=0x7fa2fc0bd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.492+0000 7fa2fbe00640 1 --2- 172.21.15.31:0/2281292338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2fc0ab7d0 0x7fa2fc0bd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.492+0000 7fa2fbe00640 1 -- 172.21.15.31:0/2281292338 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc0c15c0 con 0x7fa2e0004800 2024-09-11T00:30:20.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.492+0000 7fa301200640 1 --2- 172.21.15.31:0/2281292338 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2fc1d9450 0x7fa2fc1eac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:20.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.492+0000 7fa2fb400640 1 --2- 172.21.15.31:0/2281292338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2fc0ab7d0 0x7fa2fc0bd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:30:20.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.492+0000 7fa2fbe00640 1 --2- 172.21.15.31:0/2281292338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0004800 0x7fa2fc1e8910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400be50 tx=0x7fa2f400bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.491+0000 7f99f4c00640 1 -- 172.21.15.31:0/2186869119 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f99f0005510 con 0x7f99f8153e90 2024-09-11T00:30:20.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.491+0000 7f99f4c00640 1 -- 172.21.15.31:0/2186869119 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f99f0012940 con 0x7f99f8153e90 2024-09-11T00:30:20.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.503+0000 7f99ff600640 1 -- 172.21.15.31:0/2186869119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d4005290 con 0x7f99f8153e90 2024-09-11T00:30:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.502+0000 7fa2f3e00640 1 -- 172.21.15.31:0/2281292338 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4019070 con 0x7fa2e0004800 2024-09-11T00:30:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.506+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2fc1d9170 con 0x7fa2e0004800 2024-09-11T00:30:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.506+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2fc0c19e0 con 0x7fa2e0004800 2024-09-11T00:30:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.506+0000 7fa2f3e00640 1 -- 172.21.15.31:0/2281292338 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2f4014440 con 0x7fa2e0004800 2024-09-11T00:30:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.506+0000 7fa2f3e00640 1 -- 172.21.15.31:0/2281292338 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4014870 con 0x7fa2e0004800 2024-09-11T00:30:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.506+0000 7f99f4c00640 1 -- 172.21.15.31:0/2186869119 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f99f81f1690 con 0x7f99f8153e90 2024-09-11T00:30:20.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.509+0000 7fa2f3e00640 1 -- 172.21.15.31:0/2281292338 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fa2f4014a90 con 0x7fa2e0004800 2024-09-11T00:30:20.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.509+0000 7fa2f3e00640 1 -- 172.21.15.31:0/2281292338 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa2f401d740 con 0x7fa2e0004800 2024-09-11T00:30:20.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.520+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2d4005290 con 0x7fa2e0004800 2024-09-11T00:30:20.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.524+0000 7fa2f3e00640 1 -- 172.21.15.31:0/2281292338 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fa2f401d990 con 0x7fa2e0004800 2024-09-11T00:30:20.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.541+0000 7f1be6000640 1 -- 172.21.15.31:0/1195986104 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f1bc8005850 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.552+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f0cec0430d0 con 0x7f0cec006be0 2024-09-11T00:30:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.555+0000 7f1bf0c00640 1 -- 172.21.15.31:0/1195986104 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v13)=0 v13) ==== 121+0+0 (secure 0 0 0) 0x7f1be0018610 con 0x7f1bf40a6fb0 2024-09-11T00:30:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.556+0000 7f5cf2000640 1 -- 172.21.15.31:0/3578596779 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7f5cec0acd00 con 0x7f5cec06e4e0 2024-09-11T00:30:20.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.557+0000 7f1be6000640 1 -- 172.21.15.31:0/1195986104 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bf40a6fb0 msgr2=0x7f1bf4154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.558+0000 7f1be6000640 1 --2- 172.21.15.31:0/1195986104 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bf40a6fb0 0x7f1bf4154560 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1be000e9c0 tx=0x7f1be000ee80 comp rx=0 tx=0).stop 2024-09-11T00:30:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.558+0000 7f1be6000640 1 -- 172.21.15.31:0/1195986104 shutdown_connections 2024-09-11T00:30:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.558+0000 7f1be6000640 1 --2- 172.21.15.31:0/1195986104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bf414f990 0x7f1bf4154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.558+0000 7f1be6000640 1 --2- 172.21.15.31:0/1195986104 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bf40a6fb0 0x7f1bf4154560 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.558+0000 7f1be6000640 1 --2- 172.21.15.31:0/1195986104 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bf406e170 0x7f1bf4073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.558+0000 7f1be6000640 1 -- 172.21.15.31:0/1195986104 >> 172.21.15.31:0/1195986104 conn(0x7f1bf40a9670 msgr2=0x7f1bf40c6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.559+0000 7f1be6000640 1 -- 172.21.15.31:0/1195986104 shutdown_connections 2024-09-11T00:30:20.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.559+0000 7f1be6000640 1 -- 172.21.15.31:0/1195986104 wait complete. 2024-09-11T00:30:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.562+0000 7f0ceaa00640 1 -- 172.21.15.31:0/494291203 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v14)=0 v14) ==== 116+0+0 (secure 0 0 0) 0x7f0cf4095d70 con 0x7f0cec006be0 2024-09-11T00:30:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.562+0000 7f19e7400640 1 -- 172.21.15.31:0/1688659756 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f19d0005580 con 0x7f19f80bf440 2024-09-11T00:30:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.565+0000 7fc440e00640 1 -- 172.21.15.31:0/290482702 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc43c0cbce0 msgr2=0x7fc43c1569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.565+0000 7fc440e00640 1 --2- 172.21.15.31:0/290482702 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc43c0cbce0 0x7fc43c1569b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009860 tx=0x7fc42c02a440 comp rx=0 tx=0).stop 2024-09-11T00:30:20.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0cec006be0 msgr2=0x7f0cec044300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7fc440e00640 1 -- 172.21.15.31:0/290482702 shutdown_connections 2024-09-11T00:30:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7fc440e00640 1 --2- 172.21.15.31:0/290482702 >> v2:172.21.15.204:3300/0 conn(0x7fc43c0b13b0 0x7fc43c0c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7f0cf9c00640 1 --2- 172.21.15.31:0/494291203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0cec006be0 0x7f0cec044300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0cf406a110 tx=0x7f0cf40772e0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 shutdown_connections 2024-09-11T00:30:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7f0cf9c00640 1 --2- 172.21.15.31:0/494291203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0cec04dbd0 0x7f0cec0472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7f0cf9c00640 1 --2- 172.21.15.31:0/494291203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cec04d530 0x7f0cec17bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7f0cf9c00640 1 --2- 172.21.15.31:0/494291203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0cec006be0 0x7f0cec044300 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 >> 172.21.15.31:0/494291203 conn(0x7f0cec04c0b0 msgr2=0x7f0cec0335c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7fc440e00640 1 --2- 172.21.15.31:0/290482702 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc43c0cbce0 0x7fc43c1569b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7fc440e00640 1 -- 172.21.15.31:0/290482702 >> 172.21.15.31:0/290482702 conn(0x7fc43c0688e0 msgr2=0x7fc43c068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7fc440e00640 1 -- 172.21.15.31:0/290482702 shutdown_connections 2024-09-11T00:30:20.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.568+0000 7fc440e00640 1 -- 172.21.15.31:0/290482702 wait complete. 2024-09-11T00:30:20.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.569+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 shutdown_connections 2024-09-11T00:30:20.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.569+0000 7f0cf9c00640 1 -- 172.21.15.31:0/494291203 wait complete. 2024-09-11T00:30:20.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.571+0000 7f5ce8c00640 1 -- 172.21.15.31:0/3578596779 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v15)=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f5cec0b19a0 con 0x7f5cec06e4e0 2024-09-11T00:30:20.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.577+0000 7f5ce2000640 1 -- 172.21.15.31:0/3578596779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5cec06e4e0 msgr2=0x7f5cec0ad020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.577+0000 7f5ce2000640 1 --2- 172.21.15.31:0/3578596779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5cec06e4e0 0x7f5cec0ad020 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400be20 tx=0x7f5ce4009f70 comp rx=0 tx=0).stop 2024-09-11T00:30:20.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.577+0000 7f5ce2000640 1 -- 172.21.15.31:0/3578596779 shutdown_connections 2024-09-11T00:30:20.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.577+0000 7f5ce2000640 1 --2- 172.21.15.31:0/3578596779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cec1e05a0 0x7f5cec1587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.577+0000 7f5ce2000640 1 --2- 172.21.15.31:0/3578596779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5cec1597b0 0x7f5cec0adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.577+0000 7f5ce2000640 1 --2- 172.21.15.31:0/3578596779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5cec06e4e0 0x7f5cec0ad020 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.577+0000 7f5ce2000640 1 -- 172.21.15.31:0/3578596779 >> 172.21.15.31:0/3578596779 conn(0x7f5cec0cbaa0 msgr2=0x7f5cec157da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.580+0000 7fc440e00640 1 Processor -- start 2024-09-11T00:30:20.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.580+0000 7fc440e00640 1 -- start start 2024-09-11T00:30:20.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.581+0000 7f19f2000640 1 -- 172.21.15.31:0/1688659756 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v16)=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f19e8003910 con 0x7f19f80bf440 2024-09-11T00:30:20.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.583+0000 7f5ce2000640 1 -- 172.21.15.31:0/3578596779 shutdown_connections 2024-09-11T00:30:20.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.583+0000 7f5ce2000640 1 -- 172.21.15.31:0/3578596779 wait complete. 2024-09-11T00:30:20.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.585+0000 7f19e7400640 1 -- 172.21.15.31:0/1688659756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80bf440 msgr2=0x7f19f80bf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.585+0000 7f19e7400640 1 --2- 172.21.15.31:0/1688659756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80bf440 0x7f19f80bf820 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f19e8011bb0 tx=0x7f19e8007590 comp rx=0 tx=0).stop 2024-09-11T00:30:20.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.585+0000 7f19e7400640 1 -- 172.21.15.31:0/1688659756 shutdown_connections 2024-09-11T00:30:20.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.585+0000 7f19e7400640 1 --2- 172.21.15.31:0/1688659756 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19f80bd030 0x7f19f81f5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.585+0000 7f19e7400640 1 --2- 172.21.15.31:0/1688659756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19f81f9880 0x7f19f80bcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.585+0000 7f19e7400640 1 --2- 172.21.15.31:0/1688659756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19f80bf440 0x7f19f80bf820 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.585+0000 7f19e7400640 1 -- 172.21.15.31:0/1688659756 >> 172.21.15.31:0/1688659756 conn(0x7f19f80a9670 msgr2=0x7f19f80c6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.585+0000 7f19e7400640 1 -- 172.21.15.31:0/1688659756 shutdown_connections 2024-09-11T00:30:20.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.585+0000 7f19e7400640 1 -- 172.21.15.31:0/1688659756 wait complete. 2024-09-11T00:30:20.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.592+0000 7f8294c00640 1 -- 172.21.15.31:0/3754749349 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f8278005580 con 0x7f82a414f720 2024-09-11T00:30:20.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.582+0000 7fc440e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc43c0b13b0 0x7fc43c154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.592+0000 7fc440e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc43c0c9df0 0x7fc43c1f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.593+0000 7fc43aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc43c0c9df0 0x7fc43c1f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.594+0000 7fc43aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc43c0c9df0 0x7fc43c1f34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34452/0 (socket says 172.21.15.31:34452) 2024-09-11T00:30:20.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.594+0000 7fc43aa00640 1 -- 172.21.15.31:0/2140453256 learned_addr learned my addr 172.21.15.31:0/2140453256 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.594+0000 7fc440e00640 1 --2- 172.21.15.31:0/2140453256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc43c0cbce0 0x7fc43c1f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.594+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc43c0cc240 con 0x7fc43c0c9df0 2024-09-11T00:30:20.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.594+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc43c0cc0c0 con 0x7fc43c0b13b0 2024-09-11T00:30:20.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.594+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc43c0c86a0 con 0x7fc43c0cbce0 2024-09-11T00:30:20.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.594+0000 7fc43b400640 1 --2- 172.21.15.31:0/2140453256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc43c0b13b0 0x7fc43c154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.595+0000 7fc43aa00640 1 -- 172.21.15.31:0/2140453256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc43c0cbce0 msgr2=0x7fc43c1f3a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:30:20.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.595+0000 7fc43aa00640 1 --2- 172.21.15.31:0/2140453256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc43c0cbce0 0x7fc43c1f3a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.595+0000 7fc43aa00640 1 -- 172.21.15.31:0/2140453256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc43c0b13b0 msgr2=0x7fc43c154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.595+0000 7fc43aa00640 1 --2- 172.21.15.31:0/2140453256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc43c0b13b0 0x7fc43c154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.595+0000 7fc43aa00640 1 -- 172.21.15.31:0/2140453256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43c1e1910 con 0x7fc43c0c9df0 2024-09-11T00:30:20.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.596+0000 7fc43aa00640 1 --2- 172.21.15.31:0/2140453256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc43c0c9df0 0x7fc43c1f34e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc42800c500 tx=0x7fc42800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.596+0000 7fc433400640 1 -- 172.21.15.31:0/2140453256 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc428016020 con 0x7fc43c0c9df0 2024-09-11T00:30:20.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.596+0000 7fc433400640 1 -- 172.21.15.31:0/2140453256 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc428012920 con 0x7fc43c0c9df0 2024-09-11T00:30:20.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.596+0000 7fc433400640 1 -- 172.21.15.31:0/2140453256 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc428012d50 con 0x7fc43c0c9df0 2024-09-11T00:30:20.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.596+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc43c1f4170 con 0x7fc43c0c9df0 2024-09-11T00:30:20.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.596+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43c1e1b90 con 0x7fc43c0c9df0 2024-09-11T00:30:20.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.598+0000 7fc43b400640 1 --2- 172.21.15.31:0/2140453256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc43c0b13b0 0x7fc43c154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:20.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.600+0000 7fc433400640 1 -- 172.21.15.31:0/2140453256 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fc428023060 con 0x7fc43c0c9df0 2024-09-11T00:30:20.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.600+0000 7fc433400640 1 -- 172.21.15.31:0/2140453256 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc42801aac0 con 0x7fc43c0c9df0 2024-09-11T00:30:20.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.600+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc414005290 con 0x7fc43c0c9df0 2024-09-11T00:30:20.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.605+0000 7fc433400640 1 -- 172.21.15.31:0/2140453256 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fc43c1e1910 con 0x7fc43c0c9df0 2024-09-11T00:30:20.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.615+0000 7f8297400640 1 -- 172.21.15.31:0/3754749349 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v17)=0 v17) ==== 119+0+0 (secure 0 0 0) 0x7f829c034de0 con 0x7f82a414f720 2024-09-11T00:30:20.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.619+0000 7f8294c00640 1 -- 172.21.15.31:0/3754749349 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82a414f720 msgr2=0x7f82a4073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.619+0000 7f8294c00640 1 --2- 172.21.15.31:0/3754749349 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82a414f720 0x7f82a4073ef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f829c02cbc0 tx=0x7f829c00c230 comp rx=0 tx=0).stop 2024-09-11T00:30:20.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.619+0000 7f8294c00640 1 -- 172.21.15.31:0/3754749349 shutdown_connections 2024-09-11T00:30:20.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.619+0000 7f8294c00640 1 --2- 172.21.15.31:0/3754749349 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82a414f720 0x7f82a4073ef0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.619+0000 7f8294c00640 1 --2- 172.21.15.31:0/3754749349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82a40c22c0 0x7f82a40739b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.619+0000 7f8294c00640 1 --2- 172.21.15.31:0/3754749349 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82a406e170 0x7f82a40c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.619+0000 7f8294c00640 1 -- 172.21.15.31:0/3754749349 >> 172.21.15.31:0/3754749349 conn(0x7f82a40a9670 msgr2=0x7f82a40a9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.620+0000 7f8294c00640 1 -- 172.21.15.31:0/3754749349 shutdown_connections 2024-09-11T00:30:20.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.620+0000 7f8294c00640 1 -- 172.21.15.31:0/3754749349 wait complete. 2024-09-11T00:30:20.630 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.630 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.630 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.635 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.635 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.638+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f26a0005580 con 0x7f26ac003a00 2024-09-11T00:30:20.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.645+0000 7f26c1600640 1 -- 172.21.15.31:0/3750126278 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v18)=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f26b004c030 con 0x7f26ac003a00 2024-09-11T00:30:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.647+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26ac003a00 msgr2=0x7f26c41ea280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.647+0000 7f26ca800640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26ac003a00 0x7f26c41ea280 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f26b000e740 tx=0x7f26b0005d50 comp rx=0 tx=0).stop 2024-09-11T00:30:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.647+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 shutdown_connections 2024-09-11T00:30:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.647+0000 7f26ca800640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26c41df150 0x7f26c4150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.647+0000 7f26ca800640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26c4073eb0 0x7f26c40a6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.647+0000 7f26ca800640 1 --2- 172.21.15.31:0/3750126278 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26ac003a00 0x7f26c41ea280 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f26b000e740 tx=0x7f26b0005d50 comp rx=0 tx=0).stop 2024-09-11T00:30:20.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.647+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 >> 172.21.15.31:0/3750126278 conn(0x7f26c40c78c0 msgr2=0x7f26c4154c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.648+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 shutdown_connections 2024-09-11T00:30:20.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.648+0000 7f26ca800640 1 -- 172.21.15.31:0/3750126278 wait complete. 2024-09-11T00:30:20.652 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.649+0000 7f5575e00640 1 -- 172.21.15.31:0/1844210541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f557015a680 con 0x7f55700c22c0 2024-09-11T00:30:20.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.655+0000 7f2f81600640 1 -- 172.21.15.31:0/2184722061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f2f64005580 con 0x7f2f88049aa0 2024-09-11T00:30:20.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.656+0000 7f556cc00640 1 -- 172.21.15.31:0/1844210541 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v19)=0 v19) ==== 120+0+0 (secure 0 0 0) 0x7f555c020060 con 0x7f55700c22c0 2024-09-11T00:30:20.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.657+0000 7f5575e00640 1 -- 172.21.15.31:0/1844210541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55700c22c0 msgr2=0x7f55700bfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.658+0000 7f5575e00640 1 --2- 172.21.15.31:0/1844210541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55700c22c0 0x7f55700bfb60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f555c00c480 tx=0x7f555c00c940 comp rx=0 tx=0).stop 2024-09-11T00:30:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.658+0000 7f5575e00640 1 -- 172.21.15.31:0/1844210541 shutdown_connections 2024-09-11T00:30:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.658+0000 7f5575e00640 1 --2- 172.21.15.31:0/1844210541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55700c4700 0x7f55700c0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.658+0000 7f5575e00640 1 --2- 172.21.15.31:0/1844210541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55700c4320 0x7f55700c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.658+0000 7f5575e00640 1 --2- 172.21.15.31:0/1844210541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55700c22c0 0x7f55700bfb60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.658+0000 7f5575e00640 1 -- 172.21.15.31:0/1844210541 >> 172.21.15.31:0/1844210541 conn(0x7f55700c78c0 msgr2=0x7f5570154060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.658+0000 7f5575e00640 1 -- 172.21.15.31:0/1844210541 shutdown_connections 2024-09-11T00:30:20.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.658+0000 7f5575e00640 1 -- 172.21.15.31:0/1844210541 wait complete. 2024-09-11T00:30:20.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.663+0000 7f2f83e00640 1 -- 172.21.15.31:0/2184722061 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v20)=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f2f7c012a80 con 0x7f2f88049aa0 2024-09-11T00:30:20.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.666+0000 7f2f81600640 1 -- 172.21.15.31:0/2184722061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f88049aa0 msgr2=0x7f2f880db200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.666+0000 7f2f81600640 1 --2- 172.21.15.31:0/2184722061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f88049aa0 0x7f2f880db200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2f8804bde0 tx=0x7f2f7c00c820 comp rx=0 tx=0).stop 2024-09-11T00:30:20.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.666+0000 7f2f81600640 1 -- 172.21.15.31:0/2184722061 shutdown_connections 2024-09-11T00:30:20.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.666+0000 7f2f81600640 1 --2- 172.21.15.31:0/2184722061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f8804ba00 0x7f2f8817fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.666+0000 7f2f81600640 1 --2- 172.21.15.31:0/2184722061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f88049aa0 0x7f2f880db200 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.666+0000 7f2f81600640 1 --2- 172.21.15.31:0/2184722061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f880329d0 0x7f2f880dacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.666+0000 7f2f81600640 1 -- 172.21.15.31:0/2184722061 >> 172.21.15.31:0/2184722061 conn(0x7f2f88009aa0 msgr2=0x7f2f88008e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.666+0000 7f2f81600640 1 -- 172.21.15.31:0/2184722061 shutdown_connections 2024-09-11T00:30:20.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.666+0000 7f2f81600640 1 -- 172.21.15.31:0/2184722061 wait complete. 2024-09-11T00:30:20.671 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.681 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.702+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fc414005580 con 0x7fc43c0c9df0 2024-09-11T00:30:20.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.710+0000 7fc433400640 1 -- 172.21.15.31:0/2140453256 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v21)=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7fc43c1e1910 con 0x7fc43c0c9df0 2024-09-11T00:30:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.711+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc43c0c9df0 msgr2=0x7fc43c1f34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.711+0000 7fc440e00640 1 --2- 172.21.15.31:0/2140453256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc43c0c9df0 0x7fc43c1f34e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc42800c500 tx=0x7fc42800c9c0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.711+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 shutdown_connections 2024-09-11T00:30:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.711+0000 7fc440e00640 1 --2- 172.21.15.31:0/2140453256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc43c0cbce0 0x7fc43c1f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.711+0000 7fc440e00640 1 --2- 172.21.15.31:0/2140453256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc43c0c9df0 0x7fc43c1f34e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.711+0000 7fc440e00640 1 --2- 172.21.15.31:0/2140453256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc43c0b13b0 0x7fc43c154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.711+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 >> 172.21.15.31:0/2140453256 conn(0x7fc43c0688e0 msgr2=0x7fc43c0b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.711+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 shutdown_connections 2024-09-11T00:30:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.711+0000 7fc440e00640 1 -- 172.21.15.31:0/2140453256 wait complete. 2024-09-11T00:30:20.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.713+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7fa2d4005580 con 0x7fa2e0004800 2024-09-11T00:30:20.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.715+0000 7f99ff600640 1 -- 172.21.15.31:0/2186869119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f99d4005580 con 0x7f99f8153e90 2024-09-11T00:30:20.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.718+0000 7fa2f3e00640 1 -- 172.21.15.31:0/2281292338 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v22)=0 v22) ==== 120+0+0 (secure 0 0 0) 0x7fa2f401db70 con 0x7fa2e0004800 2024-09-11T00:30:20.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.720+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0004800 msgr2=0x7fa2fc1e8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.720+0000 7fa301c00640 1 --2- 172.21.15.31:0/2281292338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0004800 0x7fa2fc1e8910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400be50 tx=0x7fa2f400bf40 comp rx=0 tx=0).stop 2024-09-11T00:30:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.720+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 shutdown_connections 2024-09-11T00:30:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.720+0000 7fa301c00640 1 --2- 172.21.15.31:0/2281292338 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa2fc1d9450 0x7fa2fc1eac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.720+0000 7fa301c00640 1 --2- 172.21.15.31:0/2281292338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa2fc0ab7d0 0x7fa2fc0bd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.720+0000 7fa301c00640 1 --2- 172.21.15.31:0/2281292338 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2e0004800 0x7fa2fc1e8910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.720+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 >> 172.21.15.31:0/2281292338 conn(0x7fa2fc0c78c0 msgr2=0x7fa2fc0743d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.720+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 shutdown_connections 2024-09-11T00:30:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.720+0000 7fa301c00640 1 -- 172.21.15.31:0/2281292338 wait complete. 2024-09-11T00:30:20.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.725+0000 7f99f4c00640 1 -- 172.21.15.31:0/2186869119 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v23)=0 v23) ==== 125+0+0 (secure 0 0 0) 0x7f99f0012b90 con 0x7f99f8153e90 2024-09-11T00:30:20.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.725+0000 7f99ff600640 1 -- 172.21.15.31:0/2186869119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99f8153e90 msgr2=0x7f99f8147240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.725+0000 7f99ff600640 1 --2- 172.21.15.31:0/2186869119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99f8153e90 0x7f99f8147240 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f99f000c6b0 tx=0x7f99f000cb70 comp rx=0 tx=0).stop 2024-09-11T00:30:20.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.725+0000 7f99ff600640 1 -- 172.21.15.31:0/2186869119 shutdown_connections 2024-09-11T00:30:20.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.725+0000 7f99ff600640 1 --2- 172.21.15.31:0/2186869119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99f8153e90 0x7f99f8147240 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.725+0000 7f99ff600640 1 --2- 172.21.15.31:0/2186869119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99f814de40 0x7f99f8151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.725+0000 7f99ff600640 1 --2- 172.21.15.31:0/2186869119 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99f80c2330 0x7f99f81f1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.725+0000 7f99ff600640 1 -- 172.21.15.31:0/2186869119 >> 172.21.15.31:0/2186869119 conn(0x7f99f806d560 msgr2=0x7f99f80c47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.725+0000 7f99ff600640 1 -- 172.21.15.31:0/2186869119 shutdown_connections 2024-09-11T00:30:20.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.725+0000 7f99ff600640 1 -- 172.21.15.31:0/2186869119 wait complete. 2024-09-11T00:30:20.740 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.741 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.747 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi031') 2024-09-11T00:30:20.748 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T00:30:20.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T00:30:20.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.942+0000 7f57d8e00640 1 -- 172.21.15.31:0/1243099413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57d414ffb0 msgr2=0x7f57d4155f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.942+0000 7f57d8e00640 1 --2- 172.21.15.31:0/1243099413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57d414ffb0 0x7f57d4155f20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f57d4069860 tx=0x7f57cc02a8b0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.942+0000 7f57d8e00640 1 -- 172.21.15.31:0/1243099413 shutdown_connections 2024-09-11T00:30:20.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.942+0000 7f57d8e00640 1 --2- 172.21.15.31:0/1243099413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57d414ffb0 0x7f57d4155f20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.942+0000 7f57d8e00640 1 --2- 172.21.15.31:0/1243099413 >> v2:172.21.15.204:3300/0 conn(0x7f57d40cc060 0x7f57d4159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.942+0000 7f57d8e00640 1 -- 172.21.15.31:0/1243099413 >> 172.21.15.31:0/1243099413 conn(0x7f57d4068ac0 msgr2=0x7f57d4068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:20.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.942+0000 7f57d8e00640 1 -- 172.21.15.31:0/1243099413 shutdown_connections 2024-09-11T00:30:20.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.942+0000 7f57d8e00640 1 -- 172.21.15.31:0/1243099413 wait complete. 2024-09-11T00:30:20.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.944+0000 7f57d8e00640 1 Processor -- start 2024-09-11T00:30:20.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.944+0000 7f57d8e00640 1 -- start start 2024-09-11T00:30:20.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.945+0000 7f57d8e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57d40c99a0 0x7f57d4148480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.945+0000 7f57d8e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57d40cc060 0x7f57d41e18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.945+0000 7f57d8e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57d414ffb0 0x7f57d414b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.945+0000 7f57d8e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f57d40cc440 con 0x7f57d414ffb0 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.945+0000 7f57d8e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f57d40c66e0 con 0x7f57d40c99a0 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.945+0000 7f57d8e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f57d40cc5c0 con 0x7f57d40cc060 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.945+0000 7f57d3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57d414ffb0 0x7f57d414b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.945+0000 7f57d3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57d414ffb0 0x7f57d414b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34474/0 (socket says 172.21.15.31:34474) 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.945+0000 7f57d3e00640 1 -- 172.21.15.31:0/2081150095 learned_addr learned my addr 172.21.15.31:0/2081150095 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.945+0000 7f57d3400640 1 --2- 172.21.15.31:0/2081150095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57d40c99a0 0x7f57d4148480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57cbe00640 1 --2- 172.21.15.31:0/2081150095 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57d40cc060 0x7f57d41e18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57d3e00640 1 -- 172.21.15.31:0/2081150095 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57d40cc060 msgr2=0x7f57d41e18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57d3e00640 1 --2- 172.21.15.31:0/2081150095 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57d40cc060 0x7f57d41e18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57d3e00640 1 -- 172.21.15.31:0/2081150095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57d40c99a0 msgr2=0x7f57d4148480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57d3e00640 1 --2- 172.21.15.31:0/2081150095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57d40c99a0 0x7f57d4148480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57d3e00640 1 -- 172.21.15.31:0/2081150095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d41dfb70 con 0x7f57d414ffb0 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57cbe00640 1 --2- 172.21.15.31:0/2081150095 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57d40cc060 0x7f57d41e18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57d3e00640 1 --2- 172.21.15.31:0/2081150095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57d414ffb0 0x7f57d414b0b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f57cc005fd0 tx=0x7f57cc0316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57cb400640 1 -- 172.21.15.31:0/2081150095 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc040070 con 0x7f57d414ffb0 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57cb400640 1 -- 172.21.15.31:0/2081150095 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57cc004040 con 0x7f57d414ffb0 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57cb400640 1 -- 172.21.15.31:0/2081150095 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc004470 con 0x7f57d414ffb0 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.946+0000 7f57d8e00640 1 -- 172.21.15.31:0/2081150095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57d41481a0 con 0x7f57d414ffb0 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.947+0000 7f57d8e00640 1 -- 172.21.15.31:0/2081150095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57d4159070 con 0x7f57d414ffb0 2024-09-11T00:30:20.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.947+0000 7f57cb400640 1 -- 172.21.15.31:0/2081150095 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f57cc004610 con 0x7f57d414ffb0 2024-09-11T00:30:20.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.947+0000 7f57cb400640 1 -- 172.21.15.31:0/2081150095 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f57cc045e00 con 0x7f57d414ffb0 2024-09-11T00:30:20.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.948+0000 7f57d8e00640 1 -- 172.21.15.31:0/2081150095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57d4158b80 con 0x7f57d414ffb0 2024-09-11T00:30:20.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:20.950+0000 7f57cb400640 1 -- 172.21.15.31:0/2081150095 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f57cc04c030 con 0x7f57d414ffb0 2024-09-11T00:30:21.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.019+0000 7f57d8e00640 1 -- 172.21.15.31:0/2081150095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f57d4159f20 con 0x7f57d414ffb0 2024-09-11T00:30:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.021+0000 7f57cb400640 1 -- 172.21.15.31:0/2081150095 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f57cc045460 con 0x7f57d414ffb0 2024-09-11T00:30:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-09-11T00:30:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.022+0000 7f57d8e00640 1 -- 172.21.15.31:0/2081150095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57d414ffb0 msgr2=0x7f57d414b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.022+0000 7f57d8e00640 1 --2- 172.21.15.31:0/2081150095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57d414ffb0 0x7f57d414b0b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f57cc005fd0 tx=0x7f57cc0316f0 comp rx=0 tx=0).stop 2024-09-11T00:30:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.022+0000 7f57d8e00640 1 -- 172.21.15.31:0/2081150095 shutdown_connections 2024-09-11T00:30:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.022+0000 7f57d8e00640 1 --2- 172.21.15.31:0/2081150095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57d414ffb0 0x7f57d414b0b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.022+0000 7f57d8e00640 1 --2- 172.21.15.31:0/2081150095 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57d40cc060 0x7f57d41e18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.022+0000 7f57d8e00640 1 --2- 172.21.15.31:0/2081150095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57d40c99a0 0x7f57d4148480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.022+0000 7f57d8e00640 1 -- 172.21.15.31:0/2081150095 >> 172.21.15.31:0/2081150095 conn(0x7f57d4068ac0 msgr2=0x7f57d4154ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:21.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.022+0000 7f57d8e00640 1 -- 172.21.15.31:0/2081150095 shutdown_connections 2024-09-11T00:30:21.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:21.022+0000 7f57d8e00640 1 -- 172.21.15.31:0/2081150095 wait complete. 2024-09-11T00:30:21.039 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T00:30:21.039 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T00:30:21.039 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:21.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T00:30:21.095 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:21.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T00:30:21.191 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:21.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T00:30:21.287 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:30:21.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-11T00:30:21.383 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:21.384 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-11T00:30:21.442 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:21.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-11T00:30:21.539 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:21.540 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-11T00:30:21.637 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:30:21.637 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-11T00:30:21.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 33da8841-7f3f-47d7-81aa-d0775619d938 0 2024-09-11T00:30:21.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.927+0000 7fe29c600640 1 -- 172.21.15.204:0/3407477919 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe294158130 msgr2=0x7fe294072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:21.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.927+0000 7fe29c600640 1 --2- 172.21.15.204:0/3407477919 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe294158130 0x7fe294072390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe288005fd0 tx=0x7fe28802a5d0 comp rx=0 tx=0).stop 2024-09-11T00:30:21.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.927+0000 7fe29c600640 1 -- 172.21.15.204:0/3407477919 shutdown_connections 2024-09-11T00:30:21.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.927+0000 7fe29c600640 1 --2- 172.21.15.204:0/3407477919 >> v2:172.21.15.31:3300/0 conn(0x7fe2940c23a0 0x7fe2940c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:21.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.927+0000 7fe29c600640 1 --2- 172.21.15.204:0/3407477919 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe294158130 0x7fe294072390 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:21.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.927+0000 7fe29c600640 1 -- 172.21.15.204:0/3407477919 >> 172.21.15.204:0/3407477919 conn(0x7fe29406d480 msgr2=0x7fe2940a9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:21.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.927+0000 7fe29c600640 1 -- 172.21.15.204:0/3407477919 shutdown_connections 2024-09-11T00:30:21.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.927+0000 7fe29c600640 1 -- 172.21.15.204:0/3407477919 wait complete. 2024-09-11T00:30:21.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.929+0000 7fe29c600640 1 Processor -- start 2024-09-11T00:30:21.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.929+0000 7fe29c600640 1 -- start start 2024-09-11T00:30:21.933 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe29c600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2940c23a0 0x7fe2940ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:21.933 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe29c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe294157440 0x7fe29406ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:21.933 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe29c600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe294158130 0x7fe2941e06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:21.933 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe29c600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941549b0 con 0x7fe294157440 2024-09-11T00:30:21.933 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe29c600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe2941546d0 con 0x7fe2940c23a0 2024-09-11T00:30:21.933 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe29c600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe2940c3d50 con 0x7fe294158130 2024-09-11T00:30:21.933 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe299c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2940c23a0 0x7fe2940ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe29a600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe294158130 0x7fe2941e06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe299c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2940c23a0 0x7fe2940ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:60838/0 (socket says 172.21.15.204:60838) 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe299c00640 1 -- 172.21.15.204:0/2893188088 learned_addr learned my addr 172.21.15.204:0/2893188088 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.930+0000 7fe299200640 1 --2- 172.21.15.204:0/2893188088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe294157440 0x7fe29406ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe29a600640 1 -- 172.21.15.204:0/2893188088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2940c23a0 msgr2=0x7fe2940ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe29a600640 1 --2- 172.21.15.204:0/2893188088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2940c23a0 0x7fe2940ad1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe29a600640 1 -- 172.21.15.204:0/2893188088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe294157440 msgr2=0x7fe29406ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe29a600640 1 --2- 172.21.15.204:0/2893188088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe294157440 0x7fe29406ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe29a600640 1 -- 172.21.15.204:0/2893188088 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2940a8e10 con 0x7fe294158130 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe299c00640 1 --2- 172.21.15.204:0/2893188088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2940c23a0 0x7fe2940ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe29a600640 1 --2- 172.21.15.204:0/2893188088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe294158130 0x7fe2941e06b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00c6b0 tx=0x7fe28c00cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:21.934 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe291600640 1 -- 172.21.15.204:0/2893188088 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c003b70 con 0x7fe294158130 2024-09-11T00:30:21.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe291600640 1 -- 172.21.15.204:0/2893188088 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe28c003d10 con 0x7fe294158130 2024-09-11T00:30:21.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe291600640 1 -- 172.21.15.204:0/2893188088 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c005330 con 0x7fe294158130 2024-09-11T00:30:21.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe29c600640 1 -- 172.21.15.204:0/2893188088 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2941511d0 con 0x7fe294158130 2024-09-11T00:30:21.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.931+0000 7fe29c600640 1 -- 172.21.15.204:0/2893188088 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2940ad730 con 0x7fe294158130 2024-09-11T00:30:21.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.932+0000 7fe291600640 1 -- 172.21.15.204:0/2893188088 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fe28c0054d0 con 0x7fe294158130 2024-09-11T00:30:21.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.932+0000 7fe291600640 1 -- 172.21.15.204:0/2893188088 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fe28c012880 con 0x7fe294158130 2024-09-11T00:30:21.935 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.932+0000 7fe29c600640 1 -- 172.21.15.204:0/2893188088 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe294151410 con 0x7fe294158130 2024-09-11T00:30:21.936 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:21.935+0000 7fe291600640 1 -- 172.21.15.204:0/2893188088 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fe294151410 con 0x7fe294158130 2024-09-11T00:30:22.011 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.009+0000 7fe29c600640 1 -- 172.21.15.204:0/2893188088 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd new", "uuid": "33da8841-7f3f-47d7-81aa-d0775619d938", "id": 0} v 0) -- 0x7fe294150f30 con 0x7fe294158130 2024-09-11T00:30:22.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.013+0000 7fe291600640 1 -- 172.21.15.204:0/2893188088 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "33da8841-7f3f-47d7-81aa-d0775619d938", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fe28c012a80 con 0x7fe294158130 2024-09-11T00:30:22.015 INFO:teuthology.orchestra.run.smithi204.stdout:0 2024-09-11T00:30:22.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.014+0000 7fe29c600640 1 -- 172.21.15.204:0/2893188088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe294158130 msgr2=0x7fe2941e06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.014+0000 7fe29c600640 1 --2- 172.21.15.204:0/2893188088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe294158130 0x7fe2941e06b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00c6b0 tx=0x7fe28c00cb70 comp rx=0 tx=0).stop 2024-09-11T00:30:22.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.014+0000 7fe29c600640 1 -- 172.21.15.204:0/2893188088 shutdown_connections 2024-09-11T00:30:22.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.014+0000 7fe29c600640 1 --2- 172.21.15.204:0/2893188088 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe294158130 0x7fe2941e06b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.014+0000 7fe29c600640 1 --2- 172.21.15.204:0/2893188088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe294157440 0x7fe29406ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.014+0000 7fe29c600640 1 --2- 172.21.15.204:0/2893188088 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2940c23a0 0x7fe2940ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.014+0000 7fe29c600640 1 -- 172.21.15.204:0/2893188088 >> 172.21.15.204:0/2893188088 conn(0x7fe29406d480 msgr2=0x7fe2941de8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:22.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.014+0000 7fe29c600640 1 -- 172.21.15.204:0/2893188088 shutdown_connections 2024-09-11T00:30:22.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.014+0000 7fe29c600640 1 -- 172.21.15.204:0/2893188088 wait complete. 2024-09-11T00:30:22.031 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 9d32c08c-6b40-4a83-8b5f-973af86789d0 1 2024-09-11T00:30:22.185 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.183+0000 7f278b200640 1 -- 172.21.15.204:0/1426873485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f278c0c9fd0 msgr2=0x7f278c156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.185 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.183+0000 7f278b200640 1 --2- 172.21.15.204:0/1426873485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f278c0c9fd0 0x7f278c156970 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2778009830 tx=0x7f277802a3f0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.185 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.183+0000 7f278b200640 1 -- 172.21.15.204:0/1426873485 shutdown_connections 2024-09-11T00:30:22.185 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.183+0000 7f278b200640 1 --2- 172.21.15.204:0/1426873485 >> v2:172.21.15.31:3300/0 conn(0x7f278c06f280 0x7f278c0adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.185 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.183+0000 7f278b200640 1 --2- 172.21.15.204:0/1426873485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f278c0c9fd0 0x7f278c156970 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.185 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.183+0000 7f278b200640 1 -- 172.21.15.204:0/1426873485 >> 172.21.15.204:0/1426873485 conn(0x7f278c0cbcc0 msgr2=0x7f278c0cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:22.186 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.183+0000 7f278b200640 1 -- 172.21.15.204:0/1426873485 shutdown_connections 2024-09-11T00:30:22.186 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.183+0000 7f278b200640 1 -- 172.21.15.204:0/1426873485 wait complete. 2024-09-11T00:30:22.186 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.185+0000 7f278b200640 1 Processor -- start 2024-09-11T00:30:22.186 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.185+0000 7f278b200640 1 -- start start 2024-09-11T00:30:22.188 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f278b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f278c06f280 0x7f278c14fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.188 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f278b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f278c0c9fd0 0x7f278c0ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.188 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f278b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f278c153800 0x7f278c1ec1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.188 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f278b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f278c06e020 con 0x7f278c0c9fd0 2024-09-11T00:30:22.188 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f278b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f278c0c69e0 con 0x7f278c06f280 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f278b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f278c06ec50 con 0x7f278c153800 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f2789e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f278c06f280 0x7f278c14fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f278a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f278c153800 0x7f278c1ec1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f2789e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f278c06f280 0x7f278c14fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:60848/0 (socket says 172.21.15.204:60848) 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f278a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f278c153800 0x7f278c1ec1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.204:33052/0 (socket says 172.21.15.204:33052) 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f2789e00640 1 -- 172.21.15.204:0/1260461878 learned_addr learned my addr 172.21.15.204:0/1260461878 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f2789400640 1 --2- 172.21.15.204:0/1260461878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f278c0c9fd0 0x7f278c0ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f2789e00640 1 -- 172.21.15.204:0/1260461878 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f278c153800 msgr2=0x7f278c1ec1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f2789e00640 1 --2- 172.21.15.204:0/1260461878 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f278c153800 0x7f278c1ec1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f2789e00640 1 -- 172.21.15.204:0/1260461878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f278c0c9fd0 msgr2=0x7f278c0ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f2789e00640 1 --2- 172.21.15.204:0/1260461878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f278c0c9fd0 0x7f278c0ad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f2789e00640 1 -- 172.21.15.204:0/1260461878 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278c150550 con 0x7f278c06f280 2024-09-11T00:30:22.189 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.186+0000 7f278a800640 1 --2- 172.21.15.204:0/1260461878 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f278c153800 0x7f278c1ec1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:30:22.190 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.187+0000 7f2789e00640 1 --2- 172.21.15.204:0/1260461878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f278c06f280 0x7f278c14fc00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f277c00c3d0 tx=0x7f277c00c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:22.191 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.187+0000 7f2782000640 1 -- 172.21.15.204:0/1260461878 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c011790 con 0x7f278c06f280 2024-09-11T00:30:22.191 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.187+0000 7f2782000640 1 -- 172.21.15.204:0/1260461878 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f277c017070 con 0x7f278c06f280 2024-09-11T00:30:22.191 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.187+0000 7f2782000640 1 -- 172.21.15.204:0/1260461878 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c011cc0 con 0x7f278c06f280 2024-09-11T00:30:22.191 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.187+0000 7f278b200640 1 -- 172.21.15.204:0/1260461878 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278c150270 con 0x7f278c06f280 2024-09-11T00:30:22.191 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.187+0000 7f278b200640 1 -- 172.21.15.204:0/1260461878 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f278c0b1590 con 0x7f278c06f280 2024-09-11T00:30:22.192 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.187+0000 7f2782000640 1 -- 172.21.15.204:0/1260461878 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f277c01c460 con 0x7f278c06f280 2024-09-11T00:30:22.192 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.187+0000 7f2782000640 1 -- 172.21.15.204:0/1260461878 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f277c0127c0 con 0x7f278c06f280 2024-09-11T00:30:22.192 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.188+0000 7f278b200640 1 -- 172.21.15.204:0/1260461878 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278c1ebae0 con 0x7f278c06f280 2024-09-11T00:30:22.192 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.190+0000 7f2782000640 1 -- 172.21.15.204:0/1260461878 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f277c0129c0 con 0x7f278c06f280 2024-09-11T00:30:22.266 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.264+0000 7f278b200640 1 -- 172.21.15.204:0/1260461878 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9d32c08c-6b40-4a83-8b5f-973af86789d0", "id": 1} v 0) -- 0x7f278c156970 con 0x7f278c06f280 2024-09-11T00:30:22.271 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.269+0000 7f2782000640 1 -- 172.21.15.204:0/1260461878 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9d32c08c-6b40-4a83-8b5f-973af86789d0", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f277c012ba0 con 0x7f278c06f280 2024-09-11T00:30:22.271 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-09-11T00:30:22.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.270+0000 7f278b200640 1 -- 172.21.15.204:0/1260461878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f278c06f280 msgr2=0x7f278c14fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.270+0000 7f278b200640 1 --2- 172.21.15.204:0/1260461878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f278c06f280 0x7f278c14fc00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f277c00c3d0 tx=0x7f277c00c890 comp rx=0 tx=0).stop 2024-09-11T00:30:22.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.270+0000 7f278b200640 1 -- 172.21.15.204:0/1260461878 shutdown_connections 2024-09-11T00:30:22.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.270+0000 7f278b200640 1 --2- 172.21.15.204:0/1260461878 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f278c153800 0x7f278c1ec1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.270+0000 7f278b200640 1 --2- 172.21.15.204:0/1260461878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f278c0c9fd0 0x7f278c0ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.270+0000 7f278b200640 1 --2- 172.21.15.204:0/1260461878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f278c06f280 0x7f278c14fc00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.270+0000 7f278b200640 1 -- 172.21.15.204:0/1260461878 >> 172.21.15.204:0/1260461878 conn(0x7f278c0cbcc0 msgr2=0x7f278c152780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:22.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.270+0000 7f278b200640 1 -- 172.21.15.204:0/1260461878 shutdown_connections 2024-09-11T00:30:22.272 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.270+0000 7f278b200640 1 -- 172.21.15.204:0/1260461878 wait complete. 2024-09-11T00:30:22.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 16b0951a-976f-4612-8c55-8053d5a6c269 2 2024-09-11T00:30:22.441 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.439+0000 7fedfb400640 1 -- 172.21.15.204:0/3766899890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedf40c2330 msgr2=0x7fedf40c4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.441 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.439+0000 7fedfb400640 1 --2- 172.21.15.204:0/3766899890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedf40c2330 0x7fedf40c4ad0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feddc009830 tx=0x7feddc02a480 comp rx=0 tx=0).stop 2024-09-11T00:30:22.442 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.439+0000 7fedfb400640 1 -- 172.21.15.204:0/3766899890 shutdown_connections 2024-09-11T00:30:22.442 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.439+0000 7fedfb400640 1 --2- 172.21.15.204:0/3766899890 >> v2:172.21.15.31:3300/0 conn(0x7fedf4153e90 0x7fedf4154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.442 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.439+0000 7fedfb400640 1 --2- 172.21.15.204:0/3766899890 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedf40c2330 0x7fedf40c4ad0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.442 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.439+0000 7fedfb400640 1 -- 172.21.15.204:0/3766899890 >> 172.21.15.204:0/3766899890 conn(0x7fedf406d560 msgr2=0x7fedf40c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:22.442 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.439+0000 7fedfb400640 1 -- 172.21.15.204:0/3766899890 shutdown_connections 2024-09-11T00:30:22.442 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.439+0000 7fedfb400640 1 -- 172.21.15.204:0/3766899890 wait complete. 2024-09-11T00:30:22.442 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.441+0000 7fedfb400640 1 Processor -- start 2024-09-11T00:30:22.443 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.441+0000 7fedfb400640 1 -- start start 2024-09-11T00:30:22.444 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedfb400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedf40c2330 0x7fedf414ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.444 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedfb400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf414de40 0x7fedf41df680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedfb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedf4153e90 0x7fedf414e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedfb400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fedf406e380 con 0x7fedf4153e90 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedfb400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fedf406e200 con 0x7fedf414de40 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedfb400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fedf414e220 con 0x7fedf40c2330 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedebe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf414de40 0x7fedf41df680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedf8a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedf40c2330 0x7fedf414ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedebe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf414de40 0x7fedf41df680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:60854/0 (socket says 172.21.15.204:60854) 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedebe00640 1 -- 172.21.15.204:0/1606804569 learned_addr learned my addr 172.21.15.204:0/1606804569 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.442+0000 7fedf9400640 1 --2- 172.21.15.204:0/1606804569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedf4153e90 0x7fedf414e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedebe00640 1 -- 172.21.15.204:0/1606804569 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedf40c2330 msgr2=0x7fedf414ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedebe00640 1 --2- 172.21.15.204:0/1606804569 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedf40c2330 0x7fedf414ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedebe00640 1 -- 172.21.15.204:0/1606804569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedf4153e90 msgr2=0x7fedf414e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.445 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedebe00640 1 --2- 172.21.15.204:0/1606804569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedf4153e90 0x7fedf414e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.446 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedebe00640 1 -- 172.21.15.204:0/1606804569 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf4157110 con 0x7fedf414de40 2024-09-11T00:30:22.446 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedebe00640 1 --2- 172.21.15.204:0/1606804569 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf414de40 0x7fedf41df680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feddc02aa70 tx=0x7feddc032c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:22.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedf1600640 1 -- 172.21.15.204:0/1606804569 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc040070 con 0x7fedf414de40 2024-09-11T00:30:22.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedf1600640 1 -- 172.21.15.204:0/1606804569 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feddc0038e0 con 0x7fedf414de40 2024-09-11T00:30:22.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedf1600640 1 -- 172.21.15.204:0/1606804569 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc005c90 con 0x7fedf414de40 2024-09-11T00:30:22.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedfb400640 1 -- 172.21.15.204:0/1606804569 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf414b3c0 con 0x7fedf414de40 2024-09-11T00:30:22.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.443+0000 7fedfb400640 1 -- 172.21.15.204:0/1606804569 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf4157f50 con 0x7fedf414de40 2024-09-11T00:30:22.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.444+0000 7fedf1600640 1 -- 172.21.15.204:0/1606804569 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7feddc045460 con 0x7fedf414de40 2024-09-11T00:30:22.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.444+0000 7fedf1600640 1 -- 172.21.15.204:0/1606804569 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7feddc045ba0 con 0x7fedf414de40 2024-09-11T00:30:22.447 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.444+0000 7fedfb400640 1 -- 172.21.15.204:0/1606804569 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf4154990 con 0x7fedf414de40 2024-09-11T00:30:22.450 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.448+0000 7fedf1600640 1 -- 172.21.15.204:0/1606804569 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fedf4157110 con 0x7fedf414de40 2024-09-11T00:30:22.522 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.520+0000 7fedfb400640 1 -- 172.21.15.204:0/1606804569 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "16b0951a-976f-4612-8c55-8053d5a6c269", "id": 2} v 0) -- 0x7fedf40c2010 con 0x7fedf414de40 2024-09-11T00:30:22.526 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.524+0000 7fedf1600640 1 -- 172.21.15.204:0/1606804569 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "16b0951a-976f-4612-8c55-8053d5a6c269", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7feddc045780 con 0x7fedf414de40 2024-09-11T00:30:22.527 INFO:teuthology.orchestra.run.smithi204.stdout:2 2024-09-11T00:30:22.527 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.526+0000 7fedfb400640 1 -- 172.21.15.204:0/1606804569 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf414de40 msgr2=0x7fedf41df680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.528 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.526+0000 7fedfb400640 1 --2- 172.21.15.204:0/1606804569 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf414de40 0x7fedf41df680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feddc02aa70 tx=0x7feddc032c10 comp rx=0 tx=0).stop 2024-09-11T00:30:22.528 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.526+0000 7fedfb400640 1 -- 172.21.15.204:0/1606804569 shutdown_connections 2024-09-11T00:30:22.528 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.526+0000 7fedfb400640 1 --2- 172.21.15.204:0/1606804569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedf4153e90 0x7fedf414e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.528 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.526+0000 7fedfb400640 1 --2- 172.21.15.204:0/1606804569 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf414de40 0x7fedf41df680 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.528 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.526+0000 7fedfb400640 1 --2- 172.21.15.204:0/1606804569 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedf40c2330 0x7fedf414ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.528 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.526+0000 7fedfb400640 1 -- 172.21.15.204:0/1606804569 >> 172.21.15.204:0/1606804569 conn(0x7fedf406d560 msgr2=0x7fedf4074190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:22.528 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.526+0000 7fedfb400640 1 -- 172.21.15.204:0/1606804569 shutdown_connections 2024-09-11T00:30:22.528 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.526+0000 7fedfb400640 1 -- 172.21.15.204:0/1606804569 wait complete. 2024-09-11T00:30:22.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 530d98c7-9645-4a8b-b1ed-a63c60f70706 3 2024-09-11T00:30:22.699 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.697+0000 7fc27d600640 1 -- 172.21.15.204:0/2626963354 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc278153800 msgr2=0x7fc278156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.700 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.697+0000 7fc27d600640 1 --2- 172.21.15.204:0/2626963354 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc278153800 0x7fc278156970 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc268005fd0 tx=0x7fc26802a4b0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.700 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.697+0000 7fc27d600640 1 -- 172.21.15.204:0/2626963354 shutdown_connections 2024-09-11T00:30:22.700 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.697+0000 7fc27d600640 1 --2- 172.21.15.204:0/2626963354 >> v2:172.21.15.31:3300/0 conn(0x7fc2780c9fd0 0x7fc27806e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.700 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.697+0000 7fc27d600640 1 --2- 172.21.15.204:0/2626963354 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc278153800 0x7fc278156970 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.700 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.697+0000 7fc27d600640 1 -- 172.21.15.204:0/2626963354 >> 172.21.15.204:0/2626963354 conn(0x7fc2780cbcc0 msgr2=0x7fc2780cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:22.700 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.698+0000 7fc27d600640 1 -- 172.21.15.204:0/2626963354 shutdown_connections 2024-09-11T00:30:22.700 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.698+0000 7fc27d600640 1 -- 172.21.15.204:0/2626963354 wait complete. 2024-09-11T00:30:22.701 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.699+0000 7fc27d600640 1 Processor -- start 2024-09-11T00:30:22.701 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.699+0000 7fc27d600640 1 -- start start 2024-09-11T00:30:22.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc27d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc27806ea80 0x7fc27814b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.703 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc27d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2780c9fd0 0x7fc2781e17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc27d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc278153800 0x7fc2781ec6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc27d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc27806e020 con 0x7fc2780c9fd0 2024-09-11T00:30:22.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc27d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc2780c69e0 con 0x7fc27806ea80 2024-09-11T00:30:22.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc27d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc2780ae1f0 con 0x7fc278153800 2024-09-11T00:30:22.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc277e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc27806ea80 0x7fc27814b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc27cc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc278153800 0x7fc2781ec6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc26fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2780c9fd0 0x7fc2781e17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc277e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc27806ea80 0x7fc27814b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:60862/0 (socket says 172.21.15.204:60862) 2024-09-11T00:30:22.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.700+0000 7fc277e00640 1 -- 172.21.15.204:0/2586596546 learned_addr learned my addr 172.21.15.204:0/2586596546 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-09-11T00:30:22.704 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc27cc00640 1 -- 172.21.15.204:0/2586596546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc27806ea80 msgr2=0x7fc27814b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc27cc00640 1 --2- 172.21.15.204:0/2586596546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc27806ea80 0x7fc27814b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc27cc00640 1 -- 172.21.15.204:0/2586596546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2780c9fd0 msgr2=0x7fc2781e17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc27cc00640 1 --2- 172.21.15.204:0/2586596546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2780c9fd0 0x7fc2781e17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc27cc00640 1 -- 172.21.15.204:0/2586596546 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27815a420 con 0x7fc278153800 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc26fe00640 1 --2- 172.21.15.204:0/2586596546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2780c9fd0 0x7fc2781e17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc27cc00640 1 --2- 172.21.15.204:0/2586596546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc278153800 0x7fc2781ec6e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc27000c730 tx=0x7fc27000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc274c00640 1 -- 172.21.15.204:0/2586596546 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc270003b70 con 0x7fc278153800 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc274c00640 1 -- 172.21.15.204:0/2586596546 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc270003d10 con 0x7fc278153800 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc274c00640 1 -- 172.21.15.204:0/2586596546 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc270005330 con 0x7fc278153800 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.701+0000 7fc27d600640 1 -- 172.21.15.204:0/2586596546 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2780c2e00 con 0x7fc278153800 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.702+0000 7fc27d600640 1 -- 172.21.15.204:0/2586596546 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2780c2b20 con 0x7fc278153800 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.702+0000 7fc274c00640 1 -- 172.21.15.204:0/2586596546 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fc2700054d0 con 0x7fc278153800 2024-09-11T00:30:22.705 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.702+0000 7fc274c00640 1 -- 172.21.15.204:0/2586596546 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fc27000aad0 con 0x7fc278153800 2024-09-11T00:30:22.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.703+0000 7fc27d600640 1 -- 172.21.15.204:0/2586596546 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2780adfc0 con 0x7fc278153800 2024-09-11T00:30:22.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.705+0000 7fc274c00640 1 -- 172.21.15.204:0/2586596546 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fc2780adfc0 con 0x7fc278153800 2024-09-11T00:30:22.781 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.779+0000 7fc27d600640 1 -- 172.21.15.204:0/2586596546 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd new", "uuid": "530d98c7-9645-4a8b-b1ed-a63c60f70706", "id": 3} v 0) -- 0x7fc278155610 con 0x7fc278153800 2024-09-11T00:30:22.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.783+0000 7fc274c00640 1 -- 172.21.15.204:0/2586596546 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "530d98c7-9645-4a8b-b1ed-a63c60f70706", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fc2700057f0 con 0x7fc278153800 2024-09-11T00:30:22.785 INFO:teuthology.orchestra.run.smithi204.stdout:3 2024-09-11T00:30:22.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.785+0000 7fc27d600640 1 -- 172.21.15.204:0/2586596546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc278153800 msgr2=0x7fc2781ec6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.785+0000 7fc27d600640 1 --2- 172.21.15.204:0/2586596546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc278153800 0x7fc2781ec6e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc27000c730 tx=0x7fc27000cbf0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.785+0000 7fc27d600640 1 -- 172.21.15.204:0/2586596546 shutdown_connections 2024-09-11T00:30:22.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.785+0000 7fc27d600640 1 --2- 172.21.15.204:0/2586596546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc278153800 0x7fc2781ec6e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.785+0000 7fc27d600640 1 --2- 172.21.15.204:0/2586596546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2780c9fd0 0x7fc2781e17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.785+0000 7fc27d600640 1 --2- 172.21.15.204:0/2586596546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc27806ea80 0x7fc27814b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.785+0000 7fc27d600640 1 -- 172.21.15.204:0/2586596546 >> 172.21.15.204:0/2586596546 conn(0x7fc2780cbcc0 msgr2=0x7fc2781540e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:22.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.785+0000 7fc27d600640 1 -- 172.21.15.204:0/2586596546 shutdown_connections 2024-09-11T00:30:22.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.785+0000 7fc27d600640 1 -- 172.21.15.204:0/2586596546 wait complete. 2024-09-11T00:30:22.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 42fd8670-5d3a-4bb8-92b5-ff140b680b84 4 2024-09-11T00:30:22.956 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.954+0000 7ff769e00640 1 -- 172.21.15.204:0/1755177255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7640c9fd0 msgr2=0x7ff764156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.956 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.954+0000 7ff769e00640 1 --2- 172.21.15.204:0/1755177255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7640c9fd0 0x7ff764156970 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff76406df50 tx=0x7ff75002a4d0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.957 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.954+0000 7ff769e00640 1 -- 172.21.15.204:0/1755177255 shutdown_connections 2024-09-11T00:30:22.957 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.954+0000 7ff769e00640 1 --2- 172.21.15.204:0/1755177255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7640c9fd0 0x7ff764156970 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.957 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.954+0000 7ff769e00640 1 -- 172.21.15.204:0/1755177255 >> 172.21.15.204:0/1755177255 conn(0x7ff7640cbcc0 msgr2=0x7ff7640cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:22.957 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.954+0000 7ff769e00640 1 -- 172.21.15.204:0/1755177255 shutdown_connections 2024-09-11T00:30:22.957 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.954+0000 7ff769e00640 1 -- 172.21.15.204:0/1755177255 wait complete. 2024-09-11T00:30:22.957 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.956+0000 7ff769e00640 1 Processor -- start 2024-09-11T00:30:22.957 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.956+0000 7ff769e00640 1 -- start start 2024-09-11T00:30:22.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff769e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff76406f280 0x7ff76414e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff769e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7640c9fd0 0x7ff7640c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff769e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff764153800 0x7ff7640c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:22.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff769e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff76406e020 con 0x7ff7640c9fd0 2024-09-11T00:30:22.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff769e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff7640c69e0 con 0x7ff76406f280 2024-09-11T00:30:22.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff769e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff76406ec50 con 0x7ff764153800 2024-09-11T00:30:22.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff768a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff76406f280 0x7ff76414e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff768a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff76406f280 0x7ff76414e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:43170/0 (socket says 172.21.15.204:43170) 2024-09-11T00:30:22.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff768a00640 1 -- 172.21.15.204:0/1257682004 learned_addr learned my addr 172.21.15.204:0/1257682004 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-09-11T00:30:22.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff769400640 1 --2- 172.21.15.204:0/1257682004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff764153800 0x7ff7640c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff763e00640 1 --2- 172.21.15.204:0/1257682004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7640c9fd0 0x7ff7640c4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff768a00640 1 -- 172.21.15.204:0/1257682004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff764153800 msgr2=0x7ff7640c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff768a00640 1 --2- 172.21.15.204:0/1257682004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff764153800 0x7ff7640c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff768a00640 1 -- 172.21.15.204:0/1257682004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7640c9fd0 msgr2=0x7ff7640c4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff768a00640 1 --2- 172.21.15.204:0/1257682004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7640c9fd0 0x7ff7640c4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.957+0000 7ff768a00640 1 -- 172.21.15.204:0/1257682004 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76414e6c0 con 0x7ff76406f280 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.958+0000 7ff768a00640 1 --2- 172.21.15.204:0/1257682004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff76406f280 0x7ff76414e180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff75800b3f0 tx=0x7ff75800b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.958+0000 7ff760c00640 1 -- 172.21.15.204:0/1257682004 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758015020 con 0x7ff76406f280 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.958+0000 7ff760c00640 1 -- 172.21.15.204:0/1257682004 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff758011920 con 0x7ff76406f280 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.958+0000 7ff769e00640 1 -- 172.21.15.204:0/1257682004 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7641ecc40 con 0x7ff76406f280 2024-09-11T00:30:22.961 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.958+0000 7ff760c00640 1 -- 172.21.15.204:0/1257682004 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff758011d70 con 0x7ff76406f280 2024-09-11T00:30:22.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.958+0000 7ff769e00640 1 -- 172.21.15.204:0/1257682004 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7641e1820 con 0x7ff76406f280 2024-09-11T00:30:22.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.958+0000 7ff760c00640 1 -- 172.21.15.204:0/1257682004 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7ff758005910 con 0x7ff76406f280 2024-09-11T00:30:22.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.958+0000 7ff760c00640 1 -- 172.21.15.204:0/1257682004 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7ff7580043b0 con 0x7ff76406f280 2024-09-11T00:30:22.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.959+0000 7ff769e00640 1 -- 172.21.15.204:0/1257682004 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7640c2730 con 0x7ff76406f280 2024-09-11T00:30:22.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:22.961+0000 7ff760c00640 1 -- 172.21.15.204:0/1257682004 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7ff7640c2730 con 0x7ff76406f280 2024-09-11T00:30:23.037 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.035+0000 7ff769e00640 1 -- 172.21.15.204:0/1257682004 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "42fd8670-5d3a-4bb8-92b5-ff140b680b84", "id": 4} v 0) -- 0x7ff764156970 con 0x7ff76406f280 2024-09-11T00:30:23.041 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.039+0000 7ff760c00640 1 -- 172.21.15.204:0/1257682004 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42fd8670-5d3a-4bb8-92b5-ff140b680b84", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7ff75801f070 con 0x7ff76406f280 2024-09-11T00:30:23.041 INFO:teuthology.orchestra.run.smithi204.stdout:4 2024-09-11T00:30:23.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.040+0000 7ff769e00640 1 -- 172.21.15.204:0/1257682004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff76406f280 msgr2=0x7ff76414e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.040+0000 7ff769e00640 1 --2- 172.21.15.204:0/1257682004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff76406f280 0x7ff76414e180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff75800b3f0 tx=0x7ff75800b8b0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.040+0000 7ff769e00640 1 -- 172.21.15.204:0/1257682004 shutdown_connections 2024-09-11T00:30:23.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.040+0000 7ff769e00640 1 --2- 172.21.15.204:0/1257682004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff764153800 0x7ff7640c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.040+0000 7ff769e00640 1 --2- 172.21.15.204:0/1257682004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7640c9fd0 0x7ff7640c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.040+0000 7ff769e00640 1 --2- 172.21.15.204:0/1257682004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff76406f280 0x7ff76414e180 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.040+0000 7ff769e00640 1 -- 172.21.15.204:0/1257682004 >> 172.21.15.204:0/1257682004 conn(0x7ff7640cbcc0 msgr2=0x7ff7640cc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:23.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.040+0000 7ff769e00640 1 -- 172.21.15.204:0/1257682004 shutdown_connections 2024-09-11T00:30:23.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.040+0000 7ff769e00640 1 -- 172.21.15.204:0/1257682004 wait complete. 2024-09-11T00:30:23.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new ce393e08-131f-4a1a-9977-c2d4cdc48d1f 5 2024-09-11T00:30:23.212 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.209+0000 7f3909000640 1 -- 172.21.15.204:0/207194212 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3904150280 msgr2=0x7f39040c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.212 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.209+0000 7f3909000640 1 --2- 172.21.15.204:0/207194212 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3904150280 0x7f39040c8cf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f38f4005fd0 tx=0x7f38f402a540 comp rx=0 tx=0).stop 2024-09-11T00:30:23.212 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.209+0000 7f3909000640 1 -- 172.21.15.204:0/207194212 shutdown_connections 2024-09-11T00:30:23.213 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.209+0000 7f3909000640 1 --2- 172.21.15.204:0/207194212 >> v2:172.21.15.31:3300/0 conn(0x7f39040c9230 0x7f3904159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.213 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.209+0000 7f3909000640 1 --2- 172.21.15.204:0/207194212 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3904150280 0x7f39040c8cf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.213 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.209+0000 7f3909000640 1 -- 172.21.15.204:0/207194212 >> 172.21.15.204:0/207194212 conn(0x7f39040adc90 msgr2=0x7f39040ae0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:23.213 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.209+0000 7f3909000640 1 -- 172.21.15.204:0/207194212 shutdown_connections 2024-09-11T00:30:23.213 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.209+0000 7f3909000640 1 -- 172.21.15.204:0/207194212 wait complete. 2024-09-11T00:30:23.218 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.211+0000 7f3909000640 1 Processor -- start 2024-09-11T00:30:23.218 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.211+0000 7f3909000640 1 -- start start 2024-09-11T00:30:23.218 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f3909000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39040b1a30 0x7f390414e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:23.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f3909000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f39040c9230 0x7f390414ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:23.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f3909000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3904150280 0x7f390414bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:23.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f3909000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39040c8470 con 0x7f3904150280 2024-09-11T00:30:23.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f3909000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f390415a4d0 con 0x7f39040b1a30 2024-09-11T00:30:23.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f3909000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f390406eea0 con 0x7f39040c9230 2024-09-11T00:30:23.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f38fbe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f39040c9230 0x7f390414ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:23.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f3901e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39040b1a30 0x7f390414e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:23.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f3901e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39040b1a30 0x7f390414e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:43186/0 (socket says 172.21.15.204:43186) 2024-09-11T00:30:23.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f3901e00640 1 -- 172.21.15.204:0/1726566923 learned_addr learned my addr 172.21.15.204:0/1726566923 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-09-11T00:30:23.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f3902800640 1 --2- 172.21.15.204:0/1726566923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3904150280 0x7f390414bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:23.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f38fbe00640 1 -- 172.21.15.204:0/1726566923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39040b1a30 msgr2=0x7f390414e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f38fbe00640 1 --2- 172.21.15.204:0/1726566923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39040b1a30 0x7f390414e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f38fbe00640 1 -- 172.21.15.204:0/1726566923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3904150280 msgr2=0x7f390414bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f38fbe00640 1 --2- 172.21.15.204:0/1726566923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3904150280 0x7f390414bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.212+0000 7f38fbe00640 1 -- 172.21.15.204:0/1726566923 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39040c23a0 con 0x7f39040c9230 2024-09-11T00:30:23.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.213+0000 7f38fbe00640 1 --2- 172.21.15.204:0/1726566923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f39040c9230 0x7f390414ba80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f390408d630 tx=0x7f38ec00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:23.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.213+0000 7f38fa000640 1 -- 172.21.15.204:0/1726566923 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec017070 con 0x7f39040c9230 2024-09-11T00:30:23.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.213+0000 7f38fa000640 1 -- 172.21.15.204:0/1726566923 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38ec0053c0 con 0x7f39040c9230 2024-09-11T00:30:23.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.213+0000 7f38fa000640 1 -- 172.21.15.204:0/1726566923 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec003850 con 0x7f39040c9230 2024-09-11T00:30:23.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.213+0000 7f3909000640 1 -- 172.21.15.204:0/1726566923 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39041df8e0 con 0x7f39040c9230 2024-09-11T00:30:23.222 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.213+0000 7f3909000640 1 -- 172.21.15.204:0/1726566923 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f39041dfb80 con 0x7f39040c9230 2024-09-11T00:30:23.223 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.213+0000 7f38fa000640 1 -- 172.21.15.204:0/1726566923 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f38ec0039f0 con 0x7f39040c9230 2024-09-11T00:30:23.223 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.213+0000 7f38fa000640 1 -- 172.21.15.204:0/1726566923 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f38ec003d10 con 0x7f39040c9230 2024-09-11T00:30:23.223 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.214+0000 7f3909000640 1 -- 172.21.15.204:0/1726566923 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39040c8cf0 con 0x7f39040c9230 2024-09-11T00:30:23.223 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.216+0000 7f38fa000640 1 -- 172.21.15.204:0/1726566923 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f38ec021020 con 0x7f39040c9230 2024-09-11T00:30:23.290 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.288+0000 7f3909000640 1 -- 172.21.15.204:0/1726566923 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ce393e08-131f-4a1a-9977-c2d4cdc48d1f", "id": 5} v 0) -- 0x7f390406ea20 con 0x7f39040c9230 2024-09-11T00:30:23.294 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.292+0000 7f38fa000640 1 -- 172.21.15.204:0/1726566923 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce393e08-131f-4a1a-9977-c2d4cdc48d1f", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f38ec011590 con 0x7f39040c9230 2024-09-11T00:30:23.294 INFO:teuthology.orchestra.run.smithi204.stdout:5 2024-09-11T00:30:23.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.293+0000 7f3909000640 1 -- 172.21.15.204:0/1726566923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f39040c9230 msgr2=0x7f390414ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.293+0000 7f3909000640 1 --2- 172.21.15.204:0/1726566923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f39040c9230 0x7f390414ba80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f390408d630 tx=0x7f38ec00c820 comp rx=0 tx=0).stop 2024-09-11T00:30:23.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.293+0000 7f3909000640 1 -- 172.21.15.204:0/1726566923 shutdown_connections 2024-09-11T00:30:23.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.293+0000 7f3909000640 1 --2- 172.21.15.204:0/1726566923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3904150280 0x7f390414bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.293+0000 7f3909000640 1 --2- 172.21.15.204:0/1726566923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f39040c9230 0x7f390414ba80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.293+0000 7f3909000640 1 --2- 172.21.15.204:0/1726566923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39040b1a30 0x7f390414e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.293+0000 7f3909000640 1 -- 172.21.15.204:0/1726566923 >> 172.21.15.204:0/1726566923 conn(0x7f39040adc90 msgr2=0x7f3904153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:23.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.293+0000 7f3909000640 1 -- 172.21.15.204:0/1726566923 shutdown_connections 2024-09-11T00:30:23.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.293+0000 7f3909000640 1 -- 172.21.15.204:0/1726566923 wait complete. 2024-09-11T00:30:23.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 49eefb9a-2b2d-49a0-82c9-3cfdbac30336 6 2024-09-11T00:30:23.467 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.465+0000 7f9160c00640 1 -- 172.21.15.204:0/2038819565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f915c0c9fd0 msgr2=0x7f915c06e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.467 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.465+0000 7f9160c00640 1 --2- 172.21.15.204:0/2038819565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f915c0c9fd0 0x7f915c06e540 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9144005fd0 tx=0x7f914402a510 comp rx=0 tx=0).stop 2024-09-11T00:30:23.467 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.465+0000 7f9160c00640 1 -- 172.21.15.204:0/2038819565 shutdown_connections 2024-09-11T00:30:23.467 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.465+0000 7f9160c00640 1 --2- 172.21.15.204:0/2038819565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f915c0c9fd0 0x7f915c06e540 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.468 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.465+0000 7f9160c00640 1 --2- 172.21.15.204:0/2038819565 >> v2:172.21.15.31:3300/0 conn(0x7f915c153800 0x7f915c156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.468 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.465+0000 7f9160c00640 1 -- 172.21.15.204:0/2038819565 >> 172.21.15.204:0/2038819565 conn(0x7f915c0cbcc0 msgr2=0x7f915c0cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:23.468 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.465+0000 7f9160c00640 1 -- 172.21.15.204:0/2038819565 shutdown_connections 2024-09-11T00:30:23.468 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.465+0000 7f9160c00640 1 -- 172.21.15.204:0/2038819565 wait complete. 2024-09-11T00:30:23.468 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.466+0000 7f9160c00640 1 Processor -- start 2024-09-11T00:30:23.468 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.467+0000 7f9160c00640 1 -- start start 2024-09-11T00:30:23.471 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.467+0000 7f9160c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f915c06ea80 0x7f915c1e0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:23.471 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.467+0000 7f9160c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f915c0c9fd0 0x7f915c14fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:23.471 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.467+0000 7f9160c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f915c153800 0x7f915c150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:23.471 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.467+0000 7f9160c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f915c06e020 con 0x7f915c153800 2024-09-11T00:30:23.471 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.467+0000 7f9160c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f915c0c69e0 con 0x7f915c06ea80 2024-09-11T00:30:23.471 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.467+0000 7f9160c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f915c0ae1f0 con 0x7f915c0c9fd0 2024-09-11T00:30:23.471 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f915b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f915c06ea80 0x7f915c1e0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:23.471 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f9153e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f915c0c9fd0 0x7f915c14fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:23.471 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f9153e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f915c0c9fd0 0x7f915c14fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.204:51308/0 (socket says 172.21.15.204:51308) 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f9153e00640 1 -- 172.21.15.204:0/3717053632 learned_addr learned my addr 172.21.15.204:0/3717053632 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f915b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f915c06ea80 0x7f915c1e0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:43192/0 (socket says 172.21.15.204:43192) 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f915be00640 1 --2- 172.21.15.204:0/3717053632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f915c153800 0x7f915c150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f915b400640 1 -- 172.21.15.204:0/3717053632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f915c0c9fd0 msgr2=0x7f915c14fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f915b400640 1 --2- 172.21.15.204:0/3717053632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f915c0c9fd0 0x7f915c14fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f915b400640 1 -- 172.21.15.204:0/3717053632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f915c153800 msgr2=0x7f915c150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f915b400640 1 --2- 172.21.15.204:0/3717053632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f915c153800 0x7f915c150070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f915b400640 1 -- 172.21.15.204:0/3717053632 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915c1e21c0 con 0x7f915c06ea80 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.468+0000 7f915b400640 1 --2- 172.21.15.204:0/3717053632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f915c06ea80 0x7f915c1e0dc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f914c00e6d0 tx=0x7f914c00eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.469+0000 7f9153400640 1 -- 172.21.15.204:0/3717053632 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c00cab0 con 0x7f915c06ea80 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.469+0000 7f9153400640 1 -- 172.21.15.204:0/3717053632 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f914c00cd40 con 0x7f915c06ea80 2024-09-11T00:30:23.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.469+0000 7f9153400640 1 -- 172.21.15.204:0/3717053632 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c010690 con 0x7f915c06ea80 2024-09-11T00:30:23.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.469+0000 7f9160c00640 1 -- 172.21.15.204:0/3717053632 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915c150930 con 0x7f915c06ea80 2024-09-11T00:30:23.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.469+0000 7f9160c00640 1 -- 172.21.15.204:0/3717053632 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f915c1e1300 con 0x7f915c06ea80 2024-09-11T00:30:23.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.469+0000 7f9153400640 1 -- 172.21.15.204:0/3717053632 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f914c010830 con 0x7f915c06ea80 2024-09-11T00:30:23.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.469+0000 7f9153400640 1 -- 172.21.15.204:0/3717053632 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f914c018b50 con 0x7f915c06ea80 2024-09-11T00:30:23.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.470+0000 7f9160c00640 1 -- 172.21.15.204:0/3717053632 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f915c06e750 con 0x7f915c06ea80 2024-09-11T00:30:23.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.472+0000 7f9153400640 1 -- 172.21.15.204:0/3717053632 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f914c018d50 con 0x7f915c06ea80 2024-09-11T00:30:23.547 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.546+0000 7f9160c00640 1 -- 172.21.15.204:0/3717053632 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "49eefb9a-2b2d-49a0-82c9-3cfdbac30336", "id": 6} v 0) -- 0x7f915c0ad320 con 0x7f915c06ea80 2024-09-11T00:30:23.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.550+0000 7f9153400640 1 -- 172.21.15.204:0/3717053632 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "49eefb9a-2b2d-49a0-82c9-3cfdbac30336", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f914c010b50 con 0x7f915c06ea80 2024-09-11T00:30:23.551 INFO:teuthology.orchestra.run.smithi204.stdout:6 2024-09-11T00:30:23.553 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.551+0000 7f9160c00640 1 -- 172.21.15.204:0/3717053632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f915c06ea80 msgr2=0x7f915c1e0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.553 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.551+0000 7f9160c00640 1 --2- 172.21.15.204:0/3717053632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f915c06ea80 0x7f915c1e0dc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f914c00e6d0 tx=0x7f914c00eb90 comp rx=0 tx=0).stop 2024-09-11T00:30:23.553 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.551+0000 7f9160c00640 1 -- 172.21.15.204:0/3717053632 shutdown_connections 2024-09-11T00:30:23.553 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.551+0000 7f9160c00640 1 --2- 172.21.15.204:0/3717053632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f915c153800 0x7f915c150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.553 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.551+0000 7f9160c00640 1 --2- 172.21.15.204:0/3717053632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f915c0c9fd0 0x7f915c14fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.553 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.551+0000 7f9160c00640 1 --2- 172.21.15.204:0/3717053632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f915c06ea80 0x7f915c1e0dc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.553 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.551+0000 7f9160c00640 1 -- 172.21.15.204:0/3717053632 >> 172.21.15.204:0/3717053632 conn(0x7f915c0cbcc0 msgr2=0x7f915c152780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:23.554 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.551+0000 7f9160c00640 1 -- 172.21.15.204:0/3717053632 shutdown_connections 2024-09-11T00:30:23.554 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.551+0000 7f9160c00640 1 -- 172.21.15.204:0/3717053632 wait complete. 2024-09-11T00:30:23.567 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 234a7330-98fe-4736-80b8-6ef614d792be 7 2024-09-11T00:30:23.720 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.718+0000 7fe584e00640 1 -- 172.21.15.204:0/1444098568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5800c9ed0 msgr2=0x7fe5800c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.718+0000 7fe584e00640 1 --2- 172.21.15.204:0/1444098568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5800c9ed0 0x7fe5800c85e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe5800c42b0 tx=0x7fe57002a4d0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.719+0000 7fe584e00640 1 -- 172.21.15.204:0/1444098568 shutdown_connections 2024-09-11T00:30:23.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.719+0000 7fe584e00640 1 --2- 172.21.15.204:0/1444098568 >> v2:172.21.15.204:3300/0 conn(0x7fe5800c4e60 0x7fe5800ab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.719+0000 7fe584e00640 1 --2- 172.21.15.204:0/1444098568 >> v2:172.21.15.31:3300/0 conn(0x7fe5800c6630 0x7fe5800c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.719+0000 7fe584e00640 1 --2- 172.21.15.204:0/1444098568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5800c9ed0 0x7fe5800c85e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.719+0000 7fe584e00640 1 -- 172.21.15.204:0/1444098568 >> 172.21.15.204:0/1444098568 conn(0x7fe580070440 msgr2=0x7fe580070820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:23.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.719+0000 7fe584e00640 1 -- 172.21.15.204:0/1444098568 shutdown_connections 2024-09-11T00:30:23.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.719+0000 7fe584e00640 1 -- 172.21.15.204:0/1444098568 wait complete. 2024-09-11T00:30:23.722 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.720+0000 7fe584e00640 1 Processor -- start 2024-09-11T00:30:23.722 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.720+0000 7fe584e00640 1 -- start start 2024-09-11T00:30:23.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe584e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5800c4e60 0x7fe580153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:23.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe584e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5800c6630 0x7fe5800c3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:23.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe584e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5800c9ed0 0x7fe5801dfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:23.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe584e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe5800ca2b0 con 0x7fe5800c4e60 2024-09-11T00:30:23.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe584e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe5800c6da0 con 0x7fe5800c9ed0 2024-09-11T00:30:23.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe584e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe5800ca430 con 0x7fe5800c6630 2024-09-11T00:30:23.723 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe577e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5800c6630 0x7fe5800c3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:23.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe57e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5800c9ed0 0x7fe5801dfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:23.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe577e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5800c6630 0x7fe5800c3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.204:51328/0 (socket says 172.21.15.204:51328) 2024-09-11T00:30:23.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe577e00640 1 -- 172.21.15.204:0/2671073361 learned_addr learned my addr 172.21.15.204:0/2671073361 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-09-11T00:30:23.724 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.721+0000 7fe57de00640 1 --2- 172.21.15.204:0/2671073361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5800c4e60 0x7fe580153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:23.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.722+0000 7fe57e800640 1 -- 172.21.15.204:0/2671073361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5800c6630 msgr2=0x7fe5800c3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.722+0000 7fe57e800640 1 --2- 172.21.15.204:0/2671073361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5800c6630 0x7fe5800c3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.722+0000 7fe57e800640 1 -- 172.21.15.204:0/2671073361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5800c4e60 msgr2=0x7fe580153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.722+0000 7fe57e800640 1 --2- 172.21.15.204:0/2671073361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5800c4e60 0x7fe580153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.722+0000 7fe57e800640 1 -- 172.21.15.204:0/2671073361 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5801e0850 con 0x7fe5800c9ed0 2024-09-11T00:30:23.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.722+0000 7fe57e800640 1 --2- 172.21.15.204:0/2671073361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5800c9ed0 0x7fe5801dfd60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe57800ea80 tx=0x7fe57800ef40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:23.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.722+0000 7fe576000640 1 -- 172.21.15.204:0/2671073361 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57800ce40 con 0x7fe5800c9ed0 2024-09-11T00:30:23.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.722+0000 7fe576000640 1 -- 172.21.15.204:0/2671073361 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe578004b20 con 0x7fe5800c9ed0 2024-09-11T00:30:23.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.722+0000 7fe576000640 1 -- 172.21.15.204:0/2671073361 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe578005020 con 0x7fe5800c9ed0 2024-09-11T00:30:23.725 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.722+0000 7fe584e00640 1 -- 172.21.15.204:0/2671073361 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58006f8c0 con 0x7fe5800c9ed0 2024-09-11T00:30:23.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.723+0000 7fe584e00640 1 -- 172.21.15.204:0/2671073361 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5800a94a0 con 0x7fe5800c9ed0 2024-09-11T00:30:23.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.723+0000 7fe576000640 1 -- 172.21.15.204:0/2671073361 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fe578005550 con 0x7fe5800c9ed0 2024-09-11T00:30:23.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.723+0000 7fe576000640 1 -- 172.21.15.204:0/2671073361 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fe578005870 con 0x7fe5800c9ed0 2024-09-11T00:30:23.726 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.724+0000 7fe584e00640 1 -- 172.21.15.204:0/2671073361 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5800725d0 con 0x7fe5800c9ed0 2024-09-11T00:30:23.727 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.726+0000 7fe576000640 1 -- 172.21.15.204:0/2671073361 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fe5800725d0 con 0x7fe5800c9ed0 2024-09-11T00:30:23.801 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.799+0000 7fe584e00640 1 -- 172.21.15.204:0/2671073361 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd new", "uuid": "234a7330-98fe-4736-80b8-6ef614d792be", "id": 7} v 0) -- 0x7fe5800c5470 con 0x7fe5800c9ed0 2024-09-11T00:30:23.805 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.803+0000 7fe576000640 1 -- 172.21.15.204:0/2671073361 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "234a7330-98fe-4736-80b8-6ef614d792be", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fe578027030 con 0x7fe5800c9ed0 2024-09-11T00:30:23.805 INFO:teuthology.orchestra.run.smithi204.stdout:7 2024-09-11T00:30:23.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.804+0000 7fe584e00640 1 -- 172.21.15.204:0/2671073361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5800c9ed0 msgr2=0x7fe5801dfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:23.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.804+0000 7fe584e00640 1 --2- 172.21.15.204:0/2671073361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5800c9ed0 0x7fe5801dfd60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe57800ea80 tx=0x7fe57800ef40 comp rx=0 tx=0).stop 2024-09-11T00:30:23.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.804+0000 7fe584e00640 1 -- 172.21.15.204:0/2671073361 shutdown_connections 2024-09-11T00:30:23.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.804+0000 7fe584e00640 1 --2- 172.21.15.204:0/2671073361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5800c9ed0 0x7fe5801dfd60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.804+0000 7fe584e00640 1 --2- 172.21.15.204:0/2671073361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5800c6630 0x7fe5800c3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.806 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.804+0000 7fe584e00640 1 --2- 172.21.15.204:0/2671073361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5800c4e60 0x7fe580153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:23.807 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.804+0000 7fe584e00640 1 -- 172.21.15.204:0/2671073361 >> 172.21.15.204:0/2671073361 conn(0x7fe580070440 msgr2=0x7fe580157770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:23.807 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.804+0000 7fe584e00640 1 -- 172.21.15.204:0/2671073361 shutdown_connections 2024-09-11T00:30:23.807 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:30:23.804+0000 7fe584e00640 1 -- 172.21.15.204:0/2671073361 wait complete. 2024-09-11T00:30:23.820 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T00:30:23.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-11T00:30:23.825 INFO:tasks.ceph.osd.0:Started 2024-09-11T00:30:23.825 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T00:30:23.825 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-11T00:30:23.829 INFO:tasks.ceph.osd.1:Started 2024-09-11T00:30:23.829 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T00:30:23.829 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-11T00:30:23.833 INFO:tasks.ceph.osd.2:Started 2024-09-11T00:30:23.833 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-11T00:30:23.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-11T00:30:23.838 INFO:tasks.ceph.osd.3:Started 2024-09-11T00:30:23.838 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-11T00:30:23.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-11T00:30:23.842 INFO:tasks.ceph.osd.4:Started 2024-09-11T00:30:23.842 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-11T00:30:23.842 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-11T00:30:23.846 INFO:tasks.ceph.osd.5:Started 2024-09-11T00:30:23.846 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-11T00:30:23.847 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-11T00:30:23.851 INFO:tasks.ceph.osd.6:Started 2024-09-11T00:30:23.852 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-11T00:30:23.852 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-11T00:30:23.856 INFO:tasks.ceph.osd.7:Started 2024-09-11T00:30:23.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:30:24.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.060+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2758138442 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ebc06ff40 msgr2=0x7f9ebc0c23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:24.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.060+0000 7f9ec0a00640 1 --2- 172.21.15.31:0/2758138442 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ebc06ff40 0x7f9ebc0c23a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4005fd0 tx=0x7f9ea402b880 comp rx=0 tx=0).stop 2024-09-11T00:30:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.060+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2758138442 shutdown_connections 2024-09-11T00:30:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.060+0000 7f9ec0a00640 1 --2- 172.21.15.31:0/2758138442 >> v2:172.21.15.204:3300/0 conn(0x7f9ebc0c3f20 0x7f9ebc0c28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.060+0000 7f9ec0a00640 1 --2- 172.21.15.31:0/2758138442 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ebc06ff40 0x7f9ebc0c23a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.060+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2758138442 >> 172.21.15.31:0/2758138442 conn(0x7f9ebc0a9750 msgr2=0x7f9ebc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.060+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2758138442 shutdown_connections 2024-09-11T00:30:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.060+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2758138442 wait complete. 2024-09-11T00:30:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.062+0000 7f9ec0a00640 1 Processor -- start 2024-09-11T00:30:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.062+0000 7f9ec0a00640 1 -- start start 2024-09-11T00:30:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9ec0a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ebc06ff40 0x7f9ebc154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9ec0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ebc0c3f20 0x7f9ebc0c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9ec0a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ebc1532c0 0x7f9ebc14be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9ec0a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc06f340 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9ec0a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc06eb40 con 0x7f9ebc1532c0 2024-09-11T00:30:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9ec0a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9ebc0c7650 con 0x7f9ebc06ff40 2024-09-11T00:30:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ebc0c3f20 0x7f9ebc0c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9eba800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ebc1532c0 0x7f9ebc14be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ebc0c3f20 0x7f9ebc0c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34504/0 (socket says 172.21.15.31:34504) 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9eba800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ebc1532c0 0x7f9ebc14be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:34048/0 (socket says 172.21.15.31:34048) 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9eb9e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ebc06ff40 0x7f9ebc154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.063+0000 7f9eb3e00640 1 -- 172.21.15.31:0/2273440233 learned_addr learned my addr 172.21.15.31:0/2273440233 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.064+0000 7f9eb3e00640 1 -- 172.21.15.31:0/2273440233 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ebc06ff40 msgr2=0x7f9ebc154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.064+0000 7f9eb3e00640 1 --2- 172.21.15.31:0/2273440233 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ebc06ff40 0x7f9ebc154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.064+0000 7f9eb3e00640 1 -- 172.21.15.31:0/2273440233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ebc1532c0 msgr2=0x7f9ebc14be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.064+0000 7f9eb3e00640 1 --2- 172.21.15.31:0/2273440233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ebc1532c0 0x7f9ebc14be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.064+0000 7f9eb3e00640 1 -- 172.21.15.31:0/2273440233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ebc1565b0 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.064+0000 7f9eb9e00640 1 --2- 172.21.15.31:0/2273440233 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ebc06ff40 0x7f9ebc154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.064+0000 7f9eba800640 1 --2- 172.21.15.31:0/2273440233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ebc1532c0 0x7f9ebc14be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:30:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.064+0000 7f9eb3e00640 1 --2- 172.21.15.31:0/2273440233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ebc0c3f20 0x7f9ebc0c1040 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4034040 tx=0x7f9ea40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.065+0000 7f9eb2000640 1 -- 172.21.15.31:0/2273440233 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4040030 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.065+0000 7f9eb2000640 1 -- 172.21.15.31:0/2273440233 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ea4003d90 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.065+0000 7f9eb2000640 1 -- 172.21.15.31:0/2273440233 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40047a0 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.065+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2273440233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ebc151f60 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.065+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2273440233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ebc155ad0 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.065+0000 7f9eb2000640 1 -- 172.21.15.31:0/2273440233 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f9ea400b040 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.065+0000 7f9eb2000640 1 -- 172.21.15.31:0/2273440233 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f9ea4004c60 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.066+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2273440233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ebc0a7300 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.068+0000 7f9eb2000640 1 -- 172.21.15.31:0/2273440233 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f9ea402f050 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.134+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2273440233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ebc0c6d20 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.135+0000 7f9eb2000640 1 -- 172.21.15.31:0/2273440233 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f9ea4004eb0 con 0x7f9ebc0c3f20 2024-09-11T00:30:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:24.138 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:23.801176+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:30:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.136+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2273440233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ebc0c3f20 msgr2=0x7f9ebc0c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.136+0000 7f9ec0a00640 1 --2- 172.21.15.31:0/2273440233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ebc0c3f20 0x7f9ebc0c1040 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4034040 tx=0x7f9ea40038a0 comp rx=0 tx=0).stop 2024-09-11T00:30:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.136+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2273440233 shutdown_connections 2024-09-11T00:30:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.136+0000 7f9ec0a00640 1 --2- 172.21.15.31:0/2273440233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ebc1532c0 0x7f9ebc14be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.136+0000 7f9ec0a00640 1 --2- 172.21.15.31:0/2273440233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ebc0c3f20 0x7f9ebc0c1040 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.136+0000 7f9ec0a00640 1 --2- 172.21.15.31:0/2273440233 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ebc06ff40 0x7f9ebc154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.136+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2273440233 >> 172.21.15.31:0/2273440233 conn(0x7f9ebc0a9750 msgr2=0x7f9ebc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.136+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2273440233 shutdown_connections 2024-09-11T00:30:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:24.136+0000 7f9ec0a00640 1 -- 172.21.15.31:0/2273440233 wait complete. 2024-09-11T00:30:24.155 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T00:30:24.155 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T00:30:24.962 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T00:30:25.103 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.100+0000 7f5698c00640 1 -- 172.21.15.31:0/609860751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56940b11c0 msgr2=0x7f569415a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:25.103 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.100+0000 7f5698c00640 1 --2- 172.21.15.31:0/609860751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56940b11c0 0x7f569415a110 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f56940c9730 tx=0x7f568402a4d0 comp rx=0 tx=0).stop 2024-09-11T00:30:25.103 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.100+0000 7f5698c00640 1 -- 172.21.15.31:0/609860751 shutdown_connections 2024-09-11T00:30:25.103 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.100+0000 7f5698c00640 1 --2- 172.21.15.31:0/609860751 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5694156280 0x7f5694156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:25.103 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.100+0000 7f5698c00640 1 --2- 172.21.15.31:0/609860751 >> v2:172.21.15.204:3300/0 conn(0x7f56940cc200 0x7f56940cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:25.104 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.100+0000 7f5698c00640 1 --2- 172.21.15.31:0/609860751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56940b11c0 0x7f569415a110 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:25.104 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.100+0000 7f5698c00640 1 -- 172.21.15.31:0/609860751 >> 172.21.15.31:0/609860751 conn(0x7f5694068c80 msgr2=0x7f5694069090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:25.104 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.100+0000 7f5698c00640 1 -- 172.21.15.31:0/609860751 shutdown_connections 2024-09-11T00:30:25.104 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.100+0000 7f5698c00640 1 -- 172.21.15.31:0/609860751 wait complete. 2024-09-11T00:30:25.105 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.103+0000 7f5698c00640 1 Processor -- start 2024-09-11T00:30:25.105 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.103+0000 7f5698c00640 1 -- start start 2024-09-11T00:30:25.108 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.104+0000 7f5698c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56940b11c0 0x7f569414f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:25.108 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.104+0000 7f5698c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56940cc200 0x7f56940ab600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:25.108 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.104+0000 7f5698c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5694156280 0x7f569414d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:25.108 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.104+0000 7f5698c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f56940c68c0 con 0x7f5694156280 2024-09-11T00:30:25.108 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.104+0000 7f5698c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f569406eed0 con 0x7f56940cc200 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.104+0000 7f5698c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f56940b15a0 con 0x7f56940b11c0 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f5692800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5694156280 0x7f569414d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f5692800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5694156280 0x7f569414d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34508/0 (socket says 172.21.15.31:34508) 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f5692800640 1 -- 172.21.15.31:0/2118801788 learned_addr learned my addr 172.21.15.31:0/2118801788 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f5691e00640 1 --2- 172.21.15.31:0/2118801788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56940b11c0 0x7f569414f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f568be00640 1 --2- 172.21.15.31:0/2118801788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56940cc200 0x7f56940ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f5692800640 1 -- 172.21.15.31:0/2118801788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56940b11c0 msgr2=0x7f569414f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f5692800640 1 --2- 172.21.15.31:0/2118801788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56940b11c0 0x7f569414f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f5692800640 1 -- 172.21.15.31:0/2118801788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56940cc200 msgr2=0x7f56940ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f5692800640 1 --2- 172.21.15.31:0/2118801788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56940cc200 0x7f56940ab600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f5692800640 1 -- 172.21.15.31:0/2118801788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56941e25c0 con 0x7f5694156280 2024-09-11T00:30:25.109 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f5692800640 1 --2- 172.21.15.31:0/2118801788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5694156280 0x7f569414d060 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f568c00ea80 tx=0x7f568c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:25.110 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.105+0000 7f568a000640 1 -- 172.21.15.31:0/2118801788 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c004b20 con 0x7f5694156280 2024-09-11T00:30:25.110 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.106+0000 7f568a000640 1 -- 172.21.15.31:0/2118801788 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f568c004ce0 con 0x7f5694156280 2024-09-11T00:30:25.110 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.106+0000 7f568a000640 1 -- 172.21.15.31:0/2118801788 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c005280 con 0x7f5694156280 2024-09-11T00:30:25.110 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.106+0000 7f5698c00640 1 -- 172.21.15.31:0/2118801788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56941e20a0 con 0x7f5694156280 2024-09-11T00:30:25.110 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.106+0000 7f5698c00640 1 -- 172.21.15.31:0/2118801788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56940ac5d0 con 0x7f5694156280 2024-09-11T00:30:25.110 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.107+0000 7f568a000640 1 -- 172.21.15.31:0/2118801788 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 2) ==== 50324+0+0 (secure 0 0 0) 0x7f568c005420 con 0x7f5694156280 2024-09-11T00:30:25.110 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.107+0000 7f5698c00640 1 -- 172.21.15.31:0/2118801788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56940c9c60 con 0x7f5694156280 2024-09-11T00:30:25.111 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.107+0000 7f568a000640 1 -- 172.21.15.31:0/2118801788 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f568c052880 con 0x7f5694156280 2024-09-11T00:30:25.111 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-09-11T00:30:25.110+0000 7f3d033a8640 -1 Falling back to public interface 2024-09-11T00:30:25.112 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.109+0000 7f568a000640 1 -- 172.21.15.31:0/2118801788 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f56941e25c0 con 0x7f5694156280 2024-09-11T00:30:25.177 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.174+0000 7f5698c00640 1 -- 172.21.15.31:0/2118801788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5694155d80 con 0x7f5694156280 2024-09-11T00:30:25.178 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.175+0000 7f568a000640 1 -- 172.21.15.31:0/2118801788 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f568c005920 con 0x7f5694156280 2024-09-11T00:30:25.178 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-11T00:30:25.178 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":10,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:23.801176+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:30:25.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.176+0000 7f5698c00640 1 -- 172.21.15.31:0/2118801788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5694156280 msgr2=0x7f569414d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:25.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.176+0000 7f5698c00640 1 --2- 172.21.15.31:0/2118801788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5694156280 0x7f569414d060 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f568c00ea80 tx=0x7f568c00ef40 comp rx=0 tx=0).stop 2024-09-11T00:30:25.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.176+0000 7f5698c00640 1 -- 172.21.15.31:0/2118801788 shutdown_connections 2024-09-11T00:30:25.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.176+0000 7f5698c00640 1 --2- 172.21.15.31:0/2118801788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5694156280 0x7f569414d060 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:25.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.176+0000 7f5698c00640 1 --2- 172.21.15.31:0/2118801788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56940cc200 0x7f56940ab600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:25.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.176+0000 7f5698c00640 1 --2- 172.21.15.31:0/2118801788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56940b11c0 0x7f569414f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:25.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.176+0000 7f5698c00640 1 -- 172.21.15.31:0/2118801788 >> 172.21.15.31:0/2118801788 conn(0x7f5694068c80 msgr2=0x7f5694155450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:25.180 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.176+0000 7f5698c00640 1 -- 172.21.15.31:0/2118801788 shutdown_connections 2024-09-11T00:30:25.180 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:25.176+0000 7f5698c00640 1 -- 172.21.15.31:0/2118801788 wait complete. 2024-09-11T00:30:25.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-11T00:30:25.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T00:30:25.367+0000 7ff83ec44640 -1 Falling back to public interface 2024-09-11T00:30:25.378 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-09-11T00:30:25.376+0000 7f4248dbb640 -1 Falling back to public interface 2024-09-11T00:30:25.397 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T00:30:25.395+0000 7fba39334640 -1 Falling back to public interface 2024-09-11T00:30:25.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-09-11T00:30:25.400+0000 7f2477c00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-11T00:30:25.402 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-09-11T00:30:25.400+0000 7f2477c00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-11T00:30:25.403 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-09-11T00:30:25.402+0000 7fb0ed210640 -1 Falling back to public interface 2024-09-11T00:30:25.408 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-11T00:30:25.406+0000 7fc700134640 -1 Falling back to public interface 2024-09-11T00:30:25.423 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-11T00:30:25.422+0000 7fdcbd8ed640 -1 Falling back to public interface 2024-09-11T00:30:25.674 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T00:30:25.671+0000 7f34a9e62640 -1 Falling back to public interface 2024-09-11T00:30:28.875 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-09-11T00:30:28.874+0000 7f3d033a8640 -1 osd.4 0 log_to_monitors true 2024-09-11T00:30:29.048 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-09-11T00:30:29.046+0000 7f4248dbb640 -1 osd.5 0 log_to_monitors true 2024-09-11T00:30:29.286 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T00:30:29.284+0000 7ff83ec44640 -1 osd.1 0 log_to_monitors true 2024-09-11T00:30:29.363 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-11T00:30:29.361+0000 7fc700134640 -1 osd.2 0 log_to_monitors true 2024-09-11T00:30:29.624 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T00:30:29.622+0000 7fba39334640 -1 osd.0 0 log_to_monitors true 2024-09-11T00:30:29.707 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-09-11T00:30:29.706+0000 7fb0ed210640 -1 osd.6 0 log_to_monitors true 2024-09-11T00:30:29.710 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-11T00:30:29.709+0000 7fdcbd8ed640 -1 osd.7 0 log_to_monitors true 2024-09-11T00:30:29.843 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T00:30:29.842+0000 7f34a9e62640 -1 osd.3 0 log_to_monitors true 2024-09-11T00:30:31.094 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-09-11T00:30:31.093+0000 7f3cfe200640 -1 osd.4 0 waiting for initial osdmap 2024-09-11T00:30:31.112 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-09-11T00:30:31.108+0000 7f3cf4200640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T00:30:31.998 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T00:30:32.178 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.176+0000 7fca00200640 1 -- 172.21.15.31:0/4179130529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80c2750 msgr2=0x7fc9f815a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:32.178 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.176+0000 7fca00200640 1 --2- 172.21.15.31:0/4179130529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80c2750 0x7fc9f815a2e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009830 tx=0x7fc9e402a480 comp rx=0 tx=0).stop 2024-09-11T00:30:32.178 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.176+0000 7fca00200640 1 -- 172.21.15.31:0/4179130529 shutdown_connections 2024-09-11T00:30:32.178 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.176+0000 7fca00200640 1 --2- 172.21.15.31:0/4179130529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80c2750 0x7fc9f815a2e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:32.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.176+0000 7fca00200640 1 --2- 172.21.15.31:0/4179130529 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9f80c4740 0x7fc9f8159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:32.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.176+0000 7fca00200640 1 -- 172.21.15.31:0/4179130529 >> 172.21.15.31:0/4179130529 conn(0x7fc9f806d1a0 msgr2=0x7fc9f80c7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:32.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.176+0000 7fca00200640 1 -- 172.21.15.31:0/4179130529 shutdown_connections 2024-09-11T00:30:32.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.176+0000 7fca00200640 1 -- 172.21.15.31:0/4179130529 wait complete. 2024-09-11T00:30:32.179 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.178+0000 7fca00200640 1 Processor -- start 2024-09-11T00:30:32.180 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.178+0000 7fca00200640 1 -- start start 2024-09-11T00:30:32.181 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.179+0000 7fca00200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9f80c2750 0x7fc9f8156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:32.181 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.179+0000 7fca00200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9f80c2c10 0x7fc9f81ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:32.181 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.180+0000 7fca00200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80c4740 0x7fc9f81ec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:32.181 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.180+0000 7fca00200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f80c8930 con 0x7fc9f80c4740 2024-09-11T00:30:32.182 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.180+0000 7fca00200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f80c87b0 con 0x7fc9f80c2c10 2024-09-11T00:30:32.182 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.180+0000 7fca00200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f80c4b20 con 0x7fc9f80c2750 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.180+0000 7fc9fd800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9f80c2750 0x7fc9f8156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.180+0000 7fc9fd800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9f80c2750 0x7fc9f8156f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:44916/0 (socket says 172.21.15.31:44916) 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.180+0000 7fc9fd800640 1 -- 172.21.15.31:0/3946480884 learned_addr learned my addr 172.21.15.31:0/3946480884 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.180+0000 7fc9fe200640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80c4740 0x7fc9f81ec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fc9fe200640 1 -- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9f80c2750 msgr2=0x7fc9f8156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fc9fe200640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9f80c2750 0x7fc9f8156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fc9fe200640 1 -- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9f80c2c10 msgr2=0x7fc9f81ebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fc9fe200640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9f80c2c10 0x7fc9f81ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fc9fe200640 1 -- 172.21.15.31:0/3946480884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f814dd60 con 0x7fc9f80c4740 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fc9fe200640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80c4740 0x7fc9f81ec0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000caa0 tx=0x7fc9f000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:32.183 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fc9f5600640 1 -- 172.21.15.31:0/3946480884 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0016020 con 0x7fc9f80c4740 2024-09-11T00:30:32.184 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fc9f5600640 1 -- 172.21.15.31:0/3946480884 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9f000ad10 con 0x7fc9f80c4740 2024-09-11T00:30:32.184 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fca00200640 1 -- 172.21.15.31:0/3946480884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f814e340 con 0x7fc9f80c4740 2024-09-11T00:30:32.184 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fc9f5600640 1 -- 172.21.15.31:0/3946480884 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0012680 con 0x7fc9f80c4740 2024-09-11T00:30:32.184 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.181+0000 7fca00200640 1 -- 172.21.15.31:0/3946480884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f80acb10 con 0x7fc9f80c4740 2024-09-11T00:30:32.184 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.183+0000 7fca00200640 1 -- 172.21.15.31:0/3946480884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f8159da0 con 0x7fc9f80c4740 2024-09-11T00:30:32.185 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.184+0000 7fc9f5600640 1 -- 172.21.15.31:0/3946480884 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 100545+0+0 (secure 0 0 0) 0x7fc9f0012820 con 0x7fc9f80c4740 2024-09-11T00:30:32.186 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.185+0000 7fc9f5600640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc9d8077f10 0x7fc9d807a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:32.186 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.185+0000 7fc9f5600640 1 -- 172.21.15.31:0/3946480884 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2511+0+0 (secure 0 0 0) 0x7fc9f00997f0 con 0x7fc9f80c4740 2024-09-11T00:30:32.187 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.185+0000 7fc9fd800640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc9d8077f10 0x7fc9d807a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:32.187 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.186+0000 7fc9fd800640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc9d8077f10 0x7fc9d807a3d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec009bf0 tx=0x7fc9ec005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:32.193 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.191+0000 7fc9f5600640 1 -- 172.21.15.31:0/3946480884 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc9f0020020 con 0x7fc9f80c4740 2024-09-11T00:30:32.501 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.499+0000 7fca00200640 1 -- 172.21.15.31:0/3946480884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9f8157c60 con 0x7fc9f80c4740 2024-09-11T00:30:32.502 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.500+0000 7fc9f5600640 1 -- 172.21.15.31:0/3946480884 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+7555 (secure 0 0 0) 0x7fc9f0063040 con 0x7fc9f80c4740 2024-09-11T00:30:32.502 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-11T00:30:32.502 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":14,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:31.955843+0000","last_up_change":"2024-09-11T00:30:31.955843+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:30:32.505 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.504+0000 7fca00200640 1 -- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc9d8077f10 msgr2=0x7fc9d807a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:32.505 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.504+0000 7fca00200640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc9d8077f10 0x7fc9d807a3d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec009bf0 tx=0x7fc9ec005f90 comp rx=0 tx=0).stop 2024-09-11T00:30:32.506 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.504+0000 7fca00200640 1 -- 172.21.15.31:0/3946480884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80c4740 msgr2=0x7fc9f81ec0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:32.506 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.504+0000 7fca00200640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80c4740 0x7fc9f81ec0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000caa0 tx=0x7fc9f000cf60 comp rx=0 tx=0).stop 2024-09-11T00:30:32.506 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.504+0000 7fca00200640 1 -- 172.21.15.31:0/3946480884 shutdown_connections 2024-09-11T00:30:32.506 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.504+0000 7fca00200640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc9d8077f10 0x7fc9d807a3d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:32.506 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.504+0000 7fca00200640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9f80c4740 0x7fc9f81ec0e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:32.506 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.504+0000 7fca00200640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9f80c2c10 0x7fc9f81ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:32.506 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.504+0000 7fca00200640 1 --2- 172.21.15.31:0/3946480884 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc9f80c2750 0x7fc9f8156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:32.506 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.504+0000 7fca00200640 1 -- 172.21.15.31:0/3946480884 >> 172.21.15.31:0/3946480884 conn(0x7fc9f806d1a0 msgr2=0x7fc9f8072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:32.507 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.505+0000 7fca00200640 1 -- 172.21.15.31:0/3946480884 shutdown_connections 2024-09-11T00:30:32.507 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:32.505+0000 7fca00200640 1 -- 172.21.15.31:0/3946480884 wait complete. 2024-09-11T00:30:32.526 DEBUG:teuthology.misc:1 of 8 OSDs are up 2024-09-11T00:30:32.771 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-09-11T00:30:32.770+0000 7fb0e7a00640 -1 osd.6 0 waiting for initial osdmap 2024-09-11T00:30:32.779 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-09-11T00:30:32.777+0000 7fb0dda00640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T00:30:32.784 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-09-11T00:30:32.782+0000 7f4244600640 -1 osd.5 0 waiting for initial osdmap 2024-09-11T00:30:32.799 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-09-11T00:30:32.798+0000 7f4239c00640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T00:30:32.799 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-11T00:30:32.798+0000 7fdcb9200640 -1 osd.7 0 waiting for initial osdmap 2024-09-11T00:30:32.809 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-11T00:30:32.808+0000 7fdcae800640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T00:30:33.189 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T00:30:33.187+0000 7fba34c00640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T00:30:33.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T00:30:33.194+0000 7ff839a00640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T00:30:33.203 INFO:tasks.ceph.osd.0.smithi031.stderr:2024-09-11T00:30:33.201+0000 7fba2a200640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T00:30:33.203 INFO:tasks.ceph.osd.1.smithi031.stderr:2024-09-11T00:30:33.202+0000 7ff82fa00640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T00:30:33.260 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-11T00:30:33.258+0000 7fc6fb000640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T00:30:33.271 INFO:tasks.ceph.osd.2.smithi031.stderr:2024-09-11T00:30:33.269+0000 7fc6f1000640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T00:30:33.302 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T00:30:33.301+0000 7f34a4e00640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T00:30:33.309 INFO:tasks.ceph.osd.3.smithi031.stderr:2024-09-11T00:30:33.308+0000 7f349ae00640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T00:30:38.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:30:38.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:30:39.332 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T00:30:39.488 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.486+0000 7fd794400640 1 -- 172.21.15.31:0/2819299943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd78c074be0 msgr2=0x7fd78c0c44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:39.488 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.486+0000 7fd794400640 1 --2- 172.21.15.31:0/2819299943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd78c074be0 0x7fd78c0c44d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0751c0 tx=0x7fd77802a560 comp rx=0 tx=0).stop 2024-09-11T00:30:39.488 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.486+0000 7fd794400640 1 -- 172.21.15.31:0/2819299943 shutdown_connections 2024-09-11T00:30:39.488 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.486+0000 7fd794400640 1 --2- 172.21.15.31:0/2819299943 >> v2:172.21.15.204:3300/0 conn(0x7fd78c0c4a10 0x7fd78c0c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.488 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.486+0000 7fd794400640 1 --2- 172.21.15.31:0/2819299943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd78c074be0 0x7fd78c0c44d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.489 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.486+0000 7fd794400640 1 -- 172.21.15.31:0/2819299943 >> 172.21.15.31:0/2819299943 conn(0x7fd78c06d1a0 msgr2=0x7fd78c0c7a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:39.489 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.486+0000 7fd794400640 1 -- 172.21.15.31:0/2819299943 shutdown_connections 2024-09-11T00:30:39.489 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.486+0000 7fd794400640 1 -- 172.21.15.31:0/2819299943 wait complete. 2024-09-11T00:30:39.490 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.489+0000 7fd794400640 1 Processor -- start 2024-09-11T00:30:39.490 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.489+0000 7fd794400640 1 -- start start 2024-09-11T00:30:39.493 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.490+0000 7fd794400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd78c074be0 0x7fd78c156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:39.493 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.490+0000 7fd794400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd78c0c4a10 0x7fd78c150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:39.493 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd794400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd78c151840 0x7fd78c14cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:39.493 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd794400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c151da0 con 0x7fd78c074be0 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd794400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c151c20 con 0x7fd78c151840 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd794400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd78c074690 con 0x7fd78c0c4a10 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd791a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd78c074be0 0x7fd78c156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd791a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd78c074be0 0x7fd78c156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39612/0 (socket says 172.21.15.31:39612) 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd791a00640 1 -- 172.21.15.31:0/2612369419 learned_addr learned my addr 172.21.15.31:0/2612369419 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd791000640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd78c0c4a10 0x7fd78c150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd792400640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd78c151840 0x7fd78c14cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd791a00640 1 -- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd78c0c4a10 msgr2=0x7fd78c150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd791a00640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd78c0c4a10 0x7fd78c150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd791a00640 1 -- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd78c151840 msgr2=0x7fd78c14cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd791a00640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd78c151840 0x7fd78c14cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd791a00640 1 -- 172.21.15.31:0/2612369419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c157220 con 0x7fd78c074be0 2024-09-11T00:30:39.494 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.491+0000 7fd791000640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd78c0c4a10 0x7fd78c150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:30:39.495 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.492+0000 7fd791a00640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd78c074be0 0x7fd78c156b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd78000c450 tx=0x7fd78000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:39.495 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.492+0000 7fd789600640 1 -- 172.21.15.31:0/2612369419 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7800052d0 con 0x7fd78c074be0 2024-09-11T00:30:39.495 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.492+0000 7fd789600640 1 -- 172.21.15.31:0/2612369419 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd780005470 con 0x7fd78c074be0 2024-09-11T00:30:39.495 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.492+0000 7fd789600640 1 -- 172.21.15.31:0/2612369419 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780012870 con 0x7fd78c074be0 2024-09-11T00:30:39.495 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.492+0000 7fd794400640 1 -- 172.21.15.31:0/2612369419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78c0bcec0 con 0x7fd78c074be0 2024-09-11T00:30:39.496 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.492+0000 7fd794400640 1 -- 172.21.15.31:0/2612369419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78c150990 con 0x7fd78c074be0 2024-09-11T00:30:39.496 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.494+0000 7fd794400640 1 -- 172.21.15.31:0/2612369419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78c1529c0 con 0x7fd78c074be0 2024-09-11T00:30:39.497 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.496+0000 7fd789600640 1 -- 172.21.15.31:0/2612369419 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd7800035f0 con 0x7fd78c074be0 2024-09-11T00:30:39.498 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.497+0000 7fd789600640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd76c077f70 0x7fd76c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:39.498 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.497+0000 7fd789600640 1 -- 172.21.15.31:0/2612369419 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fd780099560 con 0x7fd78c074be0 2024-09-11T00:30:39.498 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.497+0000 7fd791000640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd76c077f70 0x7fd76c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:39.499 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.498+0000 7fd791000640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd76c077f70 0x7fd76c07a430 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd778033000 tx=0x7fd778004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:39.503 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.502+0000 7fd789600640 1 -- 172.21.15.31:0/2612369419 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd78c1529c0 con 0x7fd78c074be0 2024-09-11T00:30:39.698 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.697+0000 7fd794400640 1 -- 172.21.15.31:0/2612369419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd78c1ea290 con 0x7fd78c074be0 2024-09-11T00:30:39.699 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.698+0000 7fd789600640 1 -- 172.21.15.31:0/2612369419 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12884 (secure 0 0 0) 0x7fd780061cd0 con 0x7fd78c074be0 2024-09-11T00:30:39.700 INFO:teuthology.misc.health.smithi031.stdout: 2024-09-11T00:30:39.700 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":18,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:36.782072+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:30:39.702 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.700+0000 7fd794400640 1 -- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd76c077f70 msgr2=0x7fd76c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.700+0000 7fd794400640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd76c077f70 0x7fd76c07a430 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd778033000 tx=0x7fd778004730 comp rx=0 tx=0).stop 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.700+0000 7fd794400640 1 -- 172.21.15.31:0/2612369419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd78c074be0 msgr2=0x7fd78c156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.700+0000 7fd794400640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd78c074be0 0x7fd78c156b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd78000c450 tx=0x7fd78000c910 comp rx=0 tx=0).stop 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.701+0000 7fd794400640 1 -- 172.21.15.31:0/2612369419 shutdown_connections 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.701+0000 7fd794400640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd76c077f70 0x7fd76c07a430 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.701+0000 7fd794400640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd78c151840 0x7fd78c14cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.701+0000 7fd794400640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd78c0c4a10 0x7fd78c150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.701+0000 7fd794400640 1 --2- 172.21.15.31:0/2612369419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd78c074be0 0x7fd78c156b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.701+0000 7fd794400640 1 -- 172.21.15.31:0/2612369419 >> 172.21.15.31:0/2612369419 conn(0x7fd78c06d1a0 msgr2=0x7fd78c152100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.701+0000 7fd794400640 1 -- 172.21.15.31:0/2612369419 shutdown_connections 2024-09-11T00:30:39.703 INFO:teuthology.misc.health.smithi031.stderr:2024-09-11T00:30:39.701+0000 7fd794400640 1 -- 172.21.15.31:0/2612369419 wait complete. 2024-09-11T00:30:39.715 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-11T00:30:39.716 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T00:30:39.716 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-11T00:30:39.716 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-11T00:30:39.720 INFO:tasks.ceph.mds.a:Started 2024-09-11T00:30:39.720 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-11T00:30:39.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-11T00:30:39.723 INFO:tasks.ceph.mds.c:Started 2024-09-11T00:30:39.724 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-11T00:30:39.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-11T00:30:39.728 INFO:tasks.ceph.mds.b:Started 2024-09-11T00:30:39.729 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:30:39.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:30:39.733 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:30:39.733 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-11T00:30:39.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:30:39.857 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-11T00:30:39.858 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-11T00:30:39.870 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:30:39.873 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:30:39.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.907+0000 7fd9ce200640 1 -- 172.21.15.31:0/1488096554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9c8159650 msgr2=0x7fd9c8159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:39.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.907+0000 7fd9ce200640 1 --2- 172.21.15.31:0/1488096554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9c8159650 0x7fd9c8159b00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0005fd0 tx=0x7fd9c002a820 comp rx=0 tx=0).stop 2024-09-11T00:30:39.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.907+0000 7fd9ce200640 1 -- 172.21.15.31:0/1488096554 shutdown_connections 2024-09-11T00:30:39.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.907+0000 7fd9ce200640 1 --2- 172.21.15.31:0/1488096554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9c8159650 0x7fd9c8159b00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.907+0000 7fd9ce200640 1 --2- 172.21.15.31:0/1488096554 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9c81534f0 0x7fd9c8156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.907+0000 7fd9ce200640 1 -- 172.21.15.31:0/1488096554 >> 172.21.15.31:0/1488096554 conn(0x7fd9c8068f10 msgr2=0x7fd9c80cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:39.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.907+0000 7fd9ce200640 1 -- 172.21.15.31:0/1488096554 shutdown_connections 2024-09-11T00:30:39.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.907+0000 7fd9ce200640 1 -- 172.21.15.31:0/1488096554 wait complete. 2024-09-11T00:30:39.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.908+0000 7fd9ce200640 1 Processor -- start 2024-09-11T00:30:39.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.909+0000 7fd9ce200640 1 -- start start 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.909+0000 7fd9ce200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9c80c9b40 0x7fd9c8152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.909+0000 7fd9ce200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9c81534f0 0x7fd9c814de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9ce200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9c8159650 0x7fd9c814e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9ce200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c80cca60 con 0x7fd9c80c9b40 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9ce200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c81538d0 con 0x7fd9c8159650 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9ce200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c80ad680 con 0x7fd9c81534f0 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9c7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9c80c9b40 0x7fd9c8152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9bfe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9c81534f0 0x7fd9c814de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9c7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9c80c9b40 0x7fd9c8152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39652/0 (socket says 172.21.15.31:39652) 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9c7400640 1 -- 172.21.15.31:0/1355659229 learned_addr learned my addr 172.21.15.31:0/1355659229 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:39.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9c7e00640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9c8159650 0x7fd9c814e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:39.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9c7e00640 1 -- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9c81534f0 msgr2=0x7fd9c814de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:39.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9c7e00640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9c81534f0 0x7fd9c814de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9c7e00640 1 -- 172.21.15.31:0/1355659229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9c80c9b40 msgr2=0x7fd9c8152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:39.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9c7e00640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9c80c9b40 0x7fd9c8152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:39.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9c7e00640 1 -- 172.21.15.31:0/1355659229 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c80ad050 con 0x7fd9c8159650 2024-09-11T00:30:39.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.910+0000 7fd9bfe00640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9c81534f0 0x7fd9c814de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:30:39.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.911+0000 7fd9c7e00640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9c8159650 0x7fd9c814e3b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80c8620 tx=0x7fd9c0032c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:39.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.911+0000 7fd9bf400640 1 -- 172.21.15.31:0/1355659229 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0037040 con 0x7fd9c8159650 2024-09-11T00:30:39.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.911+0000 7fd9bf400640 1 -- 172.21.15.31:0/1355659229 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9c0042070 con 0x7fd9c8159650 2024-09-11T00:30:39.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.911+0000 7fd9bf400640 1 -- 172.21.15.31:0/1355659229 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00042a0 con 0x7fd9c8159650 2024-09-11T00:30:39.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.911+0000 7fd9ce200640 1 -- 172.21.15.31:0/1355659229 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c80ac5f0 con 0x7fd9c8159650 2024-09-11T00:30:39.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.912+0000 7fd9ce200640 1 -- 172.21.15.31:0/1355659229 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c80ab830 con 0x7fd9c8159650 2024-09-11T00:30:39.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.913+0000 7fd9ce200640 1 -- 172.21.15.31:0/1355659229 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c8156ae0 con 0x7fd9c8159650 2024-09-11T00:30:39.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.914+0000 7fd9bf400640 1 -- 172.21.15.31:0/1355659229 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd9c003a030 con 0x7fd9c8159650 2024-09-11T00:30:39.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.915+0000 7fd9bf400640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd9a4077f70 0x7fd9a407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:39.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.915+0000 7fd9bf400640 1 -- 172.21.15.31:0/1355659229 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fd9c00ba390 con 0x7fd9c8159650 2024-09-11T00:30:39.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.915+0000 7fd9c7400640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd9a4077f70 0x7fd9a407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:39.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.916+0000 7fd9c7400640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd9a4077f70 0x7fd9a407a430 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8005fd0 tx=0x7fd9b800b480 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:39.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:39.920+0000 7fd9bf400640 1 -- 172.21.15.31:0/1355659229 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd9c0082af0 con 0x7fd9c8159650 2024-09-11T00:30:40.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.157+0000 7fd9ce200640 1 -- 172.21.15.31:0/1355659229 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9c8154f10 con 0x7fd9c8159650 2024-09-11T00:30:40.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.158+0000 7fd9bf400640 1 -- 172.21.15.31:0/1355659229 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1434 (secure 0 0 0) 0x7fd9c0087520 con 0x7fd9c8159650 2024-09-11T00:30:40.161 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:40.161 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":2,"btime":"2024-09-11T00:30:39:865168+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-11T00:30:40.162 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 2 2024-09-11T00:30:40.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 -- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd9a4077f70 msgr2=0x7fd9a407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:40.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd9a4077f70 0x7fd9a407a430 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8005fd0 tx=0x7fd9b800b480 comp rx=0 tx=0).stop 2024-09-11T00:30:40.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 -- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9c8159650 msgr2=0x7fd9c814e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:40.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9c8159650 0x7fd9c814e3b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80c8620 tx=0x7fd9c0032c20 comp rx=0 tx=0).stop 2024-09-11T00:30:40.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 -- 172.21.15.31:0/1355659229 shutdown_connections 2024-09-11T00:30:40.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd9a4077f70 0x7fd9a407a430 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:40.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9c8159650 0x7fd9c814e3b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:40.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd9c81534f0 0x7fd9c814de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:40.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 --2- 172.21.15.31:0/1355659229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9c80c9b40 0x7fd9c8152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:40.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 -- 172.21.15.31:0/1355659229 >> 172.21.15.31:0/1355659229 conn(0x7fd9c8068f10 msgr2=0x7fd9c8155710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:40.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 -- 172.21.15.31:0/1355659229 shutdown_connections 2024-09-11T00:30:40.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:40.160+0000 7fd9ce200640 1 -- 172.21.15.31:0/1355659229 wait complete. 2024-09-11T00:30:42.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:30:42.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.347+0000 7efc1f200640 1 -- 172.21.15.31:0/862772982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc18150020 msgr2=0x7efc180ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.347+0000 7efc1f200640 1 --2- 172.21.15.31:0/862772982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc18150020 0x7efc180ad4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009830 tx=0x7efc0c02a510 comp rx=0 tx=0).stop 2024-09-11T00:30:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.348+0000 7efc1f200640 1 -- 172.21.15.31:0/862772982 shutdown_connections 2024-09-11T00:30:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.348+0000 7efc1f200640 1 --2- 172.21.15.31:0/862772982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc18150020 0x7efc180ad4e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.348+0000 7efc1f200640 1 -- 172.21.15.31:0/862772982 >> 172.21.15.31:0/862772982 conn(0x7efc18068f10 msgr2=0x7efc180cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.348+0000 7efc1f200640 1 -- 172.21.15.31:0/862772982 shutdown_connections 2024-09-11T00:30:42.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.348+0000 7efc1f200640 1 -- 172.21.15.31:0/862772982 wait complete. 2024-09-11T00:30:42.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.349+0000 7efc1f200640 1 Processor -- start 2024-09-11T00:30:42.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.349+0000 7efc1f200640 1 -- start start 2024-09-11T00:30:42.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.350+0000 7efc1f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1806e850 0x7efc181e0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:42.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.350+0000 7efc1f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc180ada20 0x7efc181ef760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:42.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.350+0000 7efc1f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc18150020 0x7efc1814acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:42.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.350+0000 7efc1f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efc18157550 con 0x7efc1806e850 2024-09-11T00:30:42.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.350+0000 7efc1f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efc181573d0 con 0x7efc18150020 2024-09-11T00:30:42.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.350+0000 7efc1f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efc18156960 con 0x7efc180ada20 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.350+0000 7efc1de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1806e850 0x7efc181e0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.350+0000 7efc1de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1806e850 0x7efc181e0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39656/0 (socket says 172.21.15.31:39656) 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.350+0000 7efc1de00640 1 -- 172.21.15.31:0/573921432 learned_addr learned my addr 172.21.15.31:0/573921432 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc1e800640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc18150020 0x7efc1814acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc1d400640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc180ada20 0x7efc181ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc1de00640 1 -- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc180ada20 msgr2=0x7efc181ef760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc1de00640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc180ada20 0x7efc181ef760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc1de00640 1 -- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc18150020 msgr2=0x7efc1814acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc1de00640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc18150020 0x7efc1814acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc1de00640 1 -- 172.21.15.31:0/573921432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc181e0bf0 con 0x7efc1806e850 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc1e800640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc18150020 0x7efc1814acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc1d400640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc180ada20 0x7efc181ef760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:42.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc1de00640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1806e850 0x7efc181e0020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009950 tx=0x7efc0c032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.351+0000 7efc12000640 1 -- 172.21.15.31:0/573921432 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c040070 con 0x7efc1806e850 2024-09-11T00:30:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.352+0000 7efc12000640 1 -- 172.21.15.31:0/573921432 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc0c003af0 con 0x7efc1806e850 2024-09-11T00:30:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.352+0000 7efc12000640 1 -- 172.21.15.31:0/573921432 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c045460 con 0x7efc1806e850 2024-09-11T00:30:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.352+0000 7efc1f200640 1 -- 172.21.15.31:0/573921432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc181e1470 con 0x7efc1806e850 2024-09-11T00:30:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.352+0000 7efc1f200640 1 -- 172.21.15.31:0/573921432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc180ad430 con 0x7efc1806e850 2024-09-11T00:30:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.353+0000 7efc1f200640 1 -- 172.21.15.31:0/573921432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc181e1260 con 0x7efc1806e850 2024-09-11T00:30:42.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.354+0000 7efc12000640 1 -- 172.21.15.31:0/573921432 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7efc0c045600 con 0x7efc1806e850 2024-09-11T00:30:42.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.355+0000 7efc12000640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efbfc077f70 0x7efbfc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.355+0000 7efc12000640 1 -- 172.21.15.31:0/573921432 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7efc0c0bb470 con 0x7efc1806e850 2024-09-11T00:30:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.355+0000 7efc1d400640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efbfc077f70 0x7efbfc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:42.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.356+0000 7efc1d400640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efbfc077f70 0x7efbfc07a430 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efc180ad950 tx=0x7efc08009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:42.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.359+0000 7efc12000640 1 -- 172.21.15.31:0/573921432 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efc0c0842b0 con 0x7efc1806e850 2024-09-11T00:30:42.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.591+0000 7efc1f200640 1 -- 172.21.15.31:0/573921432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc180cc220 con 0x7efc1806e850 2024-09-11T00:30:42.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.592+0000 7efc12000640 1 -- 172.21.15.31:0/573921432 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3912 (secure 0 0 0) 0x7efc0c083bb0 con 0x7efc1806e850 2024-09-11T00:30:42.594 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:42.594 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":3,"btime":"2024-09-11T00:30:40:866830+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4374,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-11T00:30:42.594 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 3 2024-09-11T00:30:42.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 -- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efbfc077f70 msgr2=0x7efbfc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:42.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efbfc077f70 0x7efbfc07a430 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efc180ad950 tx=0x7efc08009040 comp rx=0 tx=0).stop 2024-09-11T00:30:42.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 -- 172.21.15.31:0/573921432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1806e850 msgr2=0x7efc181e0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:42.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1806e850 0x7efc181e0020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009950 tx=0x7efc0c032d50 comp rx=0 tx=0).stop 2024-09-11T00:30:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 -- 172.21.15.31:0/573921432 shutdown_connections 2024-09-11T00:30:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efbfc077f70 0x7efbfc07a430 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc18150020 0x7efc1814acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc180ada20 0x7efc181ef760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 --2- 172.21.15.31:0/573921432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1806e850 0x7efc181e0020 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 -- 172.21.15.31:0/573921432 >> 172.21.15.31:0/573921432 conn(0x7efc18068f10 msgr2=0x7efc18159a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:42.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 -- 172.21.15.31:0/573921432 shutdown_connections 2024-09-11T00:30:42.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.594+0000 7efc1f200640 1 -- 172.21.15.31:0/573921432 wait complete. 2024-09-11T00:30:42.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-11T00:30:42.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.768+0000 7f3ef4200640 1 -- 172.21.15.31:0/1482430504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3eec0c23a0 msgr2=0x7f3eec0c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:42.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.768+0000 7f3ef4200640 1 --2- 172.21.15.31:0/1482430504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3eec0c23a0 0x7f3eec0c30c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8009830 tx=0x7f3ed802a5a0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.768+0000 7f3ef4200640 1 -- 172.21.15.31:0/1482430504 shutdown_connections 2024-09-11T00:30:42.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.768+0000 7f3ef4200640 1 --2- 172.21.15.31:0/1482430504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3eec0c23a0 0x7f3eec0c30c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.768+0000 7f3ef4200640 1 -- 172.21.15.31:0/1482430504 >> 172.21.15.31:0/1482430504 conn(0x7f3eec0a9750 msgr2=0x7f3eec0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:42.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.768+0000 7f3ef4200640 1 -- 172.21.15.31:0/1482430504 shutdown_connections 2024-09-11T00:30:42.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.768+0000 7f3ef4200640 1 -- 172.21.15.31:0/1482430504 wait complete. 2024-09-11T00:30:42.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.770+0000 7f3ef4200640 1 Processor -- start 2024-09-11T00:30:42.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.770+0000 7f3ef4200640 1 -- start start 2024-09-11T00:30:42.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.770+0000 7f3ef4200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3eec072480 0x7f3eec0c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:42.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef4200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3eec0c23a0 0x7f3eec143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:42.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef4200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3eec1518b0 0x7f3eec1440e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:42.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef4200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3eec074080 con 0x7f3eec1518b0 2024-09-11T00:30:42.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef4200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3eec0c3d50 con 0x7f3eec0c23a0 2024-09-11T00:30:42.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef4200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3eec074200 con 0x7f3eec072480 2024-09-11T00:30:42.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef2200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3eec1518b0 0x7f3eec1440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:42.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef2200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3eec1518b0 0x7f3eec1440e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39674/0 (socket says 172.21.15.31:39674) 2024-09-11T00:30:42.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef1800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3eec072480 0x7f3eec0c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:42.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef2200640 1 -- 172.21.15.31:0/2877304963 learned_addr learned my addr 172.21.15.31:0/2877304963 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:42.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef0e00640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3eec0c23a0 0x7f3eec143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:42.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef2200640 1 -- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3eec072480 msgr2=0x7f3eec0c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef2200640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3eec072480 0x7f3eec0c6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef2200640 1 -- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3eec0c23a0 msgr2=0x7f3eec143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef2200640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3eec0c23a0 0x7f3eec143ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.771+0000 7f3ef2200640 1 -- 172.21.15.31:0/2877304963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eec0adcb0 con 0x7f3eec1518b0 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.772+0000 7f3ef2200640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3eec1518b0 0x7f3eec1440e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400c7b0 tx=0x7f3ee400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.772+0000 7f3ee9600640 1 -- 172.21.15.31:0/2877304963 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4004b60 con 0x7f3eec1518b0 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.772+0000 7f3ee9600640 1 -- 172.21.15.31:0/2877304963 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ee4004df0 con 0x7f3eec1518b0 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.772+0000 7f3ef4200640 1 -- 172.21.15.31:0/2877304963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eec1474e0 con 0x7f3eec1518b0 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.772+0000 7f3ee9600640 1 -- 172.21.15.31:0/2877304963 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee400aa90 con 0x7f3eec1518b0 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.772+0000 7f3ef4200640 1 -- 172.21.15.31:0/2877304963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eec0c71e0 con 0x7f3eec1518b0 2024-09-11T00:30:42.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.773+0000 7f3ef4200640 1 -- 172.21.15.31:0/2877304963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3eec147720 con 0x7f3eec1518b0 2024-09-11T00:30:42.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.774+0000 7f3ee9600640 1 -- 172.21.15.31:0/2877304963 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3ee400ac30 con 0x7f3eec1518b0 2024-09-11T00:30:42.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.775+0000 7f3ee9600640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ecc077f70 0x7f3ecc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:42.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.775+0000 7f3ee9600640 1 -- 172.21.15.31:0/2877304963 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f3ee4099c90 con 0x7f3eec1518b0 2024-09-11T00:30:42.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.776+0000 7f3ef1800640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ecc077f70 0x7f3ecc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:42.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.777+0000 7f3ef1800640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ecc077f70 0x7f3ecc07a430 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0000f60 tx=0x7f3ee0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:42.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.779+0000 7f3ee9600640 1 -- 172.21.15.31:0/2877304963 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3eec147720 con 0x7f3eec1518b0 2024-09-11T00:30:42.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.974+0000 7f3ef4200640 1 -- 172.21.15.31:0/2877304963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3eec153f00 con 0x7f3eec1518b0 2024-09-11T00:30:42.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.974+0000 7f3ee9600640 1 -- 172.21.15.31:0/2877304963 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) ==== 84+0+65 (secure 0 0 0) 0x7f3ee4020070 con 0x7f3eec1518b0 2024-09-11T00:30:42.976 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:42.976 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-11T00:30:42.977 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T00:30:42.977 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-11T00:30:42.977 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-11T00:30:42.977 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T00:30:42.977 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-11T00:30:42.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 -- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ecc077f70 msgr2=0x7f3ecc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:42.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ecc077f70 0x7f3ecc07a430 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0000f60 tx=0x7f3ee0009040 comp rx=0 tx=0).stop 2024-09-11T00:30:42.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 -- 172.21.15.31:0/2877304963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3eec1518b0 msgr2=0x7f3eec1440e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:42.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3eec1518b0 0x7f3eec1440e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400c7b0 tx=0x7f3ee400cc70 comp rx=0 tx=0).stop 2024-09-11T00:30:42.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 -- 172.21.15.31:0/2877304963 shutdown_connections 2024-09-11T00:30:42.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ecc077f70 0x7f3ecc07a430 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3eec1518b0 0x7f3eec1440e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3eec0c23a0 0x7f3eec143ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 --2- 172.21.15.31:0/2877304963 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3eec072480 0x7f3eec0c6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 -- 172.21.15.31:0/2877304963 >> 172.21.15.31:0/2877304963 conn(0x7f3eec0a9750 msgr2=0x7f3eec14d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 -- 172.21.15.31:0/2877304963 shutdown_connections 2024-09-11T00:30:42.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:42.977+0000 7f3ef4200640 1 -- 172.21.15.31:0/2877304963 wait complete. 2024-09-11T00:30:42.994 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'a' 2024-09-11T00:30:42.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_metadata --pg_num_min 64 2024-09-11T00:30:43.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.157+0000 7f34d7c00640 1 -- 172.21.15.31:0/3496300218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34b4004d40 msgr2=0x7f34b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:43.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.157+0000 7f34d7c00640 1 --2- 172.21.15.31:0/3496300218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34b4004d40 0x7f34b40051c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009ed0 tx=0x7f34b4005830 comp rx=0 tx=0).stop 2024-09-11T00:30:43.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.157+0000 7f34d5c00640 1 -- 172.21.15.31:0/3496300218 reap_dead start 2024-09-11T00:30:43.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.157+0000 7f34d7c00640 1 -- 172.21.15.31:0/3496300218 shutdown_connections 2024-09-11T00:30:43.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.157+0000 7f34d7c00640 1 -- 172.21.15.31:0/3496300218 >> 172.21.15.31:0/3496300218 conn(0x7f34d00a9750 msgr2=0x7f34d00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:43.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.157+0000 7f34d7c00640 1 -- 172.21.15.31:0/3496300218 shutdown_connections 2024-09-11T00:30:43.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.157+0000 7f34d7c00640 1 -- 172.21.15.31:0/3496300218 wait complete. 2024-09-11T00:30:43.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.159+0000 7f34d7c00640 1 Processor -- start 2024-09-11T00:30:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.159+0000 7f34d7c00640 1 -- start start 2024-09-11T00:30:43.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.160+0000 7f34d7c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b4004d40 0x7f34d01efe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:43.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.160+0000 7f34d7c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34d00c1040 0x7f34d0143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.160+0000 7f34d7c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34d01441f0 0x7f34d006ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.160+0000 7f34d7c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f34d0074200 con 0x7f34d01441f0 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.160+0000 7f34d7c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f34d0074080 con 0x7f34d00c1040 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.160+0000 7f34d7c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f34d0072860 con 0x7f34b4004d40 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.160+0000 7f34d5c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34d01441f0 0x7f34d006ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.160+0000 7f34d5c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34d01441f0 0x7f34d006ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39688/0 (socket says 172.21.15.31:39688) 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.160+0000 7f34d5c00640 1 -- 172.21.15.31:0/908853001 learned_addr learned my addr 172.21.15.31:0/908853001 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34d5200640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b4004d40 0x7f34d01efe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34d5c00640 1 -- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b4004d40 msgr2=0x7f34d01efe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34d5c00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b4004d40 0x7f34d01efe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34d5c00640 1 -- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34d00c1040 msgr2=0x7f34d0143cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34c7e00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34d00c1040 0x7f34d0143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34d5c00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34d00c1040 0x7f34d0143cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34d5c00640 1 -- 172.21.15.31:0/908853001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d01f0350 con 0x7f34d01441f0 2024-09-11T00:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34c7e00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34d00c1040 0x7f34d0143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34d5c00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34d01441f0 0x7f34d006ec10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f34c802aec0 tx=0x7f34c8034c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34cd600640 1 -- 172.21.15.31:0/908853001 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8043070 con 0x7f34d01441f0 2024-09-11T00:30:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34cd600640 1 -- 172.21.15.31:0/908853001 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34c8034d10 con 0x7f34d01441f0 2024-09-11T00:30:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34d7c00640 1 -- 172.21.15.31:0/908853001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d006f920 con 0x7f34d01441f0 2024-09-11T00:30:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34cd600640 1 -- 172.21.15.31:0/908853001 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8038040 con 0x7f34d01441f0 2024-09-11T00:30:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.161+0000 7f34d7c00640 1 -- 172.21.15.31:0/908853001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d006fee0 con 0x7f34d01441f0 2024-09-11T00:30:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.162+0000 7f34d7c00640 1 -- 172.21.15.31:0/908853001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d0151810 con 0x7f34d01441f0 2024-09-11T00:30:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.163+0000 7f34cd600640 1 -- 172.21.15.31:0/908853001 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f34c803b050 con 0x7f34d01441f0 2024-09-11T00:30:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.164+0000 7f34cd600640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ac077f70 0x7f34ac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.164+0000 7f34cd600640 1 -- 172.21.15.31:0/908853001 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f34c80b9990 con 0x7f34d01441f0 2024-09-11T00:30:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.165+0000 7f34d5200640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ac077f70 0x7f34ac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.166+0000 7f34d5200640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ac077f70 0x7f34ac07a430 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f34d01e9cb0 tx=0x7f34c0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.169+0000 7f34cd600640 1 -- 172.21.15.31:0/908853001 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f34c8082060 con 0x7f34d01441f0 2024-09-11T00:30:43.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.366+0000 7f34d7c00640 1 -- 172.21.15.31:0/908853001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64} v 0) -- 0x7f34d014e830 con 0x7f34d01441f0 2024-09-11T00:30:43.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.884+0000 7f34cd600640 1 -- 172.21.15.31:0/908853001 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64}]=0 pool 'a_metadata' created v19) ==== 128+0+0 (secure 0 0 0) 0x7f34c8086a90 con 0x7f34d01441f0 2024-09-11T00:30:43.886 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'a_metadata' created 2024-09-11T00:30:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.888+0000 7f34d7c00640 1 -- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ac077f70 msgr2=0x7f34ac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.888+0000 7f34d7c00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ac077f70 0x7f34ac07a430 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f34d01e9cb0 tx=0x7f34c0005f90 comp rx=0 tx=0).stop 2024-09-11T00:30:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.888+0000 7f34d7c00640 1 -- 172.21.15.31:0/908853001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34d01441f0 msgr2=0x7f34d006ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.888+0000 7f34d7c00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34d01441f0 0x7f34d006ec10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f34c802aec0 tx=0x7f34c8034c20 comp rx=0 tx=0).stop 2024-09-11T00:30:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.888+0000 7f34d7c00640 1 -- 172.21.15.31:0/908853001 shutdown_connections 2024-09-11T00:30:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.888+0000 7f34d7c00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ac077f70 0x7f34ac07a430 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.888+0000 7f34d7c00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34d01441f0 0x7f34d006ec10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.888+0000 7f34d7c00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34d00c1040 0x7f34d0143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.888+0000 7f34d7c00640 1 --2- 172.21.15.31:0/908853001 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34b4004d40 0x7f34d01efe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.888+0000 7f34d7c00640 1 -- 172.21.15.31:0/908853001 >> 172.21.15.31:0/908853001 conn(0x7f34d00a9750 msgr2=0x7f34d014dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:43.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.889+0000 7f34d7c00640 1 -- 172.21.15.31:0/908853001 shutdown_connections 2024-09-11T00:30:43.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:43.889+0000 7f34d7c00640 1 -- 172.21.15.31:0/908853001 wait complete. 2024-09-11T00:30:43.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-11T00:30:44.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.084+0000 7f66cd400640 1 -- 172.21.15.31:0/266071820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac004800 msgr2=0x7f66ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.084+0000 7f66cd400640 1 --2- 172.21.15.31:0/266071820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac004800 0x7f66ac004c80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f66b000c310 tx=0x7f66ac005830 comp rx=0 tx=0).stop 2024-09-11T00:30:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.084+0000 7f66cd400640 1 -- 172.21.15.31:0/266071820 shutdown_connections 2024-09-11T00:30:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.084+0000 7f66cd400640 1 --2- 172.21.15.31:0/266071820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66ac004800 0x7f66ac004c80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.084+0000 7f66cd400640 1 -- 172.21.15.31:0/266071820 >> 172.21.15.31:0/266071820 conn(0x7f66c80aa900 msgr2=0x7f66c80c82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.084+0000 7f66cd400640 1 -- 172.21.15.31:0/266071820 shutdown_connections 2024-09-11T00:30:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.084+0000 7f66cd400640 1 -- 172.21.15.31:0/266071820 wait complete. 2024-09-11T00:30:44.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.085+0000 7f66cd400640 1 Processor -- start 2024-09-11T00:30:44.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.085+0000 7f66cd400640 1 -- start start 2024-09-11T00:30:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.086+0000 7f66cd400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66ac004800 0x7f66c81f0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.086+0000 7f66cd400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66c81ef810 0x7f66c81dd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.086+0000 7f66cd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c81dd5f0 0x7f66c81ec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.086+0000 7f66cd400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66c80c8f10 con 0x7f66c81dd5f0 2024-09-11T00:30:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.086+0000 7f66cd400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f66c80c8d90 con 0x7f66ac004800 2024-09-11T00:30:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.086+0000 7f66cd400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f66c8074ca0 con 0x7f66c81ef810 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.086+0000 7f66c7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c81dd5f0 0x7f66c81ec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66c7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c81dd5f0 0x7f66c81ec8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39694/0 (socket says 172.21.15.31:39694) 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66c7200640 1 -- 172.21.15.31:0/3931595385 learned_addr learned my addr 172.21.15.31:0/3931595385 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66bde00640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66c81ef810 0x7f66c81dd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66c6800640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66ac004800 0x7f66c81f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66c7200640 1 -- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66c81ef810 msgr2=0x7f66c81dd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66c7200640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66c81ef810 0x7f66c81dd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66c7200640 1 -- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66ac004800 msgr2=0x7f66c81f0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66c7200640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66ac004800 0x7f66c81f0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66c7200640 1 -- 172.21.15.31:0/3931595385 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c81dc5a0 con 0x7f66c81dd5f0 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66c7200640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c81dd5f0 0x7f66c81ec8e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f66c0005fd0 tx=0x7f66c000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66bf400640 1 -- 172.21.15.31:0/3931595385 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0003d80 con 0x7f66c81dd5f0 2024-09-11T00:30:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66bf400640 1 -- 172.21.15.31:0/3931595385 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66c0004b20 con 0x7f66c81dd5f0 2024-09-11T00:30:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.087+0000 7f66bf400640 1 -- 172.21.15.31:0/3931595385 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c0004050 con 0x7f66c81dd5f0 2024-09-11T00:30:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.088+0000 7f66cd400640 1 -- 172.21.15.31:0/3931595385 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c81dc880 con 0x7f66c81dd5f0 2024-09-11T00:30:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.088+0000 7f66cd400640 1 -- 172.21.15.31:0/3931595385 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c80c8920 con 0x7f66c81dd5f0 2024-09-11T00:30:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.089+0000 7f66cd400640 1 -- 172.21.15.31:0/3931595385 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c80acf80 con 0x7f66c81dd5f0 2024-09-11T00:30:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.090+0000 7f66bf400640 1 -- 172.21.15.31:0/3931595385 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f66c0018070 con 0x7f66c81dd5f0 2024-09-11T00:30:44.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.091+0000 7f66bf400640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66a4077fa0 0x7f66a407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:44.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.091+0000 7f66bf400640 1 -- 172.21.15.31:0/3931595385 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5222+0+0 (secure 0 0 0) 0x7f66c009a4b0 con 0x7f66c81dd5f0 2024-09-11T00:30:44.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.092+0000 7f66c6800640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66a4077fa0 0x7f66a407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.092+0000 7f66c6800640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66a4077fa0 0x7f66a407a460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f66b8006fd0 tx=0x7f66b8006d10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:44.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.096+0000 7f66bf400640 1 -- 172.21.15.31:0/3931595385 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f66c0062aa0 con 0x7f66c81dd5f0 2024-09-11T00:30:44.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.294+0000 7f66cd400640 1 -- 172.21.15.31:0/3931595385 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f66c81ec220 con 0x7f66c81dd5f0 2024-09-11T00:30:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.893+0000 7f66bf400640 1 -- 172.21.15.31:0/3931595385 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'a_data' created v20) ==== 160+0+0 (secure 0 0 0) 0x7f66c00674d0 con 0x7f66c81dd5f0 2024-09-11T00:30:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'a_data' created 2024-09-11T00:30:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.896+0000 7f66cd400640 1 -- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66a4077fa0 msgr2=0x7f66a407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.896+0000 7f66cd400640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66a4077fa0 0x7f66a407a460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f66b8006fd0 tx=0x7f66b8006d10 comp rx=0 tx=0).stop 2024-09-11T00:30:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.896+0000 7f66cd400640 1 -- 172.21.15.31:0/3931595385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c81dd5f0 msgr2=0x7f66c81ec8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.896+0000 7f66cd400640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c81dd5f0 0x7f66c81ec8e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f66c0005fd0 tx=0x7f66c000f790 comp rx=0 tx=0).stop 2024-09-11T00:30:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.899+0000 7f66cd400640 1 -- 172.21.15.31:0/3931595385 shutdown_connections 2024-09-11T00:30:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.899+0000 7f66cd400640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66a4077fa0 0x7f66a407a460 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.899+0000 7f66cd400640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66c81dd5f0 0x7f66c81ec8e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.899+0000 7f66cd400640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66c81ef810 0x7f66c81dd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.899+0000 7f66cd400640 1 --2- 172.21.15.31:0/3931595385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66ac004800 0x7f66c81f0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.899+0000 7f66cd400640 1 -- 172.21.15.31:0/3931595385 >> 172.21.15.31:0/3931595385 conn(0x7f66c80aa900 msgr2=0x7f66c80701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:44.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.900+0000 7f66cd400640 1 -- 172.21.15.31:0/3931595385 shutdown_connections 2024-09-11T00:30:44.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:44.900+0000 7f66cd400640 1 -- 172.21.15.31:0/3931595385 wait complete. 2024-09-11T00:30:44.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new a a_metadata a_data 2024-09-11T00:30:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.096+0000 7ffa75600640 1 -- 172.21.15.31:0/43447385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa7014de40 msgr2=0x7ffa700c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.096+0000 7ffa75600640 1 --2- 172.21.15.31:0/43447385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa7014de40 0x7ffa700c43c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa60005fd0 tx=0x7ffa6002a540 comp rx=0 tx=0).stop 2024-09-11T00:30:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.096+0000 7ffa75600640 1 -- 172.21.15.31:0/43447385 shutdown_connections 2024-09-11T00:30:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.096+0000 7ffa75600640 1 --2- 172.21.15.31:0/43447385 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa700c2330 0x7ffa700c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.096+0000 7ffa75600640 1 --2- 172.21.15.31:0/43447385 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa7014de40 0x7ffa700c43c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:45.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.096+0000 7ffa75600640 1 -- 172.21.15.31:0/43447385 >> 172.21.15.31:0/43447385 conn(0x7ffa7006d560 msgr2=0x7ffa700c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.096+0000 7ffa75600640 1 -- 172.21.15.31:0/43447385 shutdown_connections 2024-09-11T00:30:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.096+0000 7ffa75600640 1 -- 172.21.15.31:0/43447385 wait complete. 2024-09-11T00:30:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.098+0000 7ffa75600640 1 Processor -- start 2024-09-11T00:30:45.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.098+0000 7ffa75600640 1 -- start start 2024-09-11T00:30:45.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa75600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa700c2330 0x7ffa7014b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:45.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa75600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa7014de40 0x7ffa70156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:45.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa75600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa70153e90 0x7ffa701ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa75600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7014e220 con 0x7ffa7014de40 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa75600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffa700c7000 con 0x7ffa700c2330 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa75600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffa7014e3a0 con 0x7ffa70153e90 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa66000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa7014de40 0x7ffa70156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa66000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa7014de40 0x7ffa70156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39720/0 (socket says 172.21.15.31:39720) 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa66000640 1 -- 172.21.15.31:0/4216261722 learned_addr learned my addr 172.21.15.31:0/4216261722 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa6ea00640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa700c2330 0x7ffa7014b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa6f400640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa70153e90 0x7ffa701ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa66000640 1 -- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa70153e90 msgr2=0x7ffa701ed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa66000640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa70153e90 0x7ffa701ed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa66000640 1 -- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa700c2330 msgr2=0x7ffa7014b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa66000640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa700c2330 0x7ffa7014b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.099+0000 7ffa66000640 1 -- 172.21.15.31:0/4216261722 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa701ee170 con 0x7ffa7014de40 2024-09-11T00:30:45.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.100+0000 7ffa66000640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa7014de40 0x7ffa70156b00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800d720 tx=0x7ffa5800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.100+0000 7ffa67400640 1 -- 172.21.15.31:0/4216261722 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58017070 con 0x7ffa7014de40 2024-09-11T00:30:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.100+0000 7ffa67400640 1 -- 172.21.15.31:0/4216261722 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa58009cc0 con 0x7ffa7014de40 2024-09-11T00:30:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.100+0000 7ffa67400640 1 -- 172.21.15.31:0/4216261722 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58005280 con 0x7ffa7014de40 2024-09-11T00:30:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.100+0000 7ffa75600640 1 -- 172.21.15.31:0/4216261722 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa701ede90 con 0x7ffa7014de40 2024-09-11T00:30:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.100+0000 7ffa75600640 1 -- 172.21.15.31:0/4216261722 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa700a9880 con 0x7ffa7014de40 2024-09-11T00:30:45.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.101+0000 7ffa75600640 1 -- 172.21.15.31:0/4216261722 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa700a9a80 con 0x7ffa7014de40 2024-09-11T00:30:45.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.102+0000 7ffa67400640 1 -- 172.21.15.31:0/4216261722 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ffa58005420 con 0x7ffa7014de40 2024-09-11T00:30:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.103+0000 7ffa67400640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffa4c077f70 0x7ffa4c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.103+0000 7ffa67400640 1 -- 172.21.15.31:0/4216261722 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5599+0+0 (secure 0 0 0) 0x7ffa58099d50 con 0x7ffa7014de40 2024-09-11T00:30:45.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.104+0000 7ffa6ea00640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffa4c077f70 0x7ffa4c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:45.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.104+0000 7ffa6ea00640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffa4c077f70 0x7ffa4c07a430 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffa60033000 tx=0x7ffa60004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:45.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.108+0000 7ffa67400640 1 -- 172.21.15.31:0/4216261722 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffa700a9a80 con 0x7ffa7014de40 2024-09-11T00:30:45.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.341+0000 7ffa75600640 1 -- 172.21.15.31:0/4216261722 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"} v 0) -- 0x7ffa70154320 con 0x7ffa7014de40 2024-09-11T00:30:45.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.895+0000 7ffa67400640 1 -- 172.21.15.31:0/4216261722 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 157+0+0 (secure 0 0 0) 0x7ffa5809d030 con 0x7ffa7014de40 2024-09-11T00:30:45.900 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 2 and data pool 3 2024-09-11T00:30:45.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.899+0000 7ffa75600640 1 -- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffa4c077f70 msgr2=0x7ffa4c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:45.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.899+0000 7ffa75600640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffa4c077f70 0x7ffa4c07a430 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffa60033000 tx=0x7ffa60004730 comp rx=0 tx=0).stop 2024-09-11T00:30:45.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.899+0000 7ffa75600640 1 -- 172.21.15.31:0/4216261722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa7014de40 msgr2=0x7ffa70156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:45.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.899+0000 7ffa75600640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa7014de40 0x7ffa70156b00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800d720 tx=0x7ffa5800dbe0 comp rx=0 tx=0).stop 2024-09-11T00:30:45.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.900+0000 7ffa75600640 1 -- 172.21.15.31:0/4216261722 shutdown_connections 2024-09-11T00:30:45.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.900+0000 7ffa75600640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffa4c077f70 0x7ffa4c07a430 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:45.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.900+0000 7ffa75600640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffa70153e90 0x7ffa701ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:45.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.900+0000 7ffa75600640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa7014de40 0x7ffa70156b00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:45.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.900+0000 7ffa75600640 1 --2- 172.21.15.31:0/4216261722 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa700c2330 0x7ffa7014b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.900+0000 7ffa75600640 1 -- 172.21.15.31:0/4216261722 >> 172.21.15.31:0/4216261722 conn(0x7ffa7006d560 msgr2=0x7ffa7006f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.901+0000 7ffa75600640 1 -- 172.21.15.31:0/4216261722 shutdown_connections 2024-09-11T00:30:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:45.901+0000 7ffa75600640 1 -- 172.21.15.31:0/4216261722 wait complete. 2024-09-11T00:30:45.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:30:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.107+0000 7f4ab2000640 1 -- 172.21.15.31:0/3025617839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a90004800 msgr2=0x7f4a90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.107+0000 7f4ab2000640 1 --2- 172.21.15.31:0/3025617839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a90004800 0x7f4a90004c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400c290 tx=0x7f4a90005830 comp rx=0 tx=0).stop 2024-09-11T00:30:46.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.107+0000 7f4ab2000640 1 -- 172.21.15.31:0/3025617839 shutdown_connections 2024-09-11T00:30:46.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.107+0000 7f4ab2000640 1 --2- 172.21.15.31:0/3025617839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a90004800 0x7f4a90004c80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.107+0000 7f4ab2000640 1 -- 172.21.15.31:0/3025617839 >> 172.21.15.31:0/3025617839 conn(0x7f4aac06caf0 msgr2=0x7f4aac06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:46.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.108+0000 7f4ab2000640 1 -- 172.21.15.31:0/3025617839 shutdown_connections 2024-09-11T00:30:46.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.108+0000 7f4ab2000640 1 -- 172.21.15.31:0/3025617839 wait complete. 2024-09-11T00:30:46.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.109+0000 7f4ab2000640 1 Processor -- start 2024-09-11T00:30:46.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.109+0000 7f4ab2000640 1 -- start start 2024-09-11T00:30:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.110+0000 7f4ab2000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a90004800 0x7f4aac072090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.110+0000 7f4ab2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aac0aa050 0x7f4aac1486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.110+0000 7f4ab2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aac0c0d60 0x7f4aac0c11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.110+0000 7f4ab2000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4aac0aca30 con 0x7f4aac0aa050 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.110+0000 7f4ab2000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4aac0ac8b0 con 0x7f4aac0c0d60 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.110+0000 7f4ab2000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4aac0acbb0 con 0x7f4a90004800 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.110+0000 7f4aa3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aac0aa050 0x7f4aac1486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.110+0000 7f4aa3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aac0aa050 0x7f4aac1486a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39734/0 (socket says 172.21.15.31:39734) 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.110+0000 7f4aa3e00640 1 -- 172.21.15.31:0/1594625444 learned_addr learned my addr 172.21.15.31:0/1594625444 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4ab0c00640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a90004800 0x7f4aac072090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4ab1600640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aac0c0d60 0x7f4aac0c11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4aa3e00640 1 -- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a90004800 msgr2=0x7f4aac072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4aa3e00640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a90004800 0x7f4aac072090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4aa3e00640 1 -- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aac0c0d60 msgr2=0x7f4aac0c11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4aa3e00640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aac0c0d60 0x7f4aac0c11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4aa3e00640 1 -- 172.21.15.31:0/1594625444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aac148fb0 con 0x7f4aac0aa050 2024-09-11T00:30:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4aa3e00640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aac0aa050 0x7f4aac1486a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400c3b0 tx=0x7f4a94003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4aa9600640 1 -- 172.21.15.31:0/1594625444 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94001cb0 con 0x7f4aac0aa050 2024-09-11T00:30:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4aa9600640 1 -- 172.21.15.31:0/1594625444 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a940014f0 con 0x7f4aac0aa050 2024-09-11T00:30:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4aa9600640 1 -- 172.21.15.31:0/1594625444 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9402e6e0 con 0x7f4aac0aa050 2024-09-11T00:30:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.111+0000 7f4ab2000640 1 -- 172.21.15.31:0/1594625444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aac149250 con 0x7f4aac0aa050 2024-09-11T00:30:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.112+0000 7f4ab2000640 1 -- 172.21.15.31:0/1594625444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aac1efd40 con 0x7f4aac0aa050 2024-09-11T00:30:46.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.113+0000 7f4ab2000640 1 -- 172.21.15.31:0/1594625444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aac0c6600 con 0x7f4aac0aa050 2024-09-11T00:30:46.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.114+0000 7f4aa9600640 1 -- 172.21.15.31:0/1594625444 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4a9402e880 con 0x7f4aac0aa050 2024-09-11T00:30:46.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.115+0000 7f4aa9600640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a88077f70 0x7f4a8807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.115+0000 7f4aa9600640 1 -- 172.21.15.31:0/1594625444 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7f4a940b9ff0 con 0x7f4aac0aa050 2024-09-11T00:30:46.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.115+0000 7f4ab0c00640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a88077f70 0x7f4a8807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.116+0000 7f4ab0c00640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a88077f70 0x7f4a8807a430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4aac158470 tx=0x7f4a9c006b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:46.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.119+0000 7f4aa9600640 1 -- 172.21.15.31:0/1594625444 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4a94047070 con 0x7f4aac0aa050 2024-09-11T00:30:46.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.314+0000 7f4ab2000640 1 -- 172.21.15.31:0/1594625444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4aac150a00 con 0x7f4aac0aa050 2024-09-11T00:30:46.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.315+0000 7f4aa9600640 1 -- 172.21.15.31:0/1594625444 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16650 (secure 0 0 0) 0x7f4a9403e070 con 0x7f4aac0aa050 2024-09-11T00:30:46.317 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:46.317 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:45.342202+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:30:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 -- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a88077f70 msgr2=0x7f4a8807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a88077f70 0x7f4a8807a430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4aac158470 tx=0x7f4a9c006b70 comp rx=0 tx=0).stop 2024-09-11T00:30:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 -- 172.21.15.31:0/1594625444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aac0aa050 msgr2=0x7f4aac1486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aac0aa050 0x7f4aac1486a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400c3b0 tx=0x7f4a94003e40 comp rx=0 tx=0).stop 2024-09-11T00:30:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 -- 172.21.15.31:0/1594625444 shutdown_connections 2024-09-11T00:30:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a88077f70 0x7f4a8807a430 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aac0c0d60 0x7f4aac0c11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aac0aa050 0x7f4aac1486a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 --2- 172.21.15.31:0/1594625444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a90004800 0x7f4aac072090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 -- 172.21.15.31:0/1594625444 >> 172.21.15.31:0/1594625444 conn(0x7f4aac06caf0 msgr2=0x7f4aac14e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 -- 172.21.15.31:0/1594625444 shutdown_connections 2024-09-11T00:30:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.317+0000 7f4ab2000640 1 -- 172.21.15.31:0/1594625444 wait complete. 2024-09-11T00:30:46.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:30:46.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.493+0000 7fb886000640 1 -- 172.21.15.31:0/29650965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8800c23a0 msgr2=0x7fb8800c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.493+0000 7fb886000640 1 --2- 172.21.15.31:0/29650965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8800c23a0 0x7fb8800c30c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009890 tx=0x7fb86c02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.493+0000 7fb886000640 1 -- 172.21.15.31:0/29650965 shutdown_connections 2024-09-11T00:30:46.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.493+0000 7fb886000640 1 --2- 172.21.15.31:0/29650965 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8800c23a0 0x7fb8800c30c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.493+0000 7fb886000640 1 -- 172.21.15.31:0/29650965 >> 172.21.15.31:0/29650965 conn(0x7fb8800a9750 msgr2=0x7fb8800a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:46.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.493+0000 7fb886000640 1 -- 172.21.15.31:0/29650965 shutdown_connections 2024-09-11T00:30:46.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.493+0000 7fb886000640 1 -- 172.21.15.31:0/29650965 wait complete. 2024-09-11T00:30:46.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.495+0000 7fb886000640 1 Processor -- start 2024-09-11T00:30:46.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.495+0000 7fb886000640 1 -- start start 2024-09-11T00:30:46.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.495+0000 7fb886000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb880072480 0x7fb880151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb886000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8800c23a0 0x7fb8801ec610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb886000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8801518b0 0x7fb8801ecb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb886000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb880074200 con 0x7fb880072480 2024-09-11T00:30:46.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb886000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb880074080 con 0x7fb8801518b0 2024-09-11T00:30:46.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb886000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb880072860 con 0x7fb8800c23a0 2024-09-11T00:30:46.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb87f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb880072480 0x7fb880151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb87f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb880072480 0x7fb880151f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39736/0 (socket says 172.21.15.31:39736) 2024-09-11T00:30:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb87f400640 1 -- 172.21.15.31:0/395549409 learned_addr learned my addr 172.21.15.31:0/395549409 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb87ea00640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8800c23a0 0x7fb8801ec610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb87fe00640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8801518b0 0x7fb8801ecb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb87f400640 1 -- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8800c23a0 msgr2=0x7fb8801ec610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb87f400640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8800c23a0 0x7fb8801ec610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb87f400640 1 -- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8801518b0 msgr2=0x7fb8801ecb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb87f400640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8801518b0 0x7fb8801ecb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.496+0000 7fb87f400640 1 -- 172.21.15.31:0/395549409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88006ee60 con 0x7fb880072480 2024-09-11T00:30:46.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.497+0000 7fb87f400640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb880072480 0x7fb880151f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb87000c490 tx=0x7fb87000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:46.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.497+0000 7fb877400640 1 -- 172.21.15.31:0/395549409 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8700052d0 con 0x7fb880072480 2024-09-11T00:30:46.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.497+0000 7fb877400640 1 -- 172.21.15.31:0/395549409 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb870005490 con 0x7fb880072480 2024-09-11T00:30:46.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.497+0000 7fb877400640 1 -- 172.21.15.31:0/395549409 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb870012890 con 0x7fb880072480 2024-09-11T00:30:46.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.497+0000 7fb886000640 1 -- 172.21.15.31:0/395549409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8801df030 con 0x7fb880072480 2024-09-11T00:30:46.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.497+0000 7fb886000640 1 -- 172.21.15.31:0/395549409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8801524b0 con 0x7fb880072480 2024-09-11T00:30:46.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.498+0000 7fb886000640 1 -- 172.21.15.31:0/395549409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8800c15f0 con 0x7fb880072480 2024-09-11T00:30:46.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.499+0000 7fb877400640 1 -- 172.21.15.31:0/395549409 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb870012a30 con 0x7fb880072480 2024-09-11T00:30:46.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.500+0000 7fb877400640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb860077fa0 0x7fb86007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.500+0000 7fb877400640 1 -- 172.21.15.31:0/395549409 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7fb8700997b0 con 0x7fb880072480 2024-09-11T00:30:46.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.501+0000 7fb87ea00640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb860077fa0 0x7fb86007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.501+0000 7fb87ea00640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb860077fa0 0x7fb86007a460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb86c034000 tx=0x7fb86c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:46.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.504+0000 7fb877400640 1 -- 172.21.15.31:0/395549409 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb870061bd0 con 0x7fb880072480 2024-09-11T00:30:46.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.698+0000 7fb886000640 1 -- 172.21.15.31:0/395549409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb88014fcc0 con 0x7fb880072480 2024-09-11T00:30:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.699+0000 7fb877400640 1 -- 172.21.15.31:0/395549409 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16650 (secure 0 0 0) 0x7fb870066600 con 0x7fb880072480 2024-09-11T00:30:46.701 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:46.701 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":21,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:45.342202+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:30:46.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.701+0000 7fb886000640 1 -- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb860077fa0 msgr2=0x7fb86007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.701+0000 7fb886000640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb860077fa0 0x7fb86007a460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb86c034000 tx=0x7fb86c004730 comp rx=0 tx=0).stop 2024-09-11T00:30:46.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.701+0000 7fb886000640 1 -- 172.21.15.31:0/395549409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb880072480 msgr2=0x7fb880151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.701+0000 7fb886000640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb880072480 0x7fb880151f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb87000c490 tx=0x7fb87000c950 comp rx=0 tx=0).stop 2024-09-11T00:30:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.701+0000 7fb886000640 1 -- 172.21.15.31:0/395549409 shutdown_connections 2024-09-11T00:30:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.701+0000 7fb886000640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb860077fa0 0x7fb86007a460 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.701+0000 7fb886000640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8801518b0 0x7fb8801ecb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.701+0000 7fb886000640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8800c23a0 0x7fb8801ec610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.701+0000 7fb886000640 1 --2- 172.21.15.31:0/395549409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb880072480 0x7fb880151f70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.701+0000 7fb886000640 1 -- 172.21.15.31:0/395549409 >> 172.21.15.31:0/395549409 conn(0x7fb8800a9750 msgr2=0x7fb8801502d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.702+0000 7fb886000640 1 -- 172.21.15.31:0/395549409 shutdown_connections 2024-09-11T00:30:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.702+0000 7fb886000640 1 -- 172.21.15.31:0/395549409 wait complete. 2024-09-11T00:30:46.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a standby_count_wanted 0 2024-09-11T00:30:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.880+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3083395792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c58157440 msgr2=0x7f1c580c3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.880+0000 7f1c5fa00640 1 --2- 172.21.15.31:0/3083395792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c58157440 0x7f1c580c3810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1c580c73d0 tx=0x7f1c5002aab0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.880+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3083395792 shutdown_connections 2024-09-11T00:30:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.880+0000 7f1c5fa00640 1 --2- 172.21.15.31:0/3083395792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c58157440 0x7f1c580c3810 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.880+0000 7f1c5fa00640 1 --2- 172.21.15.31:0/3083395792 >> v2:172.21.15.204:3300/0 conn(0x7f1c58158130 0x7f1c58072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.880+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3083395792 >> 172.21.15.31:0/3083395792 conn(0x7f1c5806d480 msgr2=0x7f1c580a9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.880+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3083395792 shutdown_connections 2024-09-11T00:30:46.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.880+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3083395792 wait complete. 2024-09-11T00:30:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.882+0000 7f1c5fa00640 1 Processor -- start 2024-09-11T00:30:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.882+0000 7f1c5fa00640 1 -- start start 2024-09-11T00:30:46.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.882+0000 7f1c5fa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c580c23a0 0x7f1c580c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5fa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58157440 0x7f1c5814e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5fa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c58158130 0x7f1c5814eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5fa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581549b0 con 0x7f1c580c23a0 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5fa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1c581546d0 con 0x7f1c58157440 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5fa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1c580c3d50 con 0x7f1c58158130 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c580c23a0 0x7f1c580c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c580c23a0 0x7f1c580c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39752/0 (socket says 172.21.15.31:39752) 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5d000640 1 -- 172.21.15.31:0/3392449508 learned_addr learned my addr 172.21.15.31:0/3392449508 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c4fe00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58157440 0x7f1c5814e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5da00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c58158130 0x7f1c5814eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5d000640 1 -- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c58158130 msgr2=0x7f1c5814eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5d000640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c58158130 0x7f1c5814eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5d000640 1 -- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58157440 msgr2=0x7f1c5814e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5d000640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58157440 0x7f1c5814e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:46.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5d000640 1 -- 172.21.15.31:0/3392449508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c580a6db0 con 0x7f1c580c23a0 2024-09-11T00:30:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c4fe00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58157440 0x7f1c5814e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:30:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.883+0000 7f1c5d000640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c580c23a0 0x7f1c580c67c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800e790 tx=0x7f1c4800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.884+0000 7f1c55600640 1 -- 172.21.15.31:0/3392449508 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4800eea0 con 0x7f1c580c23a0 2024-09-11T00:30:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.884+0000 7f1c55600640 1 -- 172.21.15.31:0/3392449508 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c4800cdb0 con 0x7f1c580c23a0 2024-09-11T00:30:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.884+0000 7f1c55600640 1 -- 172.21.15.31:0/3392449508 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c48010690 con 0x7f1c580c23a0 2024-09-11T00:30:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.884+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3392449508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c580ad0b0 con 0x7f1c580c23a0 2024-09-11T00:30:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.884+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3392449508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c5806f9e0 con 0x7f1c580c23a0 2024-09-11T00:30:46.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.885+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3392449508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c58073b00 con 0x7f1c580c23a0 2024-09-11T00:30:46.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.886+0000 7f1c55600640 1 -- 172.21.15.31:0/3392449508 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1c48010830 con 0x7f1c580c23a0 2024-09-11T00:30:46.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.887+0000 7f1c55600640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c34077f70 0x7f1c3407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:46.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.888+0000 7f1c55600640 1 -- 172.21.15.31:0/3392449508 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7f1c4809a7e0 con 0x7f1c580c23a0 2024-09-11T00:30:46.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.888+0000 7f1c4fe00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c34077f70 0x7f1c3407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:46.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.889+0000 7f1c4fe00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c34077f70 0x7f1c3407a430 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1c580a98a0 tx=0x7f1c40009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:46.892+0000 7f1c55600640 1 -- 172.21.15.31:0/3392449508 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1c58073b00 con 0x7f1c580c23a0 2024-09-11T00:30:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.128+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3392449508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f1c58072390 con 0x7f1c580c23a0 2024-09-11T00:30:47.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.919+0000 7f1c55600640 1 -- 172.21.15.31:0/3392449508 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 113+0+0 (secure 0 0 0) 0x7f1c480632f0 con 0x7f1c580c23a0 2024-09-11T00:30:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c34077f70 msgr2=0x7f1c3407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c34077f70 0x7f1c3407a430 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1c580a98a0 tx=0x7f1c40009040 comp rx=0 tx=0).stop 2024-09-11T00:30:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3392449508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c580c23a0 msgr2=0x7f1c580c67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c580c23a0 0x7f1c580c67c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1c4800e790 tx=0x7f1c4800ec50 comp rx=0 tx=0).stop 2024-09-11T00:30:47.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3392449508 shutdown_connections 2024-09-11T00:30:47.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c34077f70 0x7f1c3407a430 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:47.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c58158130 0x7f1c5814eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:47.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c58157440 0x7f1c5814e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:47.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 --2- 172.21.15.31:0/3392449508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c580c23a0 0x7f1c580c67c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:47.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3392449508 >> 172.21.15.31:0/3392449508 conn(0x7f1c5806d480 msgr2=0x7f1c58153300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:47.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.922+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3392449508 shutdown_connections 2024-09-11T00:30:47.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:47.923+0000 7f1c5fa00640 1 -- 172.21.15.31:0/3392449508 wait complete. 2024-09-11T00:30:47.945 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-09-11T00:30:47.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false 2024-09-11T00:30:48.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.110+0000 7fd198c00640 1 -- 172.21.15.31:0/935102935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd194150490 msgr2=0x7fd19406faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:48.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.110+0000 7fd198c00640 1 --2- 172.21.15.31:0/935102935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd194150490 0x7fd19406faa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd184009830 tx=0x7fd18402a480 comp rx=0 tx=0).stop 2024-09-11T00:30:48.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.110+0000 7fd198c00640 1 -- 172.21.15.31:0/935102935 shutdown_connections 2024-09-11T00:30:48.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.110+0000 7fd198c00640 1 --2- 172.21.15.31:0/935102935 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1940c74c0 0x7fd19406ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:48.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.110+0000 7fd198c00640 1 --2- 172.21.15.31:0/935102935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd194150490 0x7fd19406faa0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:48.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.110+0000 7fd198c00640 1 -- 172.21.15.31:0/935102935 >> 172.21.15.31:0/935102935 conn(0x7fd19406d480 msgr2=0x7fd1940a9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:48.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.110+0000 7fd198c00640 1 -- 172.21.15.31:0/935102935 shutdown_connections 2024-09-11T00:30:48.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.110+0000 7fd198c00640 1 -- 172.21.15.31:0/935102935 wait complete. 2024-09-11T00:30:48.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.112+0000 7fd198c00640 1 Processor -- start 2024-09-11T00:30:48.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.112+0000 7fd198c00640 1 -- start start 2024-09-11T00:30:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd198c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd19406ffe0 0x7fd1940aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:48.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd198c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1940c74c0 0x7fd1941e08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd198c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd194150490 0x7fd194149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd198c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd1940c2050 con 0x7fd194150490 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd198c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd19406e840 con 0x7fd1940c74c0 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd198c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd194150870 con 0x7fd19406ffe0 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd192800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd194150490 0x7fd194149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd192800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd194150490 0x7fd194149950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56496/0 (socket says 172.21.15.31:56496) 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd192800640 1 -- 172.21.15.31:0/1810745353 learned_addr learned my addr 172.21.15.31:0/1810745353 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd191e00640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd19406ffe0 0x7fd1940aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd189400640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1940c74c0 0x7fd1941e08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd192800640 1 -- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd19406ffe0 msgr2=0x7fd1940aafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd192800640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd19406ffe0 0x7fd1940aafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:48.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd192800640 1 -- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1940c74c0 msgr2=0x7fd1941e08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:48.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd192800640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1940c74c0 0x7fd1941e08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:48.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.113+0000 7fd192800640 1 -- 172.21.15.31:0/1810745353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1941ec470 con 0x7fd194150490 2024-09-11T00:30:48.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.114+0000 7fd192800640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd194150490 0x7fd194149950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd18c00c6f0 tx=0x7fd18c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:48.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.114+0000 7fd18aa00640 1 -- 172.21.15.31:0/1810745353 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c017070 con 0x7fd194150490 2024-09-11T00:30:48.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.114+0000 7fd18aa00640 1 -- 172.21.15.31:0/1810745353 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd18c003c90 con 0x7fd194150490 2024-09-11T00:30:48.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.114+0000 7fd18aa00640 1 -- 172.21.15.31:0/1810745353 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c003e60 con 0x7fd194150490 2024-09-11T00:30:48.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.114+0000 7fd198c00640 1 -- 172.21.15.31:0/1810745353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1941ebef0 con 0x7fd194150490 2024-09-11T00:30:48.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.114+0000 7fd198c00640 1 -- 172.21.15.31:0/1810745353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1941ec190 con 0x7fd194150490 2024-09-11T00:30:48.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.115+0000 7fd198c00640 1 -- 172.21.15.31:0/1810745353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd19406de30 con 0x7fd194150490 2024-09-11T00:30:48.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.116+0000 7fd18aa00640 1 -- 172.21.15.31:0/1810745353 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd18c0052d0 con 0x7fd194150490 2024-09-11T00:30:48.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.117+0000 7fd18aa00640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd170077f70 0x7fd17007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:48.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.117+0000 7fd18aa00640 1 -- 172.21.15.31:0/1810745353 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7fd18c099670 con 0x7fd194150490 2024-09-11T00:30:48.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.118+0000 7fd191e00640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd170077f70 0x7fd17007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:48.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.118+0000 7fd191e00640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd170077f70 0x7fd17007a430 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd184034000 tx=0x7fd184004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:48.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.121+0000 7fd18aa00640 1 -- 172.21.15.31:0/1810745353 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd18c062170 con 0x7fd194150490 2024-09-11T00:30:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.355+0000 7fd198c00640 1 -- 172.21.15.31:0/1810745353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd1941e0e10 con 0x7fd194150490 2024-09-11T00:30:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.927+0000 7fd18aa00640 1 -- 172.21.15.31:0/1810745353 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 117+0+0 (secure 0 0 0) 0x7fd18c061a70 con 0x7fd194150490 2024-09-11T00:30:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.930+0000 7fd198c00640 1 -- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd170077f70 msgr2=0x7fd17007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.930+0000 7fd198c00640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd170077f70 0x7fd17007a430 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd184034000 tx=0x7fd184004730 comp rx=0 tx=0).stop 2024-09-11T00:30:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.930+0000 7fd198c00640 1 -- 172.21.15.31:0/1810745353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd194150490 msgr2=0x7fd194149950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.930+0000 7fd198c00640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd194150490 0x7fd194149950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd18c00c6f0 tx=0x7fd18c00cbb0 comp rx=0 tx=0).stop 2024-09-11T00:30:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.931+0000 7fd198c00640 1 -- 172.21.15.31:0/1810745353 shutdown_connections 2024-09-11T00:30:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.931+0000 7fd198c00640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd170077f70 0x7fd17007a430 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.931+0000 7fd198c00640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd194150490 0x7fd194149950 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.931+0000 7fd198c00640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1940c74c0 0x7fd1941e08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.931+0000 7fd198c00640 1 --2- 172.21.15.31:0/1810745353 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd19406ffe0 0x7fd1940aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.931+0000 7fd198c00640 1 -- 172.21.15.31:0/1810745353 >> 172.21.15.31:0/1810745353 conn(0x7fd19406d480 msgr2=0x7fd19414dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:48.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.931+0000 7fd198c00640 1 -- 172.21.15.31:0/1810745353 shutdown_connections 2024-09-11T00:30:48.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:48.931+0000 7fd198c00640 1 -- 172.21.15.31:0/1810745353 wait complete. 2024-09-11T00:30:48.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a session_timeout 300 2024-09-11T00:30:49.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.114+0000 7f2cf4a00640 1 -- 172.21.15.31:0/3728572208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cf006f6c0 msgr2=0x7f2cf006fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.114+0000 7f2cf4a00640 1 --2- 172.21.15.31:0/3728572208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cf006f6c0 0x7f2cf006fad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009830 tx=0x7f2cd802a5a0 comp rx=0 tx=0).stop 2024-09-11T00:30:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.114+0000 7f2cf4a00640 1 -- 172.21.15.31:0/3728572208 shutdown_connections 2024-09-11T00:30:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.114+0000 7f2cf4a00640 1 --2- 172.21.15.31:0/3728572208 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cf006caf0 0x7f2cf006cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.114+0000 7f2cf4a00640 1 --2- 172.21.15.31:0/3728572208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cf006f6c0 0x7f2cf006fad0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.114+0000 7f2cf4a00640 1 -- 172.21.15.31:0/3728572208 >> 172.21.15.31:0/3728572208 conn(0x7f2cf006dc00 msgr2=0x7f2cf006dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.115+0000 7f2cf4a00640 1 -- 172.21.15.31:0/3728572208 shutdown_connections 2024-09-11T00:30:49.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.115+0000 7f2cf4a00640 1 -- 172.21.15.31:0/3728572208 wait complete. 2024-09-11T00:30:49.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.116+0000 7f2cf4a00640 1 Processor -- start 2024-09-11T00:30:49.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.116+0000 7f2cf4a00640 1 -- start start 2024-09-11T00:30:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2cf4a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cf006caf0 0x7f2cf01de010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2cf4a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cf006f6c0 0x7f2cf0150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2cf4a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cf00c7460 0x7f2cf00c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2cf4a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf006ee10 con 0x7f2cf006caf0 2024-09-11T00:30:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2cf4a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf00c8050 con 0x7f2cf006f6c0 2024-09-11T00:30:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2cf4a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf0070010 con 0x7f2cf00c7460 2024-09-11T00:30:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2cee800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cf006caf0 0x7f2cf01de010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2cee800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cf006caf0 0x7f2cf01de010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56510/0 (socket says 172.21.15.31:56510) 2024-09-11T00:30:49.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2cee800640 1 -- 172.21.15.31:0/178054265 learned_addr learned my addr 172.21.15.31:0/178054265 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2cef200640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cf00c7460 0x7f2cf00c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.117+0000 7f2ce5e00640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cf006f6c0 0x7f2cf0150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2cee800640 1 -- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cf00c7460 msgr2=0x7f2cf00c20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2cee800640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cf00c7460 0x7f2cf00c20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2cee800640 1 -- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cf006f6c0 msgr2=0x7f2cf0150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2cee800640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cf006f6c0 0x7f2cf0150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2cee800640 1 -- 172.21.15.31:0/178054265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf0150880 con 0x7f2cf006caf0 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2cee800640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cf006caf0 0x7f2cf01de010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000cbf0 tx=0x7f2ce000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2ce7400640 1 -- 172.21.15.31:0/178054265 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0016020 con 0x7f2cf006caf0 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2ce7400640 1 -- 172.21.15.31:0/178054265 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ce000ad90 con 0x7f2cf006caf0 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2ce7400640 1 -- 172.21.15.31:0/178054265 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0012680 con 0x7f2cf006caf0 2024-09-11T00:30:49.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2cf4a00640 1 -- 172.21.15.31:0/178054265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf01559a0 con 0x7f2cf006caf0 2024-09-11T00:30:49.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.118+0000 7f2cf4a00640 1 -- 172.21.15.31:0/178054265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf00a98e0 con 0x7f2cf006caf0 2024-09-11T00:30:49.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.119+0000 7f2cf4a00640 1 -- 172.21.15.31:0/178054265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf00c7840 con 0x7f2cf006caf0 2024-09-11T00:30:49.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.120+0000 7f2ce7400640 1 -- 172.21.15.31:0/178054265 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2ce001b400 con 0x7f2cf006caf0 2024-09-11T00:30:49.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.121+0000 7f2ce7400640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ccc077f70 0x7f2ccc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:49.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.122+0000 7f2ce7400640 1 -- 172.21.15.31:0/178054265 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f2ce009a6c0 con 0x7f2cf006caf0 2024-09-11T00:30:49.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.122+0000 7f2ce5e00640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ccc077f70 0x7f2ccc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:49.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.122+0000 7f2ce5e00640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ccc077f70 0x7f2ccc07a430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8034000 tx=0x7f2cd8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:49.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.125+0000 7f2ce7400640 1 -- 172.21.15.31:0/178054265 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2ce0020020 con 0x7f2cf006caf0 2024-09-11T00:30:49.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.363+0000 7f2cf4a00640 1 -- 172.21.15.31:0/178054265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"} v 0) -- 0x7f2cf00ab590 con 0x7f2cf006caf0 2024-09-11T00:30:49.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.929+0000 7f2ce7400640 1 -- 172.21.15.31:0/178054265 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"}]=0 v9) ==== 110+0+0 (secure 0 0 0) 0x7f2ce00631c0 con 0x7f2cf006caf0 2024-09-11T00:30:49.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.932+0000 7f2cf4a00640 1 -- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ccc077f70 msgr2=0x7f2ccc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:49.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.932+0000 7f2cf4a00640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ccc077f70 0x7f2ccc07a430 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8034000 tx=0x7f2cd8004730 comp rx=0 tx=0).stop 2024-09-11T00:30:49.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.932+0000 7f2cf4a00640 1 -- 172.21.15.31:0/178054265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cf006caf0 msgr2=0x7f2cf01de010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:49.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.932+0000 7f2cf4a00640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cf006caf0 0x7f2cf01de010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2ce000cbf0 tx=0x7f2ce000a060 comp rx=0 tx=0).stop 2024-09-11T00:30:49.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.932+0000 7f2cf4a00640 1 -- 172.21.15.31:0/178054265 shutdown_connections 2024-09-11T00:30:49.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.932+0000 7f2cf4a00640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ccc077f70 0x7f2ccc07a430 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:49.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.932+0000 7f2cf4a00640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cf00c7460 0x7f2cf00c20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:49.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.932+0000 7f2cf4a00640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cf006f6c0 0x7f2cf0150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:49.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.932+0000 7f2cf4a00640 1 --2- 172.21.15.31:0/178054265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cf006caf0 0x7f2cf01de010 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:49.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.932+0000 7f2cf4a00640 1 -- 172.21.15.31:0/178054265 >> 172.21.15.31:0/178054265 conn(0x7f2cf006dc00 msgr2=0x7f2cf00ac740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:49.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.933+0000 7f2cf4a00640 1 -- 172.21.15.31:0/178054265 shutdown_connections 2024-09-11T00:30:49.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:49.933+0000 7f2cf4a00640 1 -- 172.21.15.31:0/178054265 wait complete. 2024-09-11T00:30:49.952 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'a' 2024-09-11T00:30:49.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:30:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.111+0000 7f29ab200640 1 -- 172.21.15.31:0/2590915147 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29ac0a6e20 msgr2=0x7f29ac1510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.111+0000 7f29ab200640 1 --2- 172.21.15.31:0/2590915147 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29ac0a6e20 0x7f29ac1510c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2998009830 tx=0x7f299802a3e0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.111+0000 7f29ab200640 1 -- 172.21.15.31:0/2590915147 shutdown_connections 2024-09-11T00:30:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.111+0000 7f29ab200640 1 --2- 172.21.15.31:0/2590915147 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29ac0a6e20 0x7f29ac1510c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.111+0000 7f29ab200640 1 --2- 172.21.15.31:0/2590915147 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29ac0ac2c0 0x7f29ac0ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.111+0000 7f29ab200640 1 -- 172.21.15.31:0/2590915147 >> 172.21.15.31:0/2590915147 conn(0x7f29ac06caf0 msgr2=0x7f29ac06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:50.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.111+0000 7f29ab200640 1 -- 172.21.15.31:0/2590915147 shutdown_connections 2024-09-11T00:30:50.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.111+0000 7f29ab200640 1 -- 172.21.15.31:0/2590915147 wait complete. 2024-09-11T00:30:50.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.113+0000 7f29ab200640 1 Processor -- start 2024-09-11T00:30:50.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.113+0000 7f29ab200640 1 -- start start 2024-09-11T00:30:50.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.113+0000 7f29ab200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29ac0a6e20 0x7f29ac1ea430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.113+0000 7f29ab200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29ac0ac2c0 0x7f29ac149370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29ab200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29ac0acbe0 0x7f29ac0c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29ab200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac151780 con 0x7f29ac0acbe0 2024-09-11T00:30:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29ab200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac151600 con 0x7f29ac0ac2c0 2024-09-11T00:30:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29ab200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f29ac151900 con 0x7f29ac0a6e20 2024-09-11T00:30:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29aa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29ac0acbe0 0x7f29ac0c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29aa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29ac0acbe0 0x7f29ac0c23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56522/0 (socket says 172.21.15.31:56522) 2024-09-11T00:30:50.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29aa800640 1 -- 172.21.15.31:0/2972298414 learned_addr learned my addr 172.21.15.31:0/2972298414 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29aa800640 1 -- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29ac0a6e20 msgr2=0x7f29ac1ea430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29aa800640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29ac0a6e20 0x7f29ac1ea430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29aa800640 1 -- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29ac0ac2c0 msgr2=0x7f29ac149370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29aa800640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29ac0ac2c0 0x7f29ac149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29aa800640 1 -- 172.21.15.31:0/2972298414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ac0c3fc0 con 0x7f29ac0acbe0 2024-09-11T00:30:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.114+0000 7f29aa800640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29ac0acbe0 0x7f29ac0c23a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f29a400c6b0 tx=0x7f29a400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.115+0000 7f29a2000640 1 -- 172.21.15.31:0/2972298414 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4003b70 con 0x7f29ac0acbe0 2024-09-11T00:30:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.115+0000 7f29a2000640 1 -- 172.21.15.31:0/2972298414 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29a4003d30 con 0x7f29ac0acbe0 2024-09-11T00:30:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.115+0000 7f29a2000640 1 -- 172.21.15.31:0/2972298414 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4005330 con 0x7f29ac0acbe0 2024-09-11T00:30:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.115+0000 7f29ab200640 1 -- 172.21.15.31:0/2972298414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ac0c4260 con 0x7f29ac0acbe0 2024-09-11T00:30:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.115+0000 7f29ab200640 1 -- 172.21.15.31:0/2972298414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ac1e9f10 con 0x7f29ac0acbe0 2024-09-11T00:30:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.116+0000 7f29ab200640 1 -- 172.21.15.31:0/2972298414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29ac149910 con 0x7f29ac0acbe0 2024-09-11T00:30:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.117+0000 7f29a2000640 1 -- 172.21.15.31:0/2972298414 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f29a40054d0 con 0x7f29ac0acbe0 2024-09-11T00:30:50.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.118+0000 7f29a2000640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f298c077f70 0x7f298c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.119+0000 7f29a2000640 1 -- 172.21.15.31:0/2972298414 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f29a4099ad0 con 0x7f29ac0acbe0 2024-09-11T00:30:50.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.119+0000 7f29a9e00640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f298c077f70 0x7f298c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:50.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.120+0000 7f29a9e00640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f298c077f70 0x7f298c07a430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f299c005ab0 tx=0x7f299c005a20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:50.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.123+0000 7f29a2000640 1 -- 172.21.15.31:0/2972298414 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f29a40057f0 con 0x7f29ac0acbe0 2024-09-11T00:30:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.353+0000 7f29ab200640 1 -- 172.21.15.31:0/2972298414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29ac1510c0 con 0x7f29ac0acbe0 2024-09-11T00:30:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.354+0000 7f29a2000640 1 -- 172.21.15.31:0/2972298414 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5219 (secure 0 0 0) 0x7f29a40625d0 con 0x7f29ac0acbe0 2024-09-11T00:30:50.356 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:50.356 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":9,"btime":"2024-09-11T00:30:49:928196+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4374,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1}]} 2024-09-11T00:30:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 9 2024-09-11T00:30:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 -- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f298c077f70 msgr2=0x7f298c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f298c077f70 0x7f298c07a430 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f299c005ab0 tx=0x7f299c005a20 comp rx=0 tx=0).stop 2024-09-11T00:30:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 -- 172.21.15.31:0/2972298414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29ac0acbe0 msgr2=0x7f29ac0c23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29ac0acbe0 0x7f29ac0c23a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f29a400c6b0 tx=0x7f29a400cb70 comp rx=0 tx=0).stop 2024-09-11T00:30:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 -- 172.21.15.31:0/2972298414 shutdown_connections 2024-09-11T00:30:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f298c077f70 0x7f298c07a430 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29ac0acbe0 0x7f29ac0c23a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29ac0ac2c0 0x7f29ac149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 --2- 172.21.15.31:0/2972298414 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29ac0a6e20 0x7f29ac1ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 -- 172.21.15.31:0/2972298414 >> 172.21.15.31:0/2972298414 conn(0x7f29ac06caf0 msgr2=0x7f29ac150140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.356+0000 7f29ab200640 1 -- 172.21.15.31:0/2972298414 shutdown_connections 2024-09-11T00:30:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.357+0000 7f29ab200640 1 -- 172.21.15.31:0/2972298414 wait complete. 2024-09-11T00:30:50.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:30:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.533+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2592899453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f3014c880 msgr2=0x7f6f300c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.533+0000 7f6f2fe00640 1 --2- 172.21.15.31:0/2592899453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f3014c880 0x7f6f300c8ac0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009830 tx=0x7f6f2002a5c0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.533+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2592899453 shutdown_connections 2024-09-11T00:30:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.533+0000 7f6f2fe00640 1 --2- 172.21.15.31:0/2592899453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f3014c880 0x7f6f300c8ac0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.533+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2592899453 >> 172.21.15.31:0/2592899453 conn(0x7f6f300adb00 msgr2=0x7f6f300adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.533+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2592899453 shutdown_connections 2024-09-11T00:30:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.533+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2592899453 wait complete. 2024-09-11T00:30:50.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.534+0000 7f6f2fe00640 1 Processor -- start 2024-09-11T00:30:50.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.535+0000 7f6f2fe00640 1 -- start start 2024-09-11T00:30:50.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.535+0000 7f6f2fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f3006cc10 0x7f6f30147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.535+0000 7f6f2fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f300c9000 0x7f6f30153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.535+0000 7f6f2fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f3014c880 0x7f6f301ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.535+0000 7f6f2fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3014cc60 con 0x7f6f3006cc10 2024-09-11T00:30:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.535+0000 7f6f2fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6f30156810 con 0x7f6f300c9000 2024-09-11T00:30:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.535+0000 7f6f2fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6f3006edc0 con 0x7f6f3014c880 2024-09-11T00:30:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f3006cc10 0x7f6f30147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f3006cc10 0x7f6f30147ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56532/0 (socket says 172.21.15.31:56532) 2024-09-11T00:30:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2ea00640 1 -- 172.21.15.31:0/2773973073 learned_addr learned my addr 172.21.15.31:0/2773973073 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2f400640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f3014c880 0x7f6f301ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2e000640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f300c9000 0x7f6f30153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2ea00640 1 -- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f3014c880 msgr2=0x7f6f301ebfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2ea00640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f3014c880 0x7f6f301ebfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2ea00640 1 -- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f300c9000 msgr2=0x7f6f30153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2ea00640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f300c9000 0x7f6f30153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2ea00640 1 -- 172.21.15.31:0/2773973073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f30158a20 con 0x7f6f3006cc10 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f2ea00640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f3006cc10 0x7f6f30147ee0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009950 tx=0x7f6f20033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.536+0000 7f6f26a00640 1 -- 172.21.15.31:0/2773973073 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20005c90 con 0x7f6f3006cc10 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.537+0000 7f6f26a00640 1 -- 172.21.15.31:0/2773973073 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f20031e50 con 0x7f6f3006cc10 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.537+0000 7f6f26a00640 1 -- 172.21.15.31:0/2773973073 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20005260 con 0x7f6f3006cc10 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.537+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2773973073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f3014ce70 con 0x7f6f3006cc10 2024-09-11T00:30:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.537+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2773973073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f301488e0 con 0x7f6f3006cc10 2024-09-11T00:30:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.538+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2773973073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f300c59a0 con 0x7f6f3006cc10 2024-09-11T00:30:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.539+0000 7f6f26a00640 1 -- 172.21.15.31:0/2773973073 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6f20005400 con 0x7f6f3006cc10 2024-09-11T00:30:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.540+0000 7f6f26a00640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f10077f70 0x7f6f1007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.540+0000 7f6f26a00640 1 -- 172.21.15.31:0/2773973073 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f6f200bada0 con 0x7f6f3006cc10 2024-09-11T00:30:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.540+0000 7f6f2e000640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f10077f70 0x7f6f1007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.541+0000 7f6f2e000640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f10077f70 0x7f6f1007a430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c007a50 tx=0x7f6f1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:50.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.544+0000 7f6f26a00640 1 -- 172.21.15.31:0/2773973073 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6f2003f030 con 0x7f6f3006cc10 2024-09-11T00:30:50.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.744+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2773973073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f30148ab0 con 0x7f6f3006cc10 2024-09-11T00:30:50.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.746+0000 7f6f26a00640 1 -- 172.21.15.31:0/2773973073 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16637 (secure 0 0 0) 0x7f6f20083810 con 0x7f6f3006cc10 2024-09-11T00:30:50.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:50.748 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":22,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:46.906141+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:30:50.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f10077f70 msgr2=0x7f6f1007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f10077f70 0x7f6f1007a430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c007a50 tx=0x7f6f1c009040 comp rx=0 tx=0).stop 2024-09-11T00:30:50.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2773973073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f3006cc10 msgr2=0x7f6f30147ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f3006cc10 0x7f6f30147ee0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009950 tx=0x7f6f20033c20 comp rx=0 tx=0).stop 2024-09-11T00:30:50.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2773973073 shutdown_connections 2024-09-11T00:30:50.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f10077f70 0x7f6f1007a430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f3014c880 0x7f6f301ebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f300c9000 0x7f6f30153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 --2- 172.21.15.31:0/2773973073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f3006cc10 0x7f6f30147ee0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2773973073 >> 172.21.15.31:0/2773973073 conn(0x7f6f300adb00 msgr2=0x7f6f300ab810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:50.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2773973073 shutdown_connections 2024-09-11T00:30:50.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.748+0000 7f6f2fe00640 1 -- 172.21.15.31:0/2773973073 wait complete. 2024-09-11T00:30:50.766 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T00:30:50.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:30:50.878 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:30:50.876+0000 7f5704400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:30:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.925+0000 7f403d600640 1 -- 172.21.15.31:0/3895432598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40380c9fe0 msgr2=0x7f40380cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.925+0000 7f403d600640 1 --2- 172.21.15.31:0/3895432598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40380c9fe0 0x7f40380cc310 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4030005fd0 tx=0x7f403002a740 comp rx=0 tx=0).stop 2024-09-11T00:30:50.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.925+0000 7f403d600640 1 -- 172.21.15.31:0/3895432598 shutdown_connections 2024-09-11T00:30:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.925+0000 7f403d600640 1 --2- 172.21.15.31:0/3895432598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40380c9fe0 0x7f40380cc310 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.925+0000 7f403d600640 1 -- 172.21.15.31:0/3895432598 >> 172.21.15.31:0/3895432598 conn(0x7f4038068f10 msgr2=0x7f40380cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.925+0000 7f403d600640 1 -- 172.21.15.31:0/3895432598 shutdown_connections 2024-09-11T00:30:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.925+0000 7f403d600640 1 -- 172.21.15.31:0/3895432598 wait complete. 2024-09-11T00:30:50.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.927+0000 7f403d600640 1 Processor -- start 2024-09-11T00:30:50.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.927+0000 7f403d600640 1 -- start start 2024-09-11T00:30:50.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f403d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40380b1230 0x7f40381e2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f403d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40380c9b00 0x7f40380adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f403d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40380c9fe0 0x7f40381efa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f403d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f40380bf390 con 0x7f40380c9b00 2024-09-11T00:30:50.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f403d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4038156540 con 0x7f40380b1230 2024-09-11T00:30:50.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f403d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f40380bf510 con 0x7f40380c9fe0 2024-09-11T00:30:50.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f402fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40380c9b00 0x7f40380adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:50.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f402fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40380c9b00 0x7f40380adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56540/0 (socket says 172.21.15.31:56540) 2024-09-11T00:30:50.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f402fe00640 1 -- 172.21.15.31:0/2124219536 learned_addr learned my addr 172.21.15.31:0/2124219536 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:50.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f4037400640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40380c9fe0 0x7f40381efa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:50.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f4036a00640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40380b1230 0x7f40381e2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:50.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f402fe00640 1 -- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40380c9fe0 msgr2=0x7f40381efa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.928+0000 7f402fe00640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40380c9fe0 0x7f40381efa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.929+0000 7f402fe00640 1 -- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40380b1230 msgr2=0x7f40381e2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.929+0000 7f402fe00640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40380b1230 0x7f40381e2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.929+0000 7f402fe00640 1 -- 172.21.15.31:0/2124219536 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40381ed440 con 0x7f40380c9b00 2024-09-11T00:30:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.929+0000 7f402fe00640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40380c9b00 0x7f40380adb90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f402000c760 tx=0x7f402000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.929+0000 7f402ea00640 1 -- 172.21.15.31:0/2124219536 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40200050d0 con 0x7f40380c9b00 2024-09-11T00:30:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.929+0000 7f402ea00640 1 -- 172.21.15.31:0/2124219536 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4020005270 con 0x7f40380c9b00 2024-09-11T00:30:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.929+0000 7f402ea00640 1 -- 172.21.15.31:0/2124219536 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402000a8a0 con 0x7f40380c9b00 2024-09-11T00:30:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.929+0000 7f403d600640 1 -- 172.21.15.31:0/2124219536 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40380c5f50 con 0x7f40380c9b00 2024-09-11T00:30:50.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.929+0000 7f403d600640 1 -- 172.21.15.31:0/2124219536 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40381e1a00 con 0x7f40380c9b00 2024-09-11T00:30:50.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.930+0000 7f403d600640 1 -- 172.21.15.31:0/2124219536 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40380766d0 con 0x7f40380c9b00 2024-09-11T00:30:50.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.932+0000 7f402ea00640 1 -- 172.21.15.31:0/2124219536 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f402000aa40 con 0x7f40380c9b00 2024-09-11T00:30:50.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.932+0000 7f402ea00640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4014077f70 0x7f401407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:50.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.933+0000 7f402ea00640 1 -- 172.21.15.31:0/2124219536 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f4020099740 con 0x7f40380c9b00 2024-09-11T00:30:50.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.933+0000 7f4036a00640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4014077f70 0x7f401407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:50.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.934+0000 7f4036a00640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4014077f70 0x7f401407a430 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4028001eb0 tx=0x7f4028009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:50.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:50.937+0000 7f402ea00640 1 -- 172.21.15.31:0/2124219536 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f40380b1610 con 0x7f40380c9b00 2024-09-11T00:30:51.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.136+0000 7f403d600640 1 -- 172.21.15.31:0/2124219536 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f40380b0000 con 0x7f4014077f70 2024-09-11T00:30:51.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.143+0000 7f402ea00640 1 -- 172.21.15.31:0/2124219536 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251933 (secure 0 0 0) 0x7f40380b0000 con 0x7f4014077f70 2024-09-11T00:30:51.148 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:51.149 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:30:51.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 -- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4014077f70 msgr2=0x7f401407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4014077f70 0x7f401407a430 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4028001eb0 tx=0x7f4028009040 comp rx=0 tx=0).stop 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 -- 172.21.15.31:0/2124219536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40380c9b00 msgr2=0x7f40380adb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40380c9b00 0x7f40380adb90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f402000c760 tx=0x7f402000cc20 comp rx=0 tx=0).stop 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 -- 172.21.15.31:0/2124219536 shutdown_connections 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4014077f70 0x7f401407a430 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40380c9fe0 0x7f40381efa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40380c9b00 0x7f40380adb90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 --2- 172.21.15.31:0/2124219536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40380b1230 0x7f40381e2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 -- 172.21.15.31:0/2124219536 >> 172.21.15.31:0/2124219536 conn(0x7f4038068f10 msgr2=0x7f40381558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 -- 172.21.15.31:0/2124219536 shutdown_connections 2024-09-11T00:30:51.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.147+0000 7f403d600640 1 -- 172.21.15.31:0/2124219536 wait complete. 2024-09-11T00:30:51.164 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-11T00:30:50.392614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219192,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755528,"statfs":{"total":773094113280,"available":772869660672,"internally_reserved":0,"allocated":3022848,"data_stored":1795188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001495"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923421+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:30:46.923421+0000","last_peered":"2024-09-11T00:30:46.923421+0000","last_clean":"2024-09-11T00:30:46.923421+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:30:46.923421+0000","last_undegraded":"2024-09-11T00:30:46.923421+0000","last_fullsized":"2024-09-11T00:30:46.923421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916735+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:30:46.916735+0000","last_peered":"2024-09-11T00:30:46.916735+0000","last_clean":"2024-09-11T00:30:46.916735+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:30:46.916735+0000","last_undegraded":"2024-09-11T00:30:46.916735+0000","last_fullsized":"2024-09-11T00:30:46.916735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916834+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:30:46.916834+0000","last_peered":"2024-09-11T00:30:46.916834+0000","last_clean":"2024-09-11T00:30:46.916834+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:30:46.916834+0000","last_undegraded":"2024-09-11T00:30:46.916834+0000","last_fullsized":"2024-09-11T00:30:46.916834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016119+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:30:47.016119+0000","last_peered":"2024-09-11T00:30:47.016119+0000","last_clean":"2024-09-11T00:30:47.016119+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:30:47.016119+0000","last_undegraded":"2024-09-11T00:30:47.016119+0000","last_fullsized":"2024-09-11T00:30:47.016119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923185+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:30:46.923185+0000","last_peered":"2024-09-11T00:30:46.923185+0000","last_clean":"2024-09-11T00:30:46.923185+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:30:46.923185+0000","last_undegraded":"2024-09-11T00:30:46.923185+0000","last_fullsized":"2024-09-11T00:30:46.923185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015860+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:30:47.015860+0000","last_peered":"2024-09-11T00:30:47.015860+0000","last_clean":"2024-09-11T00:30:47.015860+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:30:47.015860+0000","last_undegraded":"2024-09-11T00:30:47.015860+0000","last_fullsized":"2024-09-11T00:30:47.015860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923059+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:30:46.923059+0000","last_peered":"2024-09-11T00:30:46.923059+0000","last_clean":"2024-09-11T00:30:46.923059+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:30:46.923059+0000","last_undegraded":"2024-09-11T00:30:46.923059+0000","last_fullsized":"2024-09-11T00:30:46.923059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915628+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:30:46.915628+0000","last_peered":"2024-09-11T00:30:46.915628+0000","last_clean":"2024-09-11T00:30:46.915628+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:30:46.915628+0000","last_undegraded":"2024-09-11T00:30:46.915628+0000","last_fullsized":"2024-09-11T00:30:46.915628+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.914177+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:30:46.914177+0000","last_peered":"2024-09-11T00:30:46.914177+0000","last_clean":"2024-09-11T00:30:46.914177+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:30:46.914177+0000","last_undegraded":"2024-09-11T00:30:46.914177+0000","last_fullsized":"2024-09-11T00:30:46.914177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923489+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:30:46.923489+0000","last_peered":"2024-09-11T00:30:46.923489+0000","last_clean":"2024-09-11T00:30:46.923489+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:46.923489+0000","last_undegraded":"2024-09-11T00:30:46.923489+0000","last_fullsized":"2024-09-11T00:30:46.923489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923575+0000","last_change":"2024-09-11T00:30:45.941635+0000","last_active":"2024-09-11T00:30:46.923575+0000","last_peered":"2024-09-11T00:30:46.923575+0000","last_clean":"2024-09-11T00:30:46.923575+0000","last_became_active":"2024-09-11T00:30:45.941357+0000","last_became_peered":"2024-09-11T00:30:45.941357+0000","last_unstale":"2024-09-11T00:30:46.923575+0000","last_undegraded":"2024-09-11T00:30:46.923575+0000","last_fullsized":"2024-09-11T00:30:46.923575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:12.016406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923293+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:30:46.923293+0000","last_peered":"2024-09-11T00:30:46.923293+0000","last_clean":"2024-09-11T00:30:46.923293+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:30:46.923293+0000","last_undegraded":"2024-09-11T00:30:46.923293+0000","last_fullsized":"2024-09-11T00:30:46.923293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917143+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:30:46.917143+0000","last_peered":"2024-09-11T00:30:46.917143+0000","last_clean":"2024-09-11T00:30:46.917143+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:30:46.917143+0000","last_undegraded":"2024-09-11T00:30:46.917143+0000","last_fullsized":"2024-09-11T00:30:46.917143+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915945+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:30:46.915945+0000","last_peered":"2024-09-11T00:30:46.915945+0000","last_clean":"2024-09-11T00:30:46.915945+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:30:46.915945+0000","last_undegraded":"2024-09-11T00:30:46.915945+0000","last_fullsized":"2024-09-11T00:30:46.915945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915659+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:30:46.915659+0000","last_peered":"2024-09-11T00:30:46.915659+0000","last_clean":"2024-09-11T00:30:46.915659+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:30:46.915659+0000","last_undegraded":"2024-09-11T00:30:46.915659+0000","last_fullsized":"2024-09-11T00:30:46.915659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016126+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:30:47.016126+0000","last_peered":"2024-09-11T00:30:47.016126+0000","last_clean":"2024-09-11T00:30:47.016126+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:30:47.016126+0000","last_undegraded":"2024-09-11T00:30:47.016126+0000","last_fullsized":"2024-09-11T00:30:47.016126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017199+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:30:47.017199+0000","last_peered":"2024-09-11T00:30:47.017199+0000","last_clean":"2024-09-11T00:30:47.017199+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:30:47.017199+0000","last_undegraded":"2024-09-11T00:30:47.017199+0000","last_fullsized":"2024-09-11T00:30:47.017199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016204+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:30:47.016204+0000","last_peered":"2024-09-11T00:30:47.016204+0000","last_clean":"2024-09-11T00:30:47.016204+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:30:47.016204+0000","last_undegraded":"2024-09-11T00:30:47.016204+0000","last_fullsized":"2024-09-11T00:30:47.016204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017083+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:30:47.017083+0000","last_peered":"2024-09-11T00:30:47.017083+0000","last_clean":"2024-09-11T00:30:47.017083+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:30:47.017083+0000","last_undegraded":"2024-09-11T00:30:47.017083+0000","last_fullsized":"2024-09-11T00:30:47.017083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923319+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:30:46.923319+0000","last_peered":"2024-09-11T00:30:46.923319+0000","last_clean":"2024-09-11T00:30:46.923319+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:30:46.923319+0000","last_undegraded":"2024-09-11T00:30:46.923319+0000","last_fullsized":"2024-09-11T00:30:46.923319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920455+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:30:46.920455+0000","last_peered":"2024-09-11T00:30:46.920455+0000","last_clean":"2024-09-11T00:30:46.920455+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:46.920455+0000","last_undegraded":"2024-09-11T00:30:46.920455+0000","last_fullsized":"2024-09-11T00:30:46.920455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.913624+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:30:46.913624+0000","last_peered":"2024-09-11T00:30:46.913624+0000","last_clean":"2024-09-11T00:30:46.913624+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:30:46.913624+0000","last_undegraded":"2024-09-11T00:30:46.913624+0000","last_fullsized":"2024-09-11T00:30:46.913624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915955+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:30:46.915955+0000","last_peered":"2024-09-11T00:30:46.915955+0000","last_clean":"2024-09-11T00:30:46.915955+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:30:46.915955+0000","last_undegraded":"2024-09-11T00:30:46.915955+0000","last_fullsized":"2024-09-11T00:30:46.915955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016366+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:30:47.016366+0000","last_peered":"2024-09-11T00:30:47.016366+0000","last_clean":"2024-09-11T00:30:47.016366+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:30:47.016366+0000","last_undegraded":"2024-09-11T00:30:47.016366+0000","last_fullsized":"2024-09-11T00:30:47.016366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917531+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:30:46.917531+0000","last_peered":"2024-09-11T00:30:46.917531+0000","last_clean":"2024-09-11T00:30:46.917531+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:30:46.917531+0000","last_undegraded":"2024-09-11T00:30:46.917531+0000","last_fullsized":"2024-09-11T00:30:46.917531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017119+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:30:47.017119+0000","last_peered":"2024-09-11T00:30:47.017119+0000","last_clean":"2024-09-11T00:30:47.017119+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:30:47.017119+0000","last_undegraded":"2024-09-11T00:30:47.017119+0000","last_fullsized":"2024-09-11T00:30:47.017119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917612+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:30:46.917612+0000","last_peered":"2024-09-11T00:30:46.917612+0000","last_clean":"2024-09-11T00:30:46.917612+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:30:46.917612+0000","last_undegraded":"2024-09-11T00:30:46.917612+0000","last_fullsized":"2024-09-11T00:30:46.917612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917421+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:30:46.917421+0000","last_peered":"2024-09-11T00:30:46.917421+0000","last_clean":"2024-09-11T00:30:46.917421+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:30:46.917421+0000","last_undegraded":"2024-09-11T00:30:46.917421+0000","last_fullsized":"2024-09-11T00:30:46.917421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916541+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:30:46.916541+0000","last_peered":"2024-09-11T00:30:46.916541+0000","last_clean":"2024-09-11T00:30:46.916541+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:30:46.916541+0000","last_undegraded":"2024-09-11T00:30:46.916541+0000","last_fullsized":"2024-09-11T00:30:46.916541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016666+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:30:47.016666+0000","last_peered":"2024-09-11T00:30:47.016666+0000","last_clean":"2024-09-11T00:30:47.016666+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:30:47.016666+0000","last_undegraded":"2024-09-11T00:30:47.016666+0000","last_fullsized":"2024-09-11T00:30:47.016666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015898+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:30:47.015898+0000","last_peered":"2024-09-11T00:30:47.015898+0000","last_clean":"2024-09-11T00:30:47.015898+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:30:47.015898+0000","last_undegraded":"2024-09-11T00:30:47.015898+0000","last_fullsized":"2024-09-11T00:30:47.015898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015698+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:30:47.015698+0000","last_peered":"2024-09-11T00:30:47.015698+0000","last_clean":"2024-09-11T00:30:47.015698+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:30:47.015698+0000","last_undegraded":"2024-09-11T00:30:47.015698+0000","last_fullsized":"2024-09-11T00:30:47.015698+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.913839+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:30:46.913839+0000","last_peered":"2024-09-11T00:30:46.913839+0000","last_clean":"2024-09-11T00:30:46.913839+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:30:46.913839+0000","last_undegraded":"2024-09-11T00:30:46.913839+0000","last_fullsized":"2024-09-11T00:30:46.913839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916407+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:30:46.916407+0000","last_peered":"2024-09-11T00:30:46.916407+0000","last_clean":"2024-09-11T00:30:46.916407+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:30:46.916407+0000","last_undegraded":"2024-09-11T00:30:46.916407+0000","last_fullsized":"2024-09-11T00:30:46.916407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920941+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:30:46.920941+0000","last_peered":"2024-09-11T00:30:46.920941+0000","last_clean":"2024-09-11T00:30:46.920941+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:30:46.920941+0000","last_undegraded":"2024-09-11T00:30:46.920941+0000","last_fullsized":"2024-09-11T00:30:46.920941+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016502+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:30:47.016502+0000","last_peered":"2024-09-11T00:30:47.016502+0000","last_clean":"2024-09-11T00:30:47.016502+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:30:47.016502+0000","last_undegraded":"2024-09-11T00:30:47.016502+0000","last_fullsized":"2024-09-11T00:30:47.016502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915856+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:30:46.915856+0000","last_peered":"2024-09-11T00:30:46.915856+0000","last_clean":"2024-09-11T00:30:46.915856+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:30:46.915856+0000","last_undegraded":"2024-09-11T00:30:46.915856+0000","last_fullsized":"2024-09-11T00:30:46.915856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916034+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:30:46.916034+0000","last_peered":"2024-09-11T00:30:46.916034+0000","last_clean":"2024-09-11T00:30:46.916034+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:30:46.916034+0000","last_undegraded":"2024-09-11T00:30:46.916034+0000","last_fullsized":"2024-09-11T00:30:46.916034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016319+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:30:47.016319+0000","last_peered":"2024-09-11T00:30:47.016319+0000","last_clean":"2024-09-11T00:30:47.016319+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:30:47.016319+0000","last_undegraded":"2024-09-11T00:30:47.016319+0000","last_fullsized":"2024-09-11T00:30:47.016319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920321+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:30:46.920321+0000","last_peered":"2024-09-11T00:30:46.920321+0000","last_clean":"2024-09-11T00:30:46.920321+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:30:46.920321+0000","last_undegraded":"2024-09-11T00:30:46.920321+0000","last_fullsized":"2024-09-11T00:30:46.920321+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.914091+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:30:46.914091+0000","last_peered":"2024-09-11T00:30:46.914091+0000","last_clean":"2024-09-11T00:30:46.914091+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:30:46.914091+0000","last_undegraded":"2024-09-11T00:30:46.914091+0000","last_fullsized":"2024-09-11T00:30:46.914091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017157+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:30:47.017157+0000","last_peered":"2024-09-11T00:30:47.017157+0000","last_clean":"2024-09-11T00:30:47.017157+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:30:47.017157+0000","last_undegraded":"2024-09-11T00:30:47.017157+0000","last_fullsized":"2024-09-11T00:30:47.017157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923555+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:30:46.923555+0000","last_peered":"2024-09-11T00:30:46.923555+0000","last_clean":"2024-09-11T00:30:46.923555+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:30:46.923555+0000","last_undegraded":"2024-09-11T00:30:46.923555+0000","last_fullsized":"2024-09-11T00:30:46.923555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.918167+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:30:46.918167+0000","last_peered":"2024-09-11T00:30:46.918167+0000","last_clean":"2024-09-11T00:30:46.918167+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:30:46.918167+0000","last_undegraded":"2024-09-11T00:30:46.918167+0000","last_fullsized":"2024-09-11T00:30:46.918167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015954+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:30:47.015954+0000","last_peered":"2024-09-11T00:30:47.015954+0000","last_clean":"2024-09-11T00:30:47.015954+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:30:47.015954+0000","last_undegraded":"2024-09-11T00:30:47.015954+0000","last_fullsized":"2024-09-11T00:30:47.015954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016363+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:30:47.016363+0000","last_peered":"2024-09-11T00:30:47.016363+0000","last_clean":"2024-09-11T00:30:47.016363+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:30:47.016363+0000","last_undegraded":"2024-09-11T00:30:47.016363+0000","last_fullsized":"2024-09-11T00:30:47.016363+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016038+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:30:47.016038+0000","last_peered":"2024-09-11T00:30:47.016038+0000","last_clean":"2024-09-11T00:30:47.016038+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:30:47.016038+0000","last_undegraded":"2024-09-11T00:30:47.016038+0000","last_fullsized":"2024-09-11T00:30:47.016038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923687+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:30:46.923687+0000","last_peered":"2024-09-11T00:30:46.923687+0000","last_clean":"2024-09-11T00:30:46.923687+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:30:46.923687+0000","last_undegraded":"2024-09-11T00:30:46.923687+0000","last_fullsized":"2024-09-11T00:30:46.923687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015958+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:30:47.015958+0000","last_peered":"2024-09-11T00:30:47.015958+0000","last_clean":"2024-09-11T00:30:47.015958+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:30:47.015958+0000","last_undegraded":"2024-09-11T00:30:47.015958+0000","last_fullsized":"2024-09-11T00:30:47.015958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916220+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:30:46.916220+0000","last_peered":"2024-09-11T00:30:46.916220+0000","last_clean":"2024-09-11T00:30:46.916220+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:30:46.916220+0000","last_undegraded":"2024-09-11T00:30:46.916220+0000","last_fullsized":"2024-09-11T00:30:46.916220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016617+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:30:47.016617+0000","last_peered":"2024-09-11T00:30:47.016617+0000","last_clean":"2024-09-11T00:30:47.016617+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:30:47.016617+0000","last_undegraded":"2024-09-11T00:30:47.016617+0000","last_fullsized":"2024-09-11T00:30:47.016617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016343+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:30:47.016343+0000","last_peered":"2024-09-11T00:30:47.016343+0000","last_clean":"2024-09-11T00:30:47.016343+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:30:47.016343+0000","last_undegraded":"2024-09-11T00:30:47.016343+0000","last_fullsized":"2024-09-11T00:30:47.016343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.922910+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:30:46.922910+0000","last_peered":"2024-09-11T00:30:46.922910+0000","last_clean":"2024-09-11T00:30:46.922910+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:30:46.922910+0000","last_undegraded":"2024-09-11T00:30:46.922910+0000","last_fullsized":"2024-09-11T00:30:46.922910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.913562+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:30:46.913562+0000","last_peered":"2024-09-11T00:30:46.913562+0000","last_clean":"2024-09-11T00:30:46.913562+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:30:46.913562+0000","last_undegraded":"2024-09-11T00:30:46.913562+0000","last_fullsized":"2024-09-11T00:30:46.913562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920602+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:30:46.920602+0000","last_peered":"2024-09-11T00:30:46.920602+0000","last_clean":"2024-09-11T00:30:46.920602+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:30:46.920602+0000","last_undegraded":"2024-09-11T00:30:46.920602+0000","last_fullsized":"2024-09-11T00:30:46.920602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016176+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:30:47.016176+0000","last_peered":"2024-09-11T00:30:47.016176+0000","last_clean":"2024-09-11T00:30:47.016176+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:30:47.016176+0000","last_undegraded":"2024-09-11T00:30:47.016176+0000","last_fullsized":"2024-09-11T00:30:47.016176+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923499+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:30:46.923499+0000","last_peered":"2024-09-11T00:30:46.923499+0000","last_clean":"2024-09-11T00:30:46.923499+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:30:46.923499+0000","last_undegraded":"2024-09-11T00:30:46.923499+0000","last_fullsized":"2024-09-11T00:30:46.923499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017319+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:30:47.017319+0000","last_peered":"2024-09-11T00:30:47.017319+0000","last_clean":"2024-09-11T00:30:47.017319+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:30:47.017319+0000","last_undegraded":"2024-09-11T00:30:47.017319+0000","last_fullsized":"2024-09-11T00:30:47.017319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920477+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:30:46.920477+0000","last_peered":"2024-09-11T00:30:46.920477+0000","last_clean":"2024-09-11T00:30:46.920477+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:30:46.920477+0000","last_undegraded":"2024-09-11T00:30:46.920477+0000","last_fullsized":"2024-09-11T00:30:46.920477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923580+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:30:46.923580+0000","last_peered":"2024-09-11T00:30:46.923580+0000","last_clean":"2024-09-11T00:30:46.923580+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:30:46.923580+0000","last_undegraded":"2024-09-11T00:30:46.923580+0000","last_fullsized":"2024-09-11T00:30:46.923580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916613+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:30:46.916613+0000","last_peered":"2024-09-11T00:30:46.916613+0000","last_clean":"2024-09-11T00:30:46.916613+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:30:46.916613+0000","last_undegraded":"2024-09-11T00:30:46.916613+0000","last_fullsized":"2024-09-11T00:30:46.916613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923501+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:30:46.923501+0000","last_peered":"2024-09-11T00:30:46.923501+0000","last_clean":"2024-09-11T00:30:46.923501+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:30:46.923501+0000","last_undegraded":"2024-09-11T00:30:46.923501+0000","last_fullsized":"2024-09-11T00:30:46.923501+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917696+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:30:46.917696+0000","last_peered":"2024-09-11T00:30:46.917696+0000","last_clean":"2024-09-11T00:30:46.917696+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:30:46.917696+0000","last_undegraded":"2024-09-11T00:30:46.917696+0000","last_fullsized":"2024-09-11T00:30:46.917696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016883+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:30:47.016883+0000","last_peered":"2024-09-11T00:30:47.016883+0000","last_clean":"2024-09-11T00:30:47.016883+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:30:47.016883+0000","last_undegraded":"2024-09-11T00:30:47.016883+0000","last_fullsized":"2024-09-11T00:30:47.016883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.913436+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:30:46.913436+0000","last_peered":"2024-09-11T00:30:46.913436+0000","last_clean":"2024-09-11T00:30:46.913436+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:30:46.913436+0000","last_undegraded":"2024-09-11T00:30:46.913436+0000","last_fullsized":"2024-09-11T00:30:46.913436+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016451+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:30:47.016451+0000","last_peered":"2024-09-11T00:30:47.016451+0000","last_clean":"2024-09-11T00:30:47.016451+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:30:47.016451+0000","last_undegraded":"2024-09-11T00:30:47.016451+0000","last_fullsized":"2024-09-11T00:30:47.016451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920418+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:30:46.920418+0000","last_peered":"2024-09-11T00:30:46.920418+0000","last_clean":"2024-09-11T00:30:46.920418+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:30:46.920418+0000","last_undegraded":"2024-09-11T00:30:46.920418+0000","last_fullsized":"2024-09-11T00:30:46.920418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016414+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:30:47.016414+0000","last_peered":"2024-09-11T00:30:47.016414+0000","last_clean":"2024-09-11T00:30:47.016414+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:30:47.016414+0000","last_undegraded":"2024-09-11T00:30:47.016414+0000","last_fullsized":"2024-09-11T00:30:47.016414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915766+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:30:46.915766+0000","last_peered":"2024-09-11T00:30:46.915766+0000","last_clean":"2024-09-11T00:30:46.915766+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:30:46.915766+0000","last_undegraded":"2024-09-11T00:30:46.915766+0000","last_fullsized":"2024-09-11T00:30:46.915766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015721+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:30:47.015721+0000","last_peered":"2024-09-11T00:30:47.015721+0000","last_clean":"2024-09-11T00:30:47.015721+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:30:47.015721+0000","last_undegraded":"2024-09-11T00:30:47.015721+0000","last_fullsized":"2024-09-11T00:30:47.015721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916079+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:30:46.916079+0000","last_peered":"2024-09-11T00:30:46.916079+0000","last_clean":"2024-09-11T00:30:46.916079+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:30:46.916079+0000","last_undegraded":"2024-09-11T00:30:46.916079+0000","last_fullsized":"2024-09-11T00:30:46.916079+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016719+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:30:47.016719+0000","last_peered":"2024-09-11T00:30:47.016719+0000","last_clean":"2024-09-11T00:30:47.016719+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:30:47.016719+0000","last_undegraded":"2024-09-11T00:30:47.016719+0000","last_fullsized":"2024-09-11T00:30:47.016719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923278+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:30:46.923278+0000","last_peered":"2024-09-11T00:30:46.923278+0000","last_clean":"2024-09-11T00:30:46.923278+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:30:46.923278+0000","last_undegraded":"2024-09-11T00:30:46.923278+0000","last_fullsized":"2024-09-11T00:30:46.923278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916500+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:30:46.916500+0000","last_peered":"2024-09-11T00:30:46.916500+0000","last_clean":"2024-09-11T00:30:46.916500+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:30:46.916500+0000","last_undegraded":"2024-09-11T00:30:46.916500+0000","last_fullsized":"2024-09-11T00:30:46.916500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016431+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:30:47.016431+0000","last_peered":"2024-09-11T00:30:47.016431+0000","last_clean":"2024-09-11T00:30:47.016431+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:30:47.016431+0000","last_undegraded":"2024-09-11T00:30:47.016431+0000","last_fullsized":"2024-09-11T00:30:47.016431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017138+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:30:47.017138+0000","last_peered":"2024-09-11T00:30:47.017138+0000","last_clean":"2024-09-11T00:30:47.017138+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:30:47.017138+0000","last_undegraded":"2024-09-11T00:30:47.017138+0000","last_fullsized":"2024-09-11T00:30:47.017138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917635+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:30:46.917635+0000","last_peered":"2024-09-11T00:30:46.917635+0000","last_clean":"2024-09-11T00:30:46.917635+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:30:46.917635+0000","last_undegraded":"2024-09-11T00:30:46.917635+0000","last_fullsized":"2024-09-11T00:30:46.917635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.922338+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:30:46.922338+0000","last_peered":"2024-09-11T00:30:46.922338+0000","last_clean":"2024-09-11T00:30:46.922338+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:30:46.922338+0000","last_undegraded":"2024-09-11T00:30:46.922338+0000","last_fullsized":"2024-09-11T00:30:46.922338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920774+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:30:46.920774+0000","last_peered":"2024-09-11T00:30:46.920774+0000","last_clean":"2024-09-11T00:30:46.920774+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:30:46.920774+0000","last_undegraded":"2024-09-11T00:30:46.920774+0000","last_fullsized":"2024-09-11T00:30:46.920774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017260+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:30:47.017260+0000","last_peered":"2024-09-11T00:30:47.017260+0000","last_clean":"2024-09-11T00:30:47.017260+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:30:47.017260+0000","last_undegraded":"2024-09-11T00:30:47.017260+0000","last_fullsized":"2024-09-11T00:30:47.017260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920716+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:30:46.920716+0000","last_peered":"2024-09-11T00:30:46.920716+0000","last_clean":"2024-09-11T00:30:46.920716+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:30:46.920716+0000","last_undegraded":"2024-09-11T00:30:46.920716+0000","last_fullsized":"2024-09-11T00:30:46.920716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.922727+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:30:46.922727+0000","last_peered":"2024-09-11T00:30:46.922727+0000","last_clean":"2024-09-11T00:30:46.922727+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:30:46.922727+0000","last_undegraded":"2024-09-11T00:30:46.922727+0000","last_fullsized":"2024-09-11T00:30:46.922727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016872+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:30:47.016872+0000","last_peered":"2024-09-11T00:30:47.016872+0000","last_clean":"2024-09-11T00:30:47.016872+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:30:47.016872+0000","last_undegraded":"2024-09-11T00:30:47.016872+0000","last_fullsized":"2024-09-11T00:30:47.016872+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920443+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:30:46.920443+0000","last_peered":"2024-09-11T00:30:46.920443+0000","last_clean":"2024-09-11T00:30:46.920443+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:30:46.920443+0000","last_undegraded":"2024-09-11T00:30:46.920443+0000","last_fullsized":"2024-09-11T00:30:46.920443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920200+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:30:46.920200+0000","last_peered":"2024-09-11T00:30:46.920200+0000","last_clean":"2024-09-11T00:30:46.920200+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:30:46.920200+0000","last_undegraded":"2024-09-11T00:30:46.920200+0000","last_fullsized":"2024-09-11T00:30:46.920200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916277+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:30:46.916277+0000","last_peered":"2024-09-11T00:30:46.916277+0000","last_clean":"2024-09-11T00:30:46.916277+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:30:46.916277+0000","last_undegraded":"2024-09-11T00:30:46.916277+0000","last_fullsized":"2024-09-11T00:30:46.916277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016150+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:30:47.016150+0000","last_peered":"2024-09-11T00:30:47.016150+0000","last_clean":"2024-09-11T00:30:47.016150+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:30:47.016150+0000","last_undegraded":"2024-09-11T00:30:47.016150+0000","last_fullsized":"2024-09-11T00:30:47.016150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016385+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:30:47.016385+0000","last_peered":"2024-09-11T00:30:47.016385+0000","last_clean":"2024-09-11T00:30:47.016385+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:30:47.016385+0000","last_undegraded":"2024-09-11T00:30:47.016385+0000","last_fullsized":"2024-09-11T00:30:47.016385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016435+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:30:47.016435+0000","last_peered":"2024-09-11T00:30:47.016435+0000","last_clean":"2024-09-11T00:30:47.016435+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:30:47.016435+0000","last_undegraded":"2024-09-11T00:30:47.016435+0000","last_fullsized":"2024-09-11T00:30:47.016435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016678+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:30:47.016678+0000","last_peered":"2024-09-11T00:30:47.016678+0000","last_clean":"2024-09-11T00:30:47.016678+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:30:47.016678+0000","last_undegraded":"2024-09-11T00:30:47.016678+0000","last_fullsized":"2024-09-11T00:30:47.016678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923381+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:30:46.923381+0000","last_peered":"2024-09-11T00:30:46.923381+0000","last_clean":"2024-09-11T00:30:46.923381+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:30:46.923381+0000","last_undegraded":"2024-09-11T00:30:46.923381+0000","last_fullsized":"2024-09-11T00:30:46.923381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.913652+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:30:46.913652+0000","last_peered":"2024-09-11T00:30:46.913652+0000","last_clean":"2024-09-11T00:30:46.913652+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:30:46.913652+0000","last_undegraded":"2024-09-11T00:30:46.913652+0000","last_fullsized":"2024-09-11T00:30:46.913652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923730+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:30:46.923730+0000","last_peered":"2024-09-11T00:30:46.923730+0000","last_clean":"2024-09-11T00:30:46.923730+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:30:46.923730+0000","last_undegraded":"2024-09-11T00:30:46.923730+0000","last_fullsized":"2024-09-11T00:30:46.923730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917529+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:30:46.917529+0000","last_peered":"2024-09-11T00:30:46.917529+0000","last_clean":"2024-09-11T00:30:46.917529+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:30:46.917529+0000","last_undegraded":"2024-09-11T00:30:46.917529+0000","last_fullsized":"2024-09-11T00:30:46.917529+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917135+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:30:46.917135+0000","last_peered":"2024-09-11T00:30:46.917135+0000","last_clean":"2024-09-11T00:30:46.917135+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:30:46.917135+0000","last_undegraded":"2024-09-11T00:30:46.917135+0000","last_fullsized":"2024-09-11T00:30:46.917135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916981+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:30:46.916981+0000","last_peered":"2024-09-11T00:30:46.916981+0000","last_clean":"2024-09-11T00:30:46.916981+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:30:46.916981+0000","last_undegraded":"2024-09-11T00:30:46.916981+0000","last_fullsized":"2024-09-11T00:30:46.916981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920899+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:30:46.920899+0000","last_peered":"2024-09-11T00:30:46.920899+0000","last_clean":"2024-09-11T00:30:46.920899+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:30:46.920899+0000","last_undegraded":"2024-09-11T00:30:46.920899+0000","last_fullsized":"2024-09-11T00:30:46.920899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":76328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":76204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":823296,"data_stored":666664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":77816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":819200,"data_stored":666642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":76260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":78488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:30:51.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:30:51.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.363+0000 7f375e200640 1 -- 172.21.15.31:0/1875862234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37580c23a0 msgr2=0x7f37581516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:51.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.363+0000 7f375e200640 1 --2- 172.21.15.31:0/1875862234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37580c23a0 0x7f37581516e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3740005fd0 tx=0x7f374002ab20 comp rx=0 tx=0).stop 2024-09-11T00:30:51.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.363+0000 7f375e200640 1 -- 172.21.15.31:0/1875862234 shutdown_connections 2024-09-11T00:30:51.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.363+0000 7f375e200640 1 --2- 172.21.15.31:0/1875862234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37580c23a0 0x7f37581516e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.363+0000 7f375e200640 1 --2- 172.21.15.31:0/1875862234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3758072480 0x7f37581511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.363+0000 7f375e200640 1 -- 172.21.15.31:0/1875862234 >> 172.21.15.31:0/1875862234 conn(0x7f37580a9750 msgr2=0x7f37580a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:51.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.363+0000 7f375e200640 1 -- 172.21.15.31:0/1875862234 shutdown_connections 2024-09-11T00:30:51.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.363+0000 7f375e200640 1 -- 172.21.15.31:0/1875862234 wait complete. 2024-09-11T00:30:51.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.365+0000 7f375e200640 1 Processor -- start 2024-09-11T00:30:51.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.365+0000 7f375e200640 1 -- start start 2024-09-11T00:30:51.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f375e200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3758072480 0x7f37580c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:51.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f375e200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37580c23a0 0x7f37581ea730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:51.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f375e200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3758153f20 0x7f37580c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:51.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f375e200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3758074200 con 0x7f3758153f20 2024-09-11T00:30:51.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f375e200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3758074080 con 0x7f37580c23a0 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f375e200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3758072860 con 0x7f3758072480 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f3757e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3758153f20 0x7f37580c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f3757e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3758153f20 0x7f37580c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56558/0 (socket says 172.21.15.31:56558) 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f3757e00640 1 -- 172.21.15.31:0/866308371 learned_addr learned my addr 172.21.15.31:0/866308371 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f374ea00640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37580c23a0 0x7f37581ea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f3757400640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3758072480 0x7f37580c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f3757e00640 1 -- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3758072480 msgr2=0x7f37580c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f3757e00640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3758072480 0x7f37580c30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f3757e00640 1 -- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37580c23a0 msgr2=0x7f37581ea730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f3757e00640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37580c23a0 0x7f37581ea730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f3757e00640 1 -- 172.21.15.31:0/866308371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37580c3600 con 0x7f3758153f20 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.366+0000 7f3757400640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3758072480 0x7f37580c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:30:51.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.367+0000 7f3757e00640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3758153f20 0x7f37580c4530 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f375000c3c0 tx=0x7f375000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:51.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.367+0000 7f374fe00640 1 -- 172.21.15.31:0/866308371 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3750017070 con 0x7f3758153f20 2024-09-11T00:30:51.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.367+0000 7f374fe00640 1 -- 172.21.15.31:0/866308371 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f37500053c0 con 0x7f3758153f20 2024-09-11T00:30:51.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.367+0000 7f374fe00640 1 -- 172.21.15.31:0/866308371 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3750003850 con 0x7f3758153f20 2024-09-11T00:30:51.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.367+0000 7f375e200640 1 -- 172.21.15.31:0/866308371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37581dedf0 con 0x7f3758153f20 2024-09-11T00:30:51.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.367+0000 7f375e200640 1 -- 172.21.15.31:0/866308371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37581df370 con 0x7f3758153f20 2024-09-11T00:30:51.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.368+0000 7f375e200640 1 -- 172.21.15.31:0/866308371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37581511a0 con 0x7f3758153f20 2024-09-11T00:30:51.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.369+0000 7f374fe00640 1 -- 172.21.15.31:0/866308371 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f37500039f0 con 0x7f3758153f20 2024-09-11T00:30:51.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.370+0000 7f374fe00640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3734077f70 0x7f373407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:51.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.370+0000 7f374fe00640 1 -- 172.21.15.31:0/866308371 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f3750099370 con 0x7f3758153f20 2024-09-11T00:30:51.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.371+0000 7f3757400640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3734077f70 0x7f373407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:51.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.371+0000 7f3757400640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3734077f70 0x7f373407a430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3748005fd0 tx=0x7f3748005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:51.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.374+0000 7f374fe00640 1 -- 172.21.15.31:0/866308371 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f37500618e0 con 0x7f3758153f20 2024-09-11T00:30:51.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.564+0000 7f375e200640 1 -- 172.21.15.31:0/866308371 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f375814fa30 con 0x7f3734077f70 2024-09-11T00:30:51.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.571+0000 7f374fe00640 1 -- 172.21.15.31:0/866308371 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251933 (secure 0 0 0) 0x7f375814fa30 con 0x7f3734077f70 2024-09-11T00:30:51.575 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:51.576 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:30:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.574+0000 7f375e200640 1 -- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3734077f70 msgr2=0x7f373407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.574+0000 7f375e200640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3734077f70 0x7f373407a430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3748005fd0 tx=0x7f3748005850 comp rx=0 tx=0).stop 2024-09-11T00:30:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.575+0000 7f375e200640 1 -- 172.21.15.31:0/866308371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3758153f20 msgr2=0x7f37580c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.575+0000 7f375e200640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3758153f20 0x7f37580c4530 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f375000c3c0 tx=0x7f375000c880 comp rx=0 tx=0).stop 2024-09-11T00:30:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.575+0000 7f375e200640 1 -- 172.21.15.31:0/866308371 shutdown_connections 2024-09-11T00:30:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.575+0000 7f375e200640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3734077f70 0x7f373407a430 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.575+0000 7f375e200640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3758153f20 0x7f37580c4530 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.575+0000 7f375e200640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37580c23a0 0x7f37581ea730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.575+0000 7f375e200640 1 --2- 172.21.15.31:0/866308371 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3758072480 0x7f37580c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.575+0000 7f375e200640 1 -- 172.21.15.31:0/866308371 >> 172.21.15.31:0/866308371 conn(0x7f37580a9750 msgr2=0x7f37581531e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:51.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.575+0000 7f375e200640 1 -- 172.21.15.31:0/866308371 shutdown_connections 2024-09-11T00:30:51.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.575+0000 7f375e200640 1 -- 172.21.15.31:0/866308371 wait complete. 2024-09-11T00:30:51.592 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-11T00:30:50.392614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219192,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755528,"statfs":{"total":773094113280,"available":772869660672,"internally_reserved":0,"allocated":3022848,"data_stored":1795188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001495"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923421+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:30:46.923421+0000","last_peered":"2024-09-11T00:30:46.923421+0000","last_clean":"2024-09-11T00:30:46.923421+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:30:46.923421+0000","last_undegraded":"2024-09-11T00:30:46.923421+0000","last_fullsized":"2024-09-11T00:30:46.923421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916735+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:30:46.916735+0000","last_peered":"2024-09-11T00:30:46.916735+0000","last_clean":"2024-09-11T00:30:46.916735+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:30:46.916735+0000","last_undegraded":"2024-09-11T00:30:46.916735+0000","last_fullsized":"2024-09-11T00:30:46.916735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916834+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:30:46.916834+0000","last_peered":"2024-09-11T00:30:46.916834+0000","last_clean":"2024-09-11T00:30:46.916834+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:30:46.916834+0000","last_undegraded":"2024-09-11T00:30:46.916834+0000","last_fullsized":"2024-09-11T00:30:46.916834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016119+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:30:47.016119+0000","last_peered":"2024-09-11T00:30:47.016119+0000","last_clean":"2024-09-11T00:30:47.016119+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:30:47.016119+0000","last_undegraded":"2024-09-11T00:30:47.016119+0000","last_fullsized":"2024-09-11T00:30:47.016119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923185+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:30:46.923185+0000","last_peered":"2024-09-11T00:30:46.923185+0000","last_clean":"2024-09-11T00:30:46.923185+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:30:46.923185+0000","last_undegraded":"2024-09-11T00:30:46.923185+0000","last_fullsized":"2024-09-11T00:30:46.923185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015860+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:30:47.015860+0000","last_peered":"2024-09-11T00:30:47.015860+0000","last_clean":"2024-09-11T00:30:47.015860+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:30:47.015860+0000","last_undegraded":"2024-09-11T00:30:47.015860+0000","last_fullsized":"2024-09-11T00:30:47.015860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923059+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:30:46.923059+0000","last_peered":"2024-09-11T00:30:46.923059+0000","last_clean":"2024-09-11T00:30:46.923059+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:30:46.923059+0000","last_undegraded":"2024-09-11T00:30:46.923059+0000","last_fullsized":"2024-09-11T00:30:46.923059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915628+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:30:46.915628+0000","last_peered":"2024-09-11T00:30:46.915628+0000","last_clean":"2024-09-11T00:30:46.915628+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:30:46.915628+0000","last_undegraded":"2024-09-11T00:30:46.915628+0000","last_fullsized":"2024-09-11T00:30:46.915628+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.914177+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:30:46.914177+0000","last_peered":"2024-09-11T00:30:46.914177+0000","last_clean":"2024-09-11T00:30:46.914177+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:30:46.914177+0000","last_undegraded":"2024-09-11T00:30:46.914177+0000","last_fullsized":"2024-09-11T00:30:46.914177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923489+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:30:46.923489+0000","last_peered":"2024-09-11T00:30:46.923489+0000","last_clean":"2024-09-11T00:30:46.923489+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:46.923489+0000","last_undegraded":"2024-09-11T00:30:46.923489+0000","last_fullsized":"2024-09-11T00:30:46.923489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923575+0000","last_change":"2024-09-11T00:30:45.941635+0000","last_active":"2024-09-11T00:30:46.923575+0000","last_peered":"2024-09-11T00:30:46.923575+0000","last_clean":"2024-09-11T00:30:46.923575+0000","last_became_active":"2024-09-11T00:30:45.941357+0000","last_became_peered":"2024-09-11T00:30:45.941357+0000","last_unstale":"2024-09-11T00:30:46.923575+0000","last_undegraded":"2024-09-11T00:30:46.923575+0000","last_fullsized":"2024-09-11T00:30:46.923575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:12.016406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923293+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:30:46.923293+0000","last_peered":"2024-09-11T00:30:46.923293+0000","last_clean":"2024-09-11T00:30:46.923293+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:30:46.923293+0000","last_undegraded":"2024-09-11T00:30:46.923293+0000","last_fullsized":"2024-09-11T00:30:46.923293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917143+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:30:46.917143+0000","last_peered":"2024-09-11T00:30:46.917143+0000","last_clean":"2024-09-11T00:30:46.917143+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:30:46.917143+0000","last_undegraded":"2024-09-11T00:30:46.917143+0000","last_fullsized":"2024-09-11T00:30:46.917143+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915945+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:30:46.915945+0000","last_peered":"2024-09-11T00:30:46.915945+0000","last_clean":"2024-09-11T00:30:46.915945+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:30:46.915945+0000","last_undegraded":"2024-09-11T00:30:46.915945+0000","last_fullsized":"2024-09-11T00:30:46.915945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915659+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:30:46.915659+0000","last_peered":"2024-09-11T00:30:46.915659+0000","last_clean":"2024-09-11T00:30:46.915659+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:30:46.915659+0000","last_undegraded":"2024-09-11T00:30:46.915659+0000","last_fullsized":"2024-09-11T00:30:46.915659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016126+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:30:47.016126+0000","last_peered":"2024-09-11T00:30:47.016126+0000","last_clean":"2024-09-11T00:30:47.016126+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:30:47.016126+0000","last_undegraded":"2024-09-11T00:30:47.016126+0000","last_fullsized":"2024-09-11T00:30:47.016126+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017199+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:30:47.017199+0000","last_peered":"2024-09-11T00:30:47.017199+0000","last_clean":"2024-09-11T00:30:47.017199+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:30:47.017199+0000","last_undegraded":"2024-09-11T00:30:47.017199+0000","last_fullsized":"2024-09-11T00:30:47.017199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016204+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:30:47.016204+0000","last_peered":"2024-09-11T00:30:47.016204+0000","last_clean":"2024-09-11T00:30:47.016204+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:30:47.016204+0000","last_undegraded":"2024-09-11T00:30:47.016204+0000","last_fullsized":"2024-09-11T00:30:47.016204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017083+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:30:47.017083+0000","last_peered":"2024-09-11T00:30:47.017083+0000","last_clean":"2024-09-11T00:30:47.017083+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:30:47.017083+0000","last_undegraded":"2024-09-11T00:30:47.017083+0000","last_fullsized":"2024-09-11T00:30:47.017083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923319+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:30:46.923319+0000","last_peered":"2024-09-11T00:30:46.923319+0000","last_clean":"2024-09-11T00:30:46.923319+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:30:46.923319+0000","last_undegraded":"2024-09-11T00:30:46.923319+0000","last_fullsized":"2024-09-11T00:30:46.923319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920455+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:30:46.920455+0000","last_peered":"2024-09-11T00:30:46.920455+0000","last_clean":"2024-09-11T00:30:46.920455+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:46.920455+0000","last_undegraded":"2024-09-11T00:30:46.920455+0000","last_fullsized":"2024-09-11T00:30:46.920455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.913624+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:30:46.913624+0000","last_peered":"2024-09-11T00:30:46.913624+0000","last_clean":"2024-09-11T00:30:46.913624+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:30:46.913624+0000","last_undegraded":"2024-09-11T00:30:46.913624+0000","last_fullsized":"2024-09-11T00:30:46.913624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915955+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:30:46.915955+0000","last_peered":"2024-09-11T00:30:46.915955+0000","last_clean":"2024-09-11T00:30:46.915955+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:30:46.915955+0000","last_undegraded":"2024-09-11T00:30:46.915955+0000","last_fullsized":"2024-09-11T00:30:46.915955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016366+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:30:47.016366+0000","last_peered":"2024-09-11T00:30:47.016366+0000","last_clean":"2024-09-11T00:30:47.016366+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:30:47.016366+0000","last_undegraded":"2024-09-11T00:30:47.016366+0000","last_fullsized":"2024-09-11T00:30:47.016366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917531+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:30:46.917531+0000","last_peered":"2024-09-11T00:30:46.917531+0000","last_clean":"2024-09-11T00:30:46.917531+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:30:46.917531+0000","last_undegraded":"2024-09-11T00:30:46.917531+0000","last_fullsized":"2024-09-11T00:30:46.917531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017119+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:30:47.017119+0000","last_peered":"2024-09-11T00:30:47.017119+0000","last_clean":"2024-09-11T00:30:47.017119+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:30:47.017119+0000","last_undegraded":"2024-09-11T00:30:47.017119+0000","last_fullsized":"2024-09-11T00:30:47.017119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917612+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:30:46.917612+0000","last_peered":"2024-09-11T00:30:46.917612+0000","last_clean":"2024-09-11T00:30:46.917612+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:30:46.917612+0000","last_undegraded":"2024-09-11T00:30:46.917612+0000","last_fullsized":"2024-09-11T00:30:46.917612+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917421+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:30:46.917421+0000","last_peered":"2024-09-11T00:30:46.917421+0000","last_clean":"2024-09-11T00:30:46.917421+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:30:46.917421+0000","last_undegraded":"2024-09-11T00:30:46.917421+0000","last_fullsized":"2024-09-11T00:30:46.917421+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916541+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:30:46.916541+0000","last_peered":"2024-09-11T00:30:46.916541+0000","last_clean":"2024-09-11T00:30:46.916541+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:30:46.916541+0000","last_undegraded":"2024-09-11T00:30:46.916541+0000","last_fullsized":"2024-09-11T00:30:46.916541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016666+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:30:47.016666+0000","last_peered":"2024-09-11T00:30:47.016666+0000","last_clean":"2024-09-11T00:30:47.016666+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:30:47.016666+0000","last_undegraded":"2024-09-11T00:30:47.016666+0000","last_fullsized":"2024-09-11T00:30:47.016666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015898+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:30:47.015898+0000","last_peered":"2024-09-11T00:30:47.015898+0000","last_clean":"2024-09-11T00:30:47.015898+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:30:47.015898+0000","last_undegraded":"2024-09-11T00:30:47.015898+0000","last_fullsized":"2024-09-11T00:30:47.015898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015698+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:30:47.015698+0000","last_peered":"2024-09-11T00:30:47.015698+0000","last_clean":"2024-09-11T00:30:47.015698+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:30:47.015698+0000","last_undegraded":"2024-09-11T00:30:47.015698+0000","last_fullsized":"2024-09-11T00:30:47.015698+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.913839+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:30:46.913839+0000","last_peered":"2024-09-11T00:30:46.913839+0000","last_clean":"2024-09-11T00:30:46.913839+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:30:46.913839+0000","last_undegraded":"2024-09-11T00:30:46.913839+0000","last_fullsized":"2024-09-11T00:30:46.913839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916407+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:30:46.916407+0000","last_peered":"2024-09-11T00:30:46.916407+0000","last_clean":"2024-09-11T00:30:46.916407+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:30:46.916407+0000","last_undegraded":"2024-09-11T00:30:46.916407+0000","last_fullsized":"2024-09-11T00:30:46.916407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920941+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:30:46.920941+0000","last_peered":"2024-09-11T00:30:46.920941+0000","last_clean":"2024-09-11T00:30:46.920941+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:30:46.920941+0000","last_undegraded":"2024-09-11T00:30:46.920941+0000","last_fullsized":"2024-09-11T00:30:46.920941+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016502+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:30:47.016502+0000","last_peered":"2024-09-11T00:30:47.016502+0000","last_clean":"2024-09-11T00:30:47.016502+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:30:47.016502+0000","last_undegraded":"2024-09-11T00:30:47.016502+0000","last_fullsized":"2024-09-11T00:30:47.016502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915856+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:30:46.915856+0000","last_peered":"2024-09-11T00:30:46.915856+0000","last_clean":"2024-09-11T00:30:46.915856+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:30:46.915856+0000","last_undegraded":"2024-09-11T00:30:46.915856+0000","last_fullsized":"2024-09-11T00:30:46.915856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916034+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:30:46.916034+0000","last_peered":"2024-09-11T00:30:46.916034+0000","last_clean":"2024-09-11T00:30:46.916034+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:30:46.916034+0000","last_undegraded":"2024-09-11T00:30:46.916034+0000","last_fullsized":"2024-09-11T00:30:46.916034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016319+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:30:47.016319+0000","last_peered":"2024-09-11T00:30:47.016319+0000","last_clean":"2024-09-11T00:30:47.016319+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:30:47.016319+0000","last_undegraded":"2024-09-11T00:30:47.016319+0000","last_fullsized":"2024-09-11T00:30:47.016319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920321+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:30:46.920321+0000","last_peered":"2024-09-11T00:30:46.920321+0000","last_clean":"2024-09-11T00:30:46.920321+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:30:46.920321+0000","last_undegraded":"2024-09-11T00:30:46.920321+0000","last_fullsized":"2024-09-11T00:30:46.920321+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.914091+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:30:46.914091+0000","last_peered":"2024-09-11T00:30:46.914091+0000","last_clean":"2024-09-11T00:30:46.914091+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:30:46.914091+0000","last_undegraded":"2024-09-11T00:30:46.914091+0000","last_fullsized":"2024-09-11T00:30:46.914091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017157+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:30:47.017157+0000","last_peered":"2024-09-11T00:30:47.017157+0000","last_clean":"2024-09-11T00:30:47.017157+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:30:47.017157+0000","last_undegraded":"2024-09-11T00:30:47.017157+0000","last_fullsized":"2024-09-11T00:30:47.017157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923555+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:30:46.923555+0000","last_peered":"2024-09-11T00:30:46.923555+0000","last_clean":"2024-09-11T00:30:46.923555+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:30:46.923555+0000","last_undegraded":"2024-09-11T00:30:46.923555+0000","last_fullsized":"2024-09-11T00:30:46.923555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.918167+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:30:46.918167+0000","last_peered":"2024-09-11T00:30:46.918167+0000","last_clean":"2024-09-11T00:30:46.918167+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:30:46.918167+0000","last_undegraded":"2024-09-11T00:30:46.918167+0000","last_fullsized":"2024-09-11T00:30:46.918167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015954+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:30:47.015954+0000","last_peered":"2024-09-11T00:30:47.015954+0000","last_clean":"2024-09-11T00:30:47.015954+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:30:47.015954+0000","last_undegraded":"2024-09-11T00:30:47.015954+0000","last_fullsized":"2024-09-11T00:30:47.015954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016363+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:30:47.016363+0000","last_peered":"2024-09-11T00:30:47.016363+0000","last_clean":"2024-09-11T00:30:47.016363+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:30:47.016363+0000","last_undegraded":"2024-09-11T00:30:47.016363+0000","last_fullsized":"2024-09-11T00:30:47.016363+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016038+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:30:47.016038+0000","last_peered":"2024-09-11T00:30:47.016038+0000","last_clean":"2024-09-11T00:30:47.016038+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:30:47.016038+0000","last_undegraded":"2024-09-11T00:30:47.016038+0000","last_fullsized":"2024-09-11T00:30:47.016038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923687+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:30:46.923687+0000","last_peered":"2024-09-11T00:30:46.923687+0000","last_clean":"2024-09-11T00:30:46.923687+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:30:46.923687+0000","last_undegraded":"2024-09-11T00:30:46.923687+0000","last_fullsized":"2024-09-11T00:30:46.923687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015958+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:30:47.015958+0000","last_peered":"2024-09-11T00:30:47.015958+0000","last_clean":"2024-09-11T00:30:47.015958+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:30:47.015958+0000","last_undegraded":"2024-09-11T00:30:47.015958+0000","last_fullsized":"2024-09-11T00:30:47.015958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916220+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:30:46.916220+0000","last_peered":"2024-09-11T00:30:46.916220+0000","last_clean":"2024-09-11T00:30:46.916220+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:30:46.916220+0000","last_undegraded":"2024-09-11T00:30:46.916220+0000","last_fullsized":"2024-09-11T00:30:46.916220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016617+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:30:47.016617+0000","last_peered":"2024-09-11T00:30:47.016617+0000","last_clean":"2024-09-11T00:30:47.016617+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:30:47.016617+0000","last_undegraded":"2024-09-11T00:30:47.016617+0000","last_fullsized":"2024-09-11T00:30:47.016617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016343+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:30:47.016343+0000","last_peered":"2024-09-11T00:30:47.016343+0000","last_clean":"2024-09-11T00:30:47.016343+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:30:47.016343+0000","last_undegraded":"2024-09-11T00:30:47.016343+0000","last_fullsized":"2024-09-11T00:30:47.016343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.922910+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:30:46.922910+0000","last_peered":"2024-09-11T00:30:46.922910+0000","last_clean":"2024-09-11T00:30:46.922910+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:30:46.922910+0000","last_undegraded":"2024-09-11T00:30:46.922910+0000","last_fullsized":"2024-09-11T00:30:46.922910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.913562+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:30:46.913562+0000","last_peered":"2024-09-11T00:30:46.913562+0000","last_clean":"2024-09-11T00:30:46.913562+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:30:46.913562+0000","last_undegraded":"2024-09-11T00:30:46.913562+0000","last_fullsized":"2024-09-11T00:30:46.913562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920602+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:30:46.920602+0000","last_peered":"2024-09-11T00:30:46.920602+0000","last_clean":"2024-09-11T00:30:46.920602+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:30:46.920602+0000","last_undegraded":"2024-09-11T00:30:46.920602+0000","last_fullsized":"2024-09-11T00:30:46.920602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016176+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:30:47.016176+0000","last_peered":"2024-09-11T00:30:47.016176+0000","last_clean":"2024-09-11T00:30:47.016176+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:30:47.016176+0000","last_undegraded":"2024-09-11T00:30:47.016176+0000","last_fullsized":"2024-09-11T00:30:47.016176+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923499+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:30:46.923499+0000","last_peered":"2024-09-11T00:30:46.923499+0000","last_clean":"2024-09-11T00:30:46.923499+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:30:46.923499+0000","last_undegraded":"2024-09-11T00:30:46.923499+0000","last_fullsized":"2024-09-11T00:30:46.923499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017319+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:30:47.017319+0000","last_peered":"2024-09-11T00:30:47.017319+0000","last_clean":"2024-09-11T00:30:47.017319+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:30:47.017319+0000","last_undegraded":"2024-09-11T00:30:47.017319+0000","last_fullsized":"2024-09-11T00:30:47.017319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920477+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:30:46.920477+0000","last_peered":"2024-09-11T00:30:46.920477+0000","last_clean":"2024-09-11T00:30:46.920477+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:30:46.920477+0000","last_undegraded":"2024-09-11T00:30:46.920477+0000","last_fullsized":"2024-09-11T00:30:46.920477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923580+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:30:46.923580+0000","last_peered":"2024-09-11T00:30:46.923580+0000","last_clean":"2024-09-11T00:30:46.923580+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:30:46.923580+0000","last_undegraded":"2024-09-11T00:30:46.923580+0000","last_fullsized":"2024-09-11T00:30:46.923580+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916613+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:30:46.916613+0000","last_peered":"2024-09-11T00:30:46.916613+0000","last_clean":"2024-09-11T00:30:46.916613+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:30:46.916613+0000","last_undegraded":"2024-09-11T00:30:46.916613+0000","last_fullsized":"2024-09-11T00:30:46.916613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923501+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:30:46.923501+0000","last_peered":"2024-09-11T00:30:46.923501+0000","last_clean":"2024-09-11T00:30:46.923501+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:30:46.923501+0000","last_undegraded":"2024-09-11T00:30:46.923501+0000","last_fullsized":"2024-09-11T00:30:46.923501+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917696+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:30:46.917696+0000","last_peered":"2024-09-11T00:30:46.917696+0000","last_clean":"2024-09-11T00:30:46.917696+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:30:46.917696+0000","last_undegraded":"2024-09-11T00:30:46.917696+0000","last_fullsized":"2024-09-11T00:30:46.917696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016883+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:30:47.016883+0000","last_peered":"2024-09-11T00:30:47.016883+0000","last_clean":"2024-09-11T00:30:47.016883+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:30:47.016883+0000","last_undegraded":"2024-09-11T00:30:47.016883+0000","last_fullsized":"2024-09-11T00:30:47.016883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.913436+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:30:46.913436+0000","last_peered":"2024-09-11T00:30:46.913436+0000","last_clean":"2024-09-11T00:30:46.913436+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:30:46.913436+0000","last_undegraded":"2024-09-11T00:30:46.913436+0000","last_fullsized":"2024-09-11T00:30:46.913436+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016451+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:30:47.016451+0000","last_peered":"2024-09-11T00:30:47.016451+0000","last_clean":"2024-09-11T00:30:47.016451+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:30:47.016451+0000","last_undegraded":"2024-09-11T00:30:47.016451+0000","last_fullsized":"2024-09-11T00:30:47.016451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920418+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:30:46.920418+0000","last_peered":"2024-09-11T00:30:46.920418+0000","last_clean":"2024-09-11T00:30:46.920418+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:30:46.920418+0000","last_undegraded":"2024-09-11T00:30:46.920418+0000","last_fullsized":"2024-09-11T00:30:46.920418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016414+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:30:47.016414+0000","last_peered":"2024-09-11T00:30:47.016414+0000","last_clean":"2024-09-11T00:30:47.016414+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:30:47.016414+0000","last_undegraded":"2024-09-11T00:30:47.016414+0000","last_fullsized":"2024-09-11T00:30:47.016414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.915766+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:30:46.915766+0000","last_peered":"2024-09-11T00:30:46.915766+0000","last_clean":"2024-09-11T00:30:46.915766+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:30:46.915766+0000","last_undegraded":"2024-09-11T00:30:46.915766+0000","last_fullsized":"2024-09-11T00:30:46.915766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.015721+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:30:47.015721+0000","last_peered":"2024-09-11T00:30:47.015721+0000","last_clean":"2024-09-11T00:30:47.015721+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:30:47.015721+0000","last_undegraded":"2024-09-11T00:30:47.015721+0000","last_fullsized":"2024-09-11T00:30:47.015721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916079+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:30:46.916079+0000","last_peered":"2024-09-11T00:30:46.916079+0000","last_clean":"2024-09-11T00:30:46.916079+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:30:46.916079+0000","last_undegraded":"2024-09-11T00:30:46.916079+0000","last_fullsized":"2024-09-11T00:30:46.916079+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016719+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:30:47.016719+0000","last_peered":"2024-09-11T00:30:47.016719+0000","last_clean":"2024-09-11T00:30:47.016719+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:30:47.016719+0000","last_undegraded":"2024-09-11T00:30:47.016719+0000","last_fullsized":"2024-09-11T00:30:47.016719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923278+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:30:46.923278+0000","last_peered":"2024-09-11T00:30:46.923278+0000","last_clean":"2024-09-11T00:30:46.923278+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:30:46.923278+0000","last_undegraded":"2024-09-11T00:30:46.923278+0000","last_fullsized":"2024-09-11T00:30:46.923278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916500+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:30:46.916500+0000","last_peered":"2024-09-11T00:30:46.916500+0000","last_clean":"2024-09-11T00:30:46.916500+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:30:46.916500+0000","last_undegraded":"2024-09-11T00:30:46.916500+0000","last_fullsized":"2024-09-11T00:30:46.916500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016431+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:30:47.016431+0000","last_peered":"2024-09-11T00:30:47.016431+0000","last_clean":"2024-09-11T00:30:47.016431+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:30:47.016431+0000","last_undegraded":"2024-09-11T00:30:47.016431+0000","last_fullsized":"2024-09-11T00:30:47.016431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017138+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:30:47.017138+0000","last_peered":"2024-09-11T00:30:47.017138+0000","last_clean":"2024-09-11T00:30:47.017138+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:30:47.017138+0000","last_undegraded":"2024-09-11T00:30:47.017138+0000","last_fullsized":"2024-09-11T00:30:47.017138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917635+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:30:46.917635+0000","last_peered":"2024-09-11T00:30:46.917635+0000","last_clean":"2024-09-11T00:30:46.917635+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:30:46.917635+0000","last_undegraded":"2024-09-11T00:30:46.917635+0000","last_fullsized":"2024-09-11T00:30:46.917635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.922338+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:30:46.922338+0000","last_peered":"2024-09-11T00:30:46.922338+0000","last_clean":"2024-09-11T00:30:46.922338+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:30:46.922338+0000","last_undegraded":"2024-09-11T00:30:46.922338+0000","last_fullsized":"2024-09-11T00:30:46.922338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920774+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:30:46.920774+0000","last_peered":"2024-09-11T00:30:46.920774+0000","last_clean":"2024-09-11T00:30:46.920774+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:30:46.920774+0000","last_undegraded":"2024-09-11T00:30:46.920774+0000","last_fullsized":"2024-09-11T00:30:46.920774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.017260+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:30:47.017260+0000","last_peered":"2024-09-11T00:30:47.017260+0000","last_clean":"2024-09-11T00:30:47.017260+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:30:47.017260+0000","last_undegraded":"2024-09-11T00:30:47.017260+0000","last_fullsized":"2024-09-11T00:30:47.017260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920716+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:30:46.920716+0000","last_peered":"2024-09-11T00:30:46.920716+0000","last_clean":"2024-09-11T00:30:46.920716+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:30:46.920716+0000","last_undegraded":"2024-09-11T00:30:46.920716+0000","last_fullsized":"2024-09-11T00:30:46.920716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.922727+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:30:46.922727+0000","last_peered":"2024-09-11T00:30:46.922727+0000","last_clean":"2024-09-11T00:30:46.922727+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:30:46.922727+0000","last_undegraded":"2024-09-11T00:30:46.922727+0000","last_fullsized":"2024-09-11T00:30:46.922727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016872+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:30:47.016872+0000","last_peered":"2024-09-11T00:30:47.016872+0000","last_clean":"2024-09-11T00:30:47.016872+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:30:47.016872+0000","last_undegraded":"2024-09-11T00:30:47.016872+0000","last_fullsized":"2024-09-11T00:30:47.016872+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920443+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:30:46.920443+0000","last_peered":"2024-09-11T00:30:46.920443+0000","last_clean":"2024-09-11T00:30:46.920443+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:30:46.920443+0000","last_undegraded":"2024-09-11T00:30:46.920443+0000","last_fullsized":"2024-09-11T00:30:46.920443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920200+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:30:46.920200+0000","last_peered":"2024-09-11T00:30:46.920200+0000","last_clean":"2024-09-11T00:30:46.920200+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:30:46.920200+0000","last_undegraded":"2024-09-11T00:30:46.920200+0000","last_fullsized":"2024-09-11T00:30:46.920200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916277+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:30:46.916277+0000","last_peered":"2024-09-11T00:30:46.916277+0000","last_clean":"2024-09-11T00:30:46.916277+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:30:46.916277+0000","last_undegraded":"2024-09-11T00:30:46.916277+0000","last_fullsized":"2024-09-11T00:30:46.916277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016150+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:30:47.016150+0000","last_peered":"2024-09-11T00:30:47.016150+0000","last_clean":"2024-09-11T00:30:47.016150+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:30:47.016150+0000","last_undegraded":"2024-09-11T00:30:47.016150+0000","last_fullsized":"2024-09-11T00:30:47.016150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016385+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:30:47.016385+0000","last_peered":"2024-09-11T00:30:47.016385+0000","last_clean":"2024-09-11T00:30:47.016385+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:30:47.016385+0000","last_undegraded":"2024-09-11T00:30:47.016385+0000","last_fullsized":"2024-09-11T00:30:47.016385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016435+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:30:47.016435+0000","last_peered":"2024-09-11T00:30:47.016435+0000","last_clean":"2024-09-11T00:30:47.016435+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:30:47.016435+0000","last_undegraded":"2024-09-11T00:30:47.016435+0000","last_fullsized":"2024-09-11T00:30:47.016435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:47.016678+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:30:47.016678+0000","last_peered":"2024-09-11T00:30:47.016678+0000","last_clean":"2024-09-11T00:30:47.016678+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:30:47.016678+0000","last_undegraded":"2024-09-11T00:30:47.016678+0000","last_fullsized":"2024-09-11T00:30:47.016678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923381+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:30:46.923381+0000","last_peered":"2024-09-11T00:30:46.923381+0000","last_clean":"2024-09-11T00:30:46.923381+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:30:46.923381+0000","last_undegraded":"2024-09-11T00:30:46.923381+0000","last_fullsized":"2024-09-11T00:30:46.923381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.913652+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:30:46.913652+0000","last_peered":"2024-09-11T00:30:46.913652+0000","last_clean":"2024-09-11T00:30:46.913652+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:30:46.913652+0000","last_undegraded":"2024-09-11T00:30:46.913652+0000","last_fullsized":"2024-09-11T00:30:46.913652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.923730+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:30:46.923730+0000","last_peered":"2024-09-11T00:30:46.923730+0000","last_clean":"2024-09-11T00:30:46.923730+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:30:46.923730+0000","last_undegraded":"2024-09-11T00:30:46.923730+0000","last_fullsized":"2024-09-11T00:30:46.923730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917529+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:30:46.917529+0000","last_peered":"2024-09-11T00:30:46.917529+0000","last_clean":"2024-09-11T00:30:46.917529+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:30:46.917529+0000","last_undegraded":"2024-09-11T00:30:46.917529+0000","last_fullsized":"2024-09-11T00:30:46.917529+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.917135+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:30:46.917135+0000","last_peered":"2024-09-11T00:30:46.917135+0000","last_clean":"2024-09-11T00:30:46.917135+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:30:46.917135+0000","last_undegraded":"2024-09-11T00:30:46.917135+0000","last_fullsized":"2024-09-11T00:30:46.917135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.916981+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:30:46.916981+0000","last_peered":"2024-09-11T00:30:46.916981+0000","last_clean":"2024-09-11T00:30:46.916981+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:30:46.916981+0000","last_undegraded":"2024-09-11T00:30:46.916981+0000","last_fullsized":"2024-09-11T00:30:46.916981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-11T00:30:46.920899+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:30:46.920899+0000","last_peered":"2024-09-11T00:30:46.920899+0000","last_clean":"2024-09-11T00:30:46.920899+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:30:46.920899+0000","last_undegraded":"2024-09-11T00:30:46.920899+0000","last_fullsized":"2024-09-11T00:30:46.920899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":76328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":76204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":823296,"data_stored":666664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":76786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":77816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":819200,"data_stored":666642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":76260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":78488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:30:51.596 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T00:30:51.596 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-11T00:30:51.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.791+0000 7fbfc5400640 1 -- 172.21.15.31:0/876339882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfc0072480 msgr2=0x7fbfc01511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:51.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.791+0000 7fbfc5400640 1 --2- 172.21.15.31:0/876339882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfc0072480 0x7fbfc01511a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0075260 tx=0x7fbfb002a530 comp rx=0 tx=0).stop 2024-09-11T00:30:51.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.791+0000 7fbfc5400640 1 -- 172.21.15.31:0/876339882 shutdown_connections 2024-09-11T00:30:51.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.791+0000 7fbfc5400640 1 --2- 172.21.15.31:0/876339882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfc0072480 0x7fbfc01511a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.791+0000 7fbfc5400640 1 -- 172.21.15.31:0/876339882 >> 172.21.15.31:0/876339882 conn(0x7fbfc00a9750 msgr2=0x7fbfc00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:51.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.791+0000 7fbfc5400640 1 -- 172.21.15.31:0/876339882 shutdown_connections 2024-09-11T00:30:51.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.791+0000 7fbfc5400640 1 -- 172.21.15.31:0/876339882 wait complete. 2024-09-11T00:30:51.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.793+0000 7fbfc5400640 1 Processor -- start 2024-09-11T00:30:51.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.793+0000 7fbfc5400640 1 -- start start 2024-09-11T00:30:51.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.794+0000 7fbfc5400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfc0072480 0x7fbfc00c6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:51.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.794+0000 7fbfc5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfc00c23a0 0x7fbfc00c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:51.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.794+0000 7fbfc5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfc0153f20 0x7fbfc01e0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.794+0000 7fbfc5400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc0074200 con 0x7fbfc0153f20 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.794+0000 7fbfc5400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc0074080 con 0x7fbfc00c23a0 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.794+0000 7fbfc5400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc0072860 con 0x7fbfc0072480 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.794+0000 7fbfbf200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfc0153f20 0x7fbfc01e0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.794+0000 7fbfbf200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfc0153f20 0x7fbfc01e0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56564/0 (socket says 172.21.15.31:56564) 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.794+0000 7fbfbf200640 1 -- 172.21.15.31:0/2293723448 learned_addr learned my addr 172.21.15.31:0/2293723448 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfbe800640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfc0072480 0x7fbfc00c6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfbde00640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfc00c23a0 0x7fbfc00c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfbf200640 1 -- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfc0072480 msgr2=0x7fbfc00c6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfbf200640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfc0072480 0x7fbfc00c6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfbf200640 1 -- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfc00c23a0 msgr2=0x7fbfc00c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:51.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfbf200640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfc00c23a0 0x7fbfc00c6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:51.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfbf200640 1 -- 172.21.15.31:0/2293723448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc006e510 con 0x7fbfc0153f20 2024-09-11T00:30:51.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfbf200640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfc0153f20 0x7fbfc01e0d50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800c700 tx=0x7fbfb800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:51.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfb6a00640 1 -- 172.21.15.31:0/2293723448 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8003b70 con 0x7fbfc0153f20 2024-09-11T00:30:51.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfb6a00640 1 -- 172.21.15.31:0/2293723448 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfb8003d10 con 0x7fbfc0153f20 2024-09-11T00:30:51.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfc5400640 1 -- 172.21.15.31:0/2293723448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc006e7f0 con 0x7fbfc0153f20 2024-09-11T00:30:51.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfb6a00640 1 -- 172.21.15.31:0/2293723448 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb800aab0 con 0x7fbfc0153f20 2024-09-11T00:30:51.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.795+0000 7fbfc5400640 1 -- 172.21.15.31:0/2293723448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc006edb0 con 0x7fbfc0153f20 2024-09-11T00:30:51.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.797+0000 7fbfc5400640 1 -- 172.21.15.31:0/2293723448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfc01511a0 con 0x7fbfc0153f20 2024-09-11T00:30:51.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.798+0000 7fbfb6a00640 1 -- 172.21.15.31:0/2293723448 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbfb800ac50 con 0x7fbfc0153f20 2024-09-11T00:30:51.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.799+0000 7fbfb6a00640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbfa0077fa0 0x7fbfa007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:51.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.799+0000 7fbfb6a00640 1 -- 172.21.15.31:0/2293723448 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7fbfb8099b40 con 0x7fbfc0153f20 2024-09-11T00:30:51.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.799+0000 7fbfbe800640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbfa0077fa0 0x7fbfa007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:51.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.800+0000 7fbfbe800640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbfa0077fa0 0x7fbfa007a460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0034000 tx=0x7fbfb0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:51.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:51.803+0000 7fbfb6a00640 1 -- 172.21.15.31:0/2293723448 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbfb80625d0 con 0x7fbfc0153f20 2024-09-11T00:30:52.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.000+0000 7fbfc5400640 1 -- 172.21.15.31:0/2293723448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbfc00c15f0 con 0x7fbfc0153f20 2024-09-11T00:30:52.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.001+0000 7fbfb6a00640 1 -- 172.21.15.31:0/2293723448 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v22) ==== 84+0+197 (secure 0 0 0) 0x7fbfb8061ed0 con 0x7fbfc0153f20 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 2, 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": "a_metadata" 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T00:30:52.003 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-09-11T00:30:52.004 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 3, 2024-09-11T00:30:52.004 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": "a_data" 2024-09-11T00:30:52.004 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T00:30:52.004 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-09-11T00:30:52.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.003+0000 7fbfc5400640 1 -- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbfa0077fa0 msgr2=0x7fbfa007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:52.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.003+0000 7fbfc5400640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbfa0077fa0 0x7fbfa007a460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0034000 tx=0x7fbfb0004730 comp rx=0 tx=0).stop 2024-09-11T00:30:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.003+0000 7fbfc5400640 1 -- 172.21.15.31:0/2293723448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfc0153f20 msgr2=0x7fbfc01e0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.003+0000 7fbfc5400640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfc0153f20 0x7fbfc01e0d50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800c700 tx=0x7fbfb800cbc0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.003+0000 7fbfc5400640 1 -- 172.21.15.31:0/2293723448 shutdown_connections 2024-09-11T00:30:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.003+0000 7fbfc5400640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbfa0077fa0 0x7fbfa007a460 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.003+0000 7fbfc5400640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfc0153f20 0x7fbfc01e0d50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.003+0000 7fbfc5400640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfc00c23a0 0x7fbfc00c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.003+0000 7fbfc5400640 1 --2- 172.21.15.31:0/2293723448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfc0072480 0x7fbfc00c6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.003+0000 7fbfc5400640 1 -- 172.21.15.31:0/2293723448 >> 172.21.15.31:0/2293723448 conn(0x7fbfc00a9750 msgr2=0x7fbfc01e20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.004+0000 7fbfc5400640 1 -- 172.21.15.31:0/2293723448 shutdown_connections 2024-09-11T00:30:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.004+0000 7fbfc5400640 1 -- 172.21.15.31:0/2293723448 wait complete. 2024-09-11T00:30:52.021 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'b' 2024-09-11T00:30:52.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_metadata --pg_num_min 64 2024-09-11T00:30:52.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.183+0000 7fee8c800640 1 -- 172.21.15.31:0/2505093202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee840c6340 msgr2=0x7fee84159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:52.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.183+0000 7fee8c800640 1 --2- 172.21.15.31:0/2505093202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee840c6340 0x7fee84159670 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fee84070470 tx=0x7fee8002aac0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.183+0000 7fee8c800640 1 -- 172.21.15.31:0/2505093202 shutdown_connections 2024-09-11T00:30:52.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.183+0000 7fee8c800640 1 --2- 172.21.15.31:0/2505093202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee840c6340 0x7fee84159670 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.183+0000 7fee8c800640 1 --2- 172.21.15.31:0/2505093202 >> v2:172.21.15.204:3300/0 conn(0x7fee840c5b10 0x7fee84156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.183+0000 7fee8c800640 1 -- 172.21.15.31:0/2505093202 >> 172.21.15.31:0/2505093202 conn(0x7fee840a9750 msgr2=0x7fee840a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:52.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.183+0000 7fee8c800640 1 -- 172.21.15.31:0/2505093202 shutdown_connections 2024-09-11T00:30:52.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.184+0000 7fee8c800640 1 -- 172.21.15.31:0/2505093202 wait complete. 2024-09-11T00:30:52.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.185+0000 7fee8c800640 1 Processor -- start 2024-09-11T00:30:52.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.185+0000 7fee8c800640 1 -- start start 2024-09-11T00:30:52.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee8c800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee840c5b10 0x7fee841ef510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:52.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee8c800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee840c5ef0 0x7fee8414c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:52.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee8c800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee840c6340 0x7fee8414cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:52.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee8c800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fee84153b90 con 0x7fee840c6340 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee8c800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fee840c3950 con 0x7fee840c5b10 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee8c800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fee84153d10 con 0x7fee840c5ef0 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee8a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee840c6340 0x7fee8414cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee89e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee840c5b10 0x7fee841ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee8a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee840c6340 0x7fee8414cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56590/0 (socket says 172.21.15.31:56590) 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee8a800640 1 -- 172.21.15.31:0/3156251723 learned_addr learned my addr 172.21.15.31:0/3156251723 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.186+0000 7fee89400640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee840c5ef0 0x7fee8414c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee8a800640 1 -- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee840c5ef0 msgr2=0x7fee8414c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee8a800640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee840c5ef0 0x7fee8414c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee8a800640 1 -- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee840c5b10 msgr2=0x7fee841ef510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee8a800640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee840c5b10 0x7fee841ef510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee8a800640 1 -- 172.21.15.31:0/3156251723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee841efd50 con 0x7fee840c6340 2024-09-11T00:30:52.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee89400640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee840c5ef0 0x7fee8414c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:30:52.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee8a800640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee840c6340 0x7fee8414cc70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fee80009d60 tx=0x7fee80003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:52.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee7e000640 1 -- 172.21.15.31:0/3156251723 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80044070 con 0x7fee840c6340 2024-09-11T00:30:52.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee7e000640 1 -- 172.21.15.31:0/3156251723 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee80004250 con 0x7fee840c6340 2024-09-11T00:30:52.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee7e000640 1 -- 172.21.15.31:0/3156251723 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80004680 con 0x7fee840c6340 2024-09-11T00:30:52.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.187+0000 7fee8c800640 1 -- 172.21.15.31:0/3156251723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee841f0030 con 0x7fee840c6340 2024-09-11T00:30:52.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.188+0000 7fee8c800640 1 -- 172.21.15.31:0/3156251723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee8414d1b0 con 0x7fee840c6340 2024-09-11T00:30:52.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.189+0000 7fee8c800640 1 -- 172.21.15.31:0/3156251723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee841ed200 con 0x7fee840c6340 2024-09-11T00:30:52.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.190+0000 7fee7e000640 1 -- 172.21.15.31:0/3156251723 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fee8002c840 con 0x7fee840c6340 2024-09-11T00:30:52.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.191+0000 7fee7e000640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee68077f70 0x7fee6807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:52.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.191+0000 7fee7e000640 1 -- 172.21.15.31:0/3156251723 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7fee800ba180 con 0x7fee840c6340 2024-09-11T00:30:52.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.191+0000 7fee89e00640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee68077f70 0x7fee6807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:52.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.192+0000 7fee89e00640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee68077f70 0x7fee6807a430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee7800bbe0 tx=0x7fee7800b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:52.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.195+0000 7fee7e000640 1 -- 172.21.15.31:0/3156251723 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fee80032030 con 0x7fee840c6340 2024-09-11T00:30:52.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.388+0000 7fee8c800640 1 -- 172.21.15.31:0/3156251723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64} v 0) -- 0x7fee84156ad0 con 0x7fee840c6340 2024-09-11T00:30:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.952+0000 7fee7e000640 1 -- 172.21.15.31:0/3156251723 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64}]=0 pool 'b_metadata' created v23) ==== 128+0+0 (secure 0 0 0) 0x7fee80004820 con 0x7fee840c6340 2024-09-11T00:30:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'b_metadata' created 2024-09-11T00:30:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 -- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee68077f70 msgr2=0x7fee6807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee68077f70 0x7fee6807a430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee7800bbe0 tx=0x7fee7800b390 comp rx=0 tx=0).stop 2024-09-11T00:30:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 -- 172.21.15.31:0/3156251723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee840c6340 msgr2=0x7fee8414cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee840c6340 0x7fee8414cc70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fee80009d60 tx=0x7fee80003f10 comp rx=0 tx=0).stop 2024-09-11T00:30:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 -- 172.21.15.31:0/3156251723 shutdown_connections 2024-09-11T00:30:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee68077f70 0x7fee6807a430 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee840c6340 0x7fee8414cc70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee840c5ef0 0x7fee8414c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 --2- 172.21.15.31:0/3156251723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee840c5b10 0x7fee841ef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 -- 172.21.15.31:0/3156251723 >> 172.21.15.31:0/3156251723 conn(0x7fee840a9750 msgr2=0x7fee840a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.955+0000 7fee8c800640 1 -- 172.21.15.31:0/3156251723 shutdown_connections 2024-09-11T00:30:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:52.956+0000 7fee8c800640 1 -- 172.21.15.31:0/3156251723 wait complete. 2024-09-11T00:30:52.981 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-11T00:30:53.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.147+0000 7fd447c00640 1 -- 172.21.15.31:0/2470772411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4240052d0 msgr2=0x7fd42400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:53.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.147+0000 7fd447c00640 1 --2- 172.21.15.31:0/2470772411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4240052d0 0x7fd42400b9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd43800c640 tx=0x7fd424005830 comp rx=0 tx=0).stop 2024-09-11T00:30:53.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.148+0000 7fd447c00640 1 -- 172.21.15.31:0/2470772411 shutdown_connections 2024-09-11T00:30:53.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.148+0000 7fd447c00640 1 --2- 172.21.15.31:0/2470772411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4240052d0 0x7fd42400b9c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:53.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.148+0000 7fd447c00640 1 -- 172.21.15.31:0/2470772411 >> 172.21.15.31:0/2470772411 conn(0x7fd44006d560 msgr2=0x7fd4400c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:53.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.148+0000 7fd447c00640 1 -- 172.21.15.31:0/2470772411 shutdown_connections 2024-09-11T00:30:53.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.148+0000 7fd447c00640 1 -- 172.21.15.31:0/2470772411 wait complete. 2024-09-11T00:30:53.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.149+0000 7fd447c00640 1 Processor -- start 2024-09-11T00:30:53.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.149+0000 7fd447c00640 1 -- start start 2024-09-11T00:30:53.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.150+0000 7fd447c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4240052d0 0x7fd4400ab7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:53.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.150+0000 7fd447c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4401ddbd0 0x7fd4400ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:53.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.150+0000 7fd447c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4400ac730 0x7fd440148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:53.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.150+0000 7fd447c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd44006e380 con 0x7fd4400ac730 2024-09-11T00:30:53.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.150+0000 7fd447c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd44006e200 con 0x7fd4401ddbd0 2024-09-11T00:30:53.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.150+0000 7fd447c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd44014e220 con 0x7fd4240052d0 2024-09-11T00:30:53.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.150+0000 7fd445c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4400ac730 0x7fd440148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:53.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.150+0000 7fd445c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4400ac730 0x7fd440148790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56612/0 (socket says 172.21.15.31:56612) 2024-09-11T00:30:53.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.150+0000 7fd445c00640 1 -- 172.21.15.31:0/848292127 learned_addr learned my addr 172.21.15.31:0/848292127 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:53.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd445200640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4240052d0 0x7fd4400ab7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:53.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd437e00640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4401ddbd0 0x7fd4400ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd445c00640 1 -- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4240052d0 msgr2=0x7fd4400ab7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd445c00640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4240052d0 0x7fd4400ab7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd445c00640 1 -- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4401ddbd0 msgr2=0x7fd4400ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd445c00640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4401ddbd0 0x7fd4400ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd445c00640 1 -- 172.21.15.31:0/848292127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4401dd740 con 0x7fd4400ac730 2024-09-11T00:30:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd445c00640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4400ac730 0x7fd440148790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd43800c730 tx=0x7fd438001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd43d600640 1 -- 172.21.15.31:0/848292127 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43802bae0 con 0x7fd4400ac730 2024-09-11T00:30:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd43d600640 1 -- 172.21.15.31:0/848292127 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd43802bd70 con 0x7fd4400ac730 2024-09-11T00:30:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd43d600640 1 -- 172.21.15.31:0/848292127 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4380062d0 con 0x7fd4400ac730 2024-09-11T00:30:53.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.151+0000 7fd447c00640 1 -- 172.21.15.31:0/848292127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd440151790 con 0x7fd4400ac730 2024-09-11T00:30:53.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.152+0000 7fd447c00640 1 -- 172.21.15.31:0/848292127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd440149b80 con 0x7fd4400ac730 2024-09-11T00:30:53.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.153+0000 7fd447c00640 1 -- 172.21.15.31:0/848292127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd440154520 con 0x7fd4400ac730 2024-09-11T00:30:53.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.154+0000 7fd43d600640 1 -- 172.21.15.31:0/848292127 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd438006470 con 0x7fd4400ac730 2024-09-11T00:30:53.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.155+0000 7fd43d600640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd41c077f70 0x7fd41c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:53.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.155+0000 7fd43d600640 1 -- 172.21.15.31:0/848292127 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6054+0+0 (secure 0 0 0) 0x7fd4380b9fa0 con 0x7fd4400ac730 2024-09-11T00:30:53.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.155+0000 7fd445200640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd41c077f70 0x7fd41c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:53.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.156+0000 7fd445200640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd41c077f70 0x7fd41c07a430 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd430009bb0 tx=0x7fd430005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:53.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.159+0000 7fd43d600640 1 -- 172.21.15.31:0/848292127 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd438082930 con 0x7fd4400ac730 2024-09-11T00:30:53.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.355+0000 7fd447c00640 1 -- 172.21.15.31:0/848292127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd4400c6a80 con 0x7fd4400ac730 2024-09-11T00:30:53.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.959+0000 7fd43d600640 1 -- 172.21.15.31:0/848292127 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'b_data' created v24) ==== 160+0+0 (secure 0 0 0) 0x7fd438082230 con 0x7fd4400ac730 2024-09-11T00:30:53.961 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'b_data' created 2024-09-11T00:30:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.964+0000 7fd447c00640 1 -- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd41c077f70 msgr2=0x7fd41c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.964+0000 7fd447c00640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd41c077f70 0x7fd41c07a430 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd430009bb0 tx=0x7fd430005f90 comp rx=0 tx=0).stop 2024-09-11T00:30:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.964+0000 7fd447c00640 1 -- 172.21.15.31:0/848292127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4400ac730 msgr2=0x7fd440148790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.964+0000 7fd447c00640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4400ac730 0x7fd440148790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd43800c730 tx=0x7fd438001aa0 comp rx=0 tx=0).stop 2024-09-11T00:30:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.964+0000 7fd447c00640 1 -- 172.21.15.31:0/848292127 shutdown_connections 2024-09-11T00:30:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.964+0000 7fd447c00640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd41c077f70 0x7fd41c07a430 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd430009bb0 tx=0x7fd430005f90 comp rx=0 tx=0).stop 2024-09-11T00:30:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.964+0000 7fd447c00640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4400ac730 0x7fd440148790 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.964+0000 7fd447c00640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4401ddbd0 0x7fd4400ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.964+0000 7fd447c00640 1 --2- 172.21.15.31:0/848292127 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4240052d0 0x7fd4400ab7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:53.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.964+0000 7fd447c00640 1 -- 172.21.15.31:0/848292127 >> 172.21.15.31:0/848292127 conn(0x7fd44006d560 msgr2=0x7fd4401536d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:53.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.965+0000 7fd447c00640 1 -- 172.21.15.31:0/848292127 shutdown_connections 2024-09-11T00:30:53.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:53.965+0000 7fd447c00640 1 -- 172.21.15.31:0/848292127 wait complete. 2024-09-11T00:30:54.005 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new b b_metadata b_data 2024-09-11T00:30:54.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.167+0000 7f46d2000640 1 -- 172.21.15.31:0/2649030699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c3bd0 msgr2=0x7f46cc0aa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:54.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.167+0000 7f46d2000640 1 --2- 172.21.15.31:0/2649030699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c3bd0 0x7f46cc0aa9e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0a6f20 tx=0x7f46c402aa30 comp rx=0 tx=0).stop 2024-09-11T00:30:54.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.167+0000 7f46d2000640 1 -- 172.21.15.31:0/2649030699 shutdown_connections 2024-09-11T00:30:54.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.167+0000 7f46d2000640 1 --2- 172.21.15.31:0/2649030699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c3bd0 0x7f46cc0aa9e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:54.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.167+0000 7f46d2000640 1 --2- 172.21.15.31:0/2649030699 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc0c3500 0x7f46cc152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:54.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.167+0000 7f46d2000640 1 -- 172.21.15.31:0/2649030699 >> 172.21.15.31:0/2649030699 conn(0x7f46cc06d560 msgr2=0x7f46cc0c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:54.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.167+0000 7f46d2000640 1 -- 172.21.15.31:0/2649030699 shutdown_connections 2024-09-11T00:30:54.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.167+0000 7f46d2000640 1 -- 172.21.15.31:0/2649030699 wait complete. 2024-09-11T00:30:54.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.169+0000 7f46d2000640 1 Processor -- start 2024-09-11T00:30:54.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.169+0000 7f46d2000640 1 -- start start 2024-09-11T00:30:54.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46d2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c3500 0x7f46cc147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:54.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46d2000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc0c3bd0 0x7f46cc150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46d2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46cc150740 0x7f46cc151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46d2000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc0c2f00 con 0x7f46cc0c3500 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46d2000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc0c2c20 con 0x7f46cc150740 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46d2000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc0c31e0 con 0x7f46cc0c3bd0 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c3500 0x7f46cc147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c3500 0x7f46cc147ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56626/0 (socket says 172.21.15.31:56626) 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46cb400640 1 -- 172.21.15.31:0/1096871396 learned_addr learned my addr 172.21.15.31:0/1096871396 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46c2a00640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc0c3bd0 0x7f46cc150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46cbe00640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46cc150740 0x7f46cc151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46cb400640 1 -- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc0c3bd0 msgr2=0x7f46cc150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:54.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46cb400640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc0c3bd0 0x7f46cc150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:54.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46cb400640 1 -- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46cc150740 msgr2=0x7f46cc151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:54.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46cb400640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46cc150740 0x7f46cc151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:54.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.170+0000 7f46cb400640 1 -- 172.21.15.31:0/1096871396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc1566d0 con 0x7f46cc0c3500 2024-09-11T00:30:54.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.171+0000 7f46cb400640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c3500 0x7f46cc147ff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00c3a0 tx=0x7f46bc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:54.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.171+0000 7f46c3e00640 1 -- 172.21.15.31:0/1096871396 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc017070 con 0x7f46cc0c3500 2024-09-11T00:30:54.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.171+0000 7f46c3e00640 1 -- 172.21.15.31:0/1096871396 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46bc0053c0 con 0x7f46cc0c3500 2024-09-11T00:30:54.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.171+0000 7f46d2000640 1 -- 172.21.15.31:0/1096871396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc1ecc70 con 0x7f46cc0c3500 2024-09-11T00:30:54.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.171+0000 7f46c3e00640 1 -- 172.21.15.31:0/1096871396 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc003850 con 0x7f46cc0c3500 2024-09-11T00:30:54.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.171+0000 7f46d2000640 1 -- 172.21.15.31:0/1096871396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc14d0f0 con 0x7f46cc0c3500 2024-09-11T00:30:54.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.172+0000 7f46d2000640 1 -- 172.21.15.31:0/1096871396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46cc1eca80 con 0x7f46cc0c3500 2024-09-11T00:30:54.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.173+0000 7f46c3e00640 1 -- 172.21.15.31:0/1096871396 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f46bc0039f0 con 0x7f46cc0c3500 2024-09-11T00:30:54.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.174+0000 7f46c3e00640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a8077fa0 0x7f46a807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:54.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.174+0000 7f46c3e00640 1 -- 172.21.15.31:0/1096871396 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6431+0+0 (secure 0 0 0) 0x7f46bc0998f0 con 0x7f46cc0c3500 2024-09-11T00:30:54.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.175+0000 7f46c2a00640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a8077fa0 0x7f46a807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:54.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.175+0000 7f46c2a00640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a8077fa0 0x7f46a807a460 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f46cc149a40 tx=0x7f46b4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:54.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.178+0000 7f46c3e00640 1 -- 172.21.15.31:0/1096871396 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f46bc021020 con 0x7f46cc0c3500 2024-09-11T00:30:54.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.410+0000 7f46d2000640 1 -- 172.21.15.31:0/1096871396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"} v 0) -- 0x7f46cc0ab050 con 0x7f46cc0c3500 2024-09-11T00:30:54.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.977+0000 7f46c3e00640 1 -- 172.21.15.31:0/1096871396 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"}]=0 new fs with metadata pool 4 and data pool 5 v10) ==== 157+0+0 (secure 0 0 0) 0x7f46bc061c40 con 0x7f46cc0c3500 2024-09-11T00:30:54.982 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 4 and data pool 5 2024-09-11T00:30:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 -- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a8077fa0 msgr2=0x7f46a807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a8077fa0 0x7f46a807a460 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f46cc149a40 tx=0x7f46b4005f90 comp rx=0 tx=0).stop 2024-09-11T00:30:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 -- 172.21.15.31:0/1096871396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c3500 msgr2=0x7f46cc147ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c3500 0x7f46cc147ff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00c3a0 tx=0x7f46bc00c860 comp rx=0 tx=0).stop 2024-09-11T00:30:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 -- 172.21.15.31:0/1096871396 shutdown_connections 2024-09-11T00:30:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a8077fa0 0x7f46a807a460 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46cc150740 0x7f46cc151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc0c3bd0 0x7f46cc150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 --2- 172.21.15.31:0/1096871396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c3500 0x7f46cc147ff0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 -- 172.21.15.31:0/1096871396 >> 172.21.15.31:0/1096871396 conn(0x7f46cc06d560 msgr2=0x7f46cc1e2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.986+0000 7f46d2000640 1 -- 172.21.15.31:0/1096871396 shutdown_connections 2024-09-11T00:30:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:54.987+0000 7f46d2000640 1 -- 172.21.15.31:0/1096871396 wait complete. 2024-09-11T00:30:55.019 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:30:55.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.179+0000 7f7992e00640 1 -- 172.21.15.31:0/608717449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0c5b10 msgr2=0x7f798c156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.179+0000 7f7992e00640 1 --2- 172.21.15.31:0/608717449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0c5b10 0x7f798c156ad0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f798c06de50 tx=0x7f797c02a560 comp rx=0 tx=0).stop 2024-09-11T00:30:55.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.179+0000 7f7992e00640 1 -- 172.21.15.31:0/608717449 shutdown_connections 2024-09-11T00:30:55.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.179+0000 7f7992e00640 1 --2- 172.21.15.31:0/608717449 >> v2:172.21.15.204:3300/0 conn(0x7f798c0c6310 0x7f798c1539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.179+0000 7f7992e00640 1 --2- 172.21.15.31:0/608717449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f798c0c5ef0 0x7f798c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.179+0000 7f7992e00640 1 --2- 172.21.15.31:0/608717449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0c5b10 0x7f798c156ad0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.179+0000 7f7992e00640 1 -- 172.21.15.31:0/608717449 >> 172.21.15.31:0/608717449 conn(0x7f798c0a9750 msgr2=0x7f798c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:55.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.179+0000 7f7992e00640 1 -- 172.21.15.31:0/608717449 shutdown_connections 2024-09-11T00:30:55.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.179+0000 7f7992e00640 1 -- 172.21.15.31:0/608717449 wait complete. 2024-09-11T00:30:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.181+0000 7f7992e00640 1 Processor -- start 2024-09-11T00:30:55.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.181+0000 7f7992e00640 1 -- start start 2024-09-11T00:30:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f7992e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f798c0c5b10 0x7f798c073250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f7992e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0c5ef0 0x7f798c1ef5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f7992e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f798c0c6310 0x7f798c1efb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f7992e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f798c1597f0 con 0x7f798c0c5ef0 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f7992e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f798c159670 con 0x7f798c0c6310 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f7992e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f798c159970 con 0x7f798c0c5b10 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f7983400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0c5ef0 0x7f798c1ef5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f798be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f798c0c5b10 0x7f798c073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f7990e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f798c0c6310 0x7f798c1efb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f7983400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0c5ef0 0x7f798c1ef5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56650/0 (socket says 172.21.15.31:56650) 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f798be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f798c0c5b10 0x7f798c073250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:43746/0 (socket says 172.21.15.31:43746) 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.182+0000 7f7983400640 1 -- 172.21.15.31:0/1286147713 learned_addr learned my addr 172.21.15.31:0/1286147713 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7983400640 1 -- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f798c0c5b10 msgr2=0x7f798c073250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7983400640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f798c0c5b10 0x7f798c073250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7983400640 1 -- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f798c0c6310 msgr2=0x7f798c1efb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7983400640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f798c0c6310 0x7f798c1efb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7983400640 1 -- 172.21.15.31:0/1286147713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798c1548d0 con 0x7f798c0c5ef0 2024-09-11T00:30:55.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7983400640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0c5ef0 0x7f798c1ef5d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f797400b450 tx=0x7f797400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:55.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7988c00640 1 -- 172.21.15.31:0/1286147713 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974015020 con 0x7f798c0c5ef0 2024-09-11T00:30:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7988c00640 1 -- 172.21.15.31:0/1286147713 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7974011920 con 0x7f798c0c5ef0 2024-09-11T00:30:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7988c00640 1 -- 172.21.15.31:0/1286147713 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974011d70 con 0x7f798c0c5ef0 2024-09-11T00:30:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7992e00640 1 -- 172.21.15.31:0/1286147713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f798c073790 con 0x7f798c0c5ef0 2024-09-11T00:30:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.183+0000 7f7992e00640 1 -- 172.21.15.31:0/1286147713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f798c1f0180 con 0x7f798c0c5ef0 2024-09-11T00:30:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.184+0000 7f7992e00640 1 -- 172.21.15.31:0/1286147713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f798c156ad0 con 0x7f798c0c5ef0 2024-09-11T00:30:55.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.185+0000 7f7988c00640 1 -- 172.21.15.31:0/1286147713 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f797401a400 con 0x7f798c0c5ef0 2024-09-11T00:30:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.186+0000 7f7988c00640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7968077fd0 0x7f796807a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.187+0000 7f7988c00640 1 -- 172.21.15.31:0/1286147713 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f797409a1d0 con 0x7f798c0c5ef0 2024-09-11T00:30:55.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.187+0000 7f798be00640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7968077fd0 0x7f796807a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:55.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.187+0000 7f798be00640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7968077fd0 0x7f796807a490 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f797c033000 tx=0x7f797c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:55.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.191+0000 7f7988c00640 1 -- 172.21.15.31:0/1286147713 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f797401f020 con 0x7f798c0c5ef0 2024-09-11T00:30:55.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.388+0000 7f7992e00640 1 -- 172.21.15.31:0/1286147713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f798c0c1e70 con 0x7f798c0c5ef0 2024-09-11T00:30:55.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.389+0000 7f7988c00640 1 -- 172.21.15.31:0/1286147713 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20396 (secure 0 0 0) 0x7f7974062340 con 0x7f798c0c5ef0 2024-09-11T00:30:55.391 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:55.392 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:54.412004+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:30:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 -- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7968077fd0 msgr2=0x7f796807a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7968077fd0 0x7f796807a490 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f797c033000 tx=0x7f797c004730 comp rx=0 tx=0).stop 2024-09-11T00:30:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 -- 172.21.15.31:0/1286147713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0c5ef0 msgr2=0x7f798c1ef5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0c5ef0 0x7f798c1ef5d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f797400b450 tx=0x7f797400b910 comp rx=0 tx=0).stop 2024-09-11T00:30:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 -- 172.21.15.31:0/1286147713 shutdown_connections 2024-09-11T00:30:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7968077fd0 0x7f796807a490 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f798c0c6310 0x7f798c1efb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f798c0c5ef0 0x7f798c1ef5d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 --2- 172.21.15.31:0/1286147713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f798c0c5b10 0x7f798c073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 -- 172.21.15.31:0/1286147713 >> 172.21.15.31:0/1286147713 conn(0x7f798c0a9750 msgr2=0x7f798c14cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:55.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 -- 172.21.15.31:0/1286147713 shutdown_connections 2024-09-11T00:30:55.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.392+0000 7f7992e00640 1 -- 172.21.15.31:0/1286147713 wait complete. 2024-09-11T00:30:55.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:30:55.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.569+0000 7f73cb000640 1 -- 172.21.15.31:0/317862973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73c40a8bb0 msgr2=0x7f73c40bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.569+0000 7f73cb000640 1 --2- 172.21.15.31:0/317862973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73c40a8bb0 0x7f73c40bf600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f73c40c6250 tx=0x7f73b002a4a0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.569+0000 7f73cb000640 1 -- 172.21.15.31:0/317862973 shutdown_connections 2024-09-11T00:30:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.569+0000 7f73cb000640 1 --2- 172.21.15.31:0/317862973 >> v2:172.21.15.204:3300/0 conn(0x7f73c41578e0 0x7f73c4157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.569+0000 7f73cb000640 1 --2- 172.21.15.31:0/317862973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73c40a8bb0 0x7f73c40bf600 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.570+0000 7f73cb000640 1 -- 172.21.15.31:0/317862973 >> 172.21.15.31:0/317862973 conn(0x7f73c40c9af0 msgr2=0x7f73c40c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.570+0000 7f73cb000640 1 -- 172.21.15.31:0/317862973 shutdown_connections 2024-09-11T00:30:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.570+0000 7f73cb000640 1 -- 172.21.15.31:0/317862973 wait complete. 2024-09-11T00:30:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.571+0000 7f73cb000640 1 Processor -- start 2024-09-11T00:30:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.571+0000 7f73cb000640 1 -- start start 2024-09-11T00:30:55.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.572+0000 7f73cb000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73c40a8bb0 0x7f73c41503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.572+0000 7f73cb000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73c40c67f0 0x7f73c41ebcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.572+0000 7f73cb000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73c41578e0 0x7f73c4154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.572+0000 7f73cb000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73c40a9260 con 0x7f73c40c67f0 2024-09-11T00:30:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.572+0000 7f73cb000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f73c40a90e0 con 0x7f73c41578e0 2024-09-11T00:30:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.572+0000 7f73cb000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f73c40c5b20 con 0x7f73c40a8bb0 2024-09-11T00:30:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.572+0000 7f73c9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73c40c67f0 0x7f73c41ebcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.572+0000 7f73c9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73c40c67f0 0x7f73c41ebcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56668/0 (socket says 172.21.15.31:56668) 2024-09-11T00:30:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.572+0000 7f73c9200640 1 -- 172.21.15.31:0/20015360 learned_addr learned my addr 172.21.15.31:0/20015360 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73c9c00640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73c40a8bb0 0x7f73c41503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73c9200640 1 -- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73c40a8bb0 msgr2=0x7f73c41503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73c9200640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73c40a8bb0 0x7f73c41503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73c9200640 1 -- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73c41578e0 msgr2=0x7f73c4154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73c9200640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73c41578e0 0x7f73c4154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73c9200640 1 -- 172.21.15.31:0/20015360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c4154070 con 0x7f73c40c67f0 2024-09-11T00:30:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73c9200640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73c40c67f0 0x7f73c41ebcc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f73b0009920 tx=0x7f73b0032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73c1600640 1 -- 172.21.15.31:0/20015360 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b0040070 con 0x7f73c40c67f0 2024-09-11T00:30:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73c1600640 1 -- 172.21.15.31:0/20015360 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73b0005310 con 0x7f73c40c67f0 2024-09-11T00:30:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73c1600640 1 -- 172.21.15.31:0/20015360 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b0005c90 con 0x7f73c40c67f0 2024-09-11T00:30:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73cb000640 1 -- 172.21.15.31:0/20015360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c41510b0 con 0x7f73c40c67f0 2024-09-11T00:30:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.573+0000 7f73cb000640 1 -- 172.21.15.31:0/20015360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c41ec200 con 0x7f73c40c67f0 2024-09-11T00:30:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.574+0000 7f73cb000640 1 -- 172.21.15.31:0/20015360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c40c6bd0 con 0x7f73c40c67f0 2024-09-11T00:30:55.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.575+0000 7f73c1600640 1 -- 172.21.15.31:0/20015360 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f73b00049b0 con 0x7f73c40c67f0 2024-09-11T00:30:55.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.577+0000 7f73c1600640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f73a4077f70 0x7f73a407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.577+0000 7f73c1600640 1 -- 172.21.15.31:0/20015360 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f73b00bb6d0 con 0x7f73c40c67f0 2024-09-11T00:30:55.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.577+0000 7f73c9c00640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f73a4077f70 0x7f73a407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:55.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.578+0000 7f73c9c00640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f73a4077f70 0x7f73a407a430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f73c40aef40 tx=0x7f73b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:55.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.581+0000 7f73c1600640 1 -- 172.21.15.31:0/20015360 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f73c4154070 con 0x7f73c40c67f0 2024-09-11T00:30:55.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.779+0000 7f73cb000640 1 -- 172.21.15.31:0/20015360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73c4150e40 con 0x7f73c40c67f0 2024-09-11T00:30:55.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.780+0000 7f73c1600640 1 -- 172.21.15.31:0/20015360 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20396 (secure 0 0 0) 0x7f73b0083e90 con 0x7f73c40c67f0 2024-09-11T00:30:55.782 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:55.782 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:54.412004+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:30:55.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 -- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f73a4077f70 msgr2=0x7f73a407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f73a4077f70 0x7f73a407a430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f73c40aef40 tx=0x7f73b8009040 comp rx=0 tx=0).stop 2024-09-11T00:30:55.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 -- 172.21.15.31:0/20015360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73c40c67f0 msgr2=0x7f73c41ebcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73c40c67f0 0x7f73c41ebcc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f73b0009920 tx=0x7f73b0032c10 comp rx=0 tx=0).stop 2024-09-11T00:30:55.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 -- 172.21.15.31:0/20015360 shutdown_connections 2024-09-11T00:30:55.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f73a4077f70 0x7f73a407a430 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73c41578e0 0x7f73c4154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73c40c67f0 0x7f73c41ebcc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 --2- 172.21.15.31:0/20015360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73c40a8bb0 0x7f73c41503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 -- 172.21.15.31:0/20015360 >> 172.21.15.31:0/20015360 conn(0x7f73c40c9af0 msgr2=0x7f73c40c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 -- 172.21.15.31:0/20015360 shutdown_connections 2024-09-11T00:30:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.783+0000 7f73cb000640 1 -- 172.21.15.31:0/20015360 wait complete. 2024-09-11T00:30:55.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b standby_count_wanted 0 2024-09-11T00:30:55.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.963+0000 7fbfbdc00640 1 -- 172.21.15.31:0/3411876718 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfb80ad5a0 msgr2=0x7fbfb8151320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.963+0000 7fbfbdc00640 1 --2- 172.21.15.31:0/3411876718 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfb80ad5a0 0x7fbfb8151320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0005fd0 tx=0x7fbfb002a820 comp rx=0 tx=0).stop 2024-09-11T00:30:55.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.964+0000 7fbfbdc00640 1 -- 172.21.15.31:0/3411876718 shutdown_connections 2024-09-11T00:30:55.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.964+0000 7fbfbdc00640 1 --2- 172.21.15.31:0/3411876718 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfb80ad5a0 0x7fbfb8151320 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.964+0000 7fbfbdc00640 1 --2- 172.21.15.31:0/3411876718 >> v2:172.21.15.204:3300/0 conn(0x7fbfb80aab00 0x7fbfb8150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.964+0000 7fbfbdc00640 1 -- 172.21.15.31:0/3411876718 >> 172.21.15.31:0/3411876718 conn(0x7fbfb806caf0 msgr2=0x7fbfb806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:55.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.964+0000 7fbfbdc00640 1 -- 172.21.15.31:0/3411876718 shutdown_connections 2024-09-11T00:30:55.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.964+0000 7fbfbdc00640 1 -- 172.21.15.31:0/3411876718 wait complete. 2024-09-11T00:30:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.965+0000 7fbfbdc00640 1 Processor -- start 2024-09-11T00:30:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.966+0000 7fbfbdc00640 1 -- start start 2024-09-11T00:30:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfbdc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb80aab00 0x7fbfb80c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfbdc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfb80ad5a0 0x7fbfb8149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfbdc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfb80c23a0 0x7fbfb81495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfbdc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb80aaee0 con 0x7fbfb80aab00 2024-09-11T00:30:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfbdc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb80704c0 con 0x7fbfb80c23a0 2024-09-11T00:30:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfbdc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb80ab060 con 0x7fbfb80ad5a0 2024-09-11T00:30:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfb7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb80aab00 0x7fbfb80c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfb7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb80aab00 0x7fbfb80c1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56680/0 (socket says 172.21.15.31:56680) 2024-09-11T00:30:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfb7e00640 1 -- 172.21.15.31:0/803640485 learned_addr learned my addr 172.21.15.31:0/803640485 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfbd200640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfb80c23a0 0x7fbfb81495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.967+0000 7fbfb7e00640 1 -- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfb80ad5a0 msgr2=0x7fbfb8149090 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.968+0000 7fbfb7e00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfb80ad5a0 0x7fbfb8149090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.968+0000 7fbfb7e00640 1 -- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfb80c23a0 msgr2=0x7fbfb81495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.968+0000 7fbfb7e00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfb80c23a0 0x7fbfb81495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.968+0000 7fbfb7e00640 1 -- 172.21.15.31:0/803640485 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb8151f90 con 0x7fbfb80aab00 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.968+0000 7fbfb7e00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb80aab00 0x7fbfb80c1a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800e9c0 tx=0x7fbfa800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.968+0000 7fbfb4c00640 1 -- 172.21.15.31:0/803640485 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa800cde0 con 0x7fbfb80aab00 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.968+0000 7fbfb4c00640 1 -- 172.21.15.31:0/803640485 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfa8004020 con 0x7fbfb80aab00 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.968+0000 7fbfb4c00640 1 -- 172.21.15.31:0/803640485 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8004450 con 0x7fbfb80aab00 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.968+0000 7fbfbdc00640 1 -- 172.21.15.31:0/803640485 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb80c7550 con 0x7fbfb80aab00 2024-09-11T00:30:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.968+0000 7fbfbdc00640 1 -- 172.21.15.31:0/803640485 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb81536a0 con 0x7fbfb80aab00 2024-09-11T00:30:55.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.969+0000 7fbfbdc00640 1 -- 172.21.15.31:0/803640485 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfb8150de0 con 0x7fbfb80aab00 2024-09-11T00:30:55.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.970+0000 7fbfb4c00640 1 -- 172.21.15.31:0/803640485 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbfa80035f0 con 0x7fbfb80aab00 2024-09-11T00:30:55.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.971+0000 7fbfb4c00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbf94077fa0 0x7fbf9407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:55.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.971+0000 7fbfb4c00640 1 -- 172.21.15.31:0/803640485 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fbfa809aad0 con 0x7fbfb80aab00 2024-09-11T00:30:55.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.972+0000 7fbfafe00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbf94077fa0 0x7fbf9407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:55.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.973+0000 7fbfafe00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbf94077fa0 0x7fbf9407a460 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8150d40 tx=0x7fbfa00079d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:55.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:55.976+0000 7fbfb4c00640 1 -- 172.21.15.31:0/803640485 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbfa8063290 con 0x7fbfb80aab00 2024-09-11T00:30:56.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.211+0000 7fbfbdc00640 1 -- 172.21.15.31:0/803640485 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fbfb814b750 con 0x7fbfb80aab00 2024-09-11T00:30:56.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.995+0000 7fbfb4c00640 1 -- 172.21.15.31:0/803640485 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"}]=0 v13) ==== 113+0+0 (secure 0 0 0) 0x7fbfa8062b90 con 0x7fbfb80aab00 2024-09-11T00:30:57.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 -- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbf94077fa0 msgr2=0x7fbf9407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:57.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbf94077fa0 0x7fbf9407a460 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8150d40 tx=0x7fbfa00079d0 comp rx=0 tx=0).stop 2024-09-11T00:30:57.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 -- 172.21.15.31:0/803640485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb80aab00 msgr2=0x7fbfb80c1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:57.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb80aab00 0x7fbfb80c1a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800e9c0 tx=0x7fbfa800ee80 comp rx=0 tx=0).stop 2024-09-11T00:30:57.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 -- 172.21.15.31:0/803640485 shutdown_connections 2024-09-11T00:30:57.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbf94077fa0 0x7fbf9407a460 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbfb80c23a0 0x7fbfb81495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbfb80ad5a0 0x7fbfb8149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 --2- 172.21.15.31:0/803640485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb80aab00 0x7fbfb80c1a10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 -- 172.21.15.31:0/803640485 >> 172.21.15.31:0/803640485 conn(0x7fbfb806caf0 msgr2=0x7fbfb80c2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 -- 172.21.15.31:0/803640485 shutdown_connections 2024-09-11T00:30:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:56.998+0000 7fbfbdc00640 1 -- 172.21.15.31:0/803640485 wait complete. 2024-09-11T00:30:57.017 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-09-11T00:30:57.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b allow_standby_replay false 2024-09-11T00:30:57.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.183+0000 7fe350a00640 1 -- 172.21.15.31:0/2006343837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c3590 msgr2=0x7fe34c072490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:57.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.183+0000 7fe350a00640 1 --2- 172.21.15.31:0/2006343837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c3590 0x7fe34c072490 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0c5520 tx=0x7fe33802b300 comp rx=0 tx=0).stop 2024-09-11T00:30:57.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.183+0000 7fe350a00640 1 -- 172.21.15.31:0/2006343837 shutdown_connections 2024-09-11T00:30:57.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.183+0000 7fe350a00640 1 --2- 172.21.15.31:0/2006343837 >> v2:172.21.15.204:3300/0 conn(0x7fe34c074450 0x7fe34c0729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:57.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.183+0000 7fe350a00640 1 --2- 172.21.15.31:0/2006343837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c3590 0x7fe34c072490 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:57.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.183+0000 7fe350a00640 1 -- 172.21.15.31:0/2006343837 >> 172.21.15.31:0/2006343837 conn(0x7fe34c06d480 msgr2=0x7fe34c0a9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:57.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.183+0000 7fe350a00640 1 -- 172.21.15.31:0/2006343837 shutdown_connections 2024-09-11T00:30:57.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.183+0000 7fe350a00640 1 -- 172.21.15.31:0/2006343837 wait complete. 2024-09-11T00:30:57.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.184+0000 7fe350a00640 1 Processor -- start 2024-09-11T00:30:57.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.184+0000 7fe350a00640 1 -- start start 2024-09-11T00:30:57.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.185+0000 7fe350a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe34c074450 0x7fe34c1dbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:57.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.185+0000 7fe350a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe34c0c2d30 0x7fe34c14d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:57.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.185+0000 7fe350a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c3590 0x7fe34c1ef910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:57.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.185+0000 7fe350a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c0c3110 con 0x7fe34c0c3590 2024-09-11T00:30:57.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.185+0000 7fe350a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c06de30 con 0x7fe34c074450 2024-09-11T00:30:57.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.185+0000 7fe350a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe34c0c3290 con 0x7fe34c0c2d30 2024-09-11T00:30:57.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.185+0000 7fe34a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c3590 0x7fe34c1ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:57.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.185+0000 7fe34a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c3590 0x7fe34c1ef910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56694/0 (socket says 172.21.15.31:56694) 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.185+0000 7fe34a800640 1 -- 172.21.15.31:0/217254431 learned_addr learned my addr 172.21.15.31:0/217254431 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe34a800640 1 -- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe34c0c2d30 msgr2=0x7fe34c14d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe34a800640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe34c0c2d30 0x7fe34c14d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe34a800640 1 -- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe34c074450 msgr2=0x7fe34c1dbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe34a800640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe34c074450 0x7fe34c1dbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe34a800640 1 -- 172.21.15.31:0/217254431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34c0abc10 con 0x7fe34c0c3590 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe34a800640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c3590 0x7fe34c1ef910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe34400daa0 tx=0x7fe34400df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe342000640 1 -- 172.21.15.31:0/217254431 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344017070 con 0x7fe34c0c3590 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe342000640 1 -- 172.21.15.31:0/217254431 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe344003970 con 0x7fe34c0c3590 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe342000640 1 -- 172.21.15.31:0/217254431 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344003dc0 con 0x7fe34c0c3590 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe350a00640 1 -- 172.21.15.31:0/217254431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe34c1f0600 con 0x7fe34c0c3590 2024-09-11T00:30:57.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.186+0000 7fe350a00640 1 -- 172.21.15.31:0/217254431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe34c0aafa0 con 0x7fe34c0c3590 2024-09-11T00:30:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.187+0000 7fe350a00640 1 -- 172.21.15.31:0/217254431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe34c0aa640 con 0x7fe34c0c3590 2024-09-11T00:30:57.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.189+0000 7fe342000640 1 -- 172.21.15.31:0/217254431 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe344003f60 con 0x7fe34c0c3590 2024-09-11T00:30:57.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.189+0000 7fe342000640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe32c077f70 0x7fe32c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:57.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.190+0000 7fe342000640 1 -- 172.21.15.31:0/217254431 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fe34409a5d0 con 0x7fe34c0c3590 2024-09-11T00:30:57.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.190+0000 7fe349e00640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe32c077f70 0x7fe32c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:57.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.191+0000 7fe349e00640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe32c077f70 0x7fe32c07a430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe33c000f60 tx=0x7fe33c0079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:57.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.194+0000 7fe342000640 1 -- 172.21.15.31:0/217254431 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe34c0aa640 con 0x7fe34c0c3590 2024-09-11T00:30:57.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.428+0000 7fe350a00640 1 -- 172.21.15.31:0/217254431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe34c0a7870 con 0x7fe34c0c3590 2024-09-11T00:30:58.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:57.998+0000 7fe342000640 1 -- 172.21.15.31:0/217254431 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"}]=0 v14) ==== 117+0+0 (secure 0 0 0) 0x7fe34c0a7870 con 0x7fe34c0c3590 2024-09-11T00:30:58.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.001+0000 7fe350a00640 1 -- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe32c077f70 msgr2=0x7fe32c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:58.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.001+0000 7fe350a00640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe32c077f70 0x7fe32c07a430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe33c000f60 tx=0x7fe33c0079b0 comp rx=0 tx=0).stop 2024-09-11T00:30:58.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.001+0000 7fe350a00640 1 -- 172.21.15.31:0/217254431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c3590 msgr2=0x7fe34c1ef910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:58.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.001+0000 7fe350a00640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c3590 0x7fe34c1ef910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe34400daa0 tx=0x7fe34400df60 comp rx=0 tx=0).stop 2024-09-11T00:30:58.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.001+0000 7fe350a00640 1 -- 172.21.15.31:0/217254431 shutdown_connections 2024-09-11T00:30:58.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.001+0000 7fe350a00640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe32c077f70 0x7fe32c07a430 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:58.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.002+0000 7fe350a00640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe34c0c3590 0x7fe34c1ef910 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:58.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.002+0000 7fe350a00640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe34c0c2d30 0x7fe34c14d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:58.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.002+0000 7fe350a00640 1 --2- 172.21.15.31:0/217254431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe34c074450 0x7fe34c1dbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:58.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.002+0000 7fe350a00640 1 -- 172.21.15.31:0/217254431 >> 172.21.15.31:0/217254431 conn(0x7fe34c06d480 msgr2=0x7fe34c14e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:58.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.002+0000 7fe350a00640 1 -- 172.21.15.31:0/217254431 shutdown_connections 2024-09-11T00:30:58.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.002+0000 7fe350a00640 1 -- 172.21.15.31:0/217254431 wait complete. 2024-09-11T00:30:58.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b session_timeout 300 2024-09-11T00:30:58.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.185+0000 7f810e000640 1 -- 172.21.15.31:0/1641544773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80f0003a80 msgr2=0x7f80f0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.185+0000 7f810e000640 1 --2- 172.21.15.31:0/1641544773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80f0003a80 0x7f80f0003ea0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f81080adb20 tx=0x7f80f0005830 comp rx=0 tx=0).stop 2024-09-11T00:30:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.185+0000 7f810d600640 1 -- 172.21.15.31:0/1641544773 reap_dead start 2024-09-11T00:30:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.185+0000 7f810e000640 1 -- 172.21.15.31:0/1641544773 shutdown_connections 2024-09-11T00:30:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.185+0000 7f810e000640 1 -- 172.21.15.31:0/1641544773 >> 172.21.15.31:0/1641544773 conn(0x7f81080c78c0 msgr2=0x7f81080c7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.185+0000 7f810e000640 1 -- 172.21.15.31:0/1641544773 shutdown_connections 2024-09-11T00:30:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.186+0000 7f810e000640 1 -- 172.21.15.31:0/1641544773 wait complete. 2024-09-11T00:30:58.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.187+0000 7f810e000640 1 Processor -- start 2024-09-11T00:30:58.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.187+0000 7f810e000640 1 -- start start 2024-09-11T00:30:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f810e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80f0003a80 0x7f81080733f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f810e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81081580c0 0x7f81081de570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f810e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81081e9f80 0x7f81081edab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f810e000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f81080c6640 con 0x7f81081580c0 2024-09-11T00:30:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f810e000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f81080a7240 con 0x7f81081e9f80 2024-09-11T00:30:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f810e000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f81080ad390 con 0x7f80f0003a80 2024-09-11T00:30:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f8107e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81081580c0 0x7f81081de570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f8107e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81081580c0 0x7f81081de570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37120/0 (socket says 172.21.15.31:37120) 2024-09-11T00:30:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f8107e00640 1 -- 172.21.15.31:0/1168627241 learned_addr learned my addr 172.21.15.31:0/1168627241 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:58.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f810d600640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81081e9f80 0x7f81081edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:58.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.188+0000 7f810cc00640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80f0003a80 0x7f81080733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:58.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f8107e00640 1 -- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80f0003a80 msgr2=0x7f81080733f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f8107e00640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80f0003a80 0x7f81080733f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f8107e00640 1 -- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81081e9f80 msgr2=0x7f81081edab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f8107e00640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81081e9f80 0x7f81081edab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f8107e00640 1 -- 172.21.15.31:0/1168627241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81080bd6c0 con 0x7f81081580c0 2024-09-11T00:30:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f8107e00640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81081580c0 0x7f81081de570 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f80f40063b0 tx=0x7f80f40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f8104c00640 1 -- 172.21.15.31:0/1168627241 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4015070 con 0x7f81081580c0 2024-09-11T00:30:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f8104c00640 1 -- 172.21.15.31:0/1168627241 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80f4011900 con 0x7f81081580c0 2024-09-11T00:30:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f8104c00640 1 -- 172.21.15.31:0/1168627241 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4011d50 con 0x7f81081580c0 2024-09-11T00:30:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f810e000640 1 -- 172.21.15.31:0/1168627241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81080bd9a0 con 0x7f81081580c0 2024-09-11T00:30:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.189+0000 7f810e000640 1 -- 172.21.15.31:0/1168627241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f810814edb0 con 0x7f81081580c0 2024-09-11T00:30:58.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.190+0000 7f810e000640 1 -- 172.21.15.31:0/1168627241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8108153a30 con 0x7f81081580c0 2024-09-11T00:30:58.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.191+0000 7f8104c00640 1 -- 172.21.15.31:0/1168627241 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f80f40103c0 con 0x7f81081580c0 2024-09-11T00:30:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.193+0000 7f8104c00640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f80e8077f70 0x7f80e807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.193+0000 7f8104c00640 1 -- 172.21.15.31:0/1168627241 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f80f409b100 con 0x7f81081580c0 2024-09-11T00:30:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.193+0000 7f810cc00640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f80e8077f70 0x7f80e807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.194+0000 7f810cc00640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f80e8077f70 0x7f80e807a430 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f80fc036000 tx=0x7f80fc0046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:58.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.197+0000 7f8104c00640 1 -- 172.21.15.31:0/1168627241 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8108153a30 con 0x7f81081580c0 2024-09-11T00:30:58.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:58.434+0000 7f810e000640 1 -- 172.21.15.31:0/1168627241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"} v 0) -- 0x7f81080bf4e0 con 0x7f81081580c0 2024-09-11T00:30:59.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.000+0000 7f8104c00640 1 -- 172.21.15.31:0/1168627241 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"}]=0 v15) ==== 110+0+0 (secure 0 0 0) 0x7f81080bf4e0 con 0x7f81081580c0 2024-09-11T00:30:59.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.003+0000 7f810e000640 1 -- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f80e8077f70 msgr2=0x7f80e807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.003+0000 7f810e000640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f80e8077f70 0x7f80e807a430 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f80fc036000 tx=0x7f80fc0046c0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.003+0000 7f810e000640 1 -- 172.21.15.31:0/1168627241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81081580c0 msgr2=0x7f81081de570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.003+0000 7f810e000640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81081580c0 0x7f81081de570 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f80f40063b0 tx=0x7f80f40039c0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.003+0000 7f810e000640 1 -- 172.21.15.31:0/1168627241 shutdown_connections 2024-09-11T00:30:59.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.003+0000 7f810e000640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f80e8077f70 0x7f80e807a430 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.003+0000 7f810e000640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f81081e9f80 0x7f81081edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.003+0000 7f810e000640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f81081580c0 0x7f81081de570 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.003+0000 7f810e000640 1 --2- 172.21.15.31:0/1168627241 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f80f0003a80 0x7f81080733f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.003+0000 7f810e000640 1 -- 172.21.15.31:0/1168627241 >> 172.21.15.31:0/1168627241 conn(0x7f81080c78c0 msgr2=0x7f8108150700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:59.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.004+0000 7f810e000640 1 -- 172.21.15.31:0/1168627241 shutdown_connections 2024-09-11T00:30:59.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.004+0000 7f810e000640 1 -- 172.21.15.31:0/1168627241 wait complete. 2024-09-11T00:30:59.024 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'b' 2024-09-11T00:30:59.025 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:30:59.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.186+0000 7f63c7a00640 1 -- 172.21.15.31:0/713504640 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63c0072480 msgr2=0x7f63c01511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.186+0000 7f63c7a00640 1 --2- 172.21.15.31:0/713504640 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63c0072480 0x7f63c01511a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009830 tx=0x7f63b402a5a0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.186+0000 7f63c7a00640 1 -- 172.21.15.31:0/713504640 shutdown_connections 2024-09-11T00:30:59.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.186+0000 7f63c7a00640 1 --2- 172.21.15.31:0/713504640 >> v2:172.21.15.204:3300/0 conn(0x7f63c00c23a0 0x7f63c01516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.186+0000 7f63c7a00640 1 --2- 172.21.15.31:0/713504640 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63c0072480 0x7f63c01511a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.186+0000 7f63c7a00640 1 -- 172.21.15.31:0/713504640 >> 172.21.15.31:0/713504640 conn(0x7f63c00a9750 msgr2=0x7f63c00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:59.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.186+0000 7f63c7a00640 1 -- 172.21.15.31:0/713504640 shutdown_connections 2024-09-11T00:30:59.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.186+0000 7f63c7a00640 1 -- 172.21.15.31:0/713504640 wait complete. 2024-09-11T00:30:59.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.188+0000 7f63c7a00640 1 Processor -- start 2024-09-11T00:30:59.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.188+0000 7f63c7a00640 1 -- start start 2024-09-11T00:30:59.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63c7a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c0072480 0x7f63c00c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63c7a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63c00c23a0 0x7f63c01e1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63c7a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63c0153f20 0x7f63c01ebcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63c7a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f63c0074200 con 0x7f63c0072480 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63c7a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f63c0074080 con 0x7f63c00c23a0 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63c7a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f63c0072860 con 0x7f63c0153f20 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63c5000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c0072480 0x7f63c00c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63c5000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c0072480 0x7f63c00c3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37126/0 (socket says 172.21.15.31:37126) 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63c5000640 1 -- 172.21.15.31:0/1324620717 learned_addr learned my addr 172.21.15.31:0/1324620717 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63bfe00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63c00c23a0 0x7f63c01e1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.189+0000 7f63c5a00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63c0153f20 0x7f63c01ebcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.190+0000 7f63c5000640 1 -- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63c0153f20 msgr2=0x7f63c01ebcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.190+0000 7f63c5000640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63c0153f20 0x7f63c01ebcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.190+0000 7f63c5000640 1 -- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63c00c23a0 msgr2=0x7f63c01e1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.190+0000 7f63c5000640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63c00c23a0 0x7f63c01e1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.190+0000 7f63c5000640 1 -- 172.21.15.31:0/1324620717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c00c58d0 con 0x7f63c0072480 2024-09-11T00:30:59.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.190+0000 7f63c5000640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c0072480 0x7f63c00c3b60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009950 tx=0x7f63b4032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:59.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.190+0000 7f63bcc00640 1 -- 172.21.15.31:0/1324620717 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b4040070 con 0x7f63c0072480 2024-09-11T00:30:59.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.190+0000 7f63bcc00640 1 -- 172.21.15.31:0/1324620717 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63b40038e0 con 0x7f63c0072480 2024-09-11T00:30:59.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.190+0000 7f63c7a00640 1 -- 172.21.15.31:0/1324620717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c00c45d0 con 0x7f63c0072480 2024-09-11T00:30:59.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.190+0000 7f63bcc00640 1 -- 172.21.15.31:0/1324620717 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b4005c90 con 0x7f63c0072480 2024-09-11T00:30:59.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.191+0000 7f63c7a00640 1 -- 172.21.15.31:0/1324620717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c01e94a0 con 0x7f63c0072480 2024-09-11T00:30:59.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.192+0000 7f63c7a00640 1 -- 172.21.15.31:0/1324620717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c00c43c0 con 0x7f63c0072480 2024-09-11T00:30:59.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.193+0000 7f63bcc00640 1 -- 172.21.15.31:0/1324620717 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f63b4045460 con 0x7f63c0072480 2024-09-11T00:30:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.194+0000 7f63bcc00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f63a0077f70 0x7f63a007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.194+0000 7f63bcc00640 1 -- 172.21.15.31:0/1324620717 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f63b40bb720 con 0x7f63c0072480 2024-09-11T00:30:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.195+0000 7f63bfe00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f63a0077f70 0x7f63a007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:59.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.195+0000 7f63bfe00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f63a0077f70 0x7f63a007a430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f63c00a8db0 tx=0x7f63ac005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:59.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.198+0000 7f63bcc00640 1 -- 172.21.15.31:0/1324620717 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f63b4083890 con 0x7f63c0072480 2024-09-11T00:30:59.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.434+0000 7f63c7a00640 1 -- 172.21.15.31:0/1324620717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63c01511a0 con 0x7f63c0072480 2024-09-11T00:30:59.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.435+0000 7f63bcc00640 1 -- 172.21.15.31:0/1324620717 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f63b40882c0 con 0x7f63c0072480 2024-09-11T00:30:59.437 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:59.437 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:30:59.437 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:30:59.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.437+0000 7f63c7a00640 1 -- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f63a0077f70 msgr2=0x7f63a007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.437+0000 7f63c7a00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f63a0077f70 0x7f63a007a430 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f63c00a8db0 tx=0x7f63ac005f90 comp rx=0 tx=0).stop 2024-09-11T00:30:59.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.437+0000 7f63c7a00640 1 -- 172.21.15.31:0/1324620717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c0072480 msgr2=0x7f63c00c3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.437+0000 7f63c7a00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c0072480 0x7f63c00c3b60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f63b4009950 tx=0x7f63b4032c10 comp rx=0 tx=0).stop 2024-09-11T00:30:59.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.437+0000 7f63c7a00640 1 -- 172.21.15.31:0/1324620717 shutdown_connections 2024-09-11T00:30:59.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.437+0000 7f63c7a00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f63a0077f70 0x7f63a007a430 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.437+0000 7f63c7a00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63c0153f20 0x7f63c01ebcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.437+0000 7f63c7a00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63c00c23a0 0x7f63c01e1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.437+0000 7f63c7a00640 1 --2- 172.21.15.31:0/1324620717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63c0072480 0x7f63c00c3b60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.437+0000 7f63c7a00640 1 -- 172.21.15.31:0/1324620717 >> 172.21.15.31:0/1324620717 conn(0x7f63c00a9750 msgr2=0x7f63c01521e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:59.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.438+0000 7f63c7a00640 1 -- 172.21.15.31:0/1324620717 shutdown_connections 2024-09-11T00:30:59.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.438+0000 7f63c7a00640 1 -- 172.21.15.31:0/1324620717 wait complete. 2024-09-11T00:30:59.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:30:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.611+0000 7fa11b200640 1 -- 172.21.15.31:0/4110771415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa11c06cc10 msgr2=0x7fa11c159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.611+0000 7fa11b200640 1 --2- 172.21.15.31:0/4110771415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa11c06cc10 0x7fa11c159450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa11c08d930 tx=0x7fa10402a560 comp rx=0 tx=0).stop 2024-09-11T00:30:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.611+0000 7fa11b200640 1 -- 172.21.15.31:0/4110771415 shutdown_connections 2024-09-11T00:30:59.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.611+0000 7fa11b200640 1 --2- 172.21.15.31:0/4110771415 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa11c0c9000 0x7fa11c159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.611+0000 7fa11b200640 1 --2- 172.21.15.31:0/4110771415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa11c06cc10 0x7fa11c159450 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.611+0000 7fa11b200640 1 --2- 172.21.15.31:0/4110771415 >> v2:172.21.15.204:3300/0 conn(0x7fa11c14c880 0x7fa11c0c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.611+0000 7fa11b200640 1 -- 172.21.15.31:0/4110771415 >> 172.21.15.31:0/4110771415 conn(0x7fa11c0adb00 msgr2=0x7fa11c0adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:59.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.611+0000 7fa11b200640 1 -- 172.21.15.31:0/4110771415 shutdown_connections 2024-09-11T00:30:59.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.611+0000 7fa11b200640 1 -- 172.21.15.31:0/4110771415 wait complete. 2024-09-11T00:30:59.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.613+0000 7fa11b200640 1 Processor -- start 2024-09-11T00:30:59.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.613+0000 7fa11b200640 1 -- start start 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa11b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c06cc10 0x7fa11c1498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa11b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa11c0c9000 0x7fa11c1dfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa11b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa11c14c880 0x7fa11c148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa11b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c156810 con 0x7fa11c14c880 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa11b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c156530 con 0x7fa11c06cc10 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa11b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa11c14cc60 con 0x7fa11c0c9000 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa11a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa11c14c880 0x7fa11c148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa11a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa11c14c880 0x7fa11c148860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37152/0 (socket says 172.21.15.31:37152) 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa11a800640 1 -- 172.21.15.31:0/503352298 learned_addr learned my addr 172.21.15.31:0/503352298 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa119e00640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c06cc10 0x7fa11c1498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:59.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.614+0000 7fa111400640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa11c0c9000 0x7fa11c1dfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa11a800640 1 -- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa11c0c9000 msgr2=0x7fa11c1dfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa11a800640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa11c0c9000 0x7fa11c1dfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa11a800640 1 -- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c06cc10 msgr2=0x7fa11c1498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa11a800640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c06cc10 0x7fa11c1498e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa11a800640 1 -- 172.21.15.31:0/503352298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11c1578d0 con 0x7fa11c14c880 2024-09-11T00:30:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa11a800640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa11c14c880 0x7fa11c148860 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa11400ead0 tx=0x7fa11400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa112a00640 1 -- 172.21.15.31:0/503352298 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11400cde0 con 0x7fa11c14c880 2024-09-11T00:30:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa112a00640 1 -- 172.21.15.31:0/503352298 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa114004c10 con 0x7fa11c14c880 2024-09-11T00:30:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa112a00640 1 -- 172.21.15.31:0/503352298 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114005020 con 0x7fa11c14c880 2024-09-11T00:30:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa11b200640 1 -- 172.21.15.31:0/503352298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11c1575f0 con 0x7fa11c14c880 2024-09-11T00:30:59.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.615+0000 7fa11b200640 1 -- 172.21.15.31:0/503352298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11c159450 con 0x7fa11c14c880 2024-09-11T00:30:59.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.616+0000 7fa11b200640 1 -- 172.21.15.31:0/503352298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa11c1549d0 con 0x7fa11c14c880 2024-09-11T00:30:59.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.617+0000 7fa112a00640 1 -- 172.21.15.31:0/503352298 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa11400f040 con 0x7fa11c14c880 2024-09-11T00:30:59.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.618+0000 7fa112a00640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa0f8077f70 0x7fa0f807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:30:59.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.618+0000 7fa112a00640 1 -- 172.21.15.31:0/503352298 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fa11409a6d0 con 0x7fa11c14c880 2024-09-11T00:30:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.619+0000 7fa119e00640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa0f8077f70 0x7fa0f807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:30:59.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.619+0000 7fa119e00640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa0f8077f70 0x7fa0f807a430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00bad0 tx=0x7fa10c00b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:30:59.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.623+0000 7fa112a00640 1 -- 172.21.15.31:0/503352298 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa114016030 con 0x7fa11c14c880 2024-09-11T00:30:59.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.820+0000 7fa11b200640 1 -- 172.21.15.31:0/503352298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa11c0c59a0 con 0x7fa11c14c880 2024-09-11T00:30:59.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.821+0000 7fa112a00640 1 -- 172.21.15.31:0/503352298 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20383 (secure 0 0 0) 0x7fa1140627c0 con 0x7fa11c14c880 2024-09-11T00:30:59.823 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:30:59.823 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:55.962415+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:30:59.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 -- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa0f8077f70 msgr2=0x7fa0f807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa0f8077f70 0x7fa0f807a430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00bad0 tx=0x7fa10c00b2e0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 -- 172.21.15.31:0/503352298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa11c14c880 msgr2=0x7fa11c148860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa11c14c880 0x7fa11c148860 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa11400ead0 tx=0x7fa11400ef90 comp rx=0 tx=0).stop 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 -- 172.21.15.31:0/503352298 shutdown_connections 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa0f8077f70 0x7fa0f807a430 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa11c14c880 0x7fa11c148860 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa11c0c9000 0x7fa11c1dfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 --2- 172.21.15.31:0/503352298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa11c06cc10 0x7fa11c1498e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 -- 172.21.15.31:0/503352298 >> 172.21.15.31:0/503352298 conn(0x7fa11c0adb00 msgr2=0x7fa11c0ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 -- 172.21.15.31:0/503352298 shutdown_connections 2024-09-11T00:30:59.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:30:59.824+0000 7fa11b200640 1 -- 172.21.15.31:0/503352298 wait complete. 2024-09-11T00:30:59.841 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T00:30:59.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:31:00.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.002+0000 7f2826a00640 1 -- 172.21.15.31:0/4061864812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f282006e850 msgr2=0x7f2820160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.002+0000 7f2826a00640 1 --2- 172.21.15.31:0/4061864812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f282006e850 0x7f2820160860 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2820153d20 tx=0x7f281802aa90 comp rx=0 tx=0).stop 2024-09-11T00:31:00.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.002+0000 7f2826a00640 1 -- 172.21.15.31:0/4061864812 shutdown_connections 2024-09-11T00:31:00.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.002+0000 7f2826a00640 1 --2- 172.21.15.31:0/4061864812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f282006e850 0x7f2820160860 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.002+0000 7f2826a00640 1 --2- 172.21.15.31:0/4061864812 >> v2:172.21.15.204:3300/0 conn(0x7f2820150020 0x7f28200ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.002+0000 7f2826a00640 1 -- 172.21.15.31:0/4061864812 >> 172.21.15.31:0/4061864812 conn(0x7f2820068f10 msgr2=0x7f28200cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:00.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.002+0000 7f2826a00640 1 -- 172.21.15.31:0/4061864812 shutdown_connections 2024-09-11T00:31:00.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.002+0000 7f2826a00640 1 -- 172.21.15.31:0/4061864812 wait complete. 2024-09-11T00:31:00.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.003+0000 7f2826a00640 1 Processor -- start 2024-09-11T00:31:00.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.004+0000 7f2826a00640 1 -- start start 2024-09-11T00:31:00.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.004+0000 7f2826a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f282006e850 0x7f282014a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.004+0000 7f2826a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28200ada20 0x7f28200ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2826a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2820150020 0x7f28201e15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2826a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2820157550 con 0x7f28200ada20 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2826a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28201573d0 con 0x7f282006e850 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2826a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2820156960 con 0x7f2820150020 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2824c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28200ada20 0x7f28200ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2824c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28200ada20 0x7f28200ae680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37166/0 (socket says 172.21.15.31:37166) 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2824c00640 1 -- 172.21.15.31:0/3591171457 learned_addr learned my addr 172.21.15.31:0/3591171457 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2825600640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f282006e850 0x7f282014a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2826000640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2820150020 0x7f28201e15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2824c00640 1 -- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2820150020 msgr2=0x7f28201e15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2824c00640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2820150020 0x7f28201e15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2824c00640 1 -- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f282006e850 msgr2=0x7f282014a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2824c00640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f282006e850 0x7f282014a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.005+0000 7f2824c00640 1 -- 172.21.15.31:0/3591171457 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282014ac80 con 0x7f28200ada20 2024-09-11T00:31:00.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.006+0000 7f2824c00640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28200ada20 0x7f28200ae680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f280c00c4c0 tx=0x7f280c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:00.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.008+0000 7f281d600640 1 -- 172.21.15.31:0/3591171457 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c0052d0 con 0x7f28200ada20 2024-09-11T00:31:00.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.008+0000 7f281d600640 1 -- 172.21.15.31:0/3591171457 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f280c005490 con 0x7f28200ada20 2024-09-11T00:31:00.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.008+0000 7f2826a00640 1 -- 172.21.15.31:0/3591171457 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f282014b0d0 con 0x7f28200ada20 2024-09-11T00:31:00.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.008+0000 7f281d600640 1 -- 172.21.15.31:0/3591171457 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c012890 con 0x7f28200ada20 2024-09-11T00:31:00.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.009+0000 7f2826a00640 1 -- 172.21.15.31:0/3591171457 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28201ef900 con 0x7f28200ada20 2024-09-11T00:31:00.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.010+0000 7f2826a00640 1 -- 172.21.15.31:0/3591171457 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28200cc220 con 0x7f28200ada20 2024-09-11T00:31:00.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.011+0000 7f281d600640 1 -- 172.21.15.31:0/3591171457 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f280c012a30 con 0x7f28200ada20 2024-09-11T00:31:00.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.012+0000 7f281d600640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2800077f70 0x7f280007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.012+0000 7f281d600640 1 -- 172.21.15.31:0/3591171457 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f280c099b90 con 0x7f28200ada20 2024-09-11T00:31:00.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.013+0000 7f2825600640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2800077f70 0x7f280007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.013+0000 7f2825600640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2800077f70 0x7f280007a430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f28140095f0 tx=0x7f2814005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:00.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.016+0000 7f281d600640 1 -- 172.21.15.31:0/3591171457 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f280c061bf0 con 0x7f28200ada20 2024-09-11T00:31:00.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.206+0000 7f2826a00640 1 -- 172.21.15.31:0/3591171457 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f28200cc8e0 con 0x7f2800077f70 2024-09-11T00:31:00.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.217+0000 7f281d600640 1 -- 172.21.15.31:0/3591171457 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+480571 (secure 0 0 0) 0x7f28200cc8e0 con 0x7f2800077f70 2024-09-11T00:31:00.225 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:00.228 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:31:00.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.223+0000 7f2826a00640 1 -- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2800077f70 msgr2=0x7f280007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.223+0000 7f2826a00640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2800077f70 0x7f280007a430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f28140095f0 tx=0x7f2814005f90 comp rx=0 tx=0).stop 2024-09-11T00:31:00.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.223+0000 7f2826a00640 1 -- 172.21.15.31:0/3591171457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28200ada20 msgr2=0x7f28200ae680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.223+0000 7f2826a00640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28200ada20 0x7f28200ae680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f280c00c4c0 tx=0x7f280c00c980 comp rx=0 tx=0).stop 2024-09-11T00:31:00.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.224+0000 7f2826a00640 1 -- 172.21.15.31:0/3591171457 shutdown_connections 2024-09-11T00:31:00.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.224+0000 7f2826a00640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2800077f70 0x7f280007a430 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.224+0000 7f2826a00640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2820150020 0x7f28201e15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.224+0000 7f2826a00640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28200ada20 0x7f28200ae680 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.224+0000 7f2826a00640 1 --2- 172.21.15.31:0/3591171457 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f282006e850 0x7f282014a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.224+0000 7f2826a00640 1 -- 172.21.15.31:0/3591171457 >> 172.21.15.31:0/3591171457 conn(0x7f2820068f10 msgr2=0x7f282014f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.224+0000 7f2826a00640 1 -- 172.21.15.31:0/3591171457 shutdown_connections 2024-09-11T00:31:00.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.224+0000 7f2826a00640 1 -- 172.21.15.31:0/3591171457 wait complete. 2024-09-11T00:31:00.241 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-09-11T00:30:58.394703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":282,"acting":282,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":234,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219328,"kb_used_data":3100,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755392,"statfs":{"total":773094113280,"available":772869521408,"internally_reserved":0,"allocated":3174400,"data_stored":1871710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.440171"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970097+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:30:55.970097+0000","last_peered":"2024-09-11T00:30:55.970097+0000","last_clean":"2024-09-11T00:30:55.970097+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:30:55.970097+0000","last_undegraded":"2024-09-11T00:30:55.970097+0000","last_fullsized":"2024-09-11T00:30:55.970097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965322+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:30:52.965322+0000","last_peered":"2024-09-11T00:30:52.965322+0000","last_clean":"2024-09-11T00:30:52.965322+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:30:52.965322+0000","last_undegraded":"2024-09-11T00:30:52.965322+0000","last_fullsized":"2024-09-11T00:30:52.965322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.962462+0000","last_change":"2024-09-11T00:30:52.962462+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.962462+0000","last_undegraded":"2024-09-11T00:30:52.962462+0000","last_fullsized":"2024-09-11T00:30:52.962462+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968662+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:30:55.968662+0000","last_peered":"2024-09-11T00:30:55.968662+0000","last_clean":"2024-09-11T00:30:55.968662+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:30:55.968662+0000","last_undegraded":"2024-09-11T00:30:55.968662+0000","last_fullsized":"2024-09-11T00:30:55.968662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965506+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:30:52.965506+0000","last_peered":"2024-09-11T00:30:52.965506+0000","last_clean":"2024-09-11T00:30:52.965506+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:30:52.965506+0000","last_undegraded":"2024-09-11T00:30:52.965506+0000","last_fullsized":"2024-09-11T00:30:52.965506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964727+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:30:52.964727+0000","last_peered":"2024-09-11T00:30:52.964727+0000","last_clean":"2024-09-11T00:30:52.964727+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:30:52.964727+0000","last_undegraded":"2024-09-11T00:30:52.964727+0000","last_fullsized":"2024-09-11T00:30:52.964727+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.957411+0000","last_change":"2024-09-11T00:30:52.957411+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.957411+0000","last_undegraded":"2024-09-11T00:30:52.957411+0000","last_fullsized":"2024-09-11T00:30:52.957411+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965578+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:30:52.965578+0000","last_peered":"2024-09-11T00:30:52.965578+0000","last_clean":"2024-09-11T00:30:52.965578+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:30:52.965578+0000","last_undegraded":"2024-09-11T00:30:52.965578+0000","last_fullsized":"2024-09-11T00:30:52.965578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963168+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:30:52.963168+0000","last_peered":"2024-09-11T00:30:52.963168+0000","last_clean":"2024-09-11T00:30:52.963168+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:30:52.963168+0000","last_undegraded":"2024-09-11T00:30:52.963168+0000","last_fullsized":"2024-09-11T00:30:52.963168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.958372+0000","last_change":"2024-09-11T00:30:52.958372+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.958372+0000","last_undegraded":"2024-09-11T00:30:52.958372+0000","last_fullsized":"2024-09-11T00:30:52.958372+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963191+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:30:52.963191+0000","last_peered":"2024-09-11T00:30:52.963191+0000","last_clean":"2024-09-11T00:30:52.963191+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:30:52.963191+0000","last_undegraded":"2024-09-11T00:30:52.963191+0000","last_fullsized":"2024-09-11T00:30:52.963191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965827+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:30:52.965827+0000","last_peered":"2024-09-11T00:30:52.965827+0000","last_clean":"2024-09-11T00:30:52.965827+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:30:52.965827+0000","last_undegraded":"2024-09-11T00:30:52.965827+0000","last_fullsized":"2024-09-11T00:30:52.965827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.958240+0000","last_change":"2024-09-11T00:30:52.958240+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.958240+0000","last_undegraded":"2024-09-11T00:30:52.958240+0000","last_fullsized":"2024-09-11T00:30:52.958240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963354+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:30:52.963354+0000","last_peered":"2024-09-11T00:30:52.963354+0000","last_clean":"2024-09-11T00:30:52.963354+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:30:52.963354+0000","last_undegraded":"2024-09-11T00:30:52.963354+0000","last_fullsized":"2024-09-11T00:30:52.963354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963918+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:30:52.963918+0000","last_peered":"2024-09-11T00:30:52.963918+0000","last_clean":"2024-09-11T00:30:52.963918+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:30:52.963918+0000","last_undegraded":"2024-09-11T00:30:52.963918+0000","last_fullsized":"2024-09-11T00:30:52.963918+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.961001+0000","last_change":"2024-09-11T00:30:52.961001+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.961001+0000","last_undegraded":"2024-09-11T00:30:52.961001+0000","last_fullsized":"2024-09-11T00:30:52.961001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963761+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:30:52.963761+0000","last_peered":"2024-09-11T00:30:52.963761+0000","last_clean":"2024-09-11T00:30:52.963761+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:30:52.963761+0000","last_undegraded":"2024-09-11T00:30:52.963761+0000","last_fullsized":"2024-09-11T00:30:52.963761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.960285+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:30:52.960285+0000","last_peered":"2024-09-11T00:30:52.960285+0000","last_clean":"2024-09-11T00:30:52.960285+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:30:52.960285+0000","last_undegraded":"2024-09-11T00:30:52.960285+0000","last_fullsized":"2024-09-11T00:30:52.960285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.960164+0000","last_change":"2024-09-11T00:30:52.960164+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.960164+0000","last_undegraded":"2024-09-11T00:30:52.960164+0000","last_fullsized":"2024-09-11T00:30:52.960164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968190+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:30:55.968190+0000","last_peered":"2024-09-11T00:30:55.968190+0000","last_clean":"2024-09-11T00:30:55.968190+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:30:55.968190+0000","last_undegraded":"2024-09-11T00:30:55.968190+0000","last_fullsized":"2024-09-11T00:30:55.968190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968850+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:30:55.968850+0000","last_peered":"2024-09-11T00:30:55.968850+0000","last_clean":"2024-09-11T00:30:55.968850+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:30:55.968850+0000","last_undegraded":"2024-09-11T00:30:55.968850+0000","last_fullsized":"2024-09-11T00:30:55.968850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.963720+0000","last_change":"2024-09-11T00:30:52.963720+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.963720+0000","last_undegraded":"2024-09-11T00:30:52.963720+0000","last_fullsized":"2024-09-11T00:30:52.963720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.962776+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:30:52.962776+0000","last_peered":"2024-09-11T00:30:52.962776+0000","last_clean":"2024-09-11T00:30:52.962776+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:30:52.962776+0000","last_undegraded":"2024-09-11T00:30:52.962776+0000","last_fullsized":"2024-09-11T00:30:52.962776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.961346+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:30:52.961346+0000","last_peered":"2024-09-11T00:30:52.961346+0000","last_clean":"2024-09-11T00:30:52.961346+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:30:52.961346+0000","last_undegraded":"2024-09-11T00:30:52.961346+0000","last_fullsized":"2024-09-11T00:30:52.961346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.957379+0000","last_change":"2024-09-11T00:30:52.957379+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.957379+0000","last_undegraded":"2024-09-11T00:30:52.957379+0000","last_fullsized":"2024-09-11T00:30:52.957379+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968462+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:30:55.968462+0000","last_peered":"2024-09-11T00:30:55.968462+0000","last_clean":"2024-09-11T00:30:55.968462+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:30:55.968462+0000","last_undegraded":"2024-09-11T00:30:55.968462+0000","last_fullsized":"2024-09-11T00:30:55.968462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.966172+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:30:52.966172+0000","last_peered":"2024-09-11T00:30:52.966172+0000","last_clean":"2024-09-11T00:30:52.966172+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:30:52.966172+0000","last_undegraded":"2024-09-11T00:30:52.966172+0000","last_fullsized":"2024-09-11T00:30:52.966172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.957759+0000","last_change":"2024-09-11T00:30:52.957759+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.957759+0000","last_undegraded":"2024-09-11T00:30:52.957759+0000","last_fullsized":"2024-09-11T00:30:52.957759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970081+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:30:55.970081+0000","last_peered":"2024-09-11T00:30:55.970081+0000","last_clean":"2024-09-11T00:30:55.970081+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:30:55.970081+0000","last_undegraded":"2024-09-11T00:30:55.970081+0000","last_fullsized":"2024-09-11T00:30:55.970081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970376+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:30:55.970376+0000","last_peered":"2024-09-11T00:30:55.970376+0000","last_clean":"2024-09-11T00:30:55.970376+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:30:55.970376+0000","last_undegraded":"2024-09-11T00:30:55.970376+0000","last_fullsized":"2024-09-11T00:30:55.970376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968131+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:30:55.968131+0000","last_peered":"2024-09-11T00:30:55.968131+0000","last_clean":"2024-09-11T00:30:55.968131+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:30:55.968131+0000","last_undegraded":"2024-09-11T00:30:55.968131+0000","last_fullsized":"2024-09-11T00:30:55.968131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.961501+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:30:52.961501+0000","last_peered":"2024-09-11T00:30:52.961501+0000","last_clean":"2024-09-11T00:30:52.961501+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:30:52.961501+0000","last_undegraded":"2024-09-11T00:30:52.961501+0000","last_fullsized":"2024-09-11T00:30:52.961501+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965981+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:30:52.965981+0000","last_peered":"2024-09-11T00:30:52.965981+0000","last_clean":"2024-09-11T00:30:52.965981+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:30:52.965981+0000","last_undegraded":"2024-09-11T00:30:52.965981+0000","last_fullsized":"2024-09-11T00:30:52.965981+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.958407+0000","last_change":"2024-09-11T00:30:52.958407+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.958407+0000","last_undegraded":"2024-09-11T00:30:52.958407+0000","last_fullsized":"2024-09-11T00:30:52.958407+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969549+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:30:55.969549+0000","last_peered":"2024-09-11T00:30:55.969549+0000","last_clean":"2024-09-11T00:30:55.969549+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:30:55.969549+0000","last_undegraded":"2024-09-11T00:30:55.969549+0000","last_fullsized":"2024-09-11T00:30:55.969549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.955678+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:30:52.955678+0000","last_peered":"2024-09-11T00:30:52.955678+0000","last_clean":"2024-09-11T00:30:52.955678+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:30:52.955678+0000","last_undegraded":"2024-09-11T00:30:52.955678+0000","last_fullsized":"2024-09-11T00:30:52.955678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964293+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:30:52.964293+0000","last_peered":"2024-09-11T00:30:52.964293+0000","last_clean":"2024-09-11T00:30:52.964293+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:30:52.964293+0000","last_undegraded":"2024-09-11T00:30:52.964293+0000","last_fullsized":"2024-09-11T00:30:52.964293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.957866+0000","last_change":"2024-09-11T00:30:52.957866+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.957866+0000","last_undegraded":"2024-09-11T00:30:52.957866+0000","last_fullsized":"2024-09-11T00:30:52.957866+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.962890+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:30:52.962890+0000","last_peered":"2024-09-11T00:30:52.962890+0000","last_clean":"2024-09-11T00:30:52.962890+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:30:52.962890+0000","last_undegraded":"2024-09-11T00:30:52.962890+0000","last_fullsized":"2024-09-11T00:30:52.962890+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964769+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:30:52.964769+0000","last_peered":"2024-09-11T00:30:52.964769+0000","last_clean":"2024-09-11T00:30:52.964769+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:30:52.964769+0000","last_undegraded":"2024-09-11T00:30:52.964769+0000","last_fullsized":"2024-09-11T00:30:52.964769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.957961+0000","last_change":"2024-09-11T00:30:52.957961+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.957961+0000","last_undegraded":"2024-09-11T00:30:52.957961+0000","last_fullsized":"2024-09-11T00:30:52.957961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964243+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:30:52.964243+0000","last_peered":"2024-09-11T00:30:52.964243+0000","last_clean":"2024-09-11T00:30:52.964243+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:30:52.964243+0000","last_undegraded":"2024-09-11T00:30:52.964243+0000","last_fullsized":"2024-09-11T00:30:52.964243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965640+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:30:52.965640+0000","last_peered":"2024-09-11T00:30:52.965640+0000","last_clean":"2024-09-11T00:30:52.965640+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:30:52.965640+0000","last_undegraded":"2024-09-11T00:30:52.965640+0000","last_fullsized":"2024-09-11T00:30:52.965640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.958477+0000","last_change":"2024-09-11T00:30:52.958477+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.958477+0000","last_undegraded":"2024-09-11T00:30:52.958477+0000","last_fullsized":"2024-09-11T00:30:52.958477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.960744+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:30:52.960744+0000","last_peered":"2024-09-11T00:30:52.960744+0000","last_clean":"2024-09-11T00:30:52.960744+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:30:52.960744+0000","last_undegraded":"2024-09-11T00:30:52.960744+0000","last_fullsized":"2024-09-11T00:30:52.960744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963412+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:30:52.963412+0000","last_peered":"2024-09-11T00:30:52.963412+0000","last_clean":"2024-09-11T00:30:52.963412+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:30:52.963412+0000","last_undegraded":"2024-09-11T00:30:52.963412+0000","last_fullsized":"2024-09-11T00:30:52.963412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.961881+0000","last_change":"2024-09-11T00:30:52.961881+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.961881+0000","last_undegraded":"2024-09-11T00:30:52.961881+0000","last_fullsized":"2024-09-11T00:30:52.961881+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969545+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:30:55.969545+0000","last_peered":"2024-09-11T00:30:55.969545+0000","last_clean":"2024-09-11T00:30:55.969545+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:30:55.969545+0000","last_undegraded":"2024-09-11T00:30:55.969545+0000","last_fullsized":"2024-09-11T00:30:55.969545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964011+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:30:52.964011+0000","last_peered":"2024-09-11T00:30:52.964011+0000","last_clean":"2024-09-11T00:30:52.964011+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:30:52.964011+0000","last_undegraded":"2024-09-11T00:30:52.964011+0000","last_fullsized":"2024-09-11T00:30:52.964011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":113,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963075+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:30:52.963075+0000","last_peered":"2024-09-11T00:30:52.963075+0000","last_clean":"2024-09-11T00:30:52.963075+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:30:52.963075+0000","last_undegraded":"2024-09-11T00:30:52.963075+0000","last_fullsized":"2024-09-11T00:30:52.963075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.959856+0000","last_change":"2024-09-11T00:30:52.959856+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.959856+0000","last_undegraded":"2024-09-11T00:30:52.959856+0000","last_fullsized":"2024-09-11T00:30:52.959856+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965875+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:30:52.965875+0000","last_peered":"2024-09-11T00:30:52.965875+0000","last_clean":"2024-09-11T00:30:52.965875+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:30:52.965875+0000","last_undegraded":"2024-09-11T00:30:52.965875+0000","last_fullsized":"2024-09-11T00:30:52.965875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.961615+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:30:52.961615+0000","last_peered":"2024-09-11T00:30:52.961615+0000","last_clean":"2024-09-11T00:30:52.961615+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:30:52.961615+0000","last_undegraded":"2024-09-11T00:30:52.961615+0000","last_fullsized":"2024-09-11T00:30:52.961615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.957447+0000","last_change":"2024-09-11T00:30:52.957447+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.957447+0000","last_undegraded":"2024-09-11T00:30:52.957447+0000","last_fullsized":"2024-09-11T00:30:52.957447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963109+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:30:52.963109+0000","last_peered":"2024-09-11T00:30:52.963109+0000","last_clean":"2024-09-11T00:30:52.963109+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:30:52.963109+0000","last_undegraded":"2024-09-11T00:30:52.963109+0000","last_fullsized":"2024-09-11T00:30:52.963109+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964437+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:30:52.964437+0000","last_peered":"2024-09-11T00:30:52.964437+0000","last_clean":"2024-09-11T00:30:52.964437+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:30:52.964437+0000","last_undegraded":"2024-09-11T00:30:52.964437+0000","last_fullsized":"2024-09-11T00:30:52.964437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.961779+0000","last_change":"2024-09-11T00:30:52.961779+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.961779+0000","last_undegraded":"2024-09-11T00:30:52.961779+0000","last_fullsized":"2024-09-11T00:30:52.961779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965346+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:30:52.965346+0000","last_peered":"2024-09-11T00:30:52.965346+0000","last_clean":"2024-09-11T00:30:52.965346+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:30:52.965346+0000","last_undegraded":"2024-09-11T00:30:52.965346+0000","last_fullsized":"2024-09-11T00:30:52.965346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970314+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:30:55.970314+0000","last_peered":"2024-09-11T00:30:55.970314+0000","last_clean":"2024-09-11T00:30:55.970314+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:30:55.970314+0000","last_undegraded":"2024-09-11T00:30:55.970314+0000","last_fullsized":"2024-09-11T00:30:55.970314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.959852+0000","last_change":"2024-09-11T00:30:52.959852+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.959852+0000","last_undegraded":"2024-09-11T00:30:52.959852+0000","last_fullsized":"2024-09-11T00:30:52.959852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.955811+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:30:52.955811+0000","last_peered":"2024-09-11T00:30:52.955811+0000","last_clean":"2024-09-11T00:30:52.955811+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:30:52.955811+0000","last_undegraded":"2024-09-11T00:30:52.955811+0000","last_fullsized":"2024-09-11T00:30:52.955811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965428+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:30:52.965428+0000","last_peered":"2024-09-11T00:30:52.965428+0000","last_clean":"2024-09-11T00:30:52.965428+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:30:52.965428+0000","last_undegraded":"2024-09-11T00:30:52.965428+0000","last_fullsized":"2024-09-11T00:30:52.965428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.966124+0000","last_change":"2024-09-11T00:30:52.966124+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.966124+0000","last_undegraded":"2024-09-11T00:30:52.966124+0000","last_fullsized":"2024-09-11T00:30:52.966124+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970030+0000","last_change":"2024-09-11T00:30:54.987533+0000","last_active":"2024-09-11T00:30:55.970030+0000","last_peered":"2024-09-11T00:30:55.970030+0000","last_clean":"2024-09-11T00:30:55.970030+0000","last_became_active":"2024-09-11T00:30:54.987253+0000","last_became_peered":"2024-09-11T00:30:54.987253+0000","last_unstale":"2024-09-11T00:30:55.970030+0000","last_undegraded":"2024-09-11T00:30:55.970030+0000","last_fullsized":"2024-09-11T00:30:55.970030+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.923656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.962650+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:30:52.962650+0000","last_peered":"2024-09-11T00:30:52.962650+0000","last_clean":"2024-09-11T00:30:52.962650+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:30:52.962650+0000","last_undegraded":"2024-09-11T00:30:52.962650+0000","last_fullsized":"2024-09-11T00:30:52.962650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968528+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:30:55.968528+0000","last_peered":"2024-09-11T00:30:55.968528+0000","last_clean":"2024-09-11T00:30:55.968528+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:30:55.968528+0000","last_undegraded":"2024-09-11T00:30:55.968528+0000","last_fullsized":"2024-09-11T00:30:55.968528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.962335+0000","last_change":"2024-09-11T00:30:52.962335+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.962335+0000","last_undegraded":"2024-09-11T00:30:52.962335+0000","last_fullsized":"2024-09-11T00:30:52.962335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.962529+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:30:52.962529+0000","last_peered":"2024-09-11T00:30:52.962529+0000","last_clean":"2024-09-11T00:30:52.962529+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:30:52.962529+0000","last_undegraded":"2024-09-11T00:30:52.962529+0000","last_fullsized":"2024-09-11T00:30:52.962529+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.961547+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:30:52.961547+0000","last_peered":"2024-09-11T00:30:52.961547+0000","last_clean":"2024-09-11T00:30:52.961547+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:30:52.961547+0000","last_undegraded":"2024-09-11T00:30:52.961547+0000","last_fullsized":"2024-09-11T00:30:52.961547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.962331+0000","last_change":"2024-09-11T00:30:52.962331+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.962331+0000","last_undegraded":"2024-09-11T00:30:52.962331+0000","last_fullsized":"2024-09-11T00:30:52.962331+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963724+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:30:52.963724+0000","last_peered":"2024-09-11T00:30:52.963724+0000","last_clean":"2024-09-11T00:30:52.963724+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:30:52.963724+0000","last_undegraded":"2024-09-11T00:30:52.963724+0000","last_fullsized":"2024-09-11T00:30:52.963724+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963562+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:30:52.963562+0000","last_peered":"2024-09-11T00:30:52.963562+0000","last_clean":"2024-09-11T00:30:52.963562+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:30:52.963562+0000","last_undegraded":"2024-09-11T00:30:52.963562+0000","last_fullsized":"2024-09-11T00:30:52.963562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.959544+0000","last_change":"2024-09-11T00:30:52.959544+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.959544+0000","last_undegraded":"2024-09-11T00:30:52.959544+0000","last_fullsized":"2024-09-11T00:30:52.959544+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969373+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:30:55.969373+0000","last_peered":"2024-09-11T00:30:55.969373+0000","last_clean":"2024-09-11T00:30:55.969373+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:30:55.969373+0000","last_undegraded":"2024-09-11T00:30:55.969373+0000","last_fullsized":"2024-09-11T00:30:55.969373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.960192+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:30:52.960192+0000","last_peered":"2024-09-11T00:30:52.960192+0000","last_clean":"2024-09-11T00:30:52.960192+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:30:52.960192+0000","last_undegraded":"2024-09-11T00:30:52.960192+0000","last_fullsized":"2024-09-11T00:30:52.960192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963881+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:30:52.963881+0000","last_peered":"2024-09-11T00:30:52.963881+0000","last_clean":"2024-09-11T00:30:52.963881+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:30:52.963881+0000","last_undegraded":"2024-09-11T00:30:52.963881+0000","last_fullsized":"2024-09-11T00:30:52.963881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.960423+0000","last_change":"2024-09-11T00:30:52.960423+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.960423+0000","last_undegraded":"2024-09-11T00:30:52.960423+0000","last_fullsized":"2024-09-11T00:30:52.960423+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969893+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:30:55.969893+0000","last_peered":"2024-09-11T00:30:55.969893+0000","last_clean":"2024-09-11T00:30:55.969893+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:30:55.969893+0000","last_undegraded":"2024-09-11T00:30:55.969893+0000","last_fullsized":"2024-09-11T00:30:55.969893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965466+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:30:52.965466+0000","last_peered":"2024-09-11T00:30:52.965466+0000","last_clean":"2024-09-11T00:30:52.965466+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:30:52.965466+0000","last_undegraded":"2024-09-11T00:30:52.965466+0000","last_fullsized":"2024-09-11T00:30:52.965466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.955610+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:30:52.955610+0000","last_peered":"2024-09-11T00:30:52.955610+0000","last_clean":"2024-09-11T00:30:52.955610+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:30:52.955610+0000","last_undegraded":"2024-09-11T00:30:52.955610+0000","last_fullsized":"2024-09-11T00:30:52.955610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969278+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:30:55.969278+0000","last_peered":"2024-09-11T00:30:55.969278+0000","last_clean":"2024-09-11T00:30:55.969278+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:30:55.969278+0000","last_undegraded":"2024-09-11T00:30:55.969278+0000","last_fullsized":"2024-09-11T00:30:55.969278+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965758+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:30:52.965758+0000","last_peered":"2024-09-11T00:30:52.965758+0000","last_clean":"2024-09-11T00:30:52.965758+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:30:52.965758+0000","last_undegraded":"2024-09-11T00:30:52.965758+0000","last_fullsized":"2024-09-11T00:30:52.965758+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964491+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:30:52.964491+0000","last_peered":"2024-09-11T00:30:52.964491+0000","last_clean":"2024-09-11T00:30:52.964491+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:30:52.964491+0000","last_undegraded":"2024-09-11T00:30:52.964491+0000","last_fullsized":"2024-09-11T00:30:52.964491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.957336+0000","last_change":"2024-09-11T00:30:52.957336+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.957336+0000","last_undegraded":"2024-09-11T00:30:52.957336+0000","last_fullsized":"2024-09-11T00:30:52.957336+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964771+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:30:52.964771+0000","last_peered":"2024-09-11T00:30:52.964771+0000","last_clean":"2024-09-11T00:30:52.964771+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:30:52.964771+0000","last_undegraded":"2024-09-11T00:30:52.964771+0000","last_fullsized":"2024-09-11T00:30:52.964771+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.966107+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:30:52.966107+0000","last_peered":"2024-09-11T00:30:52.966107+0000","last_clean":"2024-09-11T00:30:52.966107+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:30:52.966107+0000","last_undegraded":"2024-09-11T00:30:52.966107+0000","last_fullsized":"2024-09-11T00:30:52.966107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.961391+0000","last_change":"2024-09-11T00:30:52.961391+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.961391+0000","last_undegraded":"2024-09-11T00:30:52.961391+0000","last_fullsized":"2024-09-11T00:30:52.961391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.955773+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:30:52.955773+0000","last_peered":"2024-09-11T00:30:52.955773+0000","last_clean":"2024-09-11T00:30:52.955773+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:30:52.955773+0000","last_undegraded":"2024-09-11T00:30:52.955773+0000","last_fullsized":"2024-09-11T00:30:52.955773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964224+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:30:52.964224+0000","last_peered":"2024-09-11T00:30:52.964224+0000","last_clean":"2024-09-11T00:30:52.964224+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:30:52.964224+0000","last_undegraded":"2024-09-11T00:30:52.964224+0000","last_fullsized":"2024-09-11T00:30:52.964224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968520+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:30:55.968520+0000","last_peered":"2024-09-11T00:30:55.968520+0000","last_clean":"2024-09-11T00:30:55.968520+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:30:55.968520+0000","last_undegraded":"2024-09-11T00:30:55.968520+0000","last_fullsized":"2024-09-11T00:30:55.968520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968811+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:30:55.968811+0000","last_peered":"2024-09-11T00:30:55.968811+0000","last_clean":"2024-09-11T00:30:55.968811+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:30:55.968811+0000","last_undegraded":"2024-09-11T00:30:55.968811+0000","last_fullsized":"2024-09-11T00:30:55.968811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963276+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:30:52.963276+0000","last_peered":"2024-09-11T00:30:52.963276+0000","last_clean":"2024-09-11T00:30:52.963276+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:30:52.963276+0000","last_undegraded":"2024-09-11T00:30:52.963276+0000","last_fullsized":"2024-09-11T00:30:52.963276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.960155+0000","last_change":"2024-09-11T00:30:52.960155+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.960155+0000","last_undegraded":"2024-09-11T00:30:52.960155+0000","last_fullsized":"2024-09-11T00:30:52.960155+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964418+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:30:52.964418+0000","last_peered":"2024-09-11T00:30:52.964418+0000","last_clean":"2024-09-11T00:30:52.964418+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:30:52.964418+0000","last_undegraded":"2024-09-11T00:30:52.964418+0000","last_fullsized":"2024-09-11T00:30:52.964418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.960853+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:30:52.960853+0000","last_peered":"2024-09-11T00:30:52.960853+0000","last_clean":"2024-09-11T00:30:52.960853+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:30:52.960853+0000","last_undegraded":"2024-09-11T00:30:52.960853+0000","last_fullsized":"2024-09-11T00:30:52.960853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969528+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:30:55.969528+0000","last_peered":"2024-09-11T00:30:55.969528+0000","last_clean":"2024-09-11T00:30:55.969528+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:30:55.969528+0000","last_undegraded":"2024-09-11T00:30:55.969528+0000","last_fullsized":"2024-09-11T00:30:55.969528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963448+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:30:52.963448+0000","last_peered":"2024-09-11T00:30:52.963448+0000","last_clean":"2024-09-11T00:30:52.963448+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:30:52.963448+0000","last_undegraded":"2024-09-11T00:30:52.963448+0000","last_fullsized":"2024-09-11T00:30:52.963448+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969812+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:30:55.969812+0000","last_peered":"2024-09-11T00:30:55.969812+0000","last_clean":"2024-09-11T00:30:55.969812+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:30:55.969812+0000","last_undegraded":"2024-09-11T00:30:55.969812+0000","last_fullsized":"2024-09-11T00:30:55.969812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969509+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:30:55.969509+0000","last_peered":"2024-09-11T00:30:55.969509+0000","last_clean":"2024-09-11T00:30:55.969509+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:30:55.969509+0000","last_undegraded":"2024-09-11T00:30:55.969509+0000","last_fullsized":"2024-09-11T00:30:55.969509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964196+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:30:52.964196+0000","last_peered":"2024-09-11T00:30:52.964196+0000","last_clean":"2024-09-11T00:30:52.964196+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:30:52.964196+0000","last_undegraded":"2024-09-11T00:30:52.964196+0000","last_fullsized":"2024-09-11T00:30:52.964196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.962964+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:30:52.962964+0000","last_peered":"2024-09-11T00:30:52.962964+0000","last_clean":"2024-09-11T00:30:52.962964+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:30:52.962964+0000","last_undegraded":"2024-09-11T00:30:52.962964+0000","last_fullsized":"2024-09-11T00:30:52.962964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":23,"state":"creating+peering","last_fresh":"2024-09-11T00:30:52.962614+0000","last_change":"2024-09-11T00:30:52.962614+0000","last_active":"2024-09-11T00:30:52.946095+0000","last_peered":"2024-09-11T00:30:52.946095+0000","last_clean":"2024-09-11T00:30:52.946095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:52.962614+0000","last_undegraded":"2024-09-11T00:30:52.962614+0000","last_fullsized":"2024-09-11T00:30:52.962614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970084+0000","last_change":"2024-09-11T00:30:54.989925+0000","last_active":"2024-09-11T00:30:55.970084+0000","last_peered":"2024-09-11T00:30:55.970084+0000","last_clean":"2024-09-11T00:30:55.970084+0000","last_became_active":"2024-09-11T00:30:54.989401+0000","last_became_peered":"2024-09-11T00:30:54.989401+0000","last_unstale":"2024-09-11T00:30:55.970084+0000","last_undegraded":"2024-09-11T00:30:55.970084+0000","last_fullsized":"2024-09-11T00:30:55.970084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:14.267175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.955319+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:30:52.955319+0000","last_peered":"2024-09-11T00:30:52.955319+0000","last_clean":"2024-09-11T00:30:52.955319+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:30:52.955319+0000","last_undegraded":"2024-09-11T00:30:52.955319+0000","last_fullsized":"2024-09-11T00:30:52.955319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969929+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:30:55.969929+0000","last_peered":"2024-09-11T00:30:55.969929+0000","last_clean":"2024-09-11T00:30:55.969929+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:30:55.969929+0000","last_undegraded":"2024-09-11T00:30:55.969929+0000","last_fullsized":"2024-09-11T00:30:55.969929+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964588+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:30:52.964588+0000","last_peered":"2024-09-11T00:30:52.964588+0000","last_clean":"2024-09-11T00:30:52.964588+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:30:52.964588+0000","last_undegraded":"2024-09-11T00:30:52.964588+0000","last_fullsized":"2024-09-11T00:30:52.964588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.962732+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:30:52.962732+0000","last_peered":"2024-09-11T00:30:52.962732+0000","last_clean":"2024-09-11T00:30:52.962732+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:30:52.962732+0000","last_undegraded":"2024-09-11T00:30:52.962732+0000","last_fullsized":"2024-09-11T00:30:52.962732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964177+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:30:52.964177+0000","last_peered":"2024-09-11T00:30:52.964177+0000","last_clean":"2024-09-11T00:30:52.964177+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:30:52.964177+0000","last_undegraded":"2024-09-11T00:30:52.964177+0000","last_fullsized":"2024-09-11T00:30:52.964177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.961401+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:30:52.961401+0000","last_peered":"2024-09-11T00:30:52.961401+0000","last_clean":"2024-09-11T00:30:52.961401+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:30:52.961401+0000","last_undegraded":"2024-09-11T00:30:52.961401+0000","last_fullsized":"2024-09-11T00:30:52.961401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.961250+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:30:52.961250+0000","last_peered":"2024-09-11T00:30:52.961250+0000","last_clean":"2024-09-11T00:30:52.961250+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:30:52.961250+0000","last_undegraded":"2024-09-11T00:30:52.961250+0000","last_fullsized":"2024-09-11T00:30:52.961250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.955689+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:30:52.955689+0000","last_peered":"2024-09-11T00:30:52.955689+0000","last_clean":"2024-09-11T00:30:52.955689+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:30:52.955689+0000","last_undegraded":"2024-09-11T00:30:52.955689+0000","last_fullsized":"2024-09-11T00:30:52.955689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964620+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:30:52.964620+0000","last_peered":"2024-09-11T00:30:52.964620+0000","last_clean":"2024-09-11T00:30:52.964620+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:30:52.964620+0000","last_undegraded":"2024-09-11T00:30:52.964620+0000","last_fullsized":"2024-09-11T00:30:52.964620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963184+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:30:52.963184+0000","last_peered":"2024-09-11T00:30:52.963184+0000","last_clean":"2024-09-11T00:30:52.963184+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:30:52.963184+0000","last_undegraded":"2024-09-11T00:30:52.963184+0000","last_fullsized":"2024-09-11T00:30:52.963184+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963596+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:30:52.963596+0000","last_peered":"2024-09-11T00:30:52.963596+0000","last_clean":"2024-09-11T00:30:52.963596+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:30:52.963596+0000","last_undegraded":"2024-09-11T00:30:52.963596+0000","last_fullsized":"2024-09-11T00:30:52.963596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.962455+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:30:52.962455+0000","last_peered":"2024-09-11T00:30:52.962455+0000","last_clean":"2024-09-11T00:30:52.962455+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:30:52.962455+0000","last_undegraded":"2024-09-11T00:30:52.962455+0000","last_fullsized":"2024-09-11T00:30:52.962455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969635+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:30:55.969635+0000","last_peered":"2024-09-11T00:30:55.969635+0000","last_clean":"2024-09-11T00:30:55.969635+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:55.969635+0000","last_undegraded":"2024-09-11T00:30:55.969635+0000","last_fullsized":"2024-09-11T00:30:55.969635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963026+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:30:52.963026+0000","last_peered":"2024-09-11T00:30:52.963026+0000","last_clean":"2024-09-11T00:30:52.963026+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:30:52.963026+0000","last_undegraded":"2024-09-11T00:30:52.963026+0000","last_fullsized":"2024-09-11T00:30:52.963026+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965945+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:30:52.965945+0000","last_peered":"2024-09-11T00:30:52.965945+0000","last_clean":"2024-09-11T00:30:52.965945+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:30:52.965945+0000","last_undegraded":"2024-09-11T00:30:52.965945+0000","last_fullsized":"2024-09-11T00:30:52.965945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965478+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:30:52.965478+0000","last_peered":"2024-09-11T00:30:52.965478+0000","last_clean":"2024-09-11T00:30:52.965478+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:30:52.965478+0000","last_undegraded":"2024-09-11T00:30:52.965478+0000","last_fullsized":"2024-09-11T00:30:52.965478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.966150+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:30:52.966150+0000","last_peered":"2024-09-11T00:30:52.966150+0000","last_clean":"2024-09-11T00:30:52.966150+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:30:52.966150+0000","last_undegraded":"2024-09-11T00:30:52.966150+0000","last_fullsized":"2024-09-11T00:30:52.966150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970335+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:30:55.970335+0000","last_peered":"2024-09-11T00:30:55.970335+0000","last_clean":"2024-09-11T00:30:55.970335+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:30:55.970335+0000","last_undegraded":"2024-09-11T00:30:55.970335+0000","last_fullsized":"2024-09-11T00:30:55.970335+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963089+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:30:52.963089+0000","last_peered":"2024-09-11T00:30:52.963089+0000","last_clean":"2024-09-11T00:30:52.963089+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:30:52.963089+0000","last_undegraded":"2024-09-11T00:30:52.963089+0000","last_fullsized":"2024-09-11T00:30:52.963089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.963478+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:30:52.963478+0000","last_peered":"2024-09-11T00:30:52.963478+0000","last_clean":"2024-09-11T00:30:52.963478+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:30:52.963478+0000","last_undegraded":"2024-09-11T00:30:52.963478+0000","last_fullsized":"2024-09-11T00:30:52.963478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.960085+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:30:52.960085+0000","last_peered":"2024-09-11T00:30:52.960085+0000","last_clean":"2024-09-11T00:30:52.960085+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:30:52.960085+0000","last_undegraded":"2024-09-11T00:30:52.960085+0000","last_fullsized":"2024-09-11T00:30:52.960085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.961633+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:30:52.961633+0000","last_peered":"2024-09-11T00:30:52.961633+0000","last_clean":"2024-09-11T00:30:52.961633+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:30:52.961633+0000","last_undegraded":"2024-09-11T00:30:52.961633+0000","last_fullsized":"2024-09-11T00:30:52.961633+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969452+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:30:55.969452+0000","last_peered":"2024-09-11T00:30:55.969452+0000","last_clean":"2024-09-11T00:30:55.969452+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:30:55.969452+0000","last_undegraded":"2024-09-11T00:30:55.969452+0000","last_fullsized":"2024-09-11T00:30:55.969452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.962918+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:30:52.962918+0000","last_peered":"2024-09-11T00:30:52.962918+0000","last_clean":"2024-09-11T00:30:52.962918+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:30:52.962918+0000","last_undegraded":"2024-09-11T00:30:52.962918+0000","last_fullsized":"2024-09-11T00:30:52.962918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965355+0000","last_change":"2024-09-11T00:30:45.941635+0000","last_active":"2024-09-11T00:30:52.965355+0000","last_peered":"2024-09-11T00:30:52.965355+0000","last_clean":"2024-09-11T00:30:52.965355+0000","last_became_active":"2024-09-11T00:30:45.941357+0000","last_became_peered":"2024-09-11T00:30:45.941357+0000","last_unstale":"2024-09-11T00:30:52.965355+0000","last_undegraded":"2024-09-11T00:30:52.965355+0000","last_fullsized":"2024-09-11T00:30:52.965355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:12.016406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.966014+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:30:52.966014+0000","last_peered":"2024-09-11T00:30:52.966014+0000","last_clean":"2024-09-11T00:30:52.966014+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:52.966014+0000","last_undegraded":"2024-09-11T00:30:52.966014+0000","last_fullsized":"2024-09-11T00:30:52.966014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964168+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:30:52.964168+0000","last_peered":"2024-09-11T00:30:52.964168+0000","last_clean":"2024-09-11T00:30:52.964168+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:30:52.964168+0000","last_undegraded":"2024-09-11T00:30:52.964168+0000","last_fullsized":"2024-09-11T00:30:52.964168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964954+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:30:52.964954+0000","last_peered":"2024-09-11T00:30:52.964954+0000","last_clean":"2024-09-11T00:30:52.964954+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:30:52.964954+0000","last_undegraded":"2024-09-11T00:30:52.964954+0000","last_fullsized":"2024-09-11T00:30:52.964954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.961486+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:30:52.961486+0000","last_peered":"2024-09-11T00:30:52.961486+0000","last_clean":"2024-09-11T00:30:52.961486+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:30:52.961486+0000","last_undegraded":"2024-09-11T00:30:52.961486+0000","last_fullsized":"2024-09-11T00:30:52.961486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970256+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:30:55.970256+0000","last_peered":"2024-09-11T00:30:55.970256+0000","last_clean":"2024-09-11T00:30:55.970256+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:30:55.970256+0000","last_undegraded":"2024-09-11T00:30:55.970256+0000","last_fullsized":"2024-09-11T00:30:55.970256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.955477+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:30:52.955477+0000","last_peered":"2024-09-11T00:30:52.955477+0000","last_clean":"2024-09-11T00:30:52.955477+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:30:52.955477+0000","last_undegraded":"2024-09-11T00:30:52.955477+0000","last_fullsized":"2024-09-11T00:30:52.955477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.962788+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:30:52.962788+0000","last_peered":"2024-09-11T00:30:52.962788+0000","last_clean":"2024-09-11T00:30:52.962788+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:30:52.962788+0000","last_undegraded":"2024-09-11T00:30:52.962788+0000","last_fullsized":"2024-09-11T00:30:52.962788+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.955516+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:30:52.955516+0000","last_peered":"2024-09-11T00:30:52.955516+0000","last_clean":"2024-09-11T00:30:52.955516+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:30:52.955516+0000","last_undegraded":"2024-09-11T00:30:52.955516+0000","last_fullsized":"2024-09-11T00:30:52.955516+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.964177+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:30:52.964177+0000","last_peered":"2024-09-11T00:30:52.964177+0000","last_clean":"2024-09-11T00:30:52.964177+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:30:52.964177+0000","last_undegraded":"2024-09-11T00:30:52.964177+0000","last_fullsized":"2024-09-11T00:30:52.964177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965235+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:30:52.965235+0000","last_peered":"2024-09-11T00:30:52.965235+0000","last_clean":"2024-09-11T00:30:52.965235+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:30:52.965235+0000","last_undegraded":"2024-09-11T00:30:52.965235+0000","last_fullsized":"2024-09-11T00:30:52.965235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-11T00:30:53.953658+0000","last_change":"2024-09-11T00:30:53.953658+0000","last_active":"2024-09-11T00:30:53.953658+0000","last_peered":"2024-09-11T00:30:53.953658+0000","last_clean":"2024-09-11T00:30:53.953658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-11T00:30:53.953658+0000","last_undegraded":"2024-09-11T00:30:53.953658+0000","last_fullsized":"2024-09-11T00:30:53.953658+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-09-11T00:30:52.965348+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:30:52.965348+0000","last_peered":"2024-09-11T00:30:52.965348+0000","last_clean":"2024-09-11T00:30:52.965348+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:30:52.965348+0000","last_undegraded":"2024-09-11T00:30:52.965348+0000","last_fullsized":"2024-09-11T00:30:52.965348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":1},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":82967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509448,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":82843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":835584,"data_stored":673303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":83425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":84455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":831488,"data_stored":673281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":82899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":307200,"data_stored":108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:31:00.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:31:00.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.440+0000 7f2cd2200640 1 -- 172.21.15.31:0/3292903326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ccc0c23a0 msgr2=0x7f2ccc0c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.440+0000 7f2cd2200640 1 --2- 172.21.15.31:0/3292903326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ccc0c23a0 0x7f2ccc0c30c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8005fd0 tx=0x7f2cb802a5c0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.441+0000 7f2cd2200640 1 -- 172.21.15.31:0/3292903326 shutdown_connections 2024-09-11T00:31:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.441+0000 7f2cd2200640 1 --2- 172.21.15.31:0/3292903326 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ccc1518b0 0x7f2ccc0c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.441+0000 7f2cd2200640 1 --2- 172.21.15.31:0/3292903326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ccc0c23a0 0x7f2ccc0c30c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.441+0000 7f2cd2200640 1 -- 172.21.15.31:0/3292903326 >> 172.21.15.31:0/3292903326 conn(0x7f2ccc0a9750 msgr2=0x7f2ccc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.441+0000 7f2cd2200640 1 -- 172.21.15.31:0/3292903326 shutdown_connections 2024-09-11T00:31:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.441+0000 7f2cd2200640 1 -- 172.21.15.31:0/3292903326 wait complete. 2024-09-11T00:31:00.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.442+0000 7f2cd2200640 1 Processor -- start 2024-09-11T00:31:00.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.442+0000 7f2cd2200640 1 -- start start 2024-09-11T00:31:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.443+0000 7f2cd2200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ccc072480 0x7f2ccc14cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.443+0000 7f2cd2200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ccc0c23a0 0x7f2ccc148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.443+0000 7f2cd2200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ccc1518b0 0x7f2ccc149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.443+0000 7f2cd2200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc074200 con 0x7f2ccc1518b0 2024-09-11T00:31:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.443+0000 7f2cd2200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc074080 con 0x7f2ccc0c23a0 2024-09-11T00:31:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.443+0000 7f2cd2200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2ccc072860 con 0x7f2ccc072480 2024-09-11T00:31:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.443+0000 7f2ccbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ccc1518b0 0x7f2ccc149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.443+0000 7f2ccbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ccc1518b0 0x7f2ccc149080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37178/0 (socket says 172.21.15.31:37178) 2024-09-11T00:31:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.443+0000 7f2ccbe00640 1 -- 172.21.15.31:0/2765245968 learned_addr learned my addr 172.21.15.31:0/2765245968 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:00.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2ccb400640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ccc072480 0x7f2ccc14cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2ccaa00640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ccc0c23a0 0x7f2ccc148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2ccbe00640 1 -- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ccc072480 msgr2=0x7f2ccc14cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2ccbe00640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ccc072480 0x7f2ccc14cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2ccbe00640 1 -- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ccc0c23a0 msgr2=0x7f2ccc148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2ccbe00640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ccc0c23a0 0x7f2ccc148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2ccbe00640 1 -- 172.21.15.31:0/2765245968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ccc1de6f0 con 0x7f2ccc1518b0 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2ccaa00640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ccc0c23a0 0x7f2ccc148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2ccbe00640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ccc1518b0 0x7f2ccc149080 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400cb00 tx=0x7f2cc400a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2cc3400640 1 -- 172.21.15.31:0/2765245968 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc400f450 con 0x7f2ccc1518b0 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2cc3400640 1 -- 172.21.15.31:0/2765245968 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cc400f700 con 0x7f2ccc1518b0 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2cc3400640 1 -- 172.21.15.31:0/2765245968 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc400fb50 con 0x7f2ccc1518b0 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2cd2200640 1 -- 172.21.15.31:0/2765245968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ccc1e0250 con 0x7f2ccc1518b0 2024-09-11T00:31:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.444+0000 7f2cd2200640 1 -- 172.21.15.31:0/2765245968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ccc155be0 con 0x7f2ccc1518b0 2024-09-11T00:31:00.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.445+0000 7f2cd2200640 1 -- 172.21.15.31:0/2765245968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ccc072d90 con 0x7f2ccc1518b0 2024-09-11T00:31:00.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.446+0000 7f2cc3400640 1 -- 172.21.15.31:0/2765245968 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2cc400e3b0 con 0x7f2ccc1518b0 2024-09-11T00:31:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.448+0000 7f2cc3400640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2cac077f70 0x7f2cac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.448+0000 7f2cc3400640 1 -- 172.21.15.31:0/2765245968 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f2cc409ac00 con 0x7f2ccc1518b0 2024-09-11T00:31:00.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.448+0000 7f2ccb400640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2cac077f70 0x7f2cac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.449+0000 7f2ccb400640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2cac077f70 0x7f2cac07a430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc003d90 tx=0x7f2cbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:00.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.452+0000 7f2cc3400640 1 -- 172.21.15.31:0/2765245968 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2ccc072d90 con 0x7f2ccc1518b0 2024-09-11T00:31:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.646+0000 7f2cd2200640 1 -- 172.21.15.31:0/2765245968 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2ccc0c3b90 con 0x7f2cac077f70 2024-09-11T00:31:00.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.655+0000 7f2cc3400640 1 -- 172.21.15.31:0/2765245968 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490408 (secure 0 0 0) 0x7f2ccc0c3b90 con 0x7f2cac077f70 2024-09-11T00:31:00.662 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:00.665 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:31:00.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 -- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2cac077f70 msgr2=0x7f2cac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2cac077f70 0x7f2cac07a430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc003d90 tx=0x7f2cbc009040 comp rx=0 tx=0).stop 2024-09-11T00:31:00.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 -- 172.21.15.31:0/2765245968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ccc1518b0 msgr2=0x7f2ccc149080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ccc1518b0 0x7f2ccc149080 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400cb00 tx=0x7f2cc400a250 comp rx=0 tx=0).stop 2024-09-11T00:31:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 -- 172.21.15.31:0/2765245968 shutdown_connections 2024-09-11T00:31:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2cac077f70 0x7f2cac07a430 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ccc1518b0 0x7f2ccc149080 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ccc0c23a0 0x7f2ccc148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 --2- 172.21.15.31:0/2765245968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2ccc072480 0x7f2ccc14cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 -- 172.21.15.31:0/2765245968 >> 172.21.15.31:0/2765245968 conn(0x7f2ccc0a9750 msgr2=0x7f2ccc14ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 -- 172.21.15.31:0/2765245968 shutdown_connections 2024-09-11T00:31:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.661+0000 7f2cd2200640 1 -- 172.21.15.31:0/2765245968 wait complete. 2024-09-11T00:31:00.678 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-11T00:31:00.395255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":191,"num_write_kb":1656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219644,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755076,"statfs":{"total":773094113280,"available":772869197824,"internally_reserved":0,"allocated":3481600,"data_stored":2027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.433863"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719046+0000","last_change":"2024-09-11T00:30:54.983412+0000","last_active":"2024-09-11T00:30:56.719046+0000","last_peered":"2024-09-11T00:30:56.719046+0000","last_clean":"2024-09-11T00:30:56.719046+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:30:56.719046+0000","last_undegraded":"2024-09-11T00:30:56.719046+0000","last_fullsized":"2024-09-11T00:30:56.719046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:51.807846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970097+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:30:55.970097+0000","last_peered":"2024-09-11T00:30:55.970097+0000","last_clean":"2024-09-11T00:30:55.970097+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:30:55.970097+0000","last_undegraded":"2024-09-11T00:30:55.970097+0000","last_fullsized":"2024-09-11T00:30:55.970097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998047+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:30:55.998047+0000","last_peered":"2024-09-11T00:30:55.998047+0000","last_clean":"2024-09-11T00:30:55.998047+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:30:55.998047+0000","last_undegraded":"2024-09-11T00:30:55.998047+0000","last_fullsized":"2024-09-11T00:30:55.998047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000942+0000","last_change":"2024-09-11T00:30:53.984316+0000","last_active":"2024-09-11T00:30:56.000942+0000","last_peered":"2024-09-11T00:30:56.000942+0000","last_clean":"2024-09-11T00:30:56.000942+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:30:56.000942+0000","last_undegraded":"2024-09-11T00:30:56.000942+0000","last_fullsized":"2024-09-11T00:30:56.000942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:14.425265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968662+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:30:55.968662+0000","last_peered":"2024-09-11T00:30:55.968662+0000","last_clean":"2024-09-11T00:30:55.968662+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:30:55.968662+0000","last_undegraded":"2024-09-11T00:30:55.968662+0000","last_fullsized":"2024-09-11T00:30:55.968662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998255+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:30:55.998255+0000","last_peered":"2024-09-11T00:30:55.998255+0000","last_clean":"2024-09-11T00:30:55.998255+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:30:55.998255+0000","last_undegraded":"2024-09-11T00:30:55.998255+0000","last_fullsized":"2024-09-11T00:30:55.998255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998074+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:30:55.998074+0000","last_peered":"2024-09-11T00:30:55.998074+0000","last_clean":"2024-09-11T00:30:55.998074+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:30:55.998074+0000","last_undegraded":"2024-09-11T00:30:55.998074+0000","last_fullsized":"2024-09-11T00:30:55.998074+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172549+0000","last_change":"2024-09-11T00:30:53.984940+0000","last_active":"2024-09-11T00:30:56.172549+0000","last_peered":"2024-09-11T00:30:56.172549+0000","last_clean":"2024-09-11T00:30:56.172549+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:30:56.172549+0000","last_undegraded":"2024-09-11T00:30:56.172549+0000","last_fullsized":"2024-09-11T00:30:56.172549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:19.647165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719205+0000","last_change":"2024-09-11T00:30:54.993570+0000","last_active":"2024-09-11T00:30:56.719205+0000","last_peered":"2024-09-11T00:30:56.719205+0000","last_clean":"2024-09-11T00:30:56.719205+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:30:56.719205+0000","last_undegraded":"2024-09-11T00:30:56.719205+0000","last_fullsized":"2024-09-11T00:30:56.719205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:45.211259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333273+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:30:56.333273+0000","last_peered":"2024-09-11T00:30:56.333273+0000","last_clean":"2024-09-11T00:30:56.333273+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:30:56.333273+0000","last_undegraded":"2024-09-11T00:30:56.333273+0000","last_fullsized":"2024-09-11T00:30:56.333273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719392+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:30:56.719392+0000","last_peered":"2024-09-11T00:30:56.719392+0000","last_clean":"2024-09-11T00:30:56.719392+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:30:56.719392+0000","last_undegraded":"2024-09-11T00:30:56.719392+0000","last_fullsized":"2024-09-11T00:30:56.719392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172696+0000","last_change":"2024-09-11T00:30:53.974699+0000","last_active":"2024-09-11T00:30:56.172696+0000","last_peered":"2024-09-11T00:30:56.172696+0000","last_clean":"2024-09-11T00:30:56.172696+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:30:56.172696+0000","last_undegraded":"2024-09-11T00:30:56.172696+0000","last_fullsized":"2024-09-11T00:30:56.172696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:28.792487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172498+0000","last_change":"2024-09-11T00:30:54.984890+0000","last_active":"2024-09-11T00:30:56.172498+0000","last_peered":"2024-09-11T00:30:56.172498+0000","last_clean":"2024-09-11T00:30:56.172498+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:30:56.172498+0000","last_undegraded":"2024-09-11T00:30:56.172498+0000","last_fullsized":"2024-09-11T00:30:56.172498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:21.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333166+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:30:56.333166+0000","last_peered":"2024-09-11T00:30:56.333166+0000","last_clean":"2024-09-11T00:30:56.333166+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:30:56.333166+0000","last_undegraded":"2024-09-11T00:30:56.333166+0000","last_fullsized":"2024-09-11T00:30:56.333166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968144+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:30:55.968144+0000","last_peered":"2024-09-11T00:30:55.968144+0000","last_clean":"2024-09-11T00:30:55.968144+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:30:55.968144+0000","last_undegraded":"2024-09-11T00:30:55.968144+0000","last_fullsized":"2024-09-11T00:30:55.968144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172339+0000","last_change":"2024-09-11T00:30:53.977743+0000","last_active":"2024-09-11T00:30:56.172339+0000","last_peered":"2024-09-11T00:30:56.172339+0000","last_clean":"2024-09-11T00:30:56.172339+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:30:56.172339+0000","last_undegraded":"2024-09-11T00:30:56.172339+0000","last_fullsized":"2024-09-11T00:30:56.172339+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:20.618615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997709+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:30:55.997709+0000","last_peered":"2024-09-11T00:30:55.997709+0000","last_clean":"2024-09-11T00:30:55.997709+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:30:55.997709+0000","last_undegraded":"2024-09-11T00:30:55.997709+0000","last_fullsized":"2024-09-11T00:30:55.997709+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967362+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:30:55.967362+0000","last_peered":"2024-09-11T00:30:55.967362+0000","last_clean":"2024-09-11T00:30:55.967362+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:30:55.967362+0000","last_undegraded":"2024-09-11T00:30:55.967362+0000","last_fullsized":"2024-09-11T00:30:55.967362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967452+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:30:55.967452+0000","last_peered":"2024-09-11T00:30:55.967452+0000","last_clean":"2024-09-11T00:30:55.967452+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:30:55.967452+0000","last_undegraded":"2024-09-11T00:30:55.967452+0000","last_fullsized":"2024-09-11T00:30:55.967452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967914+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:30:55.967914+0000","last_peered":"2024-09-11T00:30:55.967914+0000","last_clean":"2024-09-11T00:30:55.967914+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:30:55.967914+0000","last_undegraded":"2024-09-11T00:30:55.967914+0000","last_fullsized":"2024-09-11T00:30:55.967914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001419+0000","last_change":"2024-09-11T00:30:54.997397+0000","last_active":"2024-09-11T00:30:56.001419+0000","last_peered":"2024-09-11T00:30:56.001419+0000","last_clean":"2024-09-11T00:30:56.001419+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:30:56.001419+0000","last_undegraded":"2024-09-11T00:30:56.001419+0000","last_fullsized":"2024-09-11T00:30:56.001419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:53.322576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001116+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:30:56.001116+0000","last_peered":"2024-09-11T00:30:56.001116+0000","last_clean":"2024-09-11T00:30:56.001116+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:30:56.001116+0000","last_undegraded":"2024-09-11T00:30:56.001116+0000","last_fullsized":"2024-09-11T00:30:56.001116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970056+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:30:55.970056+0000","last_peered":"2024-09-11T00:30:55.970056+0000","last_clean":"2024-09-11T00:30:55.970056+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:30:55.970056+0000","last_undegraded":"2024-09-11T00:30:55.970056+0000","last_fullsized":"2024-09-11T00:30:55.970056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332817+0000","last_change":"2024-09-11T00:30:53.986858+0000","last_active":"2024-09-11T00:30:56.332817+0000","last_peered":"2024-09-11T00:30:56.332817+0000","last_clean":"2024-09-11T00:30:56.332817+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:30:56.332817+0000","last_undegraded":"2024-09-11T00:30:56.332817+0000","last_fullsized":"2024-09-11T00:30:56.332817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:33.626193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967911+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:30:55.967911+0000","last_peered":"2024-09-11T00:30:55.967911+0000","last_clean":"2024-09-11T00:30:55.967911+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:30:55.967911+0000","last_undegraded":"2024-09-11T00:30:55.967911+0000","last_fullsized":"2024-09-11T00:30:55.967911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968190+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:30:55.968190+0000","last_peered":"2024-09-11T00:30:55.968190+0000","last_clean":"2024-09-11T00:30:55.968190+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:30:55.968190+0000","last_undegraded":"2024-09-11T00:30:55.968190+0000","last_fullsized":"2024-09-11T00:30:55.968190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968850+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:30:55.968850+0000","last_peered":"2024-09-11T00:30:55.968850+0000","last_clean":"2024-09-11T00:30:55.968850+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:30:55.968850+0000","last_undegraded":"2024-09-11T00:30:55.968850+0000","last_fullsized":"2024-09-11T00:30:55.968850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001447+0000","last_change":"2024-09-11T00:30:53.990765+0000","last_active":"2024-09-11T00:30:56.001447+0000","last_peered":"2024-09-11T00:30:56.001447+0000","last_clean":"2024-09-11T00:30:56.001447+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:30:56.001447+0000","last_undegraded":"2024-09-11T00:30:56.001447+0000","last_fullsized":"2024-09-11T00:30:56.001447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:33.720727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719722+0000","last_change":"2024-09-11T00:30:54.984949+0000","last_active":"2024-09-11T00:30:56.719722+0000","last_peered":"2024-09-11T00:30:56.719722+0000","last_clean":"2024-09-11T00:30:56.719722+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:30:56.719722+0000","last_undegraded":"2024-09-11T00:30:56.719722+0000","last_fullsized":"2024-09-11T00:30:56.719722+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:54.346937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967033+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:30:55.967033+0000","last_peered":"2024-09-11T00:30:55.967033+0000","last_clean":"2024-09-11T00:30:55.967033+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:30:55.967033+0000","last_undegraded":"2024-09-11T00:30:55.967033+0000","last_fullsized":"2024-09-11T00:30:55.967033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332357+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:30:56.332357+0000","last_peered":"2024-09-11T00:30:56.332357+0000","last_clean":"2024-09-11T00:30:56.332357+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:30:56.332357+0000","last_undegraded":"2024-09-11T00:30:56.332357+0000","last_fullsized":"2024-09-11T00:30:56.332357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172821+0000","last_change":"2024-09-11T00:30:53.986940+0000","last_active":"2024-09-11T00:30:56.172821+0000","last_peered":"2024-09-11T00:30:56.172821+0000","last_clean":"2024-09-11T00:30:56.172821+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:30:56.172821+0000","last_undegraded":"2024-09-11T00:30:56.172821+0000","last_fullsized":"2024-09-11T00:30:56.172821+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:11.942187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968462+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:30:55.968462+0000","last_peered":"2024-09-11T00:30:55.968462+0000","last_clean":"2024-09-11T00:30:55.968462+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:30:55.968462+0000","last_undegraded":"2024-09-11T00:30:55.968462+0000","last_fullsized":"2024-09-11T00:30:55.968462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968110+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:30:55.968110+0000","last_peered":"2024-09-11T00:30:55.968110+0000","last_clean":"2024-09-11T00:30:55.968110+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:30:55.968110+0000","last_undegraded":"2024-09-11T00:30:55.968110+0000","last_fullsized":"2024-09-11T00:30:55.968110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172483+0000","last_change":"2024-09-11T00:30:53.988539+0000","last_active":"2024-09-11T00:30:56.172483+0000","last_peered":"2024-09-11T00:30:56.172483+0000","last_clean":"2024-09-11T00:30:56.172483+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:30:56.172483+0000","last_undegraded":"2024-09-11T00:30:56.172483+0000","last_fullsized":"2024-09-11T00:30:56.172483+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:19.083664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966824+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:30:55.966824+0000","last_peered":"2024-09-11T00:30:55.966824+0000","last_clean":"2024-09-11T00:30:55.966824+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:30:55.966824+0000","last_undegraded":"2024-09-11T00:30:55.966824+0000","last_fullsized":"2024-09-11T00:30:55.966824+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970081+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:30:55.970081+0000","last_peered":"2024-09-11T00:30:55.970081+0000","last_clean":"2024-09-11T00:30:55.970081+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:30:55.970081+0000","last_undegraded":"2024-09-11T00:30:55.970081+0000","last_fullsized":"2024-09-11T00:30:55.970081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970376+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:30:55.970376+0000","last_peered":"2024-09-11T00:30:55.970376+0000","last_clean":"2024-09-11T00:30:55.970376+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:30:55.970376+0000","last_undegraded":"2024-09-11T00:30:55.970376+0000","last_fullsized":"2024-09-11T00:30:55.970376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968131+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:30:55.968131+0000","last_peered":"2024-09-11T00:30:55.968131+0000","last_clean":"2024-09-11T00:30:55.968131+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:30:55.968131+0000","last_undegraded":"2024-09-11T00:30:55.968131+0000","last_fullsized":"2024-09-11T00:30:55.968131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332511+0000","last_change":"2024-09-11T00:30:54.993118+0000","last_active":"2024-09-11T00:30:56.332511+0000","last_peered":"2024-09-11T00:30:56.332511+0000","last_clean":"2024-09-11T00:30:56.332511+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:30:56.332511+0000","last_undegraded":"2024-09-11T00:30:56.332511+0000","last_fullsized":"2024-09-11T00:30:56.332511+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:07:41.148370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971378+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:30:55.971378+0000","last_peered":"2024-09-11T00:30:55.971378+0000","last_clean":"2024-09-11T00:30:55.971378+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:30:55.971378+0000","last_undegraded":"2024-09-11T00:30:55.971378+0000","last_fullsized":"2024-09-11T00:30:55.971378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968486+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:30:55.968486+0000","last_peered":"2024-09-11T00:30:55.968486+0000","last_clean":"2024-09-11T00:30:55.968486+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:30:55.968486+0000","last_undegraded":"2024-09-11T00:30:55.968486+0000","last_fullsized":"2024-09-11T00:30:55.968486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970903+0000","last_change":"2024-09-11T00:30:53.986843+0000","last_active":"2024-09-11T00:30:55.970903+0000","last_peered":"2024-09-11T00:30:55.970903+0000","last_clean":"2024-09-11T00:30:55.970903+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:30:55.970903+0000","last_undegraded":"2024-09-11T00:30:55.970903+0000","last_fullsized":"2024-09-11T00:30:55.970903+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:44.684154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969549+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:30:55.969549+0000","last_peered":"2024-09-11T00:30:55.969549+0000","last_clean":"2024-09-11T00:30:55.969549+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:30:55.969549+0000","last_undegraded":"2024-09-11T00:30:55.969549+0000","last_fullsized":"2024-09-11T00:30:55.969549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172919+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:30:56.172919+0000","last_peered":"2024-09-11T00:30:56.172919+0000","last_clean":"2024-09-11T00:30:56.172919+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:30:56.172919+0000","last_undegraded":"2024-09-11T00:30:56.172919+0000","last_fullsized":"2024-09-11T00:30:56.172919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997652+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:30:55.997652+0000","last_peered":"2024-09-11T00:30:55.997652+0000","last_clean":"2024-09-11T00:30:55.997652+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:30:55.997652+0000","last_undegraded":"2024-09-11T00:30:55.997652+0000","last_fullsized":"2024-09-11T00:30:55.997652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969390+0000","last_change":"2024-09-11T00:30:53.986896+0000","last_active":"2024-09-11T00:30:55.969390+0000","last_peered":"2024-09-11T00:30:55.969390+0000","last_clean":"2024-09-11T00:30:55.969390+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:30:55.969390+0000","last_undegraded":"2024-09-11T00:30:55.969390+0000","last_fullsized":"2024-09-11T00:30:55.969390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:06.590278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001631+0000","last_change":"2024-09-11T00:30:55.001370+0000","last_active":"2024-09-11T00:30:56.001631+0000","last_peered":"2024-09-11T00:30:56.001631+0000","last_clean":"2024-09-11T00:30:56.001631+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:30:56.001631+0000","last_undegraded":"2024-09-11T00:30:56.001631+0000","last_fullsized":"2024-09-11T00:30:56.001631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:56.592767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333262+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:30:56.333262+0000","last_peered":"2024-09-11T00:30:56.333262+0000","last_clean":"2024-09-11T00:30:56.333262+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:30:56.333262+0000","last_undegraded":"2024-09-11T00:30:56.333262+0000","last_fullsized":"2024-09-11T00:30:56.333262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967594+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:30:55.967594+0000","last_peered":"2024-09-11T00:30:55.967594+0000","last_clean":"2024-09-11T00:30:55.967594+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:30:55.967594+0000","last_undegraded":"2024-09-11T00:30:55.967594+0000","last_fullsized":"2024-09-11T00:30:55.967594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971192+0000","last_change":"2024-09-11T00:30:53.978480+0000","last_active":"2024-09-11T00:30:55.971192+0000","last_peered":"2024-09-11T00:30:55.971192+0000","last_clean":"2024-09-11T00:30:55.971192+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:30:55.971192+0000","last_undegraded":"2024-09-11T00:30:55.971192+0000","last_fullsized":"2024-09-11T00:30:55.971192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:58.900565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001528+0000","last_change":"2024-09-11T00:30:54.997641+0000","last_active":"2024-09-11T00:30:56.001528+0000","last_peered":"2024-09-11T00:30:56.001528+0000","last_clean":"2024-09-11T00:30:56.001528+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:30:56.001528+0000","last_undegraded":"2024-09-11T00:30:56.001528+0000","last_fullsized":"2024-09-11T00:30:56.001528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:18.437490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997311+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:30:55.997311+0000","last_peered":"2024-09-11T00:30:55.997311+0000","last_clean":"2024-09-11T00:30:55.997311+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:30:55.997311+0000","last_undegraded":"2024-09-11T00:30:55.997311+0000","last_fullsized":"2024-09-11T00:30:55.997311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971163+0000","last_change":"2024-09-11T00:30:53.979031+0000","last_active":"2024-09-11T00:30:55.971163+0000","last_peered":"2024-09-11T00:30:55.971163+0000","last_clean":"2024-09-11T00:30:55.971163+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:30:55.971163+0000","last_undegraded":"2024-09-11T00:30:55.971163+0000","last_fullsized":"2024-09-11T00:30:55.971163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:36.604377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967699+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:30:55.967699+0000","last_peered":"2024-09-11T00:30:55.967699+0000","last_clean":"2024-09-11T00:30:55.967699+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:30:55.967699+0000","last_undegraded":"2024-09-11T00:30:55.967699+0000","last_fullsized":"2024-09-11T00:30:55.967699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:39.629122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968089+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:30:55.968089+0000","last_peered":"2024-09-11T00:30:55.968089+0000","last_clean":"2024-09-11T00:30:55.968089+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:30:55.968089+0000","last_undegraded":"2024-09-11T00:30:55.968089+0000","last_fullsized":"2024-09-11T00:30:55.968089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966595+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:30:55.966595+0000","last_peered":"2024-09-11T00:30:55.966595+0000","last_clean":"2024-09-11T00:30:55.966595+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:30:55.966595+0000","last_undegraded":"2024-09-11T00:30:55.966595+0000","last_fullsized":"2024-09-11T00:30:55.966595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"26'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998301+0000","last_change":"2024-09-11T00:30:53.989299+0000","last_active":"2024-09-11T00:30:55.998301+0000","last_peered":"2024-09-11T00:30:55.998301+0000","last_clean":"2024-09-11T00:30:55.998301+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:30:55.998301+0000","last_undegraded":"2024-09-11T00:30:55.998301+0000","last_fullsized":"2024-09-11T00:30:55.998301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:34:49.298915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332781+0000","last_change":"2024-09-11T00:30:54.990026+0000","last_active":"2024-09-11T00:30:56.332781+0000","last_peered":"2024-09-11T00:30:56.332781+0000","last_clean":"2024-09-11T00:30:56.332781+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:30:56.332781+0000","last_undegraded":"2024-09-11T00:30:56.332781+0000","last_fullsized":"2024-09-11T00:30:56.332781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:02:10.800530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969545+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:30:55.969545+0000","last_peered":"2024-09-11T00:30:55.969545+0000","last_clean":"2024-09-11T00:30:55.969545+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:30:55.969545+0000","last_undegraded":"2024-09-11T00:30:55.969545+0000","last_fullsized":"2024-09-11T00:30:55.969545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967491+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:30:55.967491+0000","last_peered":"2024-09-11T00:30:55.967491+0000","last_clean":"2024-09-11T00:30:55.967491+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:30:55.967491+0000","last_undegraded":"2024-09-11T00:30:55.967491+0000","last_fullsized":"2024-09-11T00:30:55.967491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719269+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:30:56.719269+0000","last_peered":"2024-09-11T00:30:56.719269+0000","last_clean":"2024-09-11T00:30:56.719269+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:30:56.719269+0000","last_undegraded":"2024-09-11T00:30:56.719269+0000","last_fullsized":"2024-09-11T00:30:56.719269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968267+0000","last_change":"2024-09-11T00:30:53.982142+0000","last_active":"2024-09-11T00:30:55.968267+0000","last_peered":"2024-09-11T00:30:55.968267+0000","last_clean":"2024-09-11T00:30:55.968267+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:30:55.968267+0000","last_undegraded":"2024-09-11T00:30:55.968267+0000","last_fullsized":"2024-09-11T00:30:55.968267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:41.558944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998013+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:30:55.998013+0000","last_peered":"2024-09-11T00:30:55.998013+0000","last_clean":"2024-09-11T00:30:55.998013+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:30:55.998013+0000","last_undegraded":"2024-09-11T00:30:55.998013+0000","last_fullsized":"2024-09-11T00:30:55.998013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968362+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:30:55.968362+0000","last_peered":"2024-09-11T00:30:55.968362+0000","last_clean":"2024-09-11T00:30:55.968362+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:30:55.968362+0000","last_undegraded":"2024-09-11T00:30:55.968362+0000","last_fullsized":"2024-09-11T00:30:55.968362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971436+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:30:55.971436+0000","last_peered":"2024-09-11T00:30:55.971436+0000","last_clean":"2024-09-11T00:30:55.971436+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:30:55.971436+0000","last_undegraded":"2024-09-11T00:30:55.971436+0000","last_fullsized":"2024-09-11T00:30:55.971436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172455+0000","last_change":"2024-09-11T00:30:53.989292+0000","last_active":"2024-09-11T00:30:56.172455+0000","last_peered":"2024-09-11T00:30:56.172455+0000","last_clean":"2024-09-11T00:30:56.172455+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:30:56.172455+0000","last_undegraded":"2024-09-11T00:30:56.172455+0000","last_fullsized":"2024-09-11T00:30:56.172455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:44.071338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968473+0000","last_change":"2024-09-11T00:30:54.981630+0000","last_active":"2024-09-11T00:30:55.968473+0000","last_peered":"2024-09-11T00:30:55.968473+0000","last_clean":"2024-09-11T00:30:55.968473+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:30:55.968473+0000","last_undegraded":"2024-09-11T00:30:55.968473+0000","last_fullsized":"2024-09-11T00:30:55.968473+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:12.832327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333061+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:30:56.333061+0000","last_peered":"2024-09-11T00:30:56.333061+0000","last_clean":"2024-09-11T00:30:56.333061+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:30:56.333061+0000","last_undegraded":"2024-09-11T00:30:56.333061+0000","last_fullsized":"2024-09-11T00:30:56.333061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997856+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:30:55.997856+0000","last_peered":"2024-09-11T00:30:55.997856+0000","last_clean":"2024-09-11T00:30:55.997856+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:30:55.997856+0000","last_undegraded":"2024-09-11T00:30:55.997856+0000","last_fullsized":"2024-09-11T00:30:55.997856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997848+0000","last_change":"2024-09-11T00:30:53.984013+0000","last_active":"2024-09-11T00:30:55.997848+0000","last_peered":"2024-09-11T00:30:55.997848+0000","last_clean":"2024-09-11T00:30:55.997848+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:30:55.997848+0000","last_undegraded":"2024-09-11T00:30:55.997848+0000","last_fullsized":"2024-09-11T00:30:55.997848+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:49.963623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719176+0000","last_change":"2024-09-11T00:30:54.996860+0000","last_active":"2024-09-11T00:30:56.719176+0000","last_peered":"2024-09-11T00:30:56.719176+0000","last_clean":"2024-09-11T00:30:56.719176+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:30:56.719176+0000","last_undegraded":"2024-09-11T00:30:56.719176+0000","last_fullsized":"2024-09-11T00:30:56.719176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:43.516748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333114+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:30:56.333114+0000","last_peered":"2024-09-11T00:30:56.333114+0000","last_clean":"2024-09-11T00:30:56.333114+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:30:56.333114+0000","last_undegraded":"2024-09-11T00:30:56.333114+0000","last_fullsized":"2024-09-11T00:30:56.333114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970314+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:30:55.970314+0000","last_peered":"2024-09-11T00:30:55.970314+0000","last_clean":"2024-09-11T00:30:55.970314+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:30:55.970314+0000","last_undegraded":"2024-09-11T00:30:55.970314+0000","last_fullsized":"2024-09-11T00:30:55.970314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966317+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:30:55.966317+0000","last_peered":"2024-09-11T00:30:55.966317+0000","last_clean":"2024-09-11T00:30:55.966317+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:30:55.966317+0000","last_undegraded":"2024-09-11T00:30:55.966317+0000","last_fullsized":"2024-09-11T00:30:55.966317+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:19.891052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967870+0000","last_change":"2024-09-11T00:30:54.993736+0000","last_active":"2024-09-11T00:30:55.967870+0000","last_peered":"2024-09-11T00:30:55.967870+0000","last_clean":"2024-09-11T00:30:55.967870+0000","last_became_active":"2024-09-11T00:30:54.993470+0000","last_became_peered":"2024-09-11T00:30:54.993470+0000","last_unstale":"2024-09-11T00:30:55.967870+0000","last_undegraded":"2024-09-11T00:30:55.967870+0000","last_fullsized":"2024-09-11T00:30:55.967870+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:24.712448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173170+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:30:56.173170+0000","last_peered":"2024-09-11T00:30:56.173170+0000","last_clean":"2024-09-11T00:30:56.173170+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:30:56.173170+0000","last_undegraded":"2024-09-11T00:30:56.173170+0000","last_fullsized":"2024-09-11T00:30:56.173170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333218+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:30:56.333218+0000","last_peered":"2024-09-11T00:30:56.333218+0000","last_clean":"2024-09-11T00:30:56.333218+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:30:56.333218+0000","last_undegraded":"2024-09-11T00:30:56.333218+0000","last_fullsized":"2024-09-11T00:30:56.333218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719247+0000","last_change":"2024-09-11T00:30:53.983352+0000","last_active":"2024-09-11T00:30:56.719247+0000","last_peered":"2024-09-11T00:30:56.719247+0000","last_clean":"2024-09-11T00:30:56.719247+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:30:56.719247+0000","last_undegraded":"2024-09-11T00:30:56.719247+0000","last_fullsized":"2024-09-11T00:30:56.719247+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:43.826114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970030+0000","last_change":"2024-09-11T00:30:54.987533+0000","last_active":"2024-09-11T00:30:55.970030+0000","last_peered":"2024-09-11T00:30:55.970030+0000","last_clean":"2024-09-11T00:30:55.970030+0000","last_became_active":"2024-09-11T00:30:54.987253+0000","last_became_peered":"2024-09-11T00:30:54.987253+0000","last_unstale":"2024-09-11T00:30:55.970030+0000","last_undegraded":"2024-09-11T00:30:55.970030+0000","last_fullsized":"2024-09-11T00:30:55.970030+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.923656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966934+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:30:55.966934+0000","last_peered":"2024-09-11T00:30:55.966934+0000","last_clean":"2024-09-11T00:30:55.966934+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:30:55.966934+0000","last_undegraded":"2024-09-11T00:30:55.966934+0000","last_fullsized":"2024-09-11T00:30:55.966934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968528+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:30:55.968528+0000","last_peered":"2024-09-11T00:30:55.968528+0000","last_clean":"2024-09-11T00:30:55.968528+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:30:55.968528+0000","last_undegraded":"2024-09-11T00:30:55.968528+0000","last_fullsized":"2024-09-11T00:30:55.968528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997651+0000","last_change":"2024-09-11T00:30:53.980703+0000","last_active":"2024-09-11T00:30:55.997651+0000","last_peered":"2024-09-11T00:30:55.997651+0000","last_clean":"2024-09-11T00:30:55.997651+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:30:55.997651+0000","last_undegraded":"2024-09-11T00:30:55.997651+0000","last_fullsized":"2024-09-11T00:30:55.997651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:51.050512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967820+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:30:55.967820+0000","last_peered":"2024-09-11T00:30:55.967820+0000","last_clean":"2024-09-11T00:30:55.967820+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:30:55.967820+0000","last_undegraded":"2024-09-11T00:30:55.967820+0000","last_fullsized":"2024-09-11T00:30:55.967820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719139+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:30:56.719139+0000","last_peered":"2024-09-11T00:30:56.719139+0000","last_clean":"2024-09-11T00:30:56.719139+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:30:56.719139+0000","last_undegraded":"2024-09-11T00:30:56.719139+0000","last_fullsized":"2024-09-11T00:30:56.719139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333070+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:30:56.333070+0000","last_peered":"2024-09-11T00:30:56.333070+0000","last_clean":"2024-09-11T00:30:56.333070+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:30:56.333070+0000","last_undegraded":"2024-09-11T00:30:56.333070+0000","last_fullsized":"2024-09-11T00:30:56.333070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997937+0000","last_change":"2024-09-11T00:30:53.987829+0000","last_active":"2024-09-11T00:30:55.997937+0000","last_peered":"2024-09-11T00:30:55.997937+0000","last_clean":"2024-09-11T00:30:55.997937+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:30:55.997937+0000","last_undegraded":"2024-09-11T00:30:55.997937+0000","last_fullsized":"2024-09-11T00:30:55.997937+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:57:43.792089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967657+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:30:55.967657+0000","last_peered":"2024-09-11T00:30:55.967657+0000","last_clean":"2024-09-11T00:30:55.967657+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:30:55.967657+0000","last_undegraded":"2024-09-11T00:30:55.967657+0000","last_fullsized":"2024-09-11T00:30:55.967657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967382+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:30:55.967382+0000","last_peered":"2024-09-11T00:30:55.967382+0000","last_clean":"2024-09-11T00:30:55.967382+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:30:55.967382+0000","last_undegraded":"2024-09-11T00:30:55.967382+0000","last_fullsized":"2024-09-11T00:30:55.967382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967492+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:30:55.967492+0000","last_peered":"2024-09-11T00:30:55.967492+0000","last_clean":"2024-09-11T00:30:55.967492+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:30:55.967492+0000","last_undegraded":"2024-09-11T00:30:55.967492+0000","last_fullsized":"2024-09-11T00:30:55.967492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970951+0000","last_change":"2024-09-11T00:30:53.984057+0000","last_active":"2024-09-11T00:30:55.970951+0000","last_peered":"2024-09-11T00:30:55.970951+0000","last_clean":"2024-09-11T00:30:55.970951+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:30:55.970951+0000","last_undegraded":"2024-09-11T00:30:55.970951+0000","last_fullsized":"2024-09-11T00:30:55.970951+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:10.477916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969373+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:30:55.969373+0000","last_peered":"2024-09-11T00:30:55.969373+0000","last_clean":"2024-09-11T00:30:55.969373+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:30:55.969373+0000","last_undegraded":"2024-09-11T00:30:55.969373+0000","last_fullsized":"2024-09-11T00:30:55.969373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969960+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:30:55.969960+0000","last_peered":"2024-09-11T00:30:55.969960+0000","last_clean":"2024-09-11T00:30:55.969960+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:30:55.969960+0000","last_undegraded":"2024-09-11T00:30:55.969960+0000","last_fullsized":"2024-09-11T00:30:55.969960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001240+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:30:56.001240+0000","last_peered":"2024-09-11T00:30:56.001240+0000","last_clean":"2024-09-11T00:30:56.001240+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:30:56.001240+0000","last_undegraded":"2024-09-11T00:30:56.001240+0000","last_fullsized":"2024-09-11T00:30:56.001240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968074+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:30:55.968074+0000","last_peered":"2024-09-11T00:30:55.968074+0000","last_clean":"2024-09-11T00:30:55.968074+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:30:55.968074+0000","last_undegraded":"2024-09-11T00:30:55.968074+0000","last_fullsized":"2024-09-11T00:30:55.968074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969893+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:30:55.969893+0000","last_peered":"2024-09-11T00:30:55.969893+0000","last_clean":"2024-09-11T00:30:55.969893+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:30:55.969893+0000","last_undegraded":"2024-09-11T00:30:55.969893+0000","last_fullsized":"2024-09-11T00:30:55.969893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332978+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:30:56.332978+0000","last_peered":"2024-09-11T00:30:56.332978+0000","last_clean":"2024-09-11T00:30:56.332978+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:30:56.332978+0000","last_undegraded":"2024-09-11T00:30:56.332978+0000","last_fullsized":"2024-09-11T00:30:56.332978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173134+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:30:56.173134+0000","last_peered":"2024-09-11T00:30:56.173134+0000","last_clean":"2024-09-11T00:30:56.173134+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:30:56.173134+0000","last_undegraded":"2024-09-11T00:30:56.173134+0000","last_fullsized":"2024-09-11T00:30:56.173134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969278+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:30:55.969278+0000","last_peered":"2024-09-11T00:30:55.969278+0000","last_clean":"2024-09-11T00:30:55.969278+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:30:55.969278+0000","last_undegraded":"2024-09-11T00:30:55.969278+0000","last_fullsized":"2024-09-11T00:30:55.969278+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969704+0000","last_change":"2024-09-11T00:30:54.986017+0000","last_active":"2024-09-11T00:30:55.969704+0000","last_peered":"2024-09-11T00:30:55.969704+0000","last_clean":"2024-09-11T00:30:55.969704+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:30:55.969704+0000","last_undegraded":"2024-09-11T00:30:55.969704+0000","last_fullsized":"2024-09-11T00:30:55.969704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:58:12.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967364+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:30:55.967364+0000","last_peered":"2024-09-11T00:30:55.967364+0000","last_clean":"2024-09-11T00:30:55.967364+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:30:55.967364+0000","last_undegraded":"2024-09-11T00:30:55.967364+0000","last_fullsized":"2024-09-11T00:30:55.967364+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001702+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:30:56.001702+0000","last_peered":"2024-09-11T00:30:56.001702+0000","last_clean":"2024-09-11T00:30:56.001702+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:30:56.001702+0000","last_undegraded":"2024-09-11T00:30:56.001702+0000","last_fullsized":"2024-09-11T00:30:56.001702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172588+0000","last_change":"2024-09-11T00:30:53.988462+0000","last_active":"2024-09-11T00:30:56.172588+0000","last_peered":"2024-09-11T00:30:56.172588+0000","last_clean":"2024-09-11T00:30:56.172588+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:30:56.172588+0000","last_undegraded":"2024-09-11T00:30:56.172588+0000","last_fullsized":"2024-09-11T00:30:56.172588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:22.385557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966521+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:30:55.966521+0000","last_peered":"2024-09-11T00:30:55.966521+0000","last_clean":"2024-09-11T00:30:55.966521+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:30:55.966521+0000","last_undegraded":"2024-09-11T00:30:55.966521+0000","last_fullsized":"2024-09-11T00:30:55.966521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998206+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:30:55.998206+0000","last_peered":"2024-09-11T00:30:55.998206+0000","last_clean":"2024-09-11T00:30:55.998206+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:30:55.998206+0000","last_undegraded":"2024-09-11T00:30:55.998206+0000","last_fullsized":"2024-09-11T00:30:55.998206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332956+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:30:56.332956+0000","last_peered":"2024-09-11T00:30:56.332956+0000","last_clean":"2024-09-11T00:30:56.332956+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:30:56.332956+0000","last_undegraded":"2024-09-11T00:30:56.332956+0000","last_fullsized":"2024-09-11T00:30:56.332956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000726+0000","last_change":"2024-09-11T00:30:53.986646+0000","last_active":"2024-09-11T00:30:56.000726+0000","last_peered":"2024-09-11T00:30:56.000726+0000","last_clean":"2024-09-11T00:30:56.000726+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:30:56.000726+0000","last_undegraded":"2024-09-11T00:30:56.000726+0000","last_fullsized":"2024-09-11T00:30:56.000726+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:27.447940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172237+0000","last_change":"2024-09-11T00:30:54.982486+0000","last_active":"2024-09-11T00:30:56.172237+0000","last_peered":"2024-09-11T00:30:56.172237+0000","last_clean":"2024-09-11T00:30:56.172237+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:30:56.172237+0000","last_undegraded":"2024-09-11T00:30:56.172237+0000","last_fullsized":"2024-09-11T00:30:56.172237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:41.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173110+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:30:56.173110+0000","last_peered":"2024-09-11T00:30:56.173110+0000","last_clean":"2024-09-11T00:30:56.173110+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:30:56.173110+0000","last_undegraded":"2024-09-11T00:30:56.173110+0000","last_fullsized":"2024-09-11T00:30:56.173110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719858+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:30:56.719858+0000","last_peered":"2024-09-11T00:30:56.719858+0000","last_clean":"2024-09-11T00:30:56.719858+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:30:56.719858+0000","last_undegraded":"2024-09-11T00:30:56.719858+0000","last_fullsized":"2024-09-11T00:30:56.719858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968520+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:30:55.968520+0000","last_peered":"2024-09-11T00:30:55.968520+0000","last_clean":"2024-09-11T00:30:55.968520+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:30:55.968520+0000","last_undegraded":"2024-09-11T00:30:55.968520+0000","last_fullsized":"2024-09-11T00:30:55.968520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173273+0000","last_change":"2024-09-11T00:30:54.985091+0000","last_active":"2024-09-11T00:30:56.173273+0000","last_peered":"2024-09-11T00:30:56.173273+0000","last_clean":"2024-09-11T00:30:56.173273+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:30:56.173273+0000","last_undegraded":"2024-09-11T00:30:56.173273+0000","last_fullsized":"2024-09-11T00:30:56.173273+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:33.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968811+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:30:55.968811+0000","last_peered":"2024-09-11T00:30:55.968811+0000","last_clean":"2024-09-11T00:30:55.968811+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:30:55.968811+0000","last_undegraded":"2024-09-11T00:30:55.968811+0000","last_fullsized":"2024-09-11T00:30:55.968811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966522+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:30:55.966522+0000","last_peered":"2024-09-11T00:30:55.966522+0000","last_clean":"2024-09-11T00:30:55.966522+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:30:55.966522+0000","last_undegraded":"2024-09-11T00:30:55.966522+0000","last_fullsized":"2024-09-11T00:30:55.966522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971377+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:30:55.971377+0000","last_peered":"2024-09-11T00:30:55.971377+0000","last_clean":"2024-09-11T00:30:55.971377+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:30:55.971377+0000","last_undegraded":"2024-09-11T00:30:55.971377+0000","last_fullsized":"2024-09-11T00:30:55.971377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:11.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967648+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:30:55.967648+0000","last_peered":"2024-09-11T00:30:55.967648+0000","last_clean":"2024-09-11T00:30:55.967648+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:30:55.967648+0000","last_undegraded":"2024-09-11T00:30:55.967648+0000","last_fullsized":"2024-09-11T00:30:55.967648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968196+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:30:55.968196+0000","last_peered":"2024-09-11T00:30:55.968196+0000","last_clean":"2024-09-11T00:30:55.968196+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:30:55.968196+0000","last_undegraded":"2024-09-11T00:30:55.968196+0000","last_fullsized":"2024-09-11T00:30:55.968196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969528+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:30:55.969528+0000","last_peered":"2024-09-11T00:30:55.969528+0000","last_clean":"2024-09-11T00:30:55.969528+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:30:55.969528+0000","last_undegraded":"2024-09-11T00:30:55.969528+0000","last_fullsized":"2024-09-11T00:30:55.969528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967369+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:30:55.967369+0000","last_peered":"2024-09-11T00:30:55.967369+0000","last_clean":"2024-09-11T00:30:55.967369+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:30:55.967369+0000","last_undegraded":"2024-09-11T00:30:55.967369+0000","last_fullsized":"2024-09-11T00:30:55.967369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969812+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:30:55.969812+0000","last_peered":"2024-09-11T00:30:55.969812+0000","last_clean":"2024-09-11T00:30:55.969812+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:30:55.969812+0000","last_undegraded":"2024-09-11T00:30:55.969812+0000","last_fullsized":"2024-09-11T00:30:55.969812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969509+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:30:55.969509+0000","last_peered":"2024-09-11T00:30:55.969509+0000","last_clean":"2024-09-11T00:30:55.969509+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:30:55.969509+0000","last_undegraded":"2024-09-11T00:30:55.969509+0000","last_fullsized":"2024-09-11T00:30:55.969509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173337+0000","last_change":"2024-09-11T00:30:54.998464+0000","last_active":"2024-09-11T00:30:56.173337+0000","last_peered":"2024-09-11T00:30:56.173337+0000","last_clean":"2024-09-11T00:30:56.173337+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:30:56.173337+0000","last_undegraded":"2024-09-11T00:30:56.173337+0000","last_fullsized":"2024-09-11T00:30:56.173337+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:56.469791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998082+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:30:55.998082+0000","last_peered":"2024-09-11T00:30:55.998082+0000","last_clean":"2024-09-11T00:30:55.998082+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:30:55.998082+0000","last_undegraded":"2024-09-11T00:30:55.998082+0000","last_fullsized":"2024-09-11T00:30:55.998082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719278+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:30:56.719278+0000","last_peered":"2024-09-11T00:30:56.719278+0000","last_clean":"2024-09-11T00:30:56.719278+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:30:56.719278+0000","last_undegraded":"2024-09-11T00:30:56.719278+0000","last_fullsized":"2024-09-11T00:30:56.719278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001549+0000","last_change":"2024-09-11T00:30:53.987612+0000","last_active":"2024-09-11T00:30:56.001549+0000","last_peered":"2024-09-11T00:30:56.001549+0000","last_clean":"2024-09-11T00:30:56.001549+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:30:56.001549+0000","last_undegraded":"2024-09-11T00:30:56.001549+0000","last_fullsized":"2024-09-11T00:30:56.001549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:16.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970084+0000","last_change":"2024-09-11T00:30:54.989925+0000","last_active":"2024-09-11T00:30:55.970084+0000","last_peered":"2024-09-11T00:30:55.970084+0000","last_clean":"2024-09-11T00:30:55.970084+0000","last_became_active":"2024-09-11T00:30:54.989401+0000","last_became_peered":"2024-09-11T00:30:54.989401+0000","last_unstale":"2024-09-11T00:30:55.970084+0000","last_undegraded":"2024-09-11T00:30:55.970084+0000","last_fullsized":"2024-09-11T00:30:55.970084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:14.267175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172747+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:30:56.172747+0000","last_peered":"2024-09-11T00:30:56.172747+0000","last_clean":"2024-09-11T00:30:56.172747+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:30:56.172747+0000","last_undegraded":"2024-09-11T00:30:56.172747+0000","last_fullsized":"2024-09-11T00:30:56.172747+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969929+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:30:55.969929+0000","last_peered":"2024-09-11T00:30:55.969929+0000","last_clean":"2024-09-11T00:30:55.969929+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:30:55.969929+0000","last_undegraded":"2024-09-11T00:30:55.969929+0000","last_fullsized":"2024-09-11T00:30:55.969929+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000911+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:30:56.000911+0000","last_peered":"2024-09-11T00:30:56.000911+0000","last_clean":"2024-09-11T00:30:56.000911+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:30:56.000911+0000","last_undegraded":"2024-09-11T00:30:56.000911+0000","last_fullsized":"2024-09-11T00:30:56.000911+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172912+0000","last_change":"2024-09-11T00:30:54.979133+0000","last_active":"2024-09-11T00:30:56.172912+0000","last_peered":"2024-09-11T00:30:56.172912+0000","last_clean":"2024-09-11T00:30:56.172912+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:30:56.172912+0000","last_undegraded":"2024-09-11T00:30:56.172912+0000","last_fullsized":"2024-09-11T00:30:56.172912+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:20.961678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967251+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:30:55.967251+0000","last_peered":"2024-09-11T00:30:55.967251+0000","last_clean":"2024-09-11T00:30:55.967251+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:30:55.967251+0000","last_undegraded":"2024-09-11T00:30:55.967251+0000","last_fullsized":"2024-09-11T00:30:55.967251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997885+0000","last_change":"2024-09-11T00:30:54.988345+0000","last_active":"2024-09-11T00:30:55.997885+0000","last_peered":"2024-09-11T00:30:55.997885+0000","last_clean":"2024-09-11T00:30:55.997885+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:30:55.997885+0000","last_undegraded":"2024-09-11T00:30:55.997885+0000","last_fullsized":"2024-09-11T00:30:55.997885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.112194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998078+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:30:55.998078+0000","last_peered":"2024-09-11T00:30:55.998078+0000","last_clean":"2024-09-11T00:30:55.998078+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:30:55.998078+0000","last_undegraded":"2024-09-11T00:30:55.998078+0000","last_fullsized":"2024-09-11T00:30:55.998078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332489+0000","last_change":"2024-09-11T00:30:54.981185+0000","last_active":"2024-09-11T00:30:56.332489+0000","last_peered":"2024-09-11T00:30:56.332489+0000","last_clean":"2024-09-11T00:30:56.332489+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:30:56.332489+0000","last_undegraded":"2024-09-11T00:30:56.332489+0000","last_fullsized":"2024-09-11T00:30:56.332489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:45.740467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971241+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:30:55.971241+0000","last_peered":"2024-09-11T00:30:55.971241+0000","last_clean":"2024-09-11T00:30:55.971241+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:30:55.971241+0000","last_undegraded":"2024-09-11T00:30:55.971241+0000","last_fullsized":"2024-09-11T00:30:55.971241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332647+0000","last_change":"2024-09-11T00:30:54.992763+0000","last_active":"2024-09-11T00:30:56.332647+0000","last_peered":"2024-09-11T00:30:56.332647+0000","last_clean":"2024-09-11T00:30:56.332647+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:30:56.332647+0000","last_undegraded":"2024-09-11T00:30:56.332647+0000","last_fullsized":"2024-09-11T00:30:56.332647+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:02.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971179+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:30:55.971179+0000","last_peered":"2024-09-11T00:30:55.971179+0000","last_clean":"2024-09-11T00:30:55.971179+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:30:55.971179+0000","last_undegraded":"2024-09-11T00:30:55.971179+0000","last_fullsized":"2024-09-11T00:30:55.971179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966440+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:30:55.966440+0000","last_peered":"2024-09-11T00:30:55.966440+0000","last_clean":"2024-09-11T00:30:55.966440+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:30:55.966440+0000","last_undegraded":"2024-09-11T00:30:55.966440+0000","last_fullsized":"2024-09-11T00:30:55.966440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173066+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:30:56.173066+0000","last_peered":"2024-09-11T00:30:56.173066+0000","last_clean":"2024-09-11T00:30:56.173066+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:30:56.173066+0000","last_undegraded":"2024-09-11T00:30:56.173066+0000","last_fullsized":"2024-09-11T00:30:56.173066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172366+0000","last_change":"2024-09-11T00:30:54.984447+0000","last_active":"2024-09-11T00:30:56.172366+0000","last_peered":"2024-09-11T00:30:56.172366+0000","last_clean":"2024-09-11T00:30:56.172366+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:30:56.172366+0000","last_undegraded":"2024-09-11T00:30:56.172366+0000","last_fullsized":"2024-09-11T00:30:56.172366+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:49.024022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998072+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:30:55.998072+0000","last_peered":"2024-09-11T00:30:55.998072+0000","last_clean":"2024-09-11T00:30:55.998072+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:30:55.998072+0000","last_undegraded":"2024-09-11T00:30:55.998072+0000","last_fullsized":"2024-09-11T00:30:55.998072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001507+0000","last_change":"2024-09-11T00:30:54.996091+0000","last_active":"2024-09-11T00:30:56.001507+0000","last_peered":"2024-09-11T00:30:56.001507+0000","last_clean":"2024-09-11T00:30:56.001507+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:30:56.001507+0000","last_undegraded":"2024-09-11T00:30:56.001507+0000","last_fullsized":"2024-09-11T00:30:56.001507+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:42.987336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966504+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:30:55.966504+0000","last_peered":"2024-09-11T00:30:55.966504+0000","last_clean":"2024-09-11T00:30:55.966504+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:30:55.966504+0000","last_undegraded":"2024-09-11T00:30:55.966504+0000","last_fullsized":"2024-09-11T00:30:55.966504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968347+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:30:55.968347+0000","last_peered":"2024-09-11T00:30:55.968347+0000","last_clean":"2024-09-11T00:30:55.968347+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:30:55.968347+0000","last_undegraded":"2024-09-11T00:30:55.968347+0000","last_fullsized":"2024-09-11T00:30:55.968347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971167+0000","last_change":"2024-09-11T00:30:54.981799+0000","last_active":"2024-09-11T00:30:55.971167+0000","last_peered":"2024-09-11T00:30:55.971167+0000","last_clean":"2024-09-11T00:30:55.971167+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:30:55.971167+0000","last_undegraded":"2024-09-11T00:30:55.971167+0000","last_fullsized":"2024-09-11T00:30:55.971167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:42.670629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719169+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:30:56.719169+0000","last_peered":"2024-09-11T00:30:56.719169+0000","last_clean":"2024-09-11T00:30:56.719169+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:30:56.719169+0000","last_undegraded":"2024-09-11T00:30:56.719169+0000","last_fullsized":"2024-09-11T00:30:56.719169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001453+0000","last_change":"2024-09-11T00:30:54.992697+0000","last_active":"2024-09-11T00:30:56.001453+0000","last_peered":"2024-09-11T00:30:56.001453+0000","last_clean":"2024-09-11T00:30:56.001453+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:30:56.001453+0000","last_undegraded":"2024-09-11T00:30:56.001453+0000","last_fullsized":"2024-09-11T00:30:56.001453+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:54:07.227169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969635+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:30:55.969635+0000","last_peered":"2024-09-11T00:30:55.969635+0000","last_clean":"2024-09-11T00:30:55.969635+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:55.969635+0000","last_undegraded":"2024-09-11T00:30:55.969635+0000","last_fullsized":"2024-09-11T00:30:55.969635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172413+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:30:56.172413+0000","last_peered":"2024-09-11T00:30:56.172413+0000","last_clean":"2024-09-11T00:30:56.172413+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:30:56.172413+0000","last_undegraded":"2024-09-11T00:30:56.172413+0000","last_fullsized":"2024-09-11T00:30:56.172413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332844+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:30:56.332844+0000","last_peered":"2024-09-11T00:30:56.332844+0000","last_clean":"2024-09-11T00:30:56.332844+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:30:56.332844+0000","last_undegraded":"2024-09-11T00:30:56.332844+0000","last_fullsized":"2024-09-11T00:30:56.332844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332895+0000","last_change":"2024-09-11T00:30:54.985827+0000","last_active":"2024-09-11T00:30:56.332895+0000","last_peered":"2024-09-11T00:30:56.332895+0000","last_clean":"2024-09-11T00:30:56.332895+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:30:56.332895+0000","last_undegraded":"2024-09-11T00:30:56.332895+0000","last_fullsized":"2024-09-11T00:30:56.332895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:32.027117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968441+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:30:55.968441+0000","last_peered":"2024-09-11T00:30:55.968441+0000","last_clean":"2024-09-11T00:30:55.968441+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:30:55.968441+0000","last_undegraded":"2024-09-11T00:30:55.968441+0000","last_fullsized":"2024-09-11T00:30:55.968441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172255+0000","last_change":"2024-09-11T00:30:54.989906+0000","last_active":"2024-09-11T00:30:56.172255+0000","last_peered":"2024-09-11T00:30:56.172255+0000","last_clean":"2024-09-11T00:30:56.172255+0000","last_became_active":"2024-09-11T00:30:54.989376+0000","last_became_peered":"2024-09-11T00:30:54.989376+0000","last_unstale":"2024-09-11T00:30:56.172255+0000","last_undegraded":"2024-09-11T00:30:56.172255+0000","last_fullsized":"2024-09-11T00:30:56.172255+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:18:00.138941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001665+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:30:56.001665+0000","last_peered":"2024-09-11T00:30:56.001665+0000","last_clean":"2024-09-11T00:30:56.001665+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:30:56.001665+0000","last_undegraded":"2024-09-11T00:30:56.001665+0000","last_fullsized":"2024-09-11T00:30:56.001665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968974+0000","last_change":"2024-09-11T00:30:54.993156+0000","last_active":"2024-09-11T00:30:55.968974+0000","last_peered":"2024-09-11T00:30:55.968974+0000","last_clean":"2024-09-11T00:30:55.968974+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:30:55.968974+0000","last_undegraded":"2024-09-11T00:30:55.968974+0000","last_fullsized":"2024-09-11T00:30:55.968974+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:25.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968016+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:30:55.968016+0000","last_peered":"2024-09-11T00:30:55.968016+0000","last_clean":"2024-09-11T00:30:55.968016+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:30:55.968016+0000","last_undegraded":"2024-09-11T00:30:55.968016+0000","last_fullsized":"2024-09-11T00:30:55.968016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970335+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:30:55.970335+0000","last_peered":"2024-09-11T00:30:55.970335+0000","last_clean":"2024-09-11T00:30:55.970335+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:30:55.970335+0000","last_undegraded":"2024-09-11T00:30:55.970335+0000","last_fullsized":"2024-09-11T00:30:55.970335+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966434+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:30:55.966434+0000","last_peered":"2024-09-11T00:30:55.966434+0000","last_clean":"2024-09-11T00:30:55.966434+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:30:55.966434+0000","last_undegraded":"2024-09-11T00:30:55.966434+0000","last_fullsized":"2024-09-11T00:30:55.966434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719699+0000","last_change":"2024-09-11T00:30:54.993461+0000","last_active":"2024-09-11T00:30:56.719699+0000","last_peered":"2024-09-11T00:30:56.719699+0000","last_clean":"2024-09-11T00:30:56.719699+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:30:56.719699+0000","last_undegraded":"2024-09-11T00:30:56.719699+0000","last_fullsized":"2024-09-11T00:30:56.719699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:07.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997281+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:30:55.997281+0000","last_peered":"2024-09-11T00:30:55.997281+0000","last_clean":"2024-09-11T00:30:55.997281+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:30:55.997281+0000","last_undegraded":"2024-09-11T00:30:55.997281+0000","last_fullsized":"2024-09-11T00:30:55.997281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719307+0000","last_change":"2024-09-11T00:30:54.986265+0000","last_active":"2024-09-11T00:30:56.719307+0000","last_peered":"2024-09-11T00:30:56.719307+0000","last_clean":"2024-09-11T00:30:56.719307+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:30:56.719307+0000","last_undegraded":"2024-09-11T00:30:56.719307+0000","last_fullsized":"2024-09-11T00:30:56.719307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:49.109328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969766+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:30:55.969766+0000","last_peered":"2024-09-11T00:30:55.969766+0000","last_clean":"2024-09-11T00:30:55.969766+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:30:55.969766+0000","last_undegraded":"2024-09-11T00:30:55.969766+0000","last_fullsized":"2024-09-11T00:30:55.969766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967854+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:30:55.967854+0000","last_peered":"2024-09-11T00:30:55.967854+0000","last_clean":"2024-09-11T00:30:55.967854+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:30:55.967854+0000","last_undegraded":"2024-09-11T00:30:55.967854+0000","last_fullsized":"2024-09-11T00:30:55.967854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971368+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:30:55.971368+0000","last_peered":"2024-09-11T00:30:55.971368+0000","last_clean":"2024-09-11T00:30:55.971368+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:30:55.971368+0000","last_undegraded":"2024-09-11T00:30:55.971368+0000","last_fullsized":"2024-09-11T00:30:55.971368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969452+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:30:55.969452+0000","last_peered":"2024-09-11T00:30:55.969452+0000","last_clean":"2024-09-11T00:30:55.969452+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:30:55.969452+0000","last_undegraded":"2024-09-11T00:30:55.969452+0000","last_fullsized":"2024-09-11T00:30:55.969452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332691+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:30:56.332691+0000","last_peered":"2024-09-11T00:30:56.332691+0000","last_clean":"2024-09-11T00:30:56.332691+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:30:56.332691+0000","last_undegraded":"2024-09-11T00:30:56.332691+0000","last_fullsized":"2024-09-11T00:30:56.332691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970850+0000","last_change":"2024-09-11T00:30:54.994988+0000","last_active":"2024-09-11T00:30:55.970850+0000","last_peered":"2024-09-11T00:30:55.970850+0000","last_clean":"2024-09-11T00:30:55.970850+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:30:55.970850+0000","last_undegraded":"2024-09-11T00:30:55.970850+0000","last_fullsized":"2024-09-11T00:30:55.970850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:31.973641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332924+0000","last_change":"2024-09-11T00:30:45.941635+0000","last_active":"2024-09-11T00:30:56.332924+0000","last_peered":"2024-09-11T00:30:56.332924+0000","last_clean":"2024-09-11T00:30:56.332924+0000","last_became_active":"2024-09-11T00:30:45.941357+0000","last_became_peered":"2024-09-11T00:30:45.941357+0000","last_unstale":"2024-09-11T00:30:56.332924+0000","last_undegraded":"2024-09-11T00:30:56.332924+0000","last_fullsized":"2024-09-11T00:30:56.332924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:12.016406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332421+0000","last_change":"2024-09-11T00:30:54.984767+0000","last_active":"2024-09-11T00:30:56.332421+0000","last_peered":"2024-09-11T00:30:56.332421+0000","last_clean":"2024-09-11T00:30:56.332421+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:30:56.332421+0000","last_undegraded":"2024-09-11T00:30:56.332421+0000","last_fullsized":"2024-09-11T00:30:56.332421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:44.833677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332955+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:30:56.332955+0000","last_peered":"2024-09-11T00:30:56.332955+0000","last_clean":"2024-09-11T00:30:56.332955+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:56.332955+0000","last_undegraded":"2024-09-11T00:30:56.332955+0000","last_fullsized":"2024-09-11T00:30:56.332955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332911+0000","last_change":"2024-09-11T00:30:54.986423+0000","last_active":"2024-09-11T00:30:56.332911+0000","last_peered":"2024-09-11T00:30:56.332911+0000","last_clean":"2024-09-11T00:30:56.332911+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:30:56.332911+0000","last_undegraded":"2024-09-11T00:30:56.332911+0000","last_fullsized":"2024-09-11T00:30:56.332911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:16:50.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719859+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:30:56.719859+0000","last_peered":"2024-09-11T00:30:56.719859+0000","last_clean":"2024-09-11T00:30:56.719859+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:30:56.719859+0000","last_undegraded":"2024-09-11T00:30:56.719859+0000","last_fullsized":"2024-09-11T00:30:56.719859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998062+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:30:55.998062+0000","last_peered":"2024-09-11T00:30:55.998062+0000","last_clean":"2024-09-11T00:30:55.998062+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:30:55.998062+0000","last_undegraded":"2024-09-11T00:30:55.998062+0000","last_fullsized":"2024-09-11T00:30:55.998062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997283+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:30:55.997283+0000","last_peered":"2024-09-11T00:30:55.997283+0000","last_clean":"2024-09-11T00:30:55.997283+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:30:55.997283+0000","last_undegraded":"2024-09-11T00:30:55.997283+0000","last_fullsized":"2024-09-11T00:30:55.997283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173138+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:30:56.173138+0000","last_peered":"2024-09-11T00:30:56.173138+0000","last_clean":"2024-09-11T00:30:56.173138+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:30:56.173138+0000","last_undegraded":"2024-09-11T00:30:56.173138+0000","last_fullsized":"2024-09-11T00:30:56.173138+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333239+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:30:56.333239+0000","last_peered":"2024-09-11T00:30:56.333239+0000","last_clean":"2024-09-11T00:30:56.333239+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:30:56.333239+0000","last_undegraded":"2024-09-11T00:30:56.333239+0000","last_fullsized":"2024-09-11T00:30:56.333239+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970256+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:30:55.970256+0000","last_peered":"2024-09-11T00:30:55.970256+0000","last_clean":"2024-09-11T00:30:55.970256+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:30:55.970256+0000","last_undegraded":"2024-09-11T00:30:55.970256+0000","last_fullsized":"2024-09-11T00:30:55.970256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172539+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:30:56.172539+0000","last_peered":"2024-09-11T00:30:56.172539+0000","last_clean":"2024-09-11T00:30:56.172539+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:30:56.172539+0000","last_undegraded":"2024-09-11T00:30:56.172539+0000","last_fullsized":"2024-09-11T00:30:56.172539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719531+0000","last_change":"2024-09-11T00:30:54.990631+0000","last_active":"2024-09-11T00:30:56.719531+0000","last_peered":"2024-09-11T00:30:56.719531+0000","last_clean":"2024-09-11T00:30:56.719531+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:30:56.719531+0000","last_undegraded":"2024-09-11T00:30:56.719531+0000","last_fullsized":"2024-09-11T00:30:56.719531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:02.422471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333220+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:30:56.333220+0000","last_peered":"2024-09-11T00:30:56.333220+0000","last_clean":"2024-09-11T00:30:56.333220+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:30:56.333220+0000","last_undegraded":"2024-09-11T00:30:56.333220+0000","last_fullsized":"2024-09-11T00:30:56.333220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:54.991047+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:41:48.229261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173027+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:30:56.173027+0000","last_peered":"2024-09-11T00:30:56.173027+0000","last_clean":"2024-09-11T00:30:56.173027+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:30:56.173027+0000","last_undegraded":"2024-09-11T00:30:56.173027+0000","last_fullsized":"2024-09-11T00:30:56.173027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997710+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:30:55.997710+0000","last_peered":"2024-09-11T00:30:55.997710+0000","last_clean":"2024-09-11T00:30:55.997710+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:30:55.997710+0000","last_undegraded":"2024-09-11T00:30:55.997710+0000","last_fullsized":"2024-09-11T00:30:55.997710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997737+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:30:55.997737+0000","last_peered":"2024-09-11T00:30:55.997737+0000","last_clean":"2024-09-11T00:30:55.997737+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:30:55.997737+0000","last_undegraded":"2024-09-11T00:30:55.997737+0000","last_fullsized":"2024-09-11T00:30:55.997737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172698+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:30:56.172698+0000","last_peered":"2024-09-11T00:30:56.172698+0000","last_clean":"2024-09-11T00:30:56.172698+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:30:56.172698+0000","last_undegraded":"2024-09-11T00:30:56.172698+0000","last_fullsized":"2024-09-11T00:30:56.172698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997914+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:30:55.997914+0000","last_peered":"2024-09-11T00:30:55.997914+0000","last_clean":"2024-09-11T00:30:55.997914+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:30:55.997914+0000","last_undegraded":"2024-09-11T00:30:55.997914+0000","last_fullsized":"2024-09-11T00:30:55.997914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172531+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:30:56.172531+0000","last_peered":"2024-09-11T00:30:56.172531+0000","last_clean":"2024-09-11T00:30:56.172531+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:30:56.172531+0000","last_undegraded":"2024-09-11T00:30:56.172531+0000","last_fullsized":"2024-09-11T00:30:56.172531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333081+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:30:56.333081+0000","last_peered":"2024-09-11T00:30:56.333081+0000","last_clean":"2024-09-11T00:30:56.333081+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:30:56.333081+0000","last_undegraded":"2024-09-11T00:30:56.333081+0000","last_fullsized":"2024-09-11T00:30:56.333081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509449,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":286720,"data_stored":106265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":307200,"data_stored":108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:31:00.686 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T00:31:00.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T00:31:00.687 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T00:31:00.879 INFO:tasks.ceph.mds.b.smithi204.stderr:2024-09-11T00:31:00.876+0000 7f3d0c200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:31:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.877+0000 7f9ed6a00640 1 -- 172.21.15.31:0/1795236055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed00c6310 msgr2=0x7f9ed0159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.877+0000 7f9ed6a00640 1 --2- 172.21.15.31:0/1795236055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed00c6310 0x7f9ed0159670 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8005fd0 tx=0x7f9ec802a8a0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.877+0000 7f9ed6a00640 1 -- 172.21.15.31:0/1795236055 shutdown_connections 2024-09-11T00:31:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.877+0000 7f9ed6a00640 1 --2- 172.21.15.31:0/1795236055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed00c6310 0x7f9ed0159670 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.877+0000 7f9ed6a00640 1 --2- 172.21.15.31:0/1795236055 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ed00c5b10 0x7f9ed0156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.877+0000 7f9ed6a00640 1 -- 172.21.15.31:0/1795236055 >> 172.21.15.31:0/1795236055 conn(0x7f9ed00a9750 msgr2=0x7f9ed00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.877+0000 7f9ed6a00640 1 -- 172.21.15.31:0/1795236055 shutdown_connections 2024-09-11T00:31:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.877+0000 7f9ed6a00640 1 -- 172.21.15.31:0/1795236055 wait complete. 2024-09-11T00:31:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.879+0000 7f9ed6a00640 1 Processor -- start 2024-09-11T00:31:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.879+0000 7f9ed6a00640 1 -- start start 2024-09-11T00:31:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ed6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ed00c5b10 0x7f9ed0155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ed6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed00c5ef0 0x7f9ed014cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ed6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ed00c6310 0x7f9ed014d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ed6a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed0153b90 con 0x7f9ed00c5ef0 2024-09-11T00:31:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ed6a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed00c3950 con 0x7f9ed00c5b10 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ed6a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9ed0153d10 con 0x7f9ed00c6310 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ec7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed00c5ef0 0x7f9ed014cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ec7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed00c5ef0 0x7f9ed014cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37202/0 (socket says 172.21.15.31:37202) 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ec7e00640 1 -- 172.21.15.31:0/3050147387 learned_addr learned my addr 172.21.15.31:0/3050147387 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ecfe00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ed00c5b10 0x7f9ed0155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ed4a00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ed00c6310 0x7f9ed014d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ec7e00640 1 -- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ed00c6310 msgr2=0x7f9ed014d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ec7e00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ed00c6310 0x7f9ed014d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ec7e00640 1 -- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ed00c5b10 msgr2=0x7f9ed0155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ec7e00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ed00c5b10 0x7f9ed0155bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ec7e00640 1 -- 172.21.15.31:0/3050147387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed01e2270 con 0x7f9ed00c5ef0 2024-09-11T00:31:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ecfe00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ed00c5b10 0x7f9ed0155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.880+0000 7f9ed4a00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ed00c6310 0x7f9ed014d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.881+0000 7f9ec7e00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed00c5ef0 0x7f9ed014cc80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800c450 tx=0x7f9eb800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.881+0000 7f9eccc00640 1 -- 172.21.15.31:0/3050147387 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8016020 con 0x7f9ed00c5ef0 2024-09-11T00:31:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.881+0000 7f9eccc00640 1 -- 172.21.15.31:0/3050147387 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9eb8012920 con 0x7f9ed00c5ef0 2024-09-11T00:31:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.881+0000 7f9eccc00640 1 -- 172.21.15.31:0/3050147387 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8012d30 con 0x7f9ed00c5ef0 2024-09-11T00:31:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.881+0000 7f9ed6a00640 1 -- 172.21.15.31:0/3050147387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed00abb50 con 0x7f9ed00c5ef0 2024-09-11T00:31:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.881+0000 7f9ed6a00640 1 -- 172.21.15.31:0/3050147387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed01e9b70 con 0x7f9ed00c5ef0 2024-09-11T00:31:00.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.882+0000 7f9ed6a00640 1 -- 172.21.15.31:0/3050147387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ed0156ad0 con 0x7f9ed00c5ef0 2024-09-11T00:31:00.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.883+0000 7f9eccc00640 1 -- 172.21.15.31:0/3050147387 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9eb801b400 con 0x7f9ed00c5ef0 2024-09-11T00:31:00.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.884+0000 7f9eccc00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9eac077f70 0x7f9eac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:00.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.884+0000 7f9eccc00640 1 -- 172.21.15.31:0/3050147387 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f9eb809a3f0 con 0x7f9ed00c5ef0 2024-09-11T00:31:00.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.885+0000 7f9ecfe00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9eac077f70 0x7f9eac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:00.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.885+0000 7f9ecfe00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9eac077f70 0x7f9eac07a430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9ec000baa0 tx=0x7f9ec0005d90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:00.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:00.888+0000 7f9eccc00640 1 -- 172.21.15.31:0/3050147387 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9eb8020020 con 0x7f9ed00c5ef0 2024-09-11T00:31:01.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.086+0000 7f9ed6a00640 1 -- 172.21.15.31:0/3050147387 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f9ed00a70e0 con 0x7f9ed00c5ef0 2024-09-11T00:31:01.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.086+0000 7f9eccc00640 1 -- 172.21.15.31:0/3050147387 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v31) ==== 86+0+160 (secure 0 0 0) 0x7f9eb8062b20 con 0x7f9ed00c5ef0 2024-09-11T00:31:01.088 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:01.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 -- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9eac077f70 msgr2=0x7f9eac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9eac077f70 0x7f9eac07a430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9ec000baa0 tx=0x7f9ec0005d90 comp rx=0 tx=0).stop 2024-09-11T00:31:01.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 -- 172.21.15.31:0/3050147387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed00c5ef0 msgr2=0x7f9ed014cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed00c5ef0 0x7f9ed014cc80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9eb800c450 tx=0x7f9eb800c910 comp rx=0 tx=0).stop 2024-09-11T00:31:01.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 -- 172.21.15.31:0/3050147387 shutdown_connections 2024-09-11T00:31:01.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9eac077f70 0x7f9eac07a430 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ed00c6310 0x7f9ed014d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ed00c5ef0 0x7f9ed014cc80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 --2- 172.21.15.31:0/3050147387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ed00c5b10 0x7f9ed0155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 -- 172.21.15.31:0/3050147387 >> 172.21.15.31:0/3050147387 conn(0x7f9ed00a9750 msgr2=0x7f9ed00a76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:01.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 -- 172.21.15.31:0/3050147387 shutdown_connections 2024-09-11T00:31:01.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.089+0000 7f9ed6a00640 1 -- 172.21.15.31:0/3050147387 wait complete. 2024-09-11T00:31:01.106 INFO:teuthology.orchestra.run.smithi031.stdout:[{"version":31,"timestamp":"2024-09-11T00:30:33.302340+0000","name":"","changes":[{"name":"osd.3/osd_mclock_max_capacity_iops_ssd","new_value":"3938.350443"}]}] 2024-09-11T00:31:01.106 INFO:tasks.ceph_manager:config epoch is 31 2024-09-11T00:31:01.106 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T00:31:01.107 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T00:31:01.107 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T00:31:01.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.265+0000 7f95ae600640 1 -- 172.21.15.31:0/2435550800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f958c004d90 msgr2=0x7f958c0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.265+0000 7f95ae600640 1 --2- 172.21.15.31:0/2435550800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f958c004d90 0x7f958c0051f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f95a0001dc0 tx=0x7f958c005830 comp rx=0 tx=0).stop 2024-09-11T00:31:01.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.265+0000 7f95adc00640 1 -- 172.21.15.31:0/2435550800 reap_dead start 2024-09-11T00:31:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.265+0000 7f95ae600640 1 -- 172.21.15.31:0/2435550800 shutdown_connections 2024-09-11T00:31:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.265+0000 7f95ae600640 1 -- 172.21.15.31:0/2435550800 >> 172.21.15.31:0/2435550800 conn(0x7f95a80c4d20 msgr2=0x7f95a80c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.265+0000 7f95ae600640 1 -- 172.21.15.31:0/2435550800 shutdown_connections 2024-09-11T00:31:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.265+0000 7f95ae600640 1 -- 172.21.15.31:0/2435550800 wait complete. 2024-09-11T00:31:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.267+0000 7f95ae600640 1 Processor -- start 2024-09-11T00:31:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.267+0000 7f95ae600640 1 -- start start 2024-09-11T00:31:01.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.268+0000 7f95ae600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f958c004d90 0x7f95a81dea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:01.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.268+0000 7f95ae600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95a8156e30 0x7f95a81e1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:01.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.268+0000 7f95ae600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95a8156900 0x7f95a81ef780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:01.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.268+0000 7f95ae600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95a806e390 con 0x7f95a8156e30 2024-09-11T00:31:01.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.268+0000 7f95ae600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f95a806e210 con 0x7f95a8156900 2024-09-11T00:31:01.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.268+0000 7f95ae600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f95a814cf60 con 0x7f958c004d90 2024-09-11T00:31:01.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.268+0000 7f959fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95a8156e30 0x7f95a81e1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:01.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.268+0000 7f959fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95a8156e30 0x7f95a81e1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37216/0 (socket says 172.21.15.31:37216) 2024-09-11T00:31:01.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.268+0000 7f959fe00640 1 -- 172.21.15.31:0/811526400 learned_addr learned my addr 172.21.15.31:0/811526400 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.268+0000 7f95ad200640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f958c004d90 0x7f95a81dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f95adc00640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95a8156900 0x7f95a81ef780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f959fe00640 1 -- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f958c004d90 msgr2=0x7f95a81dea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f959fe00640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f958c004d90 0x7f95a81dea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f959fe00640 1 -- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95a8156900 msgr2=0x7f95a81ef780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f959fe00640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95a8156900 0x7f95a81ef780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f959fe00640 1 -- 172.21.15.31:0/811526400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a81f03a0 con 0x7f95a8156e30 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f95ad200640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f958c004d90 0x7f95a81dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f959fe00640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95a8156e30 0x7f95a81e1ec0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9590006680 tx=0x7f95900066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f95a5600640 1 -- 172.21.15.31:0/811526400 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590015070 con 0x7f95a8156e30 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f95a5600640 1 -- 172.21.15.31:0/811526400 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9590011920 con 0x7f95a8156e30 2024-09-11T00:31:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f95ae600640 1 -- 172.21.15.31:0/811526400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a8073fd0 con 0x7f95a8156e30 2024-09-11T00:31:01.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f95a5600640 1 -- 172.21.15.31:0/811526400 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590011d70 con 0x7f95a8156e30 2024-09-11T00:31:01.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.269+0000 7f95ae600640 1 -- 172.21.15.31:0/811526400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a8150630 con 0x7f95a8156e30 2024-09-11T00:31:01.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.270+0000 7f95ae600640 1 -- 172.21.15.31:0/811526400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a81ecb10 con 0x7f95a8156e30 2024-09-11T00:31:01.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.272+0000 7f95a5600640 1 -- 172.21.15.31:0/811526400 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9590010420 con 0x7f95a8156e30 2024-09-11T00:31:01.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.272+0000 7f95a5600640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9584077fa0 0x7f958407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:01.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.273+0000 7f95a5600640 1 -- 172.21.15.31:0/811526400 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f959009b0f0 con 0x7f95a8156e30 2024-09-11T00:31:01.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.273+0000 7f95ad200640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9584077fa0 0x7f958407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:01.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.274+0000 7f95ad200640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9584077fa0 0x7f958407a460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9598009c50 tx=0x7f9598005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.276+0000 7f95a5600640 1 -- 172.21.15.31:0/811526400 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f95900638b0 con 0x7f95a8156e30 2024-09-11T00:31:01.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.520+0000 7f95ae600640 1 -- 172.21.15.31:0/811526400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f95a8070310 con 0x7f95a8156e30 2024-09-11T00:31:01.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.523+0000 7f95a5600640 1 -- 172.21.15.31:0/811526400 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+193248 (secure 0 0 0) 0x7f95900631b0 con 0x7f95a8156e30 2024-09-11T00:31:01.527 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 -- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9584077fa0 msgr2=0x7f958407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9584077fa0 0x7f958407a460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9598009c50 tx=0x7f9598005f90 comp rx=0 tx=0).stop 2024-09-11T00:31:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 -- 172.21.15.31:0/811526400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95a8156e30 msgr2=0x7f95a81e1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95a8156e30 0x7f95a81e1ec0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9590006680 tx=0x7f95900066b0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 -- 172.21.15.31:0/811526400 shutdown_connections 2024-09-11T00:31:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9584077fa0 0x7f958407a460 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95a8156900 0x7f95a81ef780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95a8156e30 0x7f95a81e1ec0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 --2- 172.21.15.31:0/811526400 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f958c004d90 0x7f95a81dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 -- 172.21.15.31:0/811526400 >> 172.21.15.31:0/811526400 conn(0x7f95a80c4d20 msgr2=0x7f95a814bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 -- 172.21.15.31:0/811526400 shutdown_connections 2024-09-11T00:31:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.526+0000 7f95ae600640 1 -- 172.21.15.31:0/811526400 wait complete. 2024-09-11T00:31:01.544 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2097467593},{"type":"v1","addr":"172.21.15.204:6801","nonce":2097467593}]},"active_addr":"172.21.15.204:6801/2097467593","active_change":"2024-09-11T00:30:24.380225+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4098,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":4032249938}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":960761776}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":2080814969}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.204:0","nonce":221189731}]}]} 2024-09-11T00:31:01.548 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T00:31:01.548 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T00:31:01.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:31:01.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.707+0000 7f70daa00640 1 -- 172.21.15.31:0/213604482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70d41518b0 msgr2=0x7f70d40c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.707+0000 7f70daa00640 1 --2- 172.21.15.31:0/213604482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70d41518b0 0x7f70d40c3600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f70cc005fd0 tx=0x7f70cc02ab20 comp rx=0 tx=0).stop 2024-09-11T00:31:01.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.707+0000 7f70daa00640 1 -- 172.21.15.31:0/213604482 shutdown_connections 2024-09-11T00:31:01.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.708+0000 7f70daa00640 1 --2- 172.21.15.31:0/213604482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70d41518b0 0x7f70d40c3600 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.708+0000 7f70daa00640 1 --2- 172.21.15.31:0/213604482 >> v2:172.21.15.204:3300/0 conn(0x7f70d40c23a0 0x7f70d40c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.708+0000 7f70daa00640 1 -- 172.21.15.31:0/213604482 >> 172.21.15.31:0/213604482 conn(0x7f70d40a9750 msgr2=0x7f70d40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:01.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.708+0000 7f70daa00640 1 -- 172.21.15.31:0/213604482 shutdown_connections 2024-09-11T00:31:01.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.708+0000 7f70daa00640 1 -- 172.21.15.31:0/213604482 wait complete. 2024-09-11T00:31:01.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.709+0000 7f70daa00640 1 Processor -- start 2024-09-11T00:31:01.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.709+0000 7f70daa00640 1 -- start start 2024-09-11T00:31:01.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.710+0000 7f70daa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d4072480 0x7f70d41f0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:01.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.710+0000 7f70daa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70d40c23a0 0x7f70d414b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.710+0000 7f70daa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70d41518b0 0x7f70d41ec180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.710+0000 7f70daa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70d4074240 con 0x7f70d4072480 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.710+0000 7f70daa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f70d40740c0 con 0x7f70d41518b0 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.710+0000 7f70daa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f70d4072860 con 0x7f70d40c23a0 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.710+0000 7f70d3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d4072480 0x7f70d41f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.710+0000 7f70d3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d4072480 0x7f70d41f0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37226/0 (socket says 172.21.15.31:37226) 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.710+0000 7f70d3e00640 1 -- 172.21.15.31:0/1333681811 learned_addr learned my addr 172.21.15.31:0/1333681811 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.710+0000 7f70d3400640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70d40c23a0 0x7f70d414b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70d8a00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70d41518b0 0x7f70d41ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70d3e00640 1 -- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70d40c23a0 msgr2=0x7f70d414b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70d3e00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70d40c23a0 0x7f70d414b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70d3e00640 1 -- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70d41518b0 msgr2=0x7f70d41ec180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70d3e00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70d41518b0 0x7f70d41ec180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70d3e00640 1 -- 172.21.15.31:0/1333681811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d40a9070 con 0x7f70d4072480 2024-09-11T00:31:01.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70d3e00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d4072480 0x7f70d41f0340 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f70c400c480 tx=0x7f70c400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:01.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70cbe00640 1 -- 172.21.15.31:0/1333681811 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c40052d0 con 0x7f70d4072480 2024-09-11T00:31:01.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70cbe00640 1 -- 172.21.15.31:0/1333681811 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70c4005470 con 0x7f70d4072480 2024-09-11T00:31:01.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70daa00640 1 -- 172.21.15.31:0/1333681811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d4153520 con 0x7f70d4072480 2024-09-11T00:31:01.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.711+0000 7f70cbe00640 1 -- 172.21.15.31:0/1333681811 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c4012870 con 0x7f70d4072480 2024-09-11T00:31:01.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.712+0000 7f70daa00640 1 -- 172.21.15.31:0/1333681811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d41537c0 con 0x7f70d4072480 2024-09-11T00:31:01.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.713+0000 7f70daa00640 1 -- 172.21.15.31:0/1333681811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d40c3cb0 con 0x7f70d4072480 2024-09-11T00:31:01.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.714+0000 7f70cbe00640 1 -- 172.21.15.31:0/1333681811 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f70c40035f0 con 0x7f70d4072480 2024-09-11T00:31:01.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.715+0000 7f70cbe00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70b4077f70 0x7f70b407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:01.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.715+0000 7f70cbe00640 1 -- 172.21.15.31:0/1333681811 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f70c4099c40 con 0x7f70d4072480 2024-09-11T00:31:01.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.715+0000 7f70d3400640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70b4077f70 0x7f70b407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:01.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.716+0000 7f70d3400640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70b4077f70 0x7f70b407a430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f70d40c2220 tx=0x7f70c0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:01.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.719+0000 7f70cbe00640 1 -- 172.21.15.31:0/1333681811 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f70c4061d30 con 0x7f70d4072480 2024-09-11T00:31:01.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.916+0000 7f70daa00640 1 -- 172.21.15.31:0/1333681811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70d414b9e0 con 0x7f70d4072480 2024-09-11T00:31:01.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.917+0000 7f70cbe00640 1 -- 172.21.15.31:0/1333681811 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20383 (secure 0 0 0) 0x7f70c4066760 con 0x7f70d4072480 2024-09-11T00:31:01.919 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:01.919 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:55.962415+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 -- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70b4077f70 msgr2=0x7f70b407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70b4077f70 0x7f70b407a430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f70d40c2220 tx=0x7f70c0005f90 comp rx=0 tx=0).stop 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 -- 172.21.15.31:0/1333681811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d4072480 msgr2=0x7f70d41f0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d4072480 0x7f70d41f0340 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f70c400c480 tx=0x7f70c400c940 comp rx=0 tx=0).stop 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 -- 172.21.15.31:0/1333681811 shutdown_connections 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70b4077f70 0x7f70b407a430 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70d41518b0 0x7f70d41ec180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70d40c23a0 0x7f70d414b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 --2- 172.21.15.31:0/1333681811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d4072480 0x7f70d41f0340 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 -- 172.21.15.31:0/1333681811 >> 172.21.15.31:0/1333681811 conn(0x7f70d40a9750 msgr2=0x7f70d414bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 -- 172.21.15.31:0/1333681811 shutdown_connections 2024-09-11T00:31:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:01.920+0000 7f70daa00640 1 -- 172.21.15.31:0/1333681811 wait complete. 2024-09-11T00:31:01.939 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T00:31:01.939 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:31:02.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.100+0000 7fa845800640 1 -- 172.21.15.31:0/2792423269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8400c23a0 msgr2=0x7fa8400c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.100+0000 7fa845800640 1 --2- 172.21.15.31:0/2792423269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8400c23a0 0x7fa8400c30c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa828005fd0 tx=0x7fa82802a660 comp rx=0 tx=0).stop 2024-09-11T00:31:02.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.100+0000 7fa845800640 1 -- 172.21.15.31:0/2792423269 shutdown_connections 2024-09-11T00:31:02.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.100+0000 7fa845800640 1 --2- 172.21.15.31:0/2792423269 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8401518b0 0x7fa8400c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.100+0000 7fa845800640 1 --2- 172.21.15.31:0/2792423269 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8400c23a0 0x7fa8400c30c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.100+0000 7fa845800640 1 -- 172.21.15.31:0/2792423269 >> 172.21.15.31:0/2792423269 conn(0x7fa8400a9750 msgr2=0x7fa8400a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.100+0000 7fa845800640 1 -- 172.21.15.31:0/2792423269 shutdown_connections 2024-09-11T00:31:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.100+0000 7fa845800640 1 -- 172.21.15.31:0/2792423269 wait complete. 2024-09-11T00:31:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.101+0000 7fa845800640 1 Processor -- start 2024-09-11T00:31:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.102+0000 7fa845800640 1 -- start start 2024-09-11T00:31:02.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.102+0000 7fa845800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa840072480 0x7fa8401e1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.102+0000 7fa845800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8400c23a0 0x7fa840152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.102+0000 7fa845800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8401518b0 0x7fa8401533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.102+0000 7fa845800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa840074200 con 0x7fa8400c23a0 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.102+0000 7fa845800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa840074080 con 0x7fa8401518b0 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.102+0000 7fa845800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa840072860 con 0x7fa840072480 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa836000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8400c23a0 0x7fa840152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa836000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8400c23a0 0x7fa840152ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37228/0 (socket says 172.21.15.31:37228) 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa836000640 1 -- 172.21.15.31:0/786928838 learned_addr learned my addr 172.21.15.31:0/786928838 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa83f400640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8401518b0 0x7fa8401533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa83ea00640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa840072480 0x7fa8401e1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa836000640 1 -- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa840072480 msgr2=0x7fa8401e1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa836000640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa840072480 0x7fa8401e1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa836000640 1 -- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8401518b0 msgr2=0x7fa8401533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa836000640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8401518b0 0x7fa8401533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa836000640 1 -- 172.21.15.31:0/786928838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa840152a30 con 0x7fa8400c23a0 2024-09-11T00:31:02.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa83f400640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8401518b0 0x7fa8401533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:02.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.103+0000 7fa836000640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8400c23a0 0x7fa840152ea0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa828009850 tx=0x7fa8280038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.104+0000 7fa837400640 1 -- 172.21.15.31:0/786928838 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82803f070 con 0x7fa8400c23a0 2024-09-11T00:31:02.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.104+0000 7fa837400640 1 -- 172.21.15.31:0/786928838 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8280047a0 con 0x7fa8400c23a0 2024-09-11T00:31:02.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.104+0000 7fa845800640 1 -- 172.21.15.31:0/786928838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8401e24a0 con 0x7fa8400c23a0 2024-09-11T00:31:02.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.104+0000 7fa837400640 1 -- 172.21.15.31:0/786928838 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828035030 con 0x7fa8400c23a0 2024-09-11T00:31:02.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.104+0000 7fa845800640 1 -- 172.21.15.31:0/786928838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8400ace10 con 0x7fa8400c23a0 2024-09-11T00:31:02.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.105+0000 7fa845800640 1 -- 172.21.15.31:0/786928838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8400c3b90 con 0x7fa8400c23a0 2024-09-11T00:31:02.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.106+0000 7fa837400640 1 -- 172.21.15.31:0/786928838 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa828004b90 con 0x7fa8400c23a0 2024-09-11T00:31:02.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.107+0000 7fa837400640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa81c077fa0 0x7fa81c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.107+0000 7fa837400640 1 -- 172.21.15.31:0/786928838 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fa8280bb1c0 con 0x7fa8400c23a0 2024-09-11T00:31:02.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.107+0000 7fa83ea00640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa81c077fa0 0x7fa81c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.108+0000 7fa83ea00640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa81c077fa0 0x7fa81c07a460 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa830000f60 tx=0x7fa830009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.111+0000 7fa837400640 1 -- 172.21.15.31:0/786928838 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa828083900 con 0x7fa8400c23a0 2024-09-11T00:31:02.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.306+0000 7fa845800640 1 -- 172.21.15.31:0/786928838 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8400adb20 con 0x7fa8400c23a0 2024-09-11T00:31:02.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.308+0000 7fa837400640 1 -- 172.21.15.31:0/786928838 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20383 (secure 0 0 0) 0x7fa828083200 con 0x7fa8400c23a0 2024-09-11T00:31:02.310 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:02.310 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:55.962415+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:31:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.310+0000 7fa845800640 1 -- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa81c077fa0 msgr2=0x7fa81c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.310+0000 7fa845800640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa81c077fa0 0x7fa81c07a460 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa830000f60 tx=0x7fa830009040 comp rx=0 tx=0).stop 2024-09-11T00:31:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.310+0000 7fa845800640 1 -- 172.21.15.31:0/786928838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8400c23a0 msgr2=0x7fa840152ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.310+0000 7fa845800640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8400c23a0 0x7fa840152ea0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa828009850 tx=0x7fa8280038a0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.310+0000 7fa845800640 1 -- 172.21.15.31:0/786928838 shutdown_connections 2024-09-11T00:31:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.310+0000 7fa845800640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa81c077fa0 0x7fa81c07a460 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.310+0000 7fa845800640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa8401518b0 0x7fa8401533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.310+0000 7fa845800640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8400c23a0 0x7fa840152ea0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.310+0000 7fa845800640 1 --2- 172.21.15.31:0/786928838 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa840072480 0x7fa8401e1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.310+0000 7fa845800640 1 -- 172.21.15.31:0/786928838 >> 172.21.15.31:0/786928838 conn(0x7fa8400a9750 msgr2=0x7fa84014fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.311+0000 7fa845800640 1 -- 172.21.15.31:0/786928838 shutdown_connections 2024-09-11T00:31:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.311+0000 7fa845800640 1 -- 172.21.15.31:0/786928838 wait complete. 2024-09-11T00:31:02.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T00:31:02.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T00:31:02.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T00:31:02.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T00:31:02.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-11T00:31:02.331 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-11T00:31:02.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-11T00:31:02.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-11T00:31:02.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.616+0000 7faad7800640 1 -- 172.21.15.31:0/932116961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac8006bc0 msgr2=0x7faac8005d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.616+0000 7faad7800640 1 --2- 172.21.15.31:0/932116961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac8006bc0 0x7faac8005d30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faac4009830 tx=0x7faac402a250 comp rx=0 tx=0).stop 2024-09-11T00:31:02.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.616+0000 7faad7800640 1 -- 172.21.15.31:0/932116961 shutdown_connections 2024-09-11T00:31:02.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.616+0000 7faad7800640 1 --2- 172.21.15.31:0/932116961 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac8006bc0 0x7faac8005d30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.616+0000 7faad7800640 1 -- 172.21.15.31:0/932116961 >> 172.21.15.31:0/932116961 conn(0x7faac804c090 msgr2=0x7faac804c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.617+0000 7faad7800640 1 -- 172.21.15.31:0/932116961 shutdown_connections 2024-09-11T00:31:02.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.617+0000 7faad7800640 1 -- 172.21.15.31:0/932116961 wait complete. 2024-09-11T00:31:02.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.617+0000 7f11e7200640 1 -- 172.21.15.31:0/1845995733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11e8159ee0 msgr2=0x7f11e8156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.617+0000 7f11e7200640 1 --2- 172.21.15.31:0/1845995733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11e8159ee0 0x7f11e8156280 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f11e0005fd0 tx=0x7f11e002a4a0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.619+0000 7f11e7200640 1 -- 172.21.15.31:0/1845995733 shutdown_connections 2024-09-11T00:31:02.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.619+0000 7f11e7200640 1 --2- 172.21.15.31:0/1845995733 >> v2:172.21.15.204:3300/0 conn(0x7f11e80c62f0 0x7f11e80c2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.619+0000 7f11e7200640 1 --2- 172.21.15.31:0/1845995733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11e8159ee0 0x7f11e8156280 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.619+0000 7f11e7200640 1 -- 172.21.15.31:0/1845995733 >> 172.21.15.31:0/1845995733 conn(0x7f11e806caf0 msgr2=0x7f11e806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.619+0000 7fadb3e00640 1 -- 172.21.15.31:0/279547750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadb40ad750 msgr2=0x7fadb406e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.620+0000 7fadb3e00640 1 --2- 172.21.15.31:0/279547750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadb40ad750 0x7fadb406e140 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fadb408d7a0 tx=0x7fadac02aa10 comp rx=0 tx=0).stop 2024-09-11T00:31:02.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.620+0000 7f11e7200640 1 -- 172.21.15.31:0/1845995733 shutdown_connections 2024-09-11T00:31:02.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.620+0000 7f11e7200640 1 -- 172.21.15.31:0/1845995733 wait complete. 2024-09-11T00:31:02.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.620+0000 7faad7800640 1 Processor -- start 2024-09-11T00:31:02.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.621+0000 7faad7800640 1 -- start start 2024-09-11T00:31:02.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.621+0000 7fadb3e00640 1 -- 172.21.15.31:0/279547750 shutdown_connections 2024-09-11T00:31:02.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.621+0000 7fadb3e00640 1 --2- 172.21.15.31:0/279547750 >> v2:172.21.15.31:3300/0 conn(0x7fadb40cc220 0x7fadb414ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.621+0000 7fadb3e00640 1 --2- 172.21.15.31:0/279547750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadb40ad750 0x7fadb406e140 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.621+0000 7fadb3e00640 1 -- 172.21.15.31:0/279547750 >> 172.21.15.31:0/279547750 conn(0x7fadb4068f10 msgr2=0x7fadb40cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.622+0000 7faad7800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faac8006bc0 0x7faac81783d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.622+0000 7f11e7200640 1 Processor -- start 2024-09-11T00:31:02.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.622+0000 7faad7800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac804d510 0x7faac80d50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.622+0000 7faad7800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faac804dbb0 0x7faac8030650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.622+0000 7faad7800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faac80daa80 con 0x7faac8006bc0 2024-09-11T00:31:02.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.622+0000 7faad7800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faac8006fa0 con 0x7faac804d510 2024-09-11T00:31:02.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.622+0000 7faad7800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faac80d9b20 con 0x7faac804dbb0 2024-09-11T00:31:02.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7faad4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faac8006bc0 0x7faac81783d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7fadb3e00640 1 -- 172.21.15.31:0/279547750 shutdown_connections 2024-09-11T00:31:02.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7fadb3e00640 1 -- 172.21.15.31:0/279547750 wait complete. 2024-09-11T00:31:02.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7f11e7200640 1 -- start start 2024-09-11T00:31:02.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7faad4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faac8006bc0 0x7faac81783d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37268/0 (socket says 172.21.15.31:37268) 2024-09-11T00:31:02.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.624+0000 7f70eb200640 1 -- 172.21.15.31:0/1947479236 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70d0003aa0 msgr2=0x7f70d0003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.624+0000 7f70eb200640 1 --2- 172.21.15.31:0/1947479236 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70d0003aa0 0x7f70d0003ec0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f70e400c560 tx=0x7f70d0005830 comp rx=0 tx=0).stop 2024-09-11T00:31:02.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.624+0000 7f70ea800640 1 -- 172.21.15.31:0/1947479236 reap_dead start 2024-09-11T00:31:02.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7faad4e00640 1 -- 172.21.15.31:0/2900440743 learned_addr learned my addr 172.21.15.31:0/2900440743 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:02.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7faad4e00640 1 -- 172.21.15.31:0/2900440743 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faac804dbb0 msgr2=0x7faac8030650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:31:02.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7faad4e00640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faac804dbb0 0x7faac8030650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7faad4e00640 1 -- 172.21.15.31:0/2900440743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac804d510 msgr2=0x7faac80d50e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:31:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7faad4e00640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faac804d510 0x7faac80d50e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.623+0000 7faad4e00640 1 -- 172.21.15.31:0/2900440743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac8031b00 con 0x7faac8006bc0 2024-09-11T00:31:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.624+0000 7faad4e00640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faac8006bc0 0x7faac81783d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faac4009950 tx=0x7faac4003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.625+0000 7faaccc00640 1 -- 172.21.15.31:0/2900440743 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4037040 con 0x7faac8006bc0 2024-09-11T00:31:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.625+0000 7faaccc00640 1 -- 172.21.15.31:0/2900440743 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faac4003e00 con 0x7faac8006bc0 2024-09-11T00:31:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.625+0000 7faaccc00640 1 -- 172.21.15.31:0/2900440743 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4005280 con 0x7faac8006bc0 2024-09-11T00:31:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.625+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac81633b0 con 0x7faac8006bc0 2024-09-11T00:31:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.625+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac8033500 con 0x7faac8006bc0 2024-09-11T00:31:02.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.626+0000 7f11e7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11e80c4c00 0x7f11e81f9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.626+0000 7fadb3e00640 1 Processor -- start 2024-09-11T00:31:02.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.626+0000 7f11e7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11e80c62f0 0x7f11e81f8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.626+0000 7f11e7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11e8159ee0 0x7f11e80ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.625+0000 7f70eb200640 1 -- 172.21.15.31:0/1947479236 shutdown_connections 2024-09-11T00:31:02.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.625+0000 7f70eb200640 1 -- 172.21.15.31:0/1947479236 >> 172.21.15.31:0/1947479236 conn(0x7f70ec068ac0 msgr2=0x7f70ec068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.625+0000 7f70eb200640 1 -- 172.21.15.31:0/1947479236 shutdown_connections 2024-09-11T00:31:02.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.625+0000 7f70eb200640 1 -- 172.21.15.31:0/1947479236 wait complete. 2024-09-11T00:31:02.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.627+0000 7f11e5e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11e80c4c00 0x7f11e81f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.627+0000 7f11e7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f11e815a2c0 con 0x7f11e80c62f0 2024-09-11T00:31:02.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.627+0000 7fadb3e00640 1 -- start start 2024-09-11T00:31:02.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.627+0000 7f11e7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f11e80c5f40 con 0x7f11e8159ee0 2024-09-11T00:31:02.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.627+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faaac001030 con 0x7faac8006bc0 2024-09-11T00:31:02.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.627+0000 7f11e7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f11e80c2a80 con 0x7f11e80c4c00 2024-09-11T00:31:02.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.627+0000 7f11e5e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11e80c4c00 0x7f11e81f9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50660/0 (socket says 172.21.15.31:50660) 2024-09-11T00:31:02.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.627+0000 7f11e5e00640 1 -- 172.21.15.31:0/2250709565 learned_addr learned my addr 172.21.15.31:0/2250709565 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.627+0000 7f70eb200640 1 Processor -- start 2024-09-11T00:31:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.627+0000 7f11e6800640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11e8159ee0 0x7f11e80ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.628+0000 7fadb3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadb40ad750 0x7fadb41ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.628+0000 7fadb3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadb40cc220 0x7fadb4147250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.628+0000 7fadb3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadb4156960 0x7fadb4142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.628+0000 7fadb3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fadb40adb30 con 0x7fadb4156960 2024-09-11T00:31:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.628+0000 7fadb3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fadb40adcb0 con 0x7fadb40cc220 2024-09-11T00:31:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.628+0000 7fadb3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fadb406e680 con 0x7fadb40ad750 2024-09-11T00:31:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.628+0000 7f11e5400640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11e80c62f0 0x7f11e81f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.628+0000 7f70eb200640 1 -- start start 2024-09-11T00:31:02.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.628+0000 7fafb1e00640 1 -- 172.21.15.31:0/3422998117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafac0c3d10 msgr2=0x7fafac06e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7fadb2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadb40cc220 0x7fadb4147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7fadb2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadb40ad750 0x7fadb41ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7fadb2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadb40cc220 0x7fadb4147250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:36946/0 (socket says 172.21.15.31:36946) 2024-09-11T00:31:02.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7fadb2000640 1 -- 172.21.15.31:0/4191976738 learned_addr learned my addr 172.21.15.31:0/4191976738 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:02.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.628+0000 7fafb1e00640 1 --2- 172.21.15.31:0/3422998117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafac0c3d10 0x7fafac06e480 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fafa4005fd0 tx=0x7fafa402a890 comp rx=0 tx=0).stop 2024-09-11T00:31:02.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7f11e5400640 1 -- 172.21.15.31:0/2250709565 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11e80c4c00 msgr2=0x7f11e81f9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7f11e5400640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11e80c4c00 0x7f11e81f9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7f11e5400640 1 -- 172.21.15.31:0/2250709565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11e8159ee0 msgr2=0x7f11e80ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7f11e5400640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11e8159ee0 0x7f11e80ab7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7f11e5400640 1 -- 172.21.15.31:0/2250709565 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e8156af0 con 0x7f11e80c62f0 2024-09-11T00:31:02.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7fafb1e00640 1 -- 172.21.15.31:0/3422998117 shutdown_connections 2024-09-11T00:31:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7fafb1e00640 1 --2- 172.21.15.31:0/3422998117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafac0c3d10 0x7fafac06e480 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7fafb1e00640 1 --2- 172.21.15.31:0/3422998117 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac070570 0x7fafac14f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7fafb1e00640 1 -- 172.21.15.31:0/3422998117 >> 172.21.15.31:0/3422998117 conn(0x7fafac0a9750 msgr2=0x7fafac0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.629+0000 7fadb3400640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadb4156960 0x7fadb4142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7fadb2a00640 1 -- 172.21.15.31:0/4191976738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadb40cc220 msgr2=0x7fadb4147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7fadb2a00640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadb40cc220 0x7fadb4147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7fadb2a00640 1 -- 172.21.15.31:0/4191976738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadb4156960 msgr2=0x7fadb4142e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f70eb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d0003aa0 0x7f70ec0c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f70eb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70ec0c4810 0x7f70ec0ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f11e5400640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11e80c62f0 0x7f11e81f8860 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11d000c410 tx=0x7f11d000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f70eb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0ad720 0x7f70ec0c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f70eb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec159de0 con 0x7f70d0003aa0 2024-09-11T00:31:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f70eb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec156640 con 0x7f70ec0c4810 2024-09-11T00:31:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f70eb200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f70ec159f60 con 0x7f70ec0ad720 2024-09-11T00:31:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f70e9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d0003aa0 0x7f70ec0c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f70ea800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0ad720 0x7f70ec0c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f70ea800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0ad720 0x7f70ec0c2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50686/0 (socket says 172.21.15.31:50686) 2024-09-11T00:31:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7f70ea800640 1 -- 172.21.15.31:0/2784598877 learned_addr learned my addr 172.21.15.31:0/2784598877 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7fafb1e00640 1 -- 172.21.15.31:0/3422998117 shutdown_connections 2024-09-11T00:31:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f11de000640 1 -- 172.21.15.31:0/2250709565 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0017070 con 0x7f11e80c62f0 2024-09-11T00:31:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f70e9400640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70ec0c4810 0x7f70ec0ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f70ea800640 1 -- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70ec0c4810 msgr2=0x7f70ec0ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f70ea800640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70ec0c4810 0x7f70ec0ad1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f70ea800640 1 -- 172.21.15.31:0/2784598877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d0003aa0 msgr2=0x7f70ec0c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f70ea800640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70d0003aa0 0x7f70ec0c42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f70ea800640 1 -- 172.21.15.31:0/2784598877 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ec0bf870 con 0x7f70ec0ad720 2024-09-11T00:31:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7fafb1e00640 1 -- 172.21.15.31:0/3422998117 wait complete. 2024-09-11T00:31:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e80a9c10 con 0x7f11e80c62f0 2024-09-11T00:31:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e8070040 con 0x7f11e80c62f0 2024-09-11T00:31:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f11de000640 1 -- 172.21.15.31:0/2250709565 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11d0005320 con 0x7f11e80c62f0 2024-09-11T00:31:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7faaccc00640 1 -- 172.21.15.31:0/2900440743 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7faac403a070 con 0x7faac8006bc0 2024-09-11T00:31:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7fadb2a00640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadb4156960 0x7fadb4142e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.630+0000 7fadb2a00640 1 -- 172.21.15.31:0/4191976738 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb414ac10 con 0x7fadb40ad750 2024-09-11T00:31:02.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.632+0000 7fadb3400640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadb4156960 0x7fadb4142e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:02.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.632+0000 7fadb2a00640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadb40ad750 0x7fadb41ee000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fadb4153c10 tx=0x7fadac003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.631+0000 7f11de000640 1 -- 172.21.15.31:0/2250709565 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0021060 con 0x7f11e80c62f0 2024-09-11T00:31:02.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.632+0000 7f70ea800640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0ad720 0x7f70ec0c2c10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f70ec06e320 tx=0x7f70d800d9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.632+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f11c0001030 con 0x7f11e80c62f0 2024-09-11T00:31:02.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.633+0000 7fadaaa00640 1 -- 172.21.15.31:0/4191976738 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac046070 con 0x7fadb40ad750 2024-09-11T00:31:02.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.633+0000 7fadaaa00640 1 -- 172.21.15.31:0/4191976738 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadac004250 con 0x7fadb40ad750 2024-09-11T00:31:02.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.633+0000 7fadaaa00640 1 -- 172.21.15.31:0/4191976738 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadac033050 con 0x7fadb40ad750 2024-09-11T00:31:02.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.633+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb414db20 con 0x7fadb40ad750 2024-09-11T00:31:02.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.633+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb414e0e0 con 0x7fadb40ad750 2024-09-11T00:31:02.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.633+0000 7faaccc00640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faab0077fc0 0x7faab007a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.634+0000 7fafb1e00640 1 Processor -- start 2024-09-11T00:31:02.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.634+0000 7f70e2000640 1 -- 172.21.15.31:0/2784598877 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d801a070 con 0x7f70ec0ad720 2024-09-11T00:31:02.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.634+0000 7f70e2000640 1 -- 172.21.15.31:0/2784598877 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70d8004320 con 0x7f70ec0ad720 2024-09-11T00:31:02.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.634+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70ec143690 con 0x7f70ec0ad720 2024-09-11T00:31:02.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.633+0000 7faaccc00640 1 -- 172.21.15.31:0/2900440743 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7faac40ba190 con 0x7faac8006bc0 2024-09-11T00:31:02.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.633+0000 7faaccc00640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] conn(0x7faab0082860 0x7faab0084c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.633+0000 7faaccc00640 1 -- 172.21.15.31:0/2900440743 --> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faac4001d30 con 0x7faab0082860 2024-09-11T00:31:02.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.633+0000 7faaccc00640 1 -- 172.21.15.31:0/2900440743 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faac40ba670 con 0x7faac8006bc0 2024-09-11T00:31:02.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.634+0000 7faacfe00640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faab0077fc0 0x7faab007a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.634+0000 7f70e2000640 1 -- 172.21.15.31:0/2784598877 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d800f8f0 con 0x7f70ec0ad720 2024-09-11T00:31:02.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.634+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f70ec14b390 con 0x7f70ec0ad720 2024-09-11T00:31:02.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.634+0000 7faad5800640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] conn(0x7faab0082860 0x7faab0084c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.635+0000 7fd34ee00640 1 -- 172.21.15.31:0/2439866438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3480c7330 msgr2=0x7fd348161b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.636+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fadb4154970 con 0x7fadb40ad750 2024-09-11T00:31:02.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.635+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f70ec06e850 con 0x7f70ec0ad720 2024-09-11T00:31:02.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.634+0000 7fafb1e00640 1 -- start start 2024-09-11T00:31:02.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.636+0000 7faad5800640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] conn(0x7faab0082860 0x7faab0084c80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.636+0000 7fafb1e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafac070570 0x7fafac151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.636+0000 7fafb1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafac0c2ff0 0x7fafac14fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.636+0000 7fafb1e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c3d10 0x7fafac150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.636+0000 7fafb1e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fafac0c5fb0 con 0x7fafac0c2ff0 2024-09-11T00:31:02.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.636+0000 7fafb1e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fafac14fb20 con 0x7fafac070570 2024-09-11T00:31:02.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.636+0000 7fafb1e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fafac0c7690 con 0x7fafac0c3d10 2024-09-11T00:31:02.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.635+0000 7fd34ee00640 1 --2- 172.21.15.31:0/2439866438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3480c7330 0x7fd348161b70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd340005fd0 tx=0x7fd34002a830 comp rx=0 tx=0).stop 2024-09-11T00:31:02.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.637+0000 7faacfe00640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faab0077fc0 0x7faab007a480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faac8030150 tx=0x7faabc007930 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.637+0000 7faaccc00640 1 -- 172.21.15.31:0/2900440743 <== osd.0 v2:172.21.15.31:6808/2913875274 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faac4001d30 con 0x7faab0082860 2024-09-11T00:31:02.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.637+0000 7f11de000640 1 -- 172.21.15.31:0/2250709565 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f11d00047e0 con 0x7f11e80c62f0 2024-09-11T00:31:02.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.636+0000 7fafab400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c3d10 0x7fafac150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.637+0000 7fafab400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c3d10 0x7fafac150320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50688/0 (socket says 172.21.15.31:50688) 2024-09-11T00:31:02.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.637+0000 7fafab400640 1 -- 172.21.15.31:0/3249532 learned_addr learned my addr 172.21.15.31:0/3249532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:02.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.637+0000 7fafab400640 1 -- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafac070570 msgr2=0x7fafac151410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T00:31:02.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.637+0000 7fafab400640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafac070570 0x7fafac151410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.637+0000 7fafab400640 1 -- 172.21.15.31:0/3249532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafac0c2ff0 msgr2=0x7fafac14fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.637+0000 7fafab400640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafac0c2ff0 0x7fafac14fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.638+0000 7fafab400640 1 -- 172.21.15.31:0/3249532 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafac1f07b0 con 0x7fafac0c3d10 2024-09-11T00:31:02.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.638+0000 7fafab400640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c3d10 0x7fafac150320 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fafa4006010 tx=0x7fafa4003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.638+0000 7fd34ee00640 1 -- 172.21.15.31:0/2439866438 shutdown_connections 2024-09-11T00:31:02.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.638+0000 7fd34ee00640 1 --2- 172.21.15.31:0/2439866438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3480c7330 0x7fd348161b70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.638+0000 7fd34ee00640 1 -- 172.21.15.31:0/2439866438 >> 172.21.15.31:0/2439866438 conn(0x7fd3480c8cc0 msgr2=0x7fd3480ad700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.638+0000 7f11de000640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f11c4077fa0 0x7f11c407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.638+0000 7f11de000640 1 -- 172.21.15.31:0/2250709565 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f11d0099650 con 0x7f11e80c62f0 2024-09-11T00:31:02.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.638+0000 7fadaaa00640 1 -- 172.21.15.31:0/4191976738 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fadac004660 con 0x7fadb40ad750 2024-09-11T00:31:02.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.639+0000 7f11de000640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f11c40827b0 0x7f11c4084bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.639+0000 7f11de000640 1 -- 172.21.15.31:0/2250709565 --> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11d0016ec0 con 0x7f11c40827b0 2024-09-11T00:31:02.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.639+0000 7f11de000640 1 -- 172.21.15.31:0/2250709565 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f11d00a0050 con 0x7f11e80c62f0 2024-09-11T00:31:02.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.639+0000 7fd34ee00640 1 -- 172.21.15.31:0/2439866438 shutdown_connections 2024-09-11T00:31:02.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.639+0000 7fafa3400640 1 -- 172.21.15.31:0/3249532 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4031d80 con 0x7fafac0c3d10 2024-09-11T00:31:02.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.639+0000 7fadaaa00640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad94077f70 0x7fad9407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.639+0000 7fadaaa00640 1 -- 172.21.15.31:0/4191976738 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fadac0bae30 con 0x7fadb40ad750 2024-09-11T00:31:02.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.640+0000 7fadaaa00640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] conn(0x7fad94082780 0x7fad94084ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.640+0000 7fadaaa00640 1 -- 172.21.15.31:0/4191976738 --> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fadac02be90 con 0x7fad94082780 2024-09-11T00:31:02.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.640+0000 7fadaaa00640 1 -- 172.21.15.31:0/4191976738 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fadac0bd450 con 0x7fadb40ad750 2024-09-11T00:31:02.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.640+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafac14cca0 con 0x7fafac0c3d10 2024-09-11T00:31:02.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.640+0000 7f70e2000640 1 -- 172.21.15.31:0/2784598877 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f70d803bac0 con 0x7f70ec0ad720 2024-09-11T00:31:02.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.640+0000 7fadb3400640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] conn(0x7fad94082780 0x7fad94084ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.640+0000 7f11e5e00640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f11c4077fa0 0x7f11c407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.640+0000 7fafa3400640 1 -- 172.21.15.31:0/3249532 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafa4004040 con 0x7fafac0c3d10 2024-09-11T00:31:02.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.640+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fafac1f0d30 con 0x7fafac0c3d10 2024-09-11T00:31:02.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.640+0000 7fafa3400640 1 -- 172.21.15.31:0/3249532 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4004470 con 0x7fafac0c3d10 2024-09-11T00:31:02.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.641+0000 7fadb3400640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] conn(0x7fad94082780 0x7fad94084ba0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.641+0000 7f11e6800640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f11c40827b0 0x7f11c4084bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.641+0000 7f11e5e00640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f11c4077fa0 0x7f11c407a460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009230 tx=0x7f11e0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.639+0000 7fd34ee00640 1 -- 172.21.15.31:0/2439866438 wait complete. 2024-09-11T00:31:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.642+0000 7fd34ee00640 1 Processor -- start 2024-09-11T00:31:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.642+0000 7fadaaa00640 1 -- 172.21.15.31:0/4191976738 <== osd.1 v2:172.21.15.31:6800/3912540156 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadac02be90 con 0x7fad94082780 2024-09-11T00:31:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.642+0000 7fd34ee00640 1 -- start start 2024-09-11T00:31:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.642+0000 7f11e6800640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f11c40827b0 0x7f11c4084bd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.642+0000 7fec84200640 1 -- 172.21.15.31:0/141283120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec7c0c2e80 msgr2=0x7fec7c15f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.642+0000 7fec84200640 1 --2- 172.21.15.31:0/141283120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec7c0c2e80 0x7fec7c15f8b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fec68005fd0 tx=0x7fec6802a4b0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.643+0000 7fec84200640 1 -- 172.21.15.31:0/141283120 shutdown_connections 2024-09-11T00:31:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.643+0000 7fec84200640 1 --2- 172.21.15.31:0/141283120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec7c0c2e80 0x7fec7c15f8b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.643+0000 7fec84200640 1 -- 172.21.15.31:0/141283120 >> 172.21.15.31:0/141283120 conn(0x7fec7c0a9750 msgr2=0x7fec7c0a9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.643+0000 7fec84200640 1 -- 172.21.15.31:0/141283120 shutdown_connections 2024-09-11T00:31:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.643+0000 7fec84200640 1 -- 172.21.15.31:0/141283120 wait complete. 2024-09-11T00:31:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.643+0000 7f11de000640 1 -- 172.21.15.31:0/2250709565 <== osd.2 v2:172.21.15.31:6816/2777449146 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f11d0016ec0 con 0x7f11c40827b0 2024-09-11T00:31:02.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.643+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf84001030 con 0x7fafac0c3d10 2024-09-11T00:31:02.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7fd34ee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd34806cf40 0x7fd3481586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7fd34ee00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3480c7330 0x7fd3480caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7fd34ee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd348153c10 0x7fd3480c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7fd34ee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd348159f40 con 0x7fd34806cf40 2024-09-11T00:31:02.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7fd34ee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd3480c6590 con 0x7fd348153c10 2024-09-11T00:31:02.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7fd34ee00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd3480ae520 con 0x7fd3480c7330 2024-09-11T00:31:02.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7fd34ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd348153c10 0x7fd3480c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7fd34ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd348153c10 0x7fd3480c55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:36982/0 (socket says 172.21.15.31:36982) 2024-09-11T00:31:02.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7f86b9000640 1 -- 172.21.15.31:0/2166877136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86b41537c0 msgr2=0x7f86b40c31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7f86b9000640 1 --2- 172.21.15.31:0/2166877136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86b41537c0 0x7f86b40c31e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f86b4159790 tx=0x7f86a402a600 comp rx=0 tx=0).stop 2024-09-11T00:31:02.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7fadb2000640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad94077f70 0x7fad9407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.644+0000 7fd34ce00640 1 -- 172.21.15.31:0/996875078 learned_addr learned my addr 172.21.15.31:0/996875078 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:02.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7f86b9000640 1 -- 172.21.15.31:0/2166877136 shutdown_connections 2024-09-11T00:31:02.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7f86b9000640 1 --2- 172.21.15.31:0/2166877136 >> v2:172.21.15.31:3300/0 conn(0x7f86b40c2d40 0x7f86b4159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7f86b9000640 1 --2- 172.21.15.31:0/2166877136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86b41537c0 0x7f86b40c31e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7f86b9000640 1 -- 172.21.15.31:0/2166877136 >> 172.21.15.31:0/2166877136 conn(0x7f86b40a9750 msgr2=0x7f86b40a9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7fd347e00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd34806cf40 0x7fd3481586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7fd34ce00640 1 -- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3480c7330 msgr2=0x7fd3480caf20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:31:02.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7fd34ce00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd3480c7330 0x7fd3480caf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7fd34ce00640 1 -- 172.21.15.31:0/996875078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd34806cf40 msgr2=0x7fd3481586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7f86b9000640 1 -- 172.21.15.31:0/2166877136 shutdown_connections 2024-09-11T00:31:02.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7fec84200640 1 Processor -- start 2024-09-11T00:31:02.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.646+0000 7fadb2000640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad94077f70 0x7fad9407a430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fadb4148120 tx=0x7fada40090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7f86b9000640 1 -- 172.21.15.31:0/2166877136 wait complete. 2024-09-11T00:31:02.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.646+0000 7fec84200640 1 -- start start 2024-09-11T00:31:02.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.645+0000 7f70e2000640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70cc078070 0x7f70cc07a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.646+0000 7f70e2000640 1 -- 172.21.15.31:0/2784598877 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f70d8099910 con 0x7f70ec0ad720 2024-09-11T00:31:02.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.646+0000 7f70e2000640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] conn(0x7f70cc082900 0x7f70cc084d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.646+0000 7f70e2000640 1 -- 172.21.15.31:0/2784598877 --> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70d8019ec0 con 0x7f70cc082900 2024-09-11T00:31:02.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.646+0000 7f70e2000640 1 -- 172.21.15.31:0/2784598877 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f70d8099d00 con 0x7f70ec0ad720 2024-09-11T00:31:02.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.646+0000 7fafa3400640 1 -- 172.21.15.31:0/3249532 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fafa4004610 con 0x7fafac0c3d10 2024-09-11T00:31:02.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.646+0000 7fd34ce00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd34806cf40 0x7fd3481586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.646+0000 7fd34ce00640 1 -- 172.21.15.31:0/996875078 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3480c2730 con 0x7fd348153c10 2024-09-11T00:31:02.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.647+0000 7fd347e00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd34806cf40 0x7fd3481586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:02.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.647+0000 7fd34ce00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd348153c10 0x7fd3480c55e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd340009cd0 tx=0x7fd3400324d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.647+0000 7f70e9e00640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70cc078070 0x7f70cc07a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.647+0000 7fd33fe00640 1 -- 172.21.15.31:0/996875078 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340042020 con 0x7fd348153c10 2024-09-11T00:31:02.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.648+0000 7fec84200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec7c0c23a0 0x7fec7c14d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.648+0000 7f86b9000640 1 Processor -- start 2024-09-11T00:31:02.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.648+0000 7f70e9400640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] conn(0x7f70cc082900 0x7f70cc084d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.648+0000 7fec84200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec7c0c2e80 0x7fec7c1ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.648+0000 7fec84200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec7c151760 0x7fec7c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.648+0000 7fd33fe00640 1 -- 172.21.15.31:0/996875078 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd340038040 con 0x7fd348153c10 2024-09-11T00:31:02.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.648+0000 7fd33fe00640 1 -- 172.21.15.31:0/996875078 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340037070 con 0x7fd348153c10 2024-09-11T00:31:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fec80e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec7c0c2e80 0x7fec7c1ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fec80e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec7c0c2e80 0x7fec7c1ef9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37350/0 (socket says 172.21.15.31:37350) 2024-09-11T00:31:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fec80e00640 1 -- 172.21.15.31:0/4025860993 learned_addr learned my addr 172.21.15.31:0/4025860993 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.648+0000 7fafa3400640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faf8c078070 0x7faf8c07a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.648+0000 7fafa3400640 1 -- 172.21.15.31:0/3249532 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fafa40bb530 con 0x7fafac0c3d10 2024-09-11T00:31:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fafa3400640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] conn(0x7faf8c082900 0x7faf8c084d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fafa3400640 1 -- 172.21.15.31:0/3249532 --> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafa40317c0 con 0x7faf8c082900 2024-09-11T00:31:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fafa3400640 1 -- 172.21.15.31:0/3249532 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fafa40c2050 con 0x7fafac0c3d10 2024-09-11T00:31:02.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c06d920 con 0x7fec7c0c2e80 2024-09-11T00:31:02.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c151b40 con 0x7fec7c0c23a0 2024-09-11T00:31:02.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fec7c0a6f90 con 0x7fec7c151760 2024-09-11T00:31:02.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fec81800640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec7c0c23a0 0x7fec7c14d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7fec82200640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec7c151760 0x7fec7c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7f70e9400640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] conn(0x7f70cc082900 0x7f70cc084d20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.648+0000 7f86b9000640 1 -- start start 2024-09-11T00:31:02.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7f86b9000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86b40c2d40 0x7f86b40aab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7f86b9000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86b40c7460 0x7f86b41585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.649+0000 7f86b9000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86b41537c0 0x7f86b41f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7f86b9000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41560a0 con 0x7f86b40c2d40 2024-09-11T00:31:02.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7f86b9000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f86b40c6c60 con 0x7f86b41537c0 2024-09-11T00:31:02.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7f86b9000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f86b4153ba0 con 0x7f86b40c7460 2024-09-11T00:31:02.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7f86b1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86b40c2d40 0x7f86b40aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7fafaaa00640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faf8c078070 0x7faf8c07a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3481f3940 con 0x7fd348153c10 2024-09-11T00:31:02.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3481f3f00 con 0x7fd348153c10 2024-09-11T00:31:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7fafaa000640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] conn(0x7faf8c082900 0x7faf8c084d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7f86b2800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86b41537c0 0x7f86b41f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7f86b2800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86b41537c0 0x7f86b41f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:37006/0 (socket says 172.21.15.31:37006) 2024-09-11T00:31:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7f86b2800640 1 -- 172.21.15.31:0/1058234459 learned_addr learned my addr 172.21.15.31:0/1058234459 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7f70e9e00640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70cc078070 0x7f70cc07a530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f70e4033000 tx=0x7f70e4001cf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.650+0000 7f86abe00640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86b40c7460 0x7f86b41585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7fec80e00640 1 -- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec7c151760 msgr2=0x7fec7c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7fec80e00640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec7c151760 0x7fec7c06f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7fec80e00640 1 -- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec7c0c23a0 msgr2=0x7fec7c14d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7fec80e00640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec7c0c23a0 0x7fec7c14d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7fec80e00640 1 -- 172.21.15.31:0/4025860993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7c150a60 con 0x7fec7c0c2e80 2024-09-11T00:31:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7f70e2000640 1 -- 172.21.15.31:0/2784598877 <== osd.3 v2:172.21.15.31:6824/4217184896 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70d8019ec0 con 0x7f70cc082900 2024-09-11T00:31:02.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7fafaaa00640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faf8c078070 0x7faf8c07a530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fafac151070 tx=0x7faf9c009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7f86b2800640 1 -- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86b40c7460 msgr2=0x7f86b41585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7f86b2800640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86b40c7460 0x7f86b41585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7f86b2800640 1 -- 172.21.15.31:0/1058234459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86b40c2d40 msgr2=0x7f86b40aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7f86b2800640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86b40c2d40 0x7f86b40aab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7f86b2800640 1 -- 172.21.15.31:0/1058234459 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b41e2110 con 0x7f86b41537c0 2024-09-11T00:31:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7fec80e00640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec7c0c2e80 0x7fec7c1ef9d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fec7400ba10 tx=0x7fec7400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.651+0000 7fec81800640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec7c0c23a0 0x7fec7c14d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.652+0000 7fec82200640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec7c151760 0x7fec7c06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.652+0000 7fafaa000640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] conn(0x7faf8c082900 0x7faf8c084d20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.652+0000 7f86b2800640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86b41537c0 0x7f86b41f3e40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f86ac00c7a0 tx=0x7f86ac00cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.652+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd3480c79f0 con 0x7fd348153c10 2024-09-11T00:31:02.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.652+0000 7fec79600640 1 -- 172.21.15.31:0/4025860993 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec74019070 con 0x7fec7c0c2e80 2024-09-11T00:31:02.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.652+0000 7fec79600640 1 -- 172.21.15.31:0/4025860993 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec7400ccc0 con 0x7fec7c0c2e80 2024-09-11T00:31:02.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.652+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec7c1ec650 con 0x7fec7c0c2e80 2024-09-11T00:31:02.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.652+0000 7f86aa000640 1 -- 172.21.15.31:0/1058234459 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac003b70 con 0x7f86b41537c0 2024-09-11T00:31:02.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.652+0000 7f86aa000640 1 -- 172.21.15.31:0/1058234459 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86ac003d10 con 0x7f86b41537c0 2024-09-11T00:31:02.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.652+0000 7f86aa000640 1 -- 172.21.15.31:0/1058234459 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac00aab0 con 0x7f86b41537c0 2024-09-11T00:31:02.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.653+0000 7fec79600640 1 -- 172.21.15.31:0/4025860993 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec740146a0 con 0x7fec7c0c2e80 2024-09-11T00:31:02.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.654+0000 7fd33fe00640 1 -- 172.21.15.31:0/996875078 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd34004f060 con 0x7fd348153c10 2024-09-11T00:31:02.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.655+0000 7fafa3400640 1 -- 172.21.15.31:0/3249532 <== osd.4 v2:172.21.15.204:6802/635888001 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fafa40317c0 con 0x7faf8c082900 2024-09-11T00:31:02.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.654+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec7c14d690 con 0x7fec7c0c2e80 2024-09-11T00:31:02.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.653+0000 7f86b9000640 1 -- 172.21.15.31:0/1058234459 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b41f0f30 con 0x7f86b41537c0 2024-09-11T00:31:02.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.653+0000 7f86b9000640 1 -- 172.21.15.31:0/1058234459 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b4158b30 con 0x7f86b41537c0 2024-09-11T00:31:02.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.656+0000 7f86a3400640 1 -- 172.21.15.31:0/1058234459 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8684001030 con 0x7f86b41537c0 2024-09-11T00:31:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.656+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec7c1ec460 con 0x7fec7c0c2e80 2024-09-11T00:31:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.657+0000 7fd33fe00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd328077fa0 0x7fd32807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.657+0000 7fd33fe00640 1 -- 172.21.15.31:0/996875078 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fd3400badc0 con 0x7fd348153c10 2024-09-11T00:31:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.657+0000 7fd33fe00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] conn(0x7fd3280827c0 0x7fd328084be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.657+0000 7fd33fe00640 1 -- 172.21.15.31:0/996875078 --> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd340032660 con 0x7fd3280827c0 2024-09-11T00:31:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.657+0000 7fd33fe00640 1 -- 172.21.15.31:0/996875078 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd3400bd4c0 con 0x7fd348153c10 2024-09-11T00:31:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.658+0000 7fd347e00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd328077fa0 0x7fd32807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.658+0000 7fd347400640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] conn(0x7fd3280827c0 0x7fd328084be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.659+0000 7fd347400640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] conn(0x7fd3280827c0 0x7fd328084be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.661+0000 7fd347e00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd328077fa0 0x7fd32807a460 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd3380016d0 tx=0x7fd338009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.661+0000 7fd33fe00640 1 -- 172.21.15.31:0/996875078 <== osd.5 v2:172.21.15.204:6810/482649985 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd340032660 con 0x7fd3280827c0 2024-09-11T00:31:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.659+0000 7fec79600640 1 -- 172.21.15.31:0/4025860993 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fec74006ce0 con 0x7fec7c0c2e80 2024-09-11T00:31:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.661+0000 7fec79600640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec5c077fc0 0x7fec5c07a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.661+0000 7fec79600640 1 -- 172.21.15.31:0/4025860993 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fec74099cc0 con 0x7fec7c0c2e80 2024-09-11T00:31:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.661+0000 7fec79600640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] conn(0x7fec5c0827e0 0x7fec5c084c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.661+0000 7fec79600640 1 -- 172.21.15.31:0/4025860993 --> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec5c0852b0 con 0x7fec5c0827e0 2024-09-11T00:31:02.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.662+0000 7fec79600640 1 -- 172.21.15.31:0/4025860993 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fec7405e480 con 0x7fec7c0c2e80 2024-09-11T00:31:02.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.662+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 --> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faaac002dd0 con 0x7faab0082860 2024-09-11T00:31:02.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.662+0000 7fec81800640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec5c077fc0 0x7fec5c07a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.663+0000 7faaccc00640 1 -- 172.21.15.31:0/2900440743 <== osd.0 v2:172.21.15.31:6808/2913875274 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faaac002dd0 con 0x7faab0082860 2024-09-11T00:31:02.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.662+0000 7fec82200640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] conn(0x7fec5c0827e0 0x7fec5c084c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.659+0000 7f86aa000640 1 -- 172.21.15.31:0/1058234459 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f86ac00ac50 con 0x7f86b41537c0 2024-09-11T00:31:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.663+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 >> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] conn(0x7faab0082860 msgr2=0x7faab0084c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.663+0000 7faad7800640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] conn(0x7faab0082860 0x7faab0084c80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.663+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faab0077fc0 msgr2=0x7faab007a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.663+0000 7faad7800640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faab0077fc0 0x7faab007a480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faac8030150 tx=0x7faabc007930 comp rx=0 tx=0).stop 2024-09-11T00:31:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.664+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faac8006bc0 msgr2=0x7faac81783d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.664+0000 7faad7800640 1 --2- 172.21.15.31:0/2900440743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faac8006bc0 0x7faac81783d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faac4009950 tx=0x7faac4003c10 comp rx=0 tx=0).stop 2024-09-11T00:31:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.664+0000 7faad5800640 1 -- 172.21.15.31:0/2900440743 reap_dead start 2024-09-11T00:31:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.664+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 shutdown_connections 2024-09-11T00:31:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.664+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 >> 172.21.15.31:0/2900440743 conn(0x7faac804c090 msgr2=0x7faac80328f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.664+0000 7fec81800640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec5c077fc0 0x7fec5c07a480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec70000f60 tx=0x7fec70009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.664+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 shutdown_connections 2024-09-11T00:31:02.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.665+0000 7faad7800640 1 -- 172.21.15.31:0/2900440743 wait complete. 2024-09-11T00:31:02.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.665+0000 7fec82200640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] conn(0x7fec5c0827e0 0x7fec5c084c00 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.666+0000 7fec79600640 1 -- 172.21.15.31:0/4025860993 <== osd.6 v2:172.21.15.204:6818/1935657097 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fec5c0852b0 con 0x7fec5c0827e0 2024-09-11T00:31:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.666+0000 7f86aa000640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8690077ff0 0x7f869007a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.666+0000 7f86aa000640 1 -- 172.21.15.31:0/1058234459 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f86ac099c90 con 0x7f86b41537c0 2024-09-11T00:31:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.666+0000 7f86aa000640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] conn(0x7f8690082890 0x7f8690084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.666+0000 7f86aa000640 1 -- 172.21.15.31:0/1058234459 --> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8690085360 con 0x7f8690082890 2024-09-11T00:31:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.666+0000 7f86aa000640 1 -- 172.21.15.31:0/1058234459 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f86ac09c3e0 con 0x7f86b41537c0 2024-09-11T00:31:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.667+0000 7f86abe00640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] conn(0x7f8690082890 0x7f8690084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.668+0000 7f86abe00640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] conn(0x7f8690082890 0x7f8690084cb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.669+0000 7f86aa000640 1 -- 172.21.15.31:0/1058234459 <== osd.7 v2:172.21.15.204:6826/276504597 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8690085360 con 0x7f8690082890 2024-09-11T00:31:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.670+0000 7f86b1e00640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8690077ff0 0x7f869007a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:02.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.671+0000 7f86b1e00640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8690077ff0 0x7f869007a4b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f86a4033000 tx=0x7f86a4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:02.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.675+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 --> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fadb406e140 con 0x7fad94082780 2024-09-11T00:31:02.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.676+0000 7fadaaa00640 1 -- 172.21.15.31:0/4191976738 <== osd.1 v2:172.21.15.31:6800/3912540156 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fadb406e140 con 0x7fad94082780 2024-09-11T00:31:02.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.677+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 >> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] conn(0x7fad94082780 msgr2=0x7fad94084ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.677+0000 7fadb3e00640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] conn(0x7fad94082780 0x7fad94084ba0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.677+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad94077f70 msgr2=0x7fad9407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.678+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 --> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f11c0002dd0 con 0x7f11c40827b0 2024-09-11T00:31:02.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.677+0000 7fadb3e00640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad94077f70 0x7fad9407a430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fadb4148120 tx=0x7fada40090e0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.677+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadb40ad750 msgr2=0x7fadb41ee000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.677+0000 7fadb3e00640 1 --2- 172.21.15.31:0/4191976738 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadb40ad750 0x7fadb41ee000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fadb4153c10 tx=0x7fadac003f10 comp rx=0 tx=0).stop 2024-09-11T00:31:02.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.678+0000 7fadb3400640 1 -- 172.21.15.31:0/4191976738 reap_dead start 2024-09-11T00:31:02.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.678+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 shutdown_connections 2024-09-11T00:31:02.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.678+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 >> 172.21.15.31:0/4191976738 conn(0x7fadb4068f10 msgr2=0x7fadb4155f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.678+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 shutdown_connections 2024-09-11T00:31:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.678+0000 7fadb3e00640 1 -- 172.21.15.31:0/4191976738 wait complete. 2024-09-11T00:31:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.679+0000 7f11de000640 1 -- 172.21.15.31:0/2250709565 <== osd.2 v2:172.21.15.31:6816/2777449146 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f11c0002dd0 con 0x7f11c40827b0 2024-09-11T00:31:02.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.680+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f11c40827b0 msgr2=0x7f11c4084bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.680+0000 7f11e7200640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f11c40827b0 0x7f11c4084bd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.680+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f11c4077fa0 msgr2=0x7f11c407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.680+0000 7f11e7200640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f11c4077fa0 0x7f11c407a460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009230 tx=0x7f11e0004730 comp rx=0 tx=0).stop 2024-09-11T00:31:02.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.680+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11e80c62f0 msgr2=0x7f11e81f8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.680+0000 7f11e7200640 1 --2- 172.21.15.31:0/2250709565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11e80c62f0 0x7f11e81f8860 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11d000c410 tx=0x7f11d000c8d0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.680+0000 7f11e6800640 1 -- 172.21.15.31:0/2250709565 reap_dead start 2024-09-11T00:31:02.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.681+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 shutdown_connections 2024-09-11T00:31:02.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.681+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 >> 172.21.15.31:0/2250709565 conn(0x7f11e806caf0 msgr2=0x7f11e8157b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.681+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 shutdown_connections 2024-09-11T00:31:02.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.681+0000 7f11e7200640 1 -- 172.21.15.31:0/2250709565 wait complete. 2024-09-11T00:31:02.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.681+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 --> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f70ec0b06a0 con 0x7f70cc082900 2024-09-11T00:31:02.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.683+0000 7f70e2000640 1 -- 172.21.15.31:0/2784598877 <== osd.3 v2:172.21.15.31:6824/4217184896 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f70ec0b06a0 con 0x7f70cc082900 2024-09-11T00:31:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.684+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 >> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] conn(0x7f70cc082900 msgr2=0x7f70cc084d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.684+0000 7f70eb200640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] conn(0x7f70cc082900 0x7f70cc084d20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.684+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70cc078070 msgr2=0x7f70cc07a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.684+0000 7f70eb200640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70cc078070 0x7f70cc07a530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f70e4033000 tx=0x7f70e4001cf0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.684+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0ad720 msgr2=0x7f70ec0c2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.684+0000 7f70eb200640 1 --2- 172.21.15.31:0/2784598877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70ec0ad720 0x7f70ec0c2c10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f70ec06e320 tx=0x7f70d800d9a0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.684+0000 7f70ea800640 1 -- 172.21.15.31:0/2784598877 reap_dead start 2024-09-11T00:31:02.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.684+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 shutdown_connections 2024-09-11T00:31:02.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.684+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 >> 172.21.15.31:0/2784598877 conn(0x7f70ec068ac0 msgr2=0x7f70ec0ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.685+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 shutdown_connections 2024-09-11T00:31:02.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.687+0000 7f70eb200640 1 -- 172.21.15.31:0/2784598877 wait complete. 2024-09-11T00:31:02.700 INFO:teuthology.orchestra.run.smithi031.stdout:64424509449 2024-09-11T00:31:02.700 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T00:31:02.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.692+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 --> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faf84002dd0 con 0x7faf8c082900 2024-09-11T00:31:02.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.693+0000 7fafa3400640 1 -- 172.21.15.31:0/3249532 <== osd.4 v2:172.21.15.204:6802/635888001 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faf84002dd0 con 0x7faf8c082900 2024-09-11T00:31:02.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.693+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 --> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd348150f60 con 0x7fd3280827c0 2024-09-11T00:31:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.693+0000 7f86a3400640 1 -- 172.21.15.31:0/1058234459 --> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8684002dd0 con 0x7f8690082890 2024-09-11T00:31:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] conn(0x7faf8c082900 msgr2=0x7faf8c084d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7fafb1e00640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] conn(0x7faf8c082900 0x7faf8c084d20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7fd33fe00640 1 -- 172.21.15.31:0/996875078 <== osd.5 v2:172.21.15.204:6810/482649985 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd348150f60 con 0x7fd3280827c0 2024-09-11T00:31:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7f86aa000640 1 -- 172.21.15.31:0/1058234459 <== osd.7 v2:172.21.15.204:6826/276504597 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8684002dd0 con 0x7f8690082890 2024-09-11T00:31:02.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faf8c078070 msgr2=0x7faf8c07a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7fafb1e00640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faf8c078070 0x7faf8c07a530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fafac151070 tx=0x7faf9c009260 comp rx=0 tx=0).stop 2024-09-11T00:31:02.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c3d10 msgr2=0x7fafac150320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7fafb1e00640 1 --2- 172.21.15.31:0/3249532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fafac0c3d10 0x7fafac150320 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fafa4006010 tx=0x7fafa4003f10 comp rx=0 tx=0).stop 2024-09-11T00:31:02.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7fafab400640 1 -- 172.21.15.31:0/3249532 reap_dead start 2024-09-11T00:31:02.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 shutdown_connections 2024-09-11T00:31:02.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.694+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 >> 172.21.15.31:0/3249532 conn(0x7fafac0a9750 msgr2=0x7fafac1e01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 shutdown_connections 2024-09-11T00:31:02.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7fafb1e00640 1 -- 172.21.15.31:0/3249532 wait complete. 2024-09-11T00:31:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7f86a3400640 1 -- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] conn(0x7f8690082890 msgr2=0x7f8690084cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7f86a3400640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] conn(0x7f8690082890 0x7f8690084cb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7f86a3400640 1 -- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8690077ff0 msgr2=0x7f869007a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7f86a3400640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8690077ff0 0x7f869007a4b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f86a4033000 tx=0x7f86a4004730 comp rx=0 tx=0).stop 2024-09-11T00:31:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7f86a3400640 1 -- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86b41537c0 msgr2=0x7f86b41f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7f86a3400640 1 --2- 172.21.15.31:0/1058234459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86b41537c0 0x7f86b41f3e40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f86ac00c7a0 tx=0x7f86ac00cc60 comp rx=0 tx=0).stop 2024-09-11T00:31:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7f86b2800640 1 -- 172.21.15.31:0/1058234459 reap_dead start 2024-09-11T00:31:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7f86a3400640 1 -- 172.21.15.31:0/1058234459 shutdown_connections 2024-09-11T00:31:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.695+0000 7f86a3400640 1 -- 172.21.15.31:0/1058234459 >> 172.21.15.31:0/1058234459 conn(0x7f86b40a9750 msgr2=0x7f86b4154c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.696+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] conn(0x7fd3280827c0 msgr2=0x7fd328084be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.696+0000 7fd34ee00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] conn(0x7fd3280827c0 0x7fd328084be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.696+0000 7f86a3400640 1 -- 172.21.15.31:0/1058234459 shutdown_connections 2024-09-11T00:31:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.696+0000 7f86a3400640 1 -- 172.21.15.31:0/1058234459 wait complete. 2024-09-11T00:31:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.698+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd328077fa0 msgr2=0x7fd32807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.698+0000 7fd34ee00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd328077fa0 0x7fd32807a460 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd3380016d0 tx=0x7fd338009040 comp rx=0 tx=0).stop 2024-09-11T00:31:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.698+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd348153c10 msgr2=0x7fd3480c55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.698+0000 7fd34ee00640 1 --2- 172.21.15.31:0/996875078 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd348153c10 0x7fd3480c55e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd340009cd0 tx=0x7fd3400324d0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.698+0000 7fd34ce00640 1 -- 172.21.15.31:0/996875078 reap_dead start 2024-09-11T00:31:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.698+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 shutdown_connections 2024-09-11T00:31:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.698+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 >> 172.21.15.31:0/996875078 conn(0x7fd3480c8cc0 msgr2=0x7fd3481597c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.698+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 shutdown_connections 2024-09-11T00:31:02.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.699+0000 7fd34ee00640 1 -- 172.21.15.31:0/996875078 wait complete. 2024-09-11T00:31:02.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.700+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 --> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fec7c0add30 con 0x7fec5c0827e0 2024-09-11T00:31:02.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.701+0000 7fec79600640 1 -- 172.21.15.31:0/4025860993 <== osd.6 v2:172.21.15.204:6818/1935657097 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fec7c0add30 con 0x7fec5c0827e0 2024-09-11T00:31:02.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.701+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] conn(0x7fec5c0827e0 msgr2=0x7fec5c084c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.701+0000 7fec84200640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] conn(0x7fec5c0827e0 0x7fec5c084c00 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.702+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec5c077fc0 msgr2=0x7fec5c07a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.702+0000 7fec84200640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec5c077fc0 0x7fec5c07a480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec70000f60 tx=0x7fec70009040 comp rx=0 tx=0).stop 2024-09-11T00:31:02.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.702+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec7c0c2e80 msgr2=0x7fec7c1ef9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.702+0000 7fec84200640 1 --2- 172.21.15.31:0/4025860993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec7c0c2e80 0x7fec7c1ef9d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fec7400ba10 tx=0x7fec7400bed0 comp rx=0 tx=0).stop 2024-09-11T00:31:02.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.702+0000 7fec82200640 1 -- 172.21.15.31:0/4025860993 reap_dead start 2024-09-11T00:31:02.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.702+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 shutdown_connections 2024-09-11T00:31:02.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.702+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 >> 172.21.15.31:0/4025860993 conn(0x7fec7c0a9750 msgr2=0x7fec7c072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:02.710 INFO:teuthology.orchestra.run.smithi031.stdout:64424509449 2024-09-11T00:31:02.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T00:31:02.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.704+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 shutdown_connections 2024-09-11T00:31:02.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.704+0000 7fec84200640 1 -- 172.21.15.31:0/4025860993 wait complete. 2024-09-11T00:31:02.713 INFO:teuthology.orchestra.run.smithi031.stdout:64424509449 2024-09-11T00:31:02.713 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-11T00:31:02.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T00:31:02.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T00:31:02.721 INFO:teuthology.orchestra.run.smithi031.stdout:64424509449 2024-09-11T00:31:02.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T00:31:02.727 INFO:teuthology.orchestra.run.smithi031.stdout:60129542153 2024-09-11T00:31:02.727 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2024-09-11T00:31:02.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T00:31:02.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-11T00:31:02.729 INFO:teuthology.orchestra.run.smithi031.stdout:64424509450 2024-09-11T00:31:02.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-11T00:31:02.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.996+0000 7f3d80a00640 1 -- 172.21.15.31:0/1846937886 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70035020 con 0x7f3d7c046570 2024-09-11T00:31:02.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.997+0000 7f3d88c00640 1 -- 172.21.15.31:0/1846937886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d7c046570 msgr2=0x7f3d7c045b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:02.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.997+0000 7f3d88c00640 1 --2- 172.21.15.31:0/1846937886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d7c046570 0x7f3d7c045b60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3d70005fd0 tx=0x7f3d7002a670 comp rx=0 tx=0).stop 2024-09-11T00:31:02.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.998+0000 7f3d88c00640 1 -- 172.21.15.31:0/1846937886 shutdown_connections 2024-09-11T00:31:03.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.998+0000 7f3d88c00640 1 --2- 172.21.15.31:0/1846937886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d7c046570 0x7f3d7c045b60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.998+0000 7f3d88c00640 1 -- 172.21.15.31:0/1846937886 >> 172.21.15.31:0/1846937886 conn(0x7f3d7c047e40 msgr2=0x7f3d7c048250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.998+0000 7f3d88c00640 1 -- 172.21.15.31:0/1846937886 shutdown_connections 2024-09-11T00:31:03.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:02.998+0000 7f3d88c00640 1 -- 172.21.15.31:0/1846937886 wait complete. 2024-09-11T00:31:03.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.000+0000 7f3d88c00640 1 Processor -- start 2024-09-11T00:31:03.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.000+0000 7f3d88c00640 1 -- start start 2024-09-11T00:31:03.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.002+0000 7f3d88c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7c00b6e0 0x7f3d7c0d0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.002+0000 7f3d88c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d7c046570 0x7f3d7c16b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.002+0000 7f3d88c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d7c0d3430 0x7f3d7c16bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.002+0000 7f3d88c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7c049a30 con 0x7f3d7c046570 2024-09-11T00:31:03.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.002+0000 7f3d88c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7c0d3860 con 0x7f3d7c00b6e0 2024-09-11T00:31:03.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.002+0000 7f3d88c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3d7c04ae90 con 0x7f3d7c0d3430 2024-09-11T00:31:03.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.003+0000 7f3d81e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7c00b6e0 0x7f3d7c0d0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.003+0000 7f3d81400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d7c046570 0x7f3d7c16b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.003+0000 7f3d81400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d7c046570 0x7f3d7c16b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37376/0 (socket says 172.21.15.31:37376) 2024-09-11T00:31:03.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.003+0000 7f3d81400640 1 -- 172.21.15.31:0/1866572023 learned_addr learned my addr 172.21.15.31:0/1866572023 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:03.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.003+0000 7f3d82800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d7c0d3430 0x7f3d7c16bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.004+0000 7f3d81400640 1 -- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d7c0d3430 msgr2=0x7f3d7c16bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.004+0000 7f3d81400640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d7c0d3430 0x7f3d7c16bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.004+0000 7f3d81400640 1 -- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7c00b6e0 msgr2=0x7f3d7c0d0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.004+0000 7f3d81400640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7c00b6e0 0x7f3d7c0d0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.004+0000 7f3d81400640 1 -- 172.21.15.31:0/1866572023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7c0cba80 con 0x7f3d7c046570 2024-09-11T00:31:03.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.005+0000 7f3d81400640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d7c046570 0x7f3d7c16b820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3d700099f0 tx=0x7f3d700039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.006+0000 7f3d7a000640 1 -- 172.21.15.31:0/1866572023 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7003f070 con 0x7f3d7c046570 2024-09-11T00:31:03.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.006+0000 7f3d7a000640 1 -- 172.21.15.31:0/1866572023 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d700047a0 con 0x7f3d7c046570 2024-09-11T00:31:03.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.006+0000 7f3d7a000640 1 -- 172.21.15.31:0/1866572023 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70035030 con 0x7f3d7c046570 2024-09-11T00:31:03.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.007+0000 7f3d88c00640 1 -- 172.21.15.31:0/1866572023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d7c0cb2f0 con 0x7f3d7c046570 2024-09-11T00:31:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.007+0000 7f3d88c00640 1 -- 172.21.15.31:0/1866572023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d7c0d1220 con 0x7f3d7c046570 2024-09-11T00:31:03.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.011+0000 7f6760a00640 1 -- 172.21.15.31:0/264969579 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f675c1510a0 msgr2=0x7f675c1514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.011+0000 7f3d88c00640 1 -- 172.21.15.31:0/1866572023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d7c045b60 con 0x7f3d7c046570 2024-09-11T00:31:03.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.011+0000 7f6760a00640 1 --2- 172.21.15.31:0/264969579 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f675c1510a0 0x7f675c1514f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6754005fd0 tx=0x7f675402a970 comp rx=0 tx=0).stop 2024-09-11T00:31:03.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.012+0000 7f6760a00640 1 -- 172.21.15.31:0/264969579 shutdown_connections 2024-09-11T00:31:03.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.012+0000 7f6760a00640 1 --2- 172.21.15.31:0/264969579 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f675c1510a0 0x7f675c1514f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.012+0000 7f6760a00640 1 -- 172.21.15.31:0/264969579 >> 172.21.15.31:0/264969579 conn(0x7f675c06caf0 msgr2=0x7f675c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.013+0000 7f6760a00640 1 -- 172.21.15.31:0/264969579 shutdown_connections 2024-09-11T00:31:03.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.013+0000 7f3d7a000640 1 -- 172.21.15.31:0/1866572023 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3d70004bf0 con 0x7f3d7c046570 2024-09-11T00:31:03.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.013+0000 7f6760a00640 1 -- 172.21.15.31:0/264969579 wait complete. 2024-09-11T00:31:03.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.016+0000 7f6760a00640 1 Processor -- start 2024-09-11T00:31:03.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.016+0000 7f6760a00640 1 -- start start 2024-09-11T00:31:03.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.017+0000 7f3d7a000640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d64078040 0x7f3d6407a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.017+0000 7f6292e00640 1 -- 172.21.15.31:0/3090622976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f628c0b1160 msgr2=0x7f628c15a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.017+0000 7f6292e00640 1 --2- 172.21.15.31:0/3090622976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f628c0b1160 0x7f628c15a1a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6274005fd0 tx=0x7f627402a540 comp rx=0 tx=0).stop 2024-09-11T00:31:03.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.017+0000 7f6292e00640 1 -- 172.21.15.31:0/3090622976 shutdown_connections 2024-09-11T00:31:03.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.017+0000 7f6292e00640 1 --2- 172.21.15.31:0/3090622976 >> v2:172.21.15.31:3300/0 conn(0x7f628c0bf2c0 0x7f628c0ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.017+0000 7f3d7a000640 1 -- 172.21.15.31:0/1866572023 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f3d700bb120 con 0x7f3d7c046570 2024-09-11T00:31:03.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.017+0000 7f6292e00640 1 --2- 172.21.15.31:0/3090622976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f628c0b1160 0x7f628c15a1a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.017+0000 7f6292e00640 1 -- 172.21.15.31:0/3090622976 >> 172.21.15.31:0/3090622976 conn(0x7f628c0cc260 msgr2=0x7f628c0ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f6760a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f675c0a6e20 0x7f675c151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f6760a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c0c4630 0x7f675c149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f6760a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f675c1510a0 0x7f675c1533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f6760a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f675c0c1e40 con 0x7f675c0c4630 2024-09-11T00:31:03.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f6760a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f675c0a7370 con 0x7f675c0a6e20 2024-09-11T00:31:03.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f6760a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f675c0c1fc0 con 0x7f675c1510a0 2024-09-11T00:31:03.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f3d81e00640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d64078040 0x7f3d6407a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.017+0000 7f6292e00640 1 -- 172.21.15.31:0/3090622976 shutdown_connections 2024-09-11T00:31:03.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f6292e00640 1 -- 172.21.15.31:0/3090622976 wait complete. 2024-09-11T00:31:03.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f675b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f675c0a6e20 0x7f675c151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f675b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f675c0a6e20 0x7f675c151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:37060/0 (socket says 172.21.15.31:37060) 2024-09-11T00:31:03.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.018+0000 7f675b400640 1 -- 172.21.15.31:0/621284465 learned_addr learned my addr 172.21.15.31:0/621284465 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:03.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.019+0000 7f3d81e00640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d64078040 0x7f3d6407a500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d84052890 tx=0x7f3d84051dd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.019+0000 7f675aa00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c0c4630 0x7f675c149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.019+0000 7f675be00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f675c1510a0 0x7f675c1533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.019+0000 7f4647c00640 1 -- 172.21.15.31:0/1610696391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4640072480 msgr2=0x7f46401511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.019+0000 7f4647c00640 1 --2- 172.21.15.31:0/1610696391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4640072480 0x7f46401511a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4638005fd0 tx=0x7f463802ab50 comp rx=0 tx=0).stop 2024-09-11T00:31:03.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.019+0000 7f4647c00640 1 -- 172.21.15.31:0/1610696391 shutdown_connections 2024-09-11T00:31:03.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.019+0000 7f4647c00640 1 --2- 172.21.15.31:0/1610696391 >> v2:172.21.15.204:3300/0 conn(0x7f4640153f20 0x7f46401516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.019+0000 7f4647c00640 1 --2- 172.21.15.31:0/1610696391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4640072480 0x7f46401511a0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4638005fd0 tx=0x7f463802ab50 comp rx=0 tx=0).stop 2024-09-11T00:31:03.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.019+0000 7f4647c00640 1 -- 172.21.15.31:0/1610696391 >> 172.21.15.31:0/1610696391 conn(0x7f46400a9750 msgr2=0x7f46400a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f675be00640 1 -- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f675c0a6e20 msgr2=0x7f675c151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f31f9a00640 1 -- 172.21.15.31:0/3650418019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31d8003a50 msgr2=0x7f31d8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f31f9a00640 1 --2- 172.21.15.31:0/3650418019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31d8003a50 0x7f31d8003e70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f31e400adc0 tx=0x7f31d8005830 comp rx=0 tx=0).stop 2024-09-11T00:31:03.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f675be00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f675c0a6e20 0x7f675c151b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f675be00640 1 -- 172.21.15.31:0/621284465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c0c4630 msgr2=0x7f675c149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f675be00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c0c4630 0x7f675c149240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f675be00640 1 -- 172.21.15.31:0/621284465 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675c1565d0 con 0x7f675c1510a0 2024-09-11T00:31:03.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f31f9a00640 1 -- 172.21.15.31:0/3650418019 shutdown_connections 2024-09-11T00:31:03.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f31f9a00640 1 --2- 172.21.15.31:0/3650418019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31d8003a50 0x7f31d8003e70 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f31e400adc0 tx=0x7f31d8005830 comp rx=0 tx=0).stop 2024-09-11T00:31:03.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f31f9a00640 1 -- 172.21.15.31:0/3650418019 >> 172.21.15.31:0/3650418019 conn(0x7f31f40688e0 msgr2=0x7f31f4068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f675b400640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f675c0a6e20 0x7f675c151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:03.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f675aa00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c0c4630 0x7f675c149240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:03.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f4647c00640 1 -- 172.21.15.31:0/1610696391 shutdown_connections 2024-09-11T00:31:03.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f4647c00640 1 -- 172.21.15.31:0/1610696391 wait complete. 2024-09-11T00:31:03.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f6292e00640 1 Processor -- start 2024-09-11T00:31:03.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f31f9a00640 1 -- 172.21.15.31:0/3650418019 shutdown_connections 2024-09-11T00:31:03.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.021+0000 7f675be00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f675c1510a0 0x7f675c1533e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6754032000 tx=0x7f6754003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.021+0000 7f6292e00640 1 -- start start 2024-09-11T00:31:03.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.021+0000 7f6753400640 1 -- 172.21.15.31:0/621284465 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754034040 con 0x7f675c1510a0 2024-09-11T00:31:03.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.021+0000 7f6753400640 1 -- 172.21.15.31:0/621284465 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67540045c0 con 0x7f675c1510a0 2024-09-11T00:31:03.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.020+0000 7f31f9a00640 1 -- 172.21.15.31:0/3650418019 wait complete. 2024-09-11T00:31:03.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.021+0000 7f6753400640 1 -- 172.21.15.31:0/621284465 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754045480 con 0x7f675c1510a0 2024-09-11T00:31:03.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.021+0000 7f6760a00640 1 -- 172.21.15.31:0/621284465 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675c1568b0 con 0x7f675c1510a0 2024-09-11T00:31:03.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.021+0000 7f6760a00640 1 -- 172.21.15.31:0/621284465 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f675c157150 con 0x7f675c1510a0 2024-09-11T00:31:03.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.023+0000 7f4647c00640 1 Processor -- start 2024-09-11T00:31:03.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.022+0000 7f6292e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f628c0bf2c0 0x7f628c1e1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.022+0000 7f6292e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f628c0c9b30 0x7f628c1df8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.022+0000 7f6292e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f628c1efb60 0x7f628c1dfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.022+0000 7f6292e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f628c0b1540 con 0x7f628c1efb60 2024-09-11T00:31:03.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.022+0000 7f6292e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f628c06f9a0 con 0x7f628c0bf2c0 2024-09-11T00:31:03.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.022+0000 7f6292e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f628c06f300 con 0x7f628c0c9b30 2024-09-11T00:31:03.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.023+0000 7f4647c00640 1 -- start start 2024-09-11T00:31:03.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.024+0000 7f3d7a000640 1 -- 172.21.15.31:0/1866572023 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3d70083280 con 0x7f3d7c046570 2024-09-11T00:31:03.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.024+0000 7f628be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f628c0bf2c0 0x7f628c1e1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.024+0000 7f628be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f628c0bf2c0 0x7f628c1e1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:37074/0 (socket says 172.21.15.31:37074) 2024-09-11T00:31:03.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.024+0000 7f628be00640 1 -- 172.21.15.31:0/589731635 learned_addr learned my addr 172.21.15.31:0/589731635 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:03.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.023+0000 7f6760a00640 1 -- 172.21.15.31:0/621284465 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f675c0c4a10 con 0x7f675c1510a0 2024-09-11T00:31:03.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.023+0000 7f31f9a00640 1 Processor -- start 2024-09-11T00:31:03.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.023+0000 7f628b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f628c0c9b30 0x7f628c1df8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.024+0000 7f628be00640 1 -- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f628c0c9b30 msgr2=0x7f628c1df8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.024+0000 7f628be00640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f628c0c9b30 0x7f628c1df8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.024+0000 7f628be00640 1 -- 172.21.15.31:0/589731635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f628c1efb60 msgr2=0x7f628c1dfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.024+0000 7f628be00640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f628c1efb60 0x7f628c1dfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.024+0000 7f628be00640 1 -- 172.21.15.31:0/589731635 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628c156150 con 0x7f628c0bf2c0 2024-09-11T00:31:03.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.025+0000 7f31f9a00640 1 -- start start 2024-09-11T00:31:03.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.025+0000 7f628be00640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f628c0bf2c0 0x7f628c1e1810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6274009850 tx=0x7f62740038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.025+0000 7f4647c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46400c23a0 0x7f4640148020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.025+0000 7f4647c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4640153f20 0x7f464014ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.025+0000 7f4647c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46400c7540 0x7f46400c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.025+0000 7f4647c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4640074200 con 0x7f4640153f20 2024-09-11T00:31:03.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.025+0000 7f4647c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4640074080 con 0x7f46400c7540 2024-09-11T00:31:03.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.025+0000 7f4647c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4640072860 con 0x7f46400c23a0 2024-09-11T00:31:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.026+0000 7f463fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4640153f20 0x7f464014ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.026+0000 7f4645200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46400c23a0 0x7f4640148020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.026+0000 7f4645c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46400c7540 0x7f46400c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.026+0000 7f4645200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46400c23a0 0x7f4640148020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50744/0 (socket says 172.21.15.31:50744) 2024-09-11T00:31:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.026+0000 7f4645200640 1 -- 172.21.15.31:0/313215615 learned_addr learned my addr 172.21.15.31:0/313215615 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.026+0000 7f463fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4640153f20 0x7f464014ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37462/0 (socket says 172.21.15.31:37462) 2024-09-11T00:31:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f31f9a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31f40ae0f0 0x7f31f40ab710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f31f9a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31f4163a80 0x7f31f40a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f31f9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f40a9d20 0x7f31f40c0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f31f9a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f31f4158640 con 0x7f31f40a9d20 2024-09-11T00:31:03.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f31f9a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f31f40c44c0 con 0x7f31f40ae0f0 2024-09-11T00:31:03.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f31f9a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f31f40c5b90 con 0x7f31f4163a80 2024-09-11T00:31:03.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f4645c00640 1 -- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46400c23a0 msgr2=0x7f4640148020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f4645c00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46400c23a0 0x7f4640148020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f4645c00640 1 -- 172.21.15.31:0/313215615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4640153f20 msgr2=0x7f464014ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f4645c00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4640153f20 0x7f464014ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f4645c00640 1 -- 172.21.15.31:0/313215615 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46400c30c0 con 0x7f46400c7540 2024-09-11T00:31:03.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f463fe00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4640153f20 0x7f464014ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:03.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f31f2000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31f4163a80 0x7f31f40a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.027+0000 7f6753400640 1 -- 172.21.15.31:0/621284465 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6754045620 con 0x7f675c1510a0 2024-09-11T00:31:03.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.028+0000 7f4645c00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46400c7540 0x7f46400c49b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f462800d5e0 tx=0x7f462800bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.028+0000 7f31f2000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31f4163a80 0x7f31f40a97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50754/0 (socket says 172.21.15.31:50754) 2024-09-11T00:31:03.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.028+0000 7f31f2000640 1 -- 172.21.15.31:0/2418569014 learned_addr learned my addr 172.21.15.31:0/2418569014 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:03.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.028+0000 7f31f3400640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f40a9d20 0x7f31f40c0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.028+0000 7f31f2a00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31f40ae0f0 0x7f31f40ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.028+0000 7f31f2a00640 1 -- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31f4163a80 msgr2=0x7f31f40a97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.028+0000 7f31f2a00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31f4163a80 0x7f31f40a97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.028+0000 7f31f2a00640 1 -- 172.21.15.31:0/2418569014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f40a9d20 msgr2=0x7f31f40c0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.028+0000 7f31f2a00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f40a9d20 0x7f31f40c0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.028+0000 7f31f2a00640 1 -- 172.21.15.31:0/2418569014 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f40abc50 con 0x7f31f40ae0f0 2024-09-11T00:31:03.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.029+0000 7f31f2000640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31f4163a80 0x7f31f40a97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:03.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.029+0000 7f6753400640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f673c077f70 0x7f673c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.029+0000 7f31f2a00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31f40ae0f0 0x7f31f40ab710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f31e4032000 tx=0x7f31e402f560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.029+0000 7f6753400640 1 -- 172.21.15.31:0/621284465 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f67540bba30 con 0x7f675c1510a0 2024-09-11T00:31:03.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.030+0000 7f675b400640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f673c077f70 0x7f673c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.030+0000 7f845a800640 1 -- 172.21.15.31:0/1413254393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84540ada20 msgr2=0x7f845406e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.030+0000 7f845a800640 1 --2- 172.21.15.31:0/1413254393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84540ada20 0x7f845406e140 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8448009830 tx=0x7f844802a520 comp rx=0 tx=0).stop 2024-09-11T00:31:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.030+0000 7f845a800640 1 -- 172.21.15.31:0/1413254393 shutdown_connections 2024-09-11T00:31:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.030+0000 7f845a800640 1 --2- 172.21.15.31:0/1413254393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84540ada20 0x7f845406e140 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.030+0000 7f845a800640 1 --2- 172.21.15.31:0/1413254393 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8454150020 0x7f84540ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.030+0000 7f845a800640 1 -- 172.21.15.31:0/1413254393 >> 172.21.15.31:0/1413254393 conn(0x7f8454068f10 msgr2=0x7f84540cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.030+0000 7f845a800640 1 -- 172.21.15.31:0/1413254393 shutdown_connections 2024-09-11T00:31:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f6283e00640 1 -- 172.21.15.31:0/589731635 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627403f020 con 0x7f628c0bf2c0 2024-09-11T00:31:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f31eaa00640 1 -- 172.21.15.31:0/2418569014 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e4033040 con 0x7f31f40ae0f0 2024-09-11T00:31:03.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f31f9a00640 1 -- 172.21.15.31:0/2418569014 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f40ac1b0 con 0x7f31f40ae0f0 2024-09-11T00:31:03.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f463cc00640 1 -- 172.21.15.31:0/313215615 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4628017070 con 0x7f46400c7540 2024-09-11T00:31:03.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f463cc00640 1 -- 172.21.15.31:0/313215615 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4628005310 con 0x7f46400c7540 2024-09-11T00:31:03.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f463cc00640 1 -- 172.21.15.31:0/313215615 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46280047a0 con 0x7f46400c7540 2024-09-11T00:31:03.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f31eaa00640 1 -- 172.21.15.31:0/2418569014 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31e4039a60 con 0x7f31f40ae0f0 2024-09-11T00:31:03.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f31eaa00640 1 -- 172.21.15.31:0/2418569014 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e4039e50 con 0x7f31f40ae0f0 2024-09-11T00:31:03.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f31f9a00640 1 -- 172.21.15.31:0/2418569014 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f40c3210 con 0x7f31f40ae0f0 2024-09-11T00:31:03.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f6283e00640 1 -- 172.21.15.31:0/589731635 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62740047a0 con 0x7f628c0bf2c0 2024-09-11T00:31:03.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f6283e00640 1 -- 172.21.15.31:0/589731635 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6274035070 con 0x7f628c0bf2c0 2024-09-11T00:31:03.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.032+0000 7f675b400640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f673c077f70 0x7f673c07a430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f674c003f70 tx=0x7f674c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.032+0000 7f4647c00640 1 -- 172.21.15.31:0/313215615 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4640145ba0 con 0x7f46400c7540 2024-09-11T00:31:03.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.032+0000 7f4647c00640 1 -- 172.21.15.31:0/313215615 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4640146590 con 0x7f46400c7540 2024-09-11T00:31:03.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.031+0000 7f845a800640 1 -- 172.21.15.31:0/1413254393 wait complete. 2024-09-11T00:31:03.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.032+0000 7f6292e00640 1 -- 172.21.15.31:0/589731635 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f628c1ef080 con 0x7f628c0bf2c0 2024-09-11T00:31:03.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.032+0000 7f6292e00640 1 -- 172.21.15.31:0/589731635 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f628c1e1d50 con 0x7f628c0bf2c0 2024-09-11T00:31:03.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.033+0000 7f31f9a00640 1 -- 172.21.15.31:0/2418569014 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31f40c3030 con 0x7f31f40ae0f0 2024-09-11T00:31:03.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.034+0000 7f6292e00640 1 -- 172.21.15.31:0/589731635 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f628c15a180 con 0x7f628c0bf2c0 2024-09-11T00:31:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.034+0000 7f4a55200640 1 -- 172.21.15.31:0/1945133268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a50159ee0 msgr2=0x7f4a50156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.034+0000 7f4a55200640 1 --2- 172.21.15.31:0/1945133268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a50159ee0 0x7f4a50156280 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4a40009860 tx=0x7f4a4002a560 comp rx=0 tx=0).stop 2024-09-11T00:31:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.034+0000 7f4a55200640 1 -- 172.21.15.31:0/1945133268 shutdown_connections 2024-09-11T00:31:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.034+0000 7f4a55200640 1 --2- 172.21.15.31:0/1945133268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a50159ee0 0x7f4a50156280 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.034+0000 7f4a55200640 1 -- 172.21.15.31:0/1945133268 >> 172.21.15.31:0/1945133268 conn(0x7f4a5006caf0 msgr2=0x7f4a5006ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.033+0000 7f845a800640 1 Processor -- start 2024-09-11T00:31:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.034+0000 7f4a55200640 1 -- 172.21.15.31:0/1945133268 shutdown_connections 2024-09-11T00:31:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.034+0000 7f4a55200640 1 -- 172.21.15.31:0/1945133268 wait complete. 2024-09-11T00:31:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.036+0000 7f4a55200640 1 Processor -- start 2024-09-11T00:31:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.037+0000 7f845a800640 1 -- start start 2024-09-11T00:31:03.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.037+0000 7f4a55200640 1 -- start start 2024-09-11T00:31:03.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.037+0000 7f31eaa00640 1 -- 172.21.15.31:0/2418569014 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f31e4044450 con 0x7f31f40ae0f0 2024-09-11T00:31:03.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.036+0000 7f4647c00640 1 -- 172.21.15.31:0/313215615 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46401511a0 con 0x7f46400c7540 2024-09-11T00:31:03.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f4a55200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a500c4c00 0x7f4a50156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f4a55200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a500c62f0 0x7f4a500aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f4a55200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a50159ee0 0x7f4a500ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f4a55200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5015a2c0 con 0x7f4a500c4c00 2024-09-11T00:31:03.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f4a55200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4a50153b10 con 0x7f4a50159ee0 2024-09-11T00:31:03.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f4a55200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4a500c2c50 con 0x7f4a500c62f0 2024-09-11T00:31:03.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f845a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84540ada20 0x7f84540c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f845a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8454150020 0x7f84540c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f845a800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8454156960 0x7f84541e1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f845a800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8454157550 con 0x7f84540ada20 2024-09-11T00:31:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f845a800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f84541573d0 con 0x7f8454156960 2024-09-11T00:31:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f845a800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f84540cc220 con 0x7f8454150020 2024-09-11T00:31:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f6283e00640 1 -- 172.21.15.31:0/589731635 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6274004b90 con 0x7f628c0bf2c0 2024-09-11T00:31:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.038+0000 7f8459400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84540ada20 0x7f84540c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f8458a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8454150020 0x7f84540c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f8458a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8454150020 0x7f84540c4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50774/0 (socket says 172.21.15.31:50774) 2024-09-11T00:31:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f8458a00640 1 -- 172.21.15.31:0/3649638045 learned_addr learned my addr 172.21.15.31:0/3649638045 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f8459e00640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8454156960 0x7f84541e1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f4a4fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a50159ee0 0x7f4a500ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f31eaa00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31d0078110 0x7f31d007a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f31eaa00640 1 -- 172.21.15.31:0/2418569014 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f31e40bb9b0 con 0x7f31f40ae0f0 2024-09-11T00:31:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f4a4fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a50159ee0 0x7f4a500ab130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:37110/0 (socket says 172.21.15.31:37110) 2024-09-11T00:31:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f4a4fe00640 1 -- 172.21.15.31:0/2889091541 learned_addr learned my addr 172.21.15.31:0/2889091541 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f4a4f400640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a500c4c00 0x7f4a50156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f4a4ea00640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a500c62f0 0x7f4a500aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f4a4f400640 1 -- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a500c62f0 msgr2=0x7f4a500aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f4a4f400640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a500c62f0 0x7f4a500aabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f4a4f400640 1 -- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a50159ee0 msgr2=0x7f4a500ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f4a4f400640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a50159ee0 0x7f4a500ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f4a4f400640 1 -- 172.21.15.31:0/2889091541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a501eff70 con 0x7f4a500c4c00 2024-09-11T00:31:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f31f2000640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31d0078110 0x7f31d007a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f8459400640 1 -- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8454150020 msgr2=0x7f84540c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f463cc00640 1 -- 172.21.15.31:0/313215615 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4628004940 con 0x7f46400c7540 2024-09-11T00:31:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.039+0000 7f6283e00640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6268077f70 0x7f626807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f6283e00640 1 -- 172.21.15.31:0/589731635 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f62740bb350 con 0x7f628c0bf2c0 2024-09-11T00:31:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f8459400640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8454150020 0x7f84540c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f8459400640 1 -- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8454156960 msgr2=0x7f84541e1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f8459400640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8454156960 0x7f84541e1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f8459400640 1 -- 172.21.15.31:0/3649638045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84540c58c0 con 0x7f84540ada20 2024-09-11T00:31:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f8458a00640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8454150020 0x7f84540c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f4a4f400640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a500c4c00 0x7f4a50156460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4a5006d740 tx=0x7f4a40033cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f4a47400640 1 -- 172.21.15.31:0/2889091541 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a40031d60 con 0x7f4a500c4c00 2024-09-11T00:31:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f4a55200640 1 -- 172.21.15.31:0/2889091541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a500abee0 con 0x7f4a500c4c00 2024-09-11T00:31:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.041+0000 7f8459400640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84540ada20 0x7f84540c3ec0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f844400b3f0 tx=0x7f844400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.040+0000 7f31f2000640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31d0078110 0x7f31d007a5d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f31f4157f70 tx=0x7f31dc006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.041+0000 7f463cc00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f461c077f70 0x7f461c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.041+0000 7f463cc00640 1 -- 172.21.15.31:0/313215615 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f462809a360 con 0x7f46400c7540 2024-09-11T00:31:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.041+0000 7f4645200640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f461c077f70 0x7f461c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.041+0000 7f4a55200640 1 -- 172.21.15.31:0/2889091541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a501569a0 con 0x7f4a500c4c00 2024-09-11T00:31:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.041+0000 7f4a47400640 1 -- 172.21.15.31:0/2889091541 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a400317a0 con 0x7f4a500c4c00 2024-09-11T00:31:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.041+0000 7f4a47400640 1 -- 172.21.15.31:0/2889091541 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a40033310 con 0x7f4a500c4c00 2024-09-11T00:31:03.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.041+0000 7f628b400640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6268077f70 0x7f626807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.041+0000 7f8451600640 1 -- 172.21.15.31:0/3649638045 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8444015020 con 0x7f84540ada20 2024-09-11T00:31:03.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.042+0000 7f845a800640 1 -- 172.21.15.31:0/3649638045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84540c5ba0 con 0x7f84540ada20 2024-09-11T00:31:03.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.042+0000 7f845a800640 1 -- 172.21.15.31:0/3649638045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8454158500 con 0x7f84540ada20 2024-09-11T00:31:03.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.042+0000 7f4645200640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f461c077f70 0x7f461c07a430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4638000f80 tx=0x7f4638004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.042+0000 7f628b400640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6268077f70 0x7f626807a430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f628c0ae760 tx=0x7f627c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.042+0000 7f8451600640 1 -- 172.21.15.31:0/3649638045 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8444011920 con 0x7f84540ada20 2024-09-11T00:31:03.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.042+0000 7f8451600640 1 -- 172.21.15.31:0/3649638045 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8444011d70 con 0x7f84540ada20 2024-09-11T00:31:03.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.042+0000 7f6753400640 1 -- 172.21.15.31:0/621284465 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6754084200 con 0x7f675c1510a0 2024-09-11T00:31:03.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.043+0000 7f4a55200640 1 -- 172.21.15.31:0/2889091541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a28005290 con 0x7f4a500c4c00 2024-09-11T00:31:03.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.044+0000 7f845a800640 1 -- 172.21.15.31:0/3649638045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8430005290 con 0x7f84540ada20 2024-09-11T00:31:03.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.046+0000 7f8451600640 1 -- 172.21.15.31:0/3649638045 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f844401a400 con 0x7f84540ada20 2024-09-11T00:31:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.046+0000 7f3588c00640 1 -- 172.21.15.31:0/1839402989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c2ff0 msgr2=0x7f358414f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.046+0000 7f3588c00640 1 --2- 172.21.15.31:0/1839402989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c2ff0 0x7f358414f4d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3570005fd0 tx=0x7f357002a680 comp rx=0 tx=0).stop 2024-09-11T00:31:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.046+0000 7f4a47400640 1 -- 172.21.15.31:0/2889091541 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4a4004b030 con 0x7f4a500c4c00 2024-09-11T00:31:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.046+0000 7f31eaa00640 1 -- 172.21.15.31:0/2418569014 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f31f40c3030 con 0x7f31f40ae0f0 2024-09-11T00:31:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.046+0000 7f3588c00640 1 -- 172.21.15.31:0/1839402989 shutdown_connections 2024-09-11T00:31:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.046+0000 7f3588c00640 1 --2- 172.21.15.31:0/1839402989 >> v2:172.21.15.204:3300/0 conn(0x7f35840c3d10 0x7f358406e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.046+0000 7f3588c00640 1 --2- 172.21.15.31:0/1839402989 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c2ff0 0x7f358414f4d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.046+0000 7f3588c00640 1 -- 172.21.15.31:0/1839402989 >> 172.21.15.31:0/1839402989 conn(0x7f35840a9750 msgr2=0x7f35840a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.047+0000 7f3588c00640 1 -- 172.21.15.31:0/1839402989 shutdown_connections 2024-09-11T00:31:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.047+0000 7f3588c00640 1 -- 172.21.15.31:0/1839402989 wait complete. 2024-09-11T00:31:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.047+0000 7f8451600640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8438077fa0 0x7f843807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.048+0000 7f8451600640 1 -- 172.21.15.31:0/3649638045 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f844409a0e0 con 0x7f84540ada20 2024-09-11T00:31:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.048+0000 7f8458a00640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8438077fa0 0x7f843807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.049+0000 7f8458a00640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8438077fa0 0x7f843807a460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8448009830 tx=0x7f8448004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.047+0000 7f4a47400640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a30077f70 0x7f4a3007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.048+0000 7f4a47400640 1 -- 172.21.15.31:0/2889091541 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f4a400baf00 con 0x7f4a500c4c00 2024-09-11T00:31:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.048+0000 7f4a4ea00640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a30077f70 0x7f4a3007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.049+0000 7f4a4ea00640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a30077f70 0x7f4a3007a430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a501e1a60 tx=0x7f4a3c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.049+0000 7f3588c00640 1 Processor -- start 2024-09-11T00:31:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.049+0000 7f3588c00640 1 -- start start 2024-09-11T00:31:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.050+0000 7f6283e00640 1 -- 172.21.15.31:0/589731635 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6274049030 con 0x7f628c0bf2c0 2024-09-11T00:31:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.051+0000 7f3588c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584070570 0x7f358414aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.051+0000 7f3588c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c2ff0 0x7f3584140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.051+0000 7f3588c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c3d10 0x7f35841410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.052+0000 7f3588c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f35840c5fb0 con 0x7f35840c3d10 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.052+0000 7f3588c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f358414fb20 con 0x7f3584070570 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.052+0000 7f3588c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f35840c7690 con 0x7f35840c2ff0 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.052+0000 7f3581400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c2ff0 0x7f3584140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.052+0000 7f3581400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c2ff0 0x7f3584140b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50802/0 (socket says 172.21.15.31:50802) 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.052+0000 7f3581400640 1 -- 172.21.15.31:0/3422657169 learned_addr learned my addr 172.21.15.31:0/3422657169 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.052+0000 7f3581e00640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584070570 0x7f358414aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.053+0000 7f3581400640 1 -- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584070570 msgr2=0x7f358414aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.053+0000 7f3581400640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584070570 0x7f358414aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.053+0000 7f3581400640 1 -- 172.21.15.31:0/3422657169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c3d10 msgr2=0x7f35841410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.052+0000 7f3582800640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c3d10 0x7f35841410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.053+0000 7f3581400640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c3d10 0x7f35841410b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.053+0000 7f3581400640 1 -- 172.21.15.31:0/3422657169 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35841492e0 con 0x7f35840c2ff0 2024-09-11T00:31:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.053+0000 7f3582800640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c3d10 0x7f35841410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.054+0000 7f3581400640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c2ff0 0x7f3584140b70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3570034040 tx=0x7f3570003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.055+0000 7f4a47400640 1 -- 172.21.15.31:0/2889091541 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4a4003f030 con 0x7f4a500c4c00 2024-09-11T00:31:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.055+0000 7f357a000640 1 -- 172.21.15.31:0/3422657169 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357003e030 con 0x7f35840c2ff0 2024-09-11T00:31:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.055+0000 7f463cc00640 1 -- 172.21.15.31:0/313215615 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4628062650 con 0x7f46400c7540 2024-09-11T00:31:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.055+0000 7f357a000640 1 -- 172.21.15.31:0/3422657169 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3570003c20 con 0x7f35840c2ff0 2024-09-11T00:31:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.055+0000 7f357a000640 1 -- 172.21.15.31:0/3422657169 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570035040 con 0x7f35840c2ff0 2024-09-11T00:31:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.057+0000 7f3588c00640 1 -- 172.21.15.31:0/3422657169 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3584148e90 con 0x7f35840c2ff0 2024-09-11T00:31:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.057+0000 7f3588c00640 1 -- 172.21.15.31:0/3422657169 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f358414b410 con 0x7f35840c2ff0 2024-09-11T00:31:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.059+0000 7f3588c00640 1 -- 172.21.15.31:0/3422657169 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35840a9550 con 0x7f35840c2ff0 2024-09-11T00:31:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.061+0000 7f357a000640 1 -- 172.21.15.31:0/3422657169 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3570043410 con 0x7f35840c2ff0 2024-09-11T00:31:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.063+0000 7f357a000640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564078060 0x7f356407a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.063+0000 7f357a000640 1 -- 172.21.15.31:0/3422657169 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f35700bb100 con 0x7f35840c2ff0 2024-09-11T00:31:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.064+0000 7f8451600640 1 -- 172.21.15.31:0/3649638045 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f844401f020 con 0x7f84540ada20 2024-09-11T00:31:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.065+0000 7f3581e00640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564078060 0x7f356407a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.065+0000 7f357a000640 1 -- 172.21.15.31:0/3422657169 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f35840a9550 con 0x7f35840c2ff0 2024-09-11T00:31:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.068+0000 7f3581e00640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564078060 0x7f356407a520 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3574000f60 tx=0x7f3574009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:03.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.473+0000 7f3d88c00640 1 -- 172.21.15.31:0/1866572023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f3d7c16f800 con 0x7f3d7c046570 2024-09-11T00:31:03.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.474+0000 7f3d7a000640 1 -- 172.21.15.31:0/1866572023 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3d70087cb0 con 0x7f3d7c046570 2024-09-11T00:31:03.477 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2024-09-11T00:31:03.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.478+0000 7f3d88c00640 1 -- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d64078040 msgr2=0x7f3d6407a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.478+0000 7f3d88c00640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d64078040 0x7f3d6407a500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d84052890 tx=0x7f3d84051dd0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.479+0000 7f3d88c00640 1 -- 172.21.15.31:0/1866572023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d7c046570 msgr2=0x7f3d7c16b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.479+0000 7f3d88c00640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d7c046570 0x7f3d7c16b820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3d700099f0 tx=0x7f3d700039e0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.480+0000 7f3d88c00640 1 -- 172.21.15.31:0/1866572023 shutdown_connections 2024-09-11T00:31:03.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.480+0000 7f3d88c00640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d64078040 0x7f3d6407a500 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.480+0000 7f3d88c00640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d7c0d3430 0x7f3d7c16bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.480+0000 7f3d88c00640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d7c046570 0x7f3d7c16b820 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.480+0000 7f3d88c00640 1 --2- 172.21.15.31:0/1866572023 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d7c00b6e0 0x7f3d7c0d0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.480+0000 7f3d88c00640 1 -- 172.21.15.31:0/1866572023 >> 172.21.15.31:0/1866572023 conn(0x7f3d7c047e40 msgr2=0x7f3d7c02fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.480+0000 7f3d88c00640 1 -- 172.21.15.31:0/1866572023 shutdown_connections 2024-09-11T00:31:03.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.480+0000 7f3d88c00640 1 -- 172.21.15.31:0/1866572023 wait complete. 2024-09-11T00:31:03.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.488+0000 7f6760a00640 1 -- 172.21.15.31:0/621284465 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f675c0c4a10 con 0x7f675c1510a0 2024-09-11T00:31:03.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.489+0000 7f6753400640 1 -- 172.21.15.31:0/621284465 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6754083b00 con 0x7f675c1510a0 2024-09-11T00:31:03.494 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2024-09-11T00:31:03.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.493+0000 7f6750c00640 1 -- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f673c077f70 msgr2=0x7f673c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.493+0000 7f6750c00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f673c077f70 0x7f673c07a430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f674c003f70 tx=0x7f674c009040 comp rx=0 tx=0).stop 2024-09-11T00:31:03.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.493+0000 7f6750c00640 1 -- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f675c1510a0 msgr2=0x7f675c1533e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.493+0000 7f6750c00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f675c1510a0 0x7f675c1533e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6754032000 tx=0x7f6754003f10 comp rx=0 tx=0).stop 2024-09-11T00:31:03.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.494+0000 7f6750c00640 1 -- 172.21.15.31:0/621284465 shutdown_connections 2024-09-11T00:31:03.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.494+0000 7f6750c00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f673c077f70 0x7f673c07a430 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.494+0000 7f6750c00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f675c1510a0 0x7f675c1533e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.494+0000 7f6750c00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f675c0c4630 0x7f675c149240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.494+0000 7f6750c00640 1 --2- 172.21.15.31:0/621284465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f675c0a6e20 0x7f675c151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.494+0000 7f6750c00640 1 -- 172.21.15.31:0/621284465 >> 172.21.15.31:0/621284465 conn(0x7f675c06caf0 msgr2=0x7f675c0c36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.495+0000 7f6750c00640 1 -- 172.21.15.31:0/621284465 shutdown_connections 2024-09-11T00:31:03.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.495+0000 7f6750c00640 1 -- 172.21.15.31:0/621284465 wait complete. 2024-09-11T00:31:03.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.501+0000 7f6292e00640 1 -- 172.21.15.31:0/589731635 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f628c1eee30 con 0x7f628c0bf2c0 2024-09-11T00:31:03.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.502+0000 7f6283e00640 1 -- 172.21.15.31:0/589731635 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6274083b90 con 0x7f628c0bf2c0 2024-09-11T00:31:03.504 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2024-09-11T00:31:03.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.504+0000 7f4647c00640 1 -- 172.21.15.31:0/313215615 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f464014b8e0 con 0x7f46400c7540 2024-09-11T00:31:03.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.504+0000 7f31f9a00640 1 -- 172.21.15.31:0/2418569014 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f31f40c5210 con 0x7f31f40ae0f0 2024-09-11T00:31:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.504+0000 7f463cc00640 1 -- 172.21.15.31:0/313215615 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f462806b3c0 con 0x7f46400c7540 2024-09-11T00:31:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.504+0000 7f4a55200640 1 -- 172.21.15.31:0/2889091541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f4a28005580 con 0x7f4a500c4c00 2024-09-11T00:31:03.507 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2024-09-11T00:31:03.507 INFO:teuthology.orchestra.run.smithi031.stdout:64424509447 2024-09-11T00:31:03.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.505+0000 7f4a47400640 1 -- 172.21.15.31:0/2889091541 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4a40083070 con 0x7f4a500c4c00 2024-09-11T00:31:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.505+0000 7f31eaa00640 1 -- 172.21.15.31:0/2418569014 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f31e4053080 con 0x7f31f40ae0f0 2024-09-11T00:31:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.506+0000 7f6292e00640 1 -- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6268077f70 msgr2=0x7f626807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.506+0000 7f6292e00640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6268077f70 0x7f626807a430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f628c0ae760 tx=0x7f627c009040 comp rx=0 tx=0).stop 2024-09-11T00:31:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.506+0000 7f6292e00640 1 -- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f628c0bf2c0 msgr2=0x7f628c1e1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.506+0000 7f6292e00640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f628c0bf2c0 0x7f628c1e1810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6274009850 tx=0x7f62740038a0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.509 INFO:teuthology.orchestra.run.smithi031.stdout:64424509446 2024-09-11T00:31:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.506+0000 7f6292e00640 1 -- 172.21.15.31:0/589731635 shutdown_connections 2024-09-11T00:31:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.506+0000 7f6292e00640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6268077f70 0x7f626807a430 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.506+0000 7f6292e00640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f628c1efb60 0x7f628c1dfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.506+0000 7f6292e00640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f628c0c9b30 0x7f628c1df8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.506+0000 7f6292e00640 1 --2- 172.21.15.31:0/589731635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f628c0bf2c0 0x7f628c1e1810 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.506+0000 7f6292e00640 1 -- 172.21.15.31:0/589731635 >> 172.21.15.31:0/589731635 conn(0x7f628c0cc260 msgr2=0x7f628c0c9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.510 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.0 2024-09-11T00:31:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.508+0000 7f6292e00640 1 -- 172.21.15.31:0/589731635 shutdown_connections 2024-09-11T00:31:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.508+0000 7f6292e00640 1 -- 172.21.15.31:0/589731635 wait complete. 2024-09-11T00:31:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f4647c00640 1 -- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f461c077f70 msgr2=0x7f461c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f4647c00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f461c077f70 0x7f461c07a430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4638000f80 tx=0x7f4638004ab0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f4647c00640 1 -- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46400c7540 msgr2=0x7f46400c49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 -- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31d0078110 msgr2=0x7f31d007a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31d0078110 0x7f31d007a5d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f31f4157f70 tx=0x7f31dc006040 comp rx=0 tx=0).stop 2024-09-11T00:31:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 -- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31f40ae0f0 msgr2=0x7f31f40ab710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31f40ae0f0 0x7f31f40ab710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f31e4032000 tx=0x7f31e402f560 comp rx=0 tx=0).stop 2024-09-11T00:31:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f4647c00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46400c7540 0x7f46400c49b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f462800d5e0 tx=0x7f462800bf90 comp rx=0 tx=0).stop 2024-09-11T00:31:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 -- 172.21.15.31:0/2418569014 shutdown_connections 2024-09-11T00:31:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31d0078110 0x7f31d007a5d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f40a9d20 0x7f31f40c0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31f4163a80 0x7f31f40a97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 --2- 172.21.15.31:0/2418569014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31f40ae0f0 0x7f31f40ab710 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 -- 172.21.15.31:0/2418569014 >> 172.21.15.31:0/2418569014 conn(0x7f31f40688e0 msgr2=0x7f31f415ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f4647c00640 1 -- 172.21.15.31:0/313215615 shutdown_connections 2024-09-11T00:31:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f31f9a00640 1 -- 172.21.15.31:0/2418569014 shutdown_connections 2024-09-11T00:31:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f4647c00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f461c077f70 0x7f461c07a430 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f4647c00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46400c7540 0x7f46400c49b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f4647c00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4640153f20 0x7f464014ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f4647c00640 1 --2- 172.21.15.31:0/313215615 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46400c23a0 0x7f4640148020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.509+0000 7f4647c00640 1 -- 172.21.15.31:0/313215615 >> 172.21.15.31:0/313215615 conn(0x7f46400a9750 msgr2=0x7f464014f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4647c00640 1 -- 172.21.15.31:0/313215615 shutdown_connections 2024-09-11T00:31:03.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f31f9a00640 1 -- 172.21.15.31:0/2418569014 wait complete. 2024-09-11T00:31:03.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4a55200640 1 -- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a30077f70 msgr2=0x7f4a3007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4a55200640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a30077f70 0x7f4a3007a430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a501e1a60 tx=0x7f4a3c009040 comp rx=0 tx=0).stop 2024-09-11T00:31:03.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4a55200640 1 -- 172.21.15.31:0/2889091541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a500c4c00 msgr2=0x7f4a50156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4a55200640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a500c4c00 0x7f4a50156460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4a5006d740 tx=0x7f4a40033cb0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4647c00640 1 -- 172.21.15.31:0/313215615 wait complete. 2024-09-11T00:31:03.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4a55200640 1 -- 172.21.15.31:0/2889091541 shutdown_connections 2024-09-11T00:31:03.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4a55200640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a30077f70 0x7f4a3007a430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4a55200640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a50159ee0 0x7f4a500ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4a55200640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a500c62f0 0x7f4a500aabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4a55200640 1 --2- 172.21.15.31:0/2889091541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a500c4c00 0x7f4a50156460 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.510+0000 7f4a55200640 1 -- 172.21.15.31:0/2889091541 >> 172.21.15.31:0/2889091541 conn(0x7f4a5006caf0 msgr2=0x7f4a50152c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.511+0000 7f4a55200640 1 -- 172.21.15.31:0/2889091541 shutdown_connections 2024-09-11T00:31:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.511+0000 7f4a55200640 1 -- 172.21.15.31:0/2889091541 wait complete. 2024-09-11T00:31:03.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.519+0000 7f845a800640 1 -- 172.21.15.31:0/3649638045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f8430005580 con 0x7f84540ada20 2024-09-11T00:31:03.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.522+0000 7f8451600640 1 -- 172.21.15.31:0/3649638045 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8444062250 con 0x7f84540ada20 2024-09-11T00:31:03.524 INFO:teuthology.orchestra.run.smithi031.stdout:60129542152 2024-09-11T00:31:03.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.2 2024-09-11T00:31:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.524+0000 7f3588c00640 1 -- 172.21.15.31:0/3422657169 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3584151230 con 0x7f35840c2ff0 2024-09-11T00:31:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.524+0000 7f357a000640 1 -- 172.21.15.31:0/3422657169 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f357008c060 con 0x7f35840c2ff0 2024-09-11T00:31:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 -- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8438077fa0 msgr2=0x7f843807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8438077fa0 0x7f843807a460 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8448009830 tx=0x7f8448004730 comp rx=0 tx=0).stop 2024-09-11T00:31:03.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 -- 172.21.15.31:0/3649638045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84540ada20 msgr2=0x7f84540c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84540ada20 0x7f84540c3ec0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f844400b3f0 tx=0x7f844400b8b0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.527 INFO:teuthology.orchestra.run.smithi031.stdout:64424509449 2024-09-11T00:31:03.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 -- 172.21.15.31:0/3649638045 shutdown_connections 2024-09-11T00:31:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8438077fa0 0x7f843807a460 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8454156960 0x7f84541e1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8454150020 0x7f84540c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 --2- 172.21.15.31:0/3649638045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84540ada20 0x7f84540c3ec0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 -- 172.21.15.31:0/3649638045 >> 172.21.15.31:0/3649638045 conn(0x7f8454068f10 msgr2=0x7f8454157d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.525+0000 7f845a800640 1 -- 172.21.15.31:0/3649638045 shutdown_connections 2024-09-11T00:31:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.526+0000 7f845a800640 1 -- 172.21.15.31:0/3649638045 wait complete. 2024-09-11T00:31:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 -- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564078060 msgr2=0x7f356407a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564078060 0x7f356407a520 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3574000f60 tx=0x7f3574009040 comp rx=0 tx=0).stop 2024-09-11T00:31:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 -- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c2ff0 msgr2=0x7f3584140b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c2ff0 0x7f3584140b70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3570034040 tx=0x7f3570003820 comp rx=0 tx=0).stop 2024-09-11T00:31:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 -- 172.21.15.31:0/3422657169 shutdown_connections 2024-09-11T00:31:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564078060 0x7f356407a520 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c3d10 0x7f35841410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c2ff0 0x7f3584140b70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 --2- 172.21.15.31:0/3422657169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584070570 0x7f358414aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 -- 172.21.15.31:0/3422657169 >> 172.21.15.31:0/3422657169 conn(0x7f35840a9750 msgr2=0x7f358414c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.528+0000 7f3588c00640 1 -- 172.21.15.31:0/3422657169 shutdown_connections 2024-09-11T00:31:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:03.529+0000 7f3588c00640 1 -- 172.21.15.31:0/3422657169 wait complete. 2024-09-11T00:31:03.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.3 2024-09-11T00:31:03.536 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.7 2024-09-11T00:31:03.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.1 2024-09-11T00:31:03.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.5 2024-09-11T00:31:03.551 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.4 2024-09-11T00:31:03.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.6 2024-09-11T00:31:04.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T00:31:04.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T00:31:04.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T00:31:04.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-11T00:31:04.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T00:31:04.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T00:31:04.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T00:31:04.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-11T00:31:04.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.768+0000 7fac8a800640 1 -- 172.21.15.31:0/151637599 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac84156970 msgr2=0x7fac840738c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.768+0000 7fac8a800640 1 --2- 172.21.15.31:0/151637599 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac84156970 0x7fac840738c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fac70009830 tx=0x7fac7002a440 comp rx=0 tx=0).stop 2024-09-11T00:31:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.768+0000 7fac8a800640 1 -- 172.21.15.31:0/151637599 shutdown_connections 2024-09-11T00:31:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.768+0000 7fac8a800640 1 --2- 172.21.15.31:0/151637599 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac84156970 0x7fac840738c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.768+0000 7fac8a800640 1 --2- 172.21.15.31:0/151637599 >> v2:172.21.15.31:3300/0 conn(0x7fac8406f7e0 0x7fac84070810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.769+0000 7fac8a800640 1 -- 172.21.15.31:0/151637599 >> 172.21.15.31:0/151637599 conn(0x7fac840c7ec0 msgr2=0x7fac840c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.769+0000 7fac8a800640 1 -- 172.21.15.31:0/151637599 shutdown_connections 2024-09-11T00:31:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.769+0000 7fac8a800640 1 -- 172.21.15.31:0/151637599 wait complete. 2024-09-11T00:31:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.771+0000 7fac8a800640 1 Processor -- start 2024-09-11T00:31:04.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.772+0000 7fac8a800640 1 -- start start 2024-09-11T00:31:04.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.773+0000 7fac8a800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac8406f7e0 0x7fac841482b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.773+0000 7fac8a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac84073e00 0x7fac8414b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.773+0000 7fac8a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac84148ab0 0x7fac84150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.773+0000 7fac8a800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fac84070110 con 0x7fac84073e00 2024-09-11T00:31:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.773+0000 7fac8a800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fac840adb60 con 0x7fac8406f7e0 2024-09-11T00:31:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.773+0000 7fac8a800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fac84070290 con 0x7fac84148ab0 2024-09-11T00:31:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.774+0000 7fac89e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac84148ab0 0x7fac84150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.774+0000 7fac89400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac8406f7e0 0x7fac841482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.774+0000 7fac89e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac84148ab0 0x7fac84150760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50818/0 (socket says 172.21.15.31:50818) 2024-09-11T00:31:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.774+0000 7fac89e00640 1 -- 172.21.15.31:0/725990357 learned_addr learned my addr 172.21.15.31:0/725990357 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.774+0000 7fac88a00640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac84073e00 0x7fac8414b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.775+0000 7fac89e00640 1 -- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac8406f7e0 msgr2=0x7fac841482b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.775+0000 7fac89e00640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac8406f7e0 0x7fac841482b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.775+0000 7fac89e00640 1 -- 172.21.15.31:0/725990357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac84073e00 msgr2=0x7fac8414b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.775+0000 7fac89e00640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac84073e00 0x7fac8414b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.775+0000 7fac89e00640 1 -- 172.21.15.31:0/725990357 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac841f48a0 con 0x7fac84148ab0 2024-09-11T00:31:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.775+0000 7fac88a00640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac84073e00 0x7fac8414b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.776+0000 7fac89e00640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac84148ab0 0x7fac84150760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fac7800c700 tx=0x7fac7800cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.776+0000 7fac81600640 1 -- 172.21.15.31:0/725990357 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac78005270 con 0x7fac84148ab0 2024-09-11T00:31:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.777+0000 7fac81600640 1 -- 172.21.15.31:0/725990357 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac78005410 con 0x7fac84148ab0 2024-09-11T00:31:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.777+0000 7fac81600640 1 -- 172.21.15.31:0/725990357 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac78005270 con 0x7fac84148ab0 2024-09-11T00:31:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.777+0000 7fac8a800640 1 -- 172.21.15.31:0/725990357 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac8414d490 con 0x7fac84148ab0 2024-09-11T00:31:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.777+0000 7fac8a800640 1 -- 172.21.15.31:0/725990357 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac840ac970 con 0x7fac84148ab0 2024-09-11T00:31:04.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.779+0000 7fac8a800640 1 -- 172.21.15.31:0/725990357 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac84074270 con 0x7fac84148ab0 2024-09-11T00:31:04.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.782+0000 7fac81600640 1 -- 172.21.15.31:0/725990357 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fac78007030 con 0x7fac84148ab0 2024-09-11T00:31:04.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.784+0000 7fac81600640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fac640781a0 0x7fac6407a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.784+0000 7fac89400640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fac640781a0 0x7fac6407a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.785+0000 7fac81600640 1 -- 172.21.15.31:0/725990357 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fac78015e00 con 0x7fac84148ab0 2024-09-11T00:31:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.786+0000 7fac89400640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fac640781a0 0x7fac6407a660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fac84074790 tx=0x7fac7c0094d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.791+0000 7fac81600640 1 -- 172.21.15.31:0/725990357 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fac840ae080 con 0x7fac84148ab0 2024-09-11T00:31:04.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.849+0000 7fdc77200640 1 -- 172.21.15.31:0/2023012875 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc780c6420 msgr2=0x7fdc78153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.849+0000 7fdc77200640 1 --2- 172.21.15.31:0/2023012875 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc780c6420 0x7fdc78153310 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdc64005fd0 tx=0x7fdc6402a560 comp rx=0 tx=0).stop 2024-09-11T00:31:04.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.849+0000 7f795e800640 1 -- 172.21.15.31:0/3541847447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580c2ff0 msgr2=0x7f795806e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.850+0000 7fdc77200640 1 -- 172.21.15.31:0/2023012875 shutdown_connections 2024-09-11T00:31:04.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.850+0000 7fdc77200640 1 --2- 172.21.15.31:0/2023012875 >> v2:172.21.15.31:3300/0 conn(0x7fdc78153850 0x7fdc78156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.850+0000 7fdc77200640 1 --2- 172.21.15.31:0/2023012875 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc780c6420 0x7fdc78153310 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdc64005fd0 tx=0x7fdc6402a560 comp rx=0 tx=0).stop 2024-09-11T00:31:04.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.850+0000 7fdc77200640 1 -- 172.21.15.31:0/2023012875 >> 172.21.15.31:0/2023012875 conn(0x7fdc7806caf0 msgr2=0x7fdc7806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:04.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.850+0000 7f795e800640 1 --2- 172.21.15.31:0/3541847447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580c2ff0 0x7f795806e480 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7948005fd0 tx=0x7f794802a660 comp rx=0 tx=0).stop 2024-09-11T00:31:04.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.850+0000 7f795e800640 1 -- 172.21.15.31:0/3541847447 shutdown_connections 2024-09-11T00:31:04.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.850+0000 7f795e800640 1 --2- 172.21.15.31:0/3541847447 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f795806e9c0 0x7f79581568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.850+0000 7f795e800640 1 --2- 172.21.15.31:0/3541847447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580c2ff0 0x7f795806e480 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.850+0000 7f795e800640 1 -- 172.21.15.31:0/3541847447 >> 172.21.15.31:0/3541847447 conn(0x7f79580a9750 msgr2=0x7f79580a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:04.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.851+0000 7f3c63200640 1 -- 172.21.15.31:0/1895292469 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c64159ee0 msgr2=0x7f3c64156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.851+0000 7f3c63200640 1 --2- 172.21.15.31:0/1895292469 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c64159ee0 0x7f3c64156280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c005fd0 tx=0x7f3c4c02a540 comp rx=0 tx=0).stop 2024-09-11T00:31:04.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.851+0000 7f3c63200640 1 -- 172.21.15.31:0/1895292469 shutdown_connections 2024-09-11T00:31:04.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.851+0000 7f3c63200640 1 --2- 172.21.15.31:0/1895292469 >> v2:172.21.15.31:3300/0 conn(0x7f3c640c4c00 0x7f3c641567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.851+0000 7f3c63200640 1 --2- 172.21.15.31:0/1895292469 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c64159ee0 0x7f3c64156280 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c005fd0 tx=0x7f3c4c02a540 comp rx=0 tx=0).stop 2024-09-11T00:31:04.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.852+0000 7f3c63200640 1 -- 172.21.15.31:0/1895292469 >> 172.21.15.31:0/1895292469 conn(0x7f3c6406caf0 msgr2=0x7f3c6406ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:04.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.852+0000 7f795e800640 1 -- 172.21.15.31:0/3541847447 shutdown_connections 2024-09-11T00:31:04.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.852+0000 7f795e800640 1 -- 172.21.15.31:0/3541847447 wait complete. 2024-09-11T00:31:04.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.852+0000 7f3c63200640 1 -- 172.21.15.31:0/1895292469 shutdown_connections 2024-09-11T00:31:04.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.852+0000 7f3c63200640 1 -- 172.21.15.31:0/1895292469 wait complete. 2024-09-11T00:31:04.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.853+0000 7fdc77200640 1 -- 172.21.15.31:0/2023012875 shutdown_connections 2024-09-11T00:31:04.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.855+0000 7f3c63200640 1 Processor -- start 2024-09-11T00:31:04.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.855+0000 7f795e800640 1 Processor -- start 2024-09-11T00:31:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.855+0000 7f795e800640 1 -- start start 2024-09-11T00:31:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.855+0000 7f3c63200640 1 -- start start 2024-09-11T00:31:04.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.856+0000 7fdc77200640 1 -- 172.21.15.31:0/2023012875 wait complete. 2024-09-11T00:31:04.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.856+0000 7f3c63200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c640c4c00 0x7f3c640743a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f795e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795806e9c0 0x7f795815fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f795e800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7958070570 0x7f7958141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f795e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580c2ff0 0x7f7958141c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f795e800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f795814fb20 con 0x7f795806e9c0 2024-09-11T00:31:04.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f795e800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f795806ef90 con 0x7f79580c2ff0 2024-09-11T00:31:04.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f795e800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f79580c5fb0 con 0x7f7958070570 2024-09-11T00:31:04.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f3c63200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c640c62f0 0x7f3c640abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f7957400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795806e9c0 0x7f795815fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f7957400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795806e9c0 0x7f795815fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37560/0 (socket says 172.21.15.31:37560) 2024-09-11T00:31:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f3c63200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6406fba0 0x7f3c64152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f7957400640 1 -- 172.21.15.31:0/2339200057 learned_addr learned my addr 172.21.15.31:0/2339200057 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:04.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.857+0000 7f7957e00640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580c2ff0 0x7f7958141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f7956a00640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7958070570 0x7f7958141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f1fbcc00640 1 -- 172.21.15.31:0/2723953254 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fb80c4c00 msgr2=0x7f1fb8156990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c63200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6415a2c0 con 0x7f3c6406fba0 2024-09-11T00:31:04.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c63200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3c640c5f40 con 0x7f3c640c4c00 2024-09-11T00:31:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c63200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3c64156d00 con 0x7f3c640c62f0 2024-09-11T00:31:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c61e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c640c4c00 0x7f3c640743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c61e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c640c4c00 0x7f3c640743a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:37176/0 (socket says 172.21.15.31:37176) 2024-09-11T00:31:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c61e00640 1 -- 172.21.15.31:0/2874359733 learned_addr learned my addr 172.21.15.31:0/2874359733 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c62800640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6406fba0 0x7f3c64152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f1fbcc00640 1 --2- 172.21.15.31:0/2723953254 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fb80c4c00 0x7f1fb8156990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009860 tx=0x7f1fa402a510 comp rx=0 tx=0).stop 2024-09-11T00:31:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7fdc77200640 1 Processor -- start 2024-09-11T00:31:04.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f1fbcc00640 1 -- 172.21.15.31:0/2723953254 shutdown_connections 2024-09-11T00:31:04.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f1fbcc00640 1 --2- 172.21.15.31:0/2723953254 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fb80c4c00 0x7f1fb8156990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c61e00640 1 -- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c640c62f0 msgr2=0x7f3c640abbc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:31:04.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c61e00640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c640c62f0 0x7f3c640abbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c61e00640 1 -- 172.21.15.31:0/2874359733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6406fba0 msgr2=0x7f3c64152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c61e00640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6406fba0 0x7f3c64152c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7fdc77200640 1 -- start start 2024-09-11T00:31:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f1fbcc00640 1 -- 172.21.15.31:0/2723953254 >> 172.21.15.31:0/2723953254 conn(0x7f1fb806caf0 msgr2=0x7f1fb806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.858+0000 7f3c61e00640 1 -- 172.21.15.31:0/2874359733 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c640748e0 con 0x7f3c640c4c00 2024-09-11T00:31:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f1fbcc00640 1 -- 172.21.15.31:0/2723953254 shutdown_connections 2024-09-11T00:31:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f7956a00640 1 -- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580c2ff0 msgr2=0x7f7958141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f7956a00640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580c2ff0 0x7f7958141c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f7956a00640 1 -- 172.21.15.31:0/2339200057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795806e9c0 msgr2=0x7f795815fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f7956a00640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795806e9c0 0x7f795815fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f7956a00640 1 -- 172.21.15.31:0/2339200057 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79581421c0 con 0x7f7958070570 2024-09-11T00:31:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f7957400640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795806e9c0 0x7f795815fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f1fbcc00640 1 -- 172.21.15.31:0/2723953254 wait complete. 2024-09-11T00:31:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f9b5d000640 1 -- 172.21.15.31:0/2716945434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b581537c0 msgr2=0x7f9b580c31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f9b5d000640 1 --2- 172.21.15.31:0/2716945434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b581537c0 0x7f9b580c31e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009830 tx=0x7f9b4802a520 comp rx=0 tx=0).stop 2024-09-11T00:31:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f9b5d000640 1 -- 172.21.15.31:0/2716945434 shutdown_connections 2024-09-11T00:31:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f9b5d000640 1 --2- 172.21.15.31:0/2716945434 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b580c7460 0x7f9b5815a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f9b5d000640 1 --2- 172.21.15.31:0/2716945434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b581537c0 0x7f9b580c31e0 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009830 tx=0x7f9b4802a520 comp rx=0 tx=0).stop 2024-09-11T00:31:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.859+0000 7f9b5d000640 1 -- 172.21.15.31:0/2716945434 >> 172.21.15.31:0/2716945434 conn(0x7f9b580a9750 msgr2=0x7f9b580a9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:04.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.860+0000 7f9b5d000640 1 -- 172.21.15.31:0/2716945434 shutdown_connections 2024-09-11T00:31:04.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.860+0000 7f8207200640 1 -- 172.21.15.31:0/432103280 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82000c99e0 msgr2=0x7f820006eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.860+0000 7f8207200640 1 --2- 172.21.15.31:0/432103280 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82000c99e0 0x7f820006eb40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81ec009860 tx=0x7f81ec02a460 comp rx=0 tx=0).stop 2024-09-11T00:31:04.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.860+0000 7f9b5d000640 1 -- 172.21.15.31:0/2716945434 wait complete. 2024-09-11T00:31:04.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.860+0000 7fdc77200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc780c5aa0 0x7fdc78155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.860+0000 7fdc77200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc78153850 0x7fdc781fc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.860+0000 7fdc77200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc780abaf0 0x7fdc781fc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc75e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc780c5aa0 0x7fdc78155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc75e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc780c5aa0 0x7fdc78155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37578/0 (socket says 172.21.15.31:37578) 2024-09-11T00:31:04.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7f7956a00640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7958070570 0x7f7958141740 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7948031000 tx=0x7f7948003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7f3c61e00640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c640c4c00 0x7f3c640743a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c006010 tx=0x7f3c4c0039e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc77200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdc780c5e80 con 0x7fdc780c5aa0 2024-09-11T00:31:04.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc77200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc780c2e80 con 0x7fdc78153850 2024-09-11T00:31:04.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc77200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdc780c6000 con 0x7fdc780abaf0 2024-09-11T00:31:04.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc75e00640 1 -- 172.21.15.31:0/3523163760 learned_addr learned my addr 172.21.15.31:0/3523163760 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:04.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7f8207200640 1 -- 172.21.15.31:0/432103280 shutdown_connections 2024-09-11T00:31:04.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7f8207200640 1 --2- 172.21.15.31:0/432103280 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82000c99e0 0x7f820006eb40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7f8207200640 1 -- 172.21.15.31:0/432103280 >> 172.21.15.31:0/432103280 conn(0x7f82000ad890 msgr2=0x7f82000adca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:04.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7f1fbcc00640 1 Processor -- start 2024-09-11T00:31:04.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7f8207200640 1 -- 172.21.15.31:0/432103280 shutdown_connections 2024-09-11T00:31:04.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc76800640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc780abaf0 0x7fdc781fc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc75e00640 1 -- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc780abaf0 msgr2=0x7fdc781fc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc75e00640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc780abaf0 0x7fdc781fc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc75e00640 1 -- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc78153850 msgr2=0x7fdc781fc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc75e00640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc78153850 0x7fdc781fc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7fdc75e00640 1 -- 172.21.15.31:0/3523163760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc781fbd00 con 0x7fdc780c5aa0 2024-09-11T00:31:04.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7fdc76800640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc780abaf0 0x7fdc781fc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:04.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7f794f400640 1 -- 172.21.15.31:0/2339200057 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794803e030 con 0x7f7958070570 2024-09-11T00:31:04.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7f794f400640 1 -- 172.21.15.31:0/2339200057 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7948003dd0 con 0x7f7958070570 2024-09-11T00:31:04.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7f794f400640 1 -- 172.21.15.31:0/2339200057 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948035040 con 0x7f7958070570 2024-09-11T00:31:04.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7f1fbcc00640 1 -- start start 2024-09-11T00:31:04.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7f9b5d000640 1 Processor -- start 2024-09-11T00:31:04.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.861+0000 7f3c5a000640 1 -- 172.21.15.31:0/2874359733 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c03f020 con 0x7f3c640c4c00 2024-09-11T00:31:04.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7f3c5a000640 1 -- 172.21.15.31:0/2874359733 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c4c004890 con 0x7f3c640c4c00 2024-09-11T00:31:04.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7f3c5a000640 1 -- 172.21.15.31:0/2874359733 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c035070 con 0x7f3c640c4c00 2024-09-11T00:31:04.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7f3c63200640 1 -- 172.21.15.31:0/2874359733 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c64072a80 con 0x7f3c640c4c00 2024-09-11T00:31:04.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7f9b5d000640 1 -- start start 2024-09-11T00:31:04.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7fdc75e00640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc780c5aa0 0x7fdc78155540 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800c430 tx=0x7fdc6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.862+0000 7f8207200640 1 -- 172.21.15.31:0/432103280 wait complete. 2024-09-11T00:31:04.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.863+0000 7f795e800640 1 -- 172.21.15.31:0/2339200057 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7958074d90 con 0x7f7958070570 2024-09-11T00:31:04.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.863+0000 7f795e800640 1 -- 172.21.15.31:0/2339200057 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7958072740 con 0x7f7958070570 2024-09-11T00:31:04.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.863+0000 7fdc6e000640 1 -- 172.21.15.31:0/3523163760 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc68011790 con 0x7fdc780c5aa0 2024-09-11T00:31:04.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.863+0000 7fdc6e000640 1 -- 172.21.15.31:0/3523163760 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc68017070 con 0x7fdc780c5aa0 2024-09-11T00:31:04.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.863+0000 7fdc6e000640 1 -- 172.21.15.31:0/3523163760 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc68011d00 con 0x7fdc780c5aa0 2024-09-11T00:31:04.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.863+0000 7f4074e00640 1 -- 172.21.15.31:0/1422806475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40700acaf0 msgr2=0x7f407006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f9b5d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b580c2d40 0x7f9b581f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f9b5d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b580c7460 0x7f9b580c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.863+0000 7f4074e00640 1 --2- 172.21.15.31:0/1422806475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40700acaf0 0x7f407006dc60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f406800afd0 tx=0x7f406802b560 comp rx=0 tx=0).stop 2024-09-11T00:31:04.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7fdc77200640 1 -- 172.21.15.31:0/3523163760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc78073890 con 0x7fdc780c5aa0 2024-09-11T00:31:04.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f9b5d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b580ab730 0x7f9b5814b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f9b5d000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581560a0 con 0x7f9b580ab730 2024-09-11T00:31:04.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f9b5d000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9b580c6c60 con 0x7f9b580c2d40 2024-09-11T00:31:04.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f9b5d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9b58153ba0 con 0x7f9b580c7460 2024-09-11T00:31:04.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.863+0000 7f3c63200640 1 -- 172.21.15.31:0/2874359733 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c64073040 con 0x7f3c640c4c00 2024-09-11T00:31:04.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7fdc77200640 1 -- 172.21.15.31:0/3523163760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc78155a80 con 0x7fdc780c5aa0 2024-09-11T00:31:04.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f9b56800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b580ab730 0x7f9b5814b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f9b56800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b580ab730 0x7f9b5814b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37580/0 (socket says 172.21.15.31:37580) 2024-09-11T00:31:04.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f9b56800640 1 -- 172.21.15.31:0/1143929635 learned_addr learned my addr 172.21.15.31:0/1143929635 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:04.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f9b55400640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b580c7460 0x7f9b580c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f9b55e00640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b580c2d40 0x7f9b581f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f9b56800640 1 -- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b580c7460 msgr2=0x7f9b580c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f9b56800640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b580c7460 0x7f9b580c5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f9b56800640 1 -- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b580c2d40 msgr2=0x7f9b581f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f9b56800640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b580c2d40 0x7f9b581f40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f9b56800640 1 -- 172.21.15.31:0/1143929635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b581f0270 con 0x7f9b580ab730 2024-09-11T00:31:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f9b55e00640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b580c2d40 0x7f9b581f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fb80c4c00 0x7f1fb80a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f3c63200640 1 -- 172.21.15.31:0/2874359733 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c640c5830 con 0x7f3c640c4c00 2024-09-11T00:31:04.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb80c62f0 0x7f1fb80a7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.865+0000 7f9b56800640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b580ab730 0x7f9b5814b950 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9b50009fd0 tx=0x7f9b5000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f4074e00640 1 -- 172.21.15.31:0/1422806475 shutdown_connections 2024-09-11T00:31:04.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f4074e00640 1 --2- 172.21.15.31:0/1422806475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40700acaf0 0x7f407006dc60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.864+0000 7f4074e00640 1 -- 172.21.15.31:0/1422806475 >> 172.21.15.31:0/1422806475 conn(0x7f407006caf0 msgr2=0x7f407006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:04.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7fdc77200640 1 -- 172.21.15.31:0/3523163760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc781fbed0 con 0x7fdc780c5aa0 2024-09-11T00:31:04.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fb8159ee0 0x7f1fb80aba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f1fb7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fb80c4c00 0x7f1fb80a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f9b4e000640 1 -- 172.21.15.31:0/1143929635 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50017070 con 0x7f9b580ab730 2024-09-11T00:31:04.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f8207200640 1 Processor -- start 2024-09-11T00:31:04.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f8207200640 1 -- start start 2024-09-11T00:31:04.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f1fbcc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb815a2c0 con 0x7f1fb80c62f0 2024-09-11T00:31:04.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f1fbcc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb8153b10 con 0x7f1fb8159ee0 2024-09-11T00:31:04.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f1fbcc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1fb80c2c50 con 0x7f1fb80c4c00 2024-09-11T00:31:04.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f1fb7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fb80c4c00 0x7f1fb80a9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50910/0 (socket says 172.21.15.31:50910) 2024-09-11T00:31:04.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f1fb7400640 1 -- 172.21.15.31:0/1715408546 learned_addr learned my addr 172.21.15.31:0/1715408546 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:04.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.866+0000 7f795e800640 1 -- 172.21.15.31:0/2339200057 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7958072280 con 0x7f7958070570 2024-09-11T00:31:04.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f4074e00640 1 -- 172.21.15.31:0/1422806475 shutdown_connections 2024-09-11T00:31:04.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f9b4e000640 1 -- 172.21.15.31:0/1143929635 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b50003970 con 0x7f9b580ab730 2024-09-11T00:31:04.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f9b4e000640 1 -- 172.21.15.31:0/1143929635 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50003dc0 con 0x7f9b580ab730 2024-09-11T00:31:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f1fb6a00640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb80c62f0 0x7f1fb80a7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f4074e00640 1 -- 172.21.15.31:0/1422806475 wait complete. 2024-09-11T00:31:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f9b5d000640 1 -- 172.21.15.31:0/1143929635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b581f3610 con 0x7f9b580ab730 2024-09-11T00:31:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f9b5d000640 1 -- 172.21.15.31:0/1143929635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b581f3bd0 con 0x7f9b580ab730 2024-09-11T00:31:04.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f1fb6a00640 1 -- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fb80c4c00 msgr2=0x7f1fb80a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f1fb6a00640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fb80c4c00 0x7f1fb80a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f1fb6a00640 1 -- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fb8159ee0 msgr2=0x7f1fb80aba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f1fb6a00640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fb8159ee0 0x7f1fb80aba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.867+0000 7f1fb6a00640 1 -- 172.21.15.31:0/1715408546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb80743c0 con 0x7f1fb80c62f0 2024-09-11T00:31:04.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.868+0000 7f1fb6a00640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb80c62f0 0x7f1fb80a7960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4031000 tx=0x7f1fa4004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.868+0000 7f9b5d000640 1 -- 172.21.15.31:0/1143929635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b581f3dd0 con 0x7f9b580ab730 2024-09-11T00:31:04.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.869+0000 7f1faf400640 1 -- 172.21.15.31:0/1715408546 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4040070 con 0x7f1fb80c62f0 2024-09-11T00:31:04.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.869+0000 7f1faf400640 1 -- 172.21.15.31:0/1715408546 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fa4005310 con 0x7f1fb80c62f0 2024-09-11T00:31:04.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.869+0000 7f1faf400640 1 -- 172.21.15.31:0/1715408546 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa40047a0 con 0x7f1fb80c62f0 2024-09-11T00:31:04.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.869+0000 7f794f400640 1 -- 172.21.15.31:0/2339200057 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7948004a00 con 0x7f7958070570 2024-09-11T00:31:04.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.869+0000 7f4074e00640 1 Processor -- start 2024-09-11T00:31:04.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f4074e00640 1 -- start start 2024-09-11T00:31:04.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f8207200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82000c84f0 0x7f82000c3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f8207200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82000c99e0 0x7f820014d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f8207200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82000c9ef0 0x7f820014d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f8207200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f82000c68e0 con 0x7f82000c84f0 2024-09-11T00:31:04.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7fdc6e000640 1 -- 172.21.15.31:0/3523163760 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdc6801c460 con 0x7fdc780c5aa0 2024-09-11T00:31:04.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f8207200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f820006dcb0 con 0x7f82000c9ef0 2024-09-11T00:31:04.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.869+0000 7f1fbcc00640 1 -- 172.21.15.31:0/1715408546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb80a7170 con 0x7f1fb80c62f0 2024-09-11T00:31:04.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.869+0000 7f1fbcc00640 1 -- 172.21.15.31:0/1715408546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb80734e0 con 0x7f1fb80c62f0 2024-09-11T00:31:04.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.869+0000 7f3c5a000640 1 -- 172.21.15.31:0/2874359733 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3c4c004c40 con 0x7f3c640c4c00 2024-09-11T00:31:04.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.871+0000 7f1fbcc00640 1 -- 172.21.15.31:0/1715408546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f90005290 con 0x7f1fb80c62f0 2024-09-11T00:31:04.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.871+0000 7f4074e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40700acaf0 0x7f40701f09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.871+0000 7f4074e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40700c64b0 0x7f40700a9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.871+0000 7f4074e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40700c7240 0x7f4070151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.872+0000 7f4074e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4070155b90 con 0x7f40700acaf0 2024-09-11T00:31:04.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.872+0000 7f4074e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f40700700f0 con 0x7f40700c64b0 2024-09-11T00:31:04.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.872+0000 7f4074e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f40700c4ad0 con 0x7f40700c7240 2024-09-11T00:31:04.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f8207200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8200156850 con 0x7f82000c99e0 2024-09-11T00:31:04.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f81ffe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82000c84f0 0x7f82000c3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f81ffe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82000c84f0 0x7f82000c3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37600/0 (socket says 172.21.15.31:37600) 2024-09-11T00:31:04.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f81ffe00640 1 -- 172.21.15.31:0/752317933 learned_addr learned my addr 172.21.15.31:0/752317933 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:04.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.870+0000 7f8205200640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82000c9ef0 0x7f820014d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.872+0000 7f406ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40700c64b0 0x7f40700a9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.872+0000 7f406ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40700c64b0 0x7f40700a9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:37238/0 (socket says 172.21.15.31:37238) 2024-09-11T00:31:04.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.872+0000 7f406ea00640 1 -- 172.21.15.31:0/1448085234 learned_addr learned my addr 172.21.15.31:0/1448085234 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:04.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.872+0000 7f81ff400640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82000c99e0 0x7f820014d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.872+0000 7f406fe00640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40700c7240 0x7f4070151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.872+0000 7f406f400640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40700acaf0 0x7f40701f09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.873+0000 7f406fe00640 1 -- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40700c64b0 msgr2=0x7f40700a9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.873+0000 7f406fe00640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40700c64b0 0x7f40700a9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.873+0000 7f406fe00640 1 -- 172.21.15.31:0/1448085234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40700acaf0 msgr2=0x7f40701f09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f8205200640 1 -- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82000c99e0 msgr2=0x7f820014d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f8205200640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82000c99e0 0x7f820014d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f8205200640 1 -- 172.21.15.31:0/752317933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82000c84f0 msgr2=0x7f82000c3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:04.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f8205200640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82000c84f0 0x7f82000c3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f8205200640 1 -- 172.21.15.31:0/752317933 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8200157660 con 0x7f82000c9ef0 2024-09-11T00:31:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f81ffe00640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82000c84f0 0x7f82000c3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.873+0000 7f794f400640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7938077fa0 0x7f793807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f81ff400640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82000c99e0 0x7f820014d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.873+0000 7fdc6e000640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc58078040 0x7fdc5807a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.873+0000 7fdc6e000640 1 -- 172.21.15.31:0/3523163760 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fdc68099e80 con 0x7fdc780c5aa0 2024-09-11T00:31:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f794f400640 1 -- 172.21.15.31:0/2339200057 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f79480bab90 con 0x7f7958070570 2024-09-11T00:31:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f8205200640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82000c9ef0 0x7f820014d6e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f81f800c6b0 tx=0x7f81f800cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f7957400640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7938077fa0 0x7f793807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f3c5a000640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c40078040 0x7f3c4007a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f81f7e00640 1 -- 172.21.15.31:0/752317933 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f8003b70 con 0x7f82000c9ef0 2024-09-11T00:31:04.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.875+0000 7f3c5a000640 1 -- 172.21.15.31:0/2874359733 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f3c4c0bb5f0 con 0x7f3c640c4c00 2024-09-11T00:31:04.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f81f7e00640 1 -- 172.21.15.31:0/752317933 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81f8003d10 con 0x7f82000c9ef0 2024-09-11T00:31:04.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.875+0000 7f3c5a000640 1 -- 172.21.15.31:0/2874359733 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3c640c5830 con 0x7f3c640c4c00 2024-09-11T00:31:04.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f81f7e00640 1 -- 172.21.15.31:0/752317933 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f8005330 con 0x7f82000c9ef0 2024-09-11T00:31:04.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.874+0000 7f8207200640 1 -- 172.21.15.31:0/752317933 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8200159a80 con 0x7f82000c9ef0 2024-09-11T00:31:04.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.875+0000 7f8207200640 1 -- 172.21.15.31:0/752317933 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8200157350 con 0x7f82000c9ef0 2024-09-11T00:31:04.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.875+0000 7f406fe00640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40700acaf0 0x7f40701f09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:04.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.875+0000 7f406fe00640 1 -- 172.21.15.31:0/1448085234 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40700a9d30 con 0x7f40700c7240 2024-09-11T00:31:04.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.875+0000 7fdc75400640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc58078040 0x7fdc5807a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.875+0000 7f406f400640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40700acaf0 0x7f40701f09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:04.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.876+0000 7f3c61400640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c40078040 0x7f3c4007a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.877+0000 7f7957400640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7938077fa0 0x7f793807a460 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7950002230 tx=0x7f7950009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.877+0000 7f8207200640 1 -- 172.21.15.31:0/752317933 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d8005290 con 0x7f82000c9ef0 2024-09-11T00:31:04.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.877+0000 7f794f400640 1 -- 172.21.15.31:0/2339200057 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7958072280 con 0x7f7958070570 2024-09-11T00:31:04.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.875+0000 7f9b4e000640 1 -- 172.21.15.31:0/1143929635 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9b50004b20 con 0x7f9b580ab730 2024-09-11T00:31:04.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.877+0000 7f9b4e000640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b34077f70 0x7f9b3407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.877+0000 7f9b4e000640 1 -- 172.21.15.31:0/1143929635 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f9b5009aa60 con 0x7f9b580ab730 2024-09-11T00:31:04.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.877+0000 7f3c61400640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c40078040 0x7f3c4007a500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c64153a70 tx=0x7f3c5c0075f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.877+0000 7f406fe00640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40700c7240 0x7f4070151c60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f40700c6960 tx=0x7f406800a5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.878+0000 7fdc75400640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc58078040 0x7fdc5807a500 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdc64032000 tx=0x7fdc640046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.878+0000 7f9b55e00640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b34077f70 0x7f9b3407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.878+0000 7f4067400640 1 -- 172.21.15.31:0/1448085234 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068037040 con 0x7f40700c7240 2024-09-11T00:31:04.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.878+0000 7f4067400640 1 -- 172.21.15.31:0/1448085234 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f406800a9f0 con 0x7f40700c7240 2024-09-11T00:31:04.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.878+0000 7f4067400640 1 -- 172.21.15.31:0/1448085234 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068003bd0 con 0x7f40700c7240 2024-09-11T00:31:04.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.878+0000 7f4074e00640 1 -- 172.21.15.31:0/1448085234 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4070152af0 con 0x7f40700c7240 2024-09-11T00:31:04.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.878+0000 7f4074e00640 1 -- 172.21.15.31:0/1448085234 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4070163b70 con 0x7f40700c7240 2024-09-11T00:31:04.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.880+0000 7f9b55e00640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b34077f70 0x7f9b3407a430 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9b48034000 tx=0x7f9b48004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.880+0000 7f4074e00640 1 -- 172.21.15.31:0/1448085234 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4070148890 con 0x7f40700c7240 2024-09-11T00:31:04.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.880+0000 7f9b4e000640 1 -- 172.21.15.31:0/1143929635 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9b581f3dd0 con 0x7f9b580ab730 2024-09-11T00:31:04.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.881+0000 7f81f7e00640 1 -- 172.21.15.31:0/752317933 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f81f80054d0 con 0x7f82000c9ef0 2024-09-11T00:31:04.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.881+0000 7f4067400640 1 -- 172.21.15.31:0/1448085234 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4068004050 con 0x7f40700c7240 2024-09-11T00:31:04.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.882+0000 7f81f7e00640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f81e0077f70 0x7f81e007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.882+0000 7f81f7e00640 1 -- 172.21.15.31:0/752317933 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f81f8099ff0 con 0x7f82000c9ef0 2024-09-11T00:31:04.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.883+0000 7f81ffe00640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f81e0077f70 0x7f81e007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.883+0000 7f81f7e00640 1 -- 172.21.15.31:0/752317933 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f81f80057f0 con 0x7f82000c9ef0 2024-09-11T00:31:04.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.883+0000 7f4067400640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4050077f70 0x7f405007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.883+0000 7f81ffe00640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f81e0077f70 0x7f81e007a430 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f81f0000f60 tx=0x7f81f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.883+0000 7f4067400640 1 -- 172.21.15.31:0/1448085234 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f40680ba0f0 con 0x7f40700c7240 2024-09-11T00:31:04.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.885+0000 7f406f400640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4050077f70 0x7f405007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.887+0000 7fdc6e000640 1 -- 172.21.15.31:0/3523163760 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdc680626c0 con 0x7fdc780c5aa0 2024-09-11T00:31:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.890+0000 7f406f400640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4050077f70 0x7f405007a430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4060000f60 tx=0x7f4060009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.890+0000 7f4067400640 1 -- 172.21.15.31:0/1448085234 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4068082820 con 0x7f40700c7240 2024-09-11T00:31:04.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.894+0000 7f1faf400640 1 -- 172.21.15.31:0/1715408546 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1fa4004940 con 0x7f1fb80c62f0 2024-09-11T00:31:04.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.895+0000 7f1faf400640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1f98078070 0x7f1f9807a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:04.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.896+0000 7f1faf400640 1 -- 172.21.15.31:0/1715408546 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f1fa40bb680 con 0x7f1fb80c62f0 2024-09-11T00:31:04.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.897+0000 7f1fb7400640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1f98078070 0x7f1f9807a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:04.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.897+0000 7f1faf400640 1 -- 172.21.15.31:0/1715408546 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1fa4049060 con 0x7f1fb80c62f0 2024-09-11T00:31:04.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:04.897+0000 7f1fb7400640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1f98078070 0x7f1f9807a530 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80046b0 tx=0x7f1fa8007f50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:05.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.219+0000 7fac8a800640 1 -- 172.21.15.31:0/725990357 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fac84074e30 con 0x7fac84148ab0 2024-09-11T00:31:05.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.220+0000 7fac81600640 1 -- 172.21.15.31:0/725990357 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fac78062280 con 0x7fac84148ab0 2024-09-11T00:31:05.222 INFO:teuthology.orchestra.run.smithi031.stdout:64424509450 2024-09-11T00:31:05.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.224+0000 7fac8a800640 1 -- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fac640781a0 msgr2=0x7fac6407a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.224+0000 7fac8a800640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fac640781a0 0x7fac6407a660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fac84074790 tx=0x7fac7c0094d0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.224+0000 7fac8a800640 1 -- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac84148ab0 msgr2=0x7fac84150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.224+0000 7fac8a800640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac84148ab0 0x7fac84150760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fac7800c700 tx=0x7fac7800cbc0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.225+0000 7fac8a800640 1 -- 172.21.15.31:0/725990357 shutdown_connections 2024-09-11T00:31:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.225+0000 7fac8a800640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fac640781a0 0x7fac6407a660 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.225+0000 7fac8a800640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fac84148ab0 0x7fac84150760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.225+0000 7fac8a800640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac84073e00 0x7fac8414b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.225+0000 7fac8a800640 1 --2- 172.21.15.31:0/725990357 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac8406f7e0 0x7fac841482b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.225+0000 7fac8a800640 1 -- 172.21.15.31:0/725990357 >> 172.21.15.31:0/725990357 conn(0x7fac840c7ec0 msgr2=0x7fac840c82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:05.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.225+0000 7fac8a800640 1 -- 172.21.15.31:0/725990357 shutdown_connections 2024-09-11T00:31:05.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.226+0000 7fac8a800640 1 -- 172.21.15.31:0/725990357 wait complete. 2024-09-11T00:31:05.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509450 for osd.0 2024-09-11T00:31:05.254 DEBUG:teuthology.parallel:result is None 2024-09-11T00:31:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.286+0000 7f4074e00640 1 -- 172.21.15.31:0/1448085234 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4070152cc0 con 0x7f40700c7240 2024-09-11T00:31:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.287+0000 7f4067400640 1 -- 172.21.15.31:0/1448085234 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4070152cc0 con 0x7f40700c7240 2024-09-11T00:31:05.290 INFO:teuthology.orchestra.run.smithi031.stdout:64424509451 2024-09-11T00:31:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 -- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4050077f70 msgr2=0x7f405007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4050077f70 0x7f405007a430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4060000f60 tx=0x7f4060009040 comp rx=0 tx=0).stop 2024-09-11T00:31:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 -- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40700c7240 msgr2=0x7f4070151c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40700c7240 0x7f4070151c60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f40700c6960 tx=0x7f406800a5c0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 -- 172.21.15.31:0/1448085234 shutdown_connections 2024-09-11T00:31:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4050077f70 0x7f405007a430 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40700c7240 0x7f4070151c60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40700c64b0 0x7f40700a9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 --2- 172.21.15.31:0/1448085234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40700acaf0 0x7f40701f09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 -- 172.21.15.31:0/1448085234 >> 172.21.15.31:0/1448085234 conn(0x7f407006caf0 msgr2=0x7f4070154d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 -- 172.21.15.31:0/1448085234 shutdown_connections 2024-09-11T00:31:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.289+0000 7f4074e00640 1 -- 172.21.15.31:0/1448085234 wait complete. 2024-09-11T00:31:05.307 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509451 for osd.6 2024-09-11T00:31:05.307 DEBUG:teuthology.parallel:result is None 2024-09-11T00:31:05.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.332+0000 7f8207200640 1 -- 172.21.15.31:0/752317933 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f81d8005850 con 0x7f82000c9ef0 2024-09-11T00:31:05.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.332+0000 7f3c63200640 1 -- 172.21.15.31:0/2874359733 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f3c640acec0 con 0x7f3c640c4c00 2024-09-11T00:31:05.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.333+0000 7f81f7e00640 1 -- 172.21.15.31:0/752317933 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f81f80621e0 con 0x7f82000c9ef0 2024-09-11T00:31:05.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.333+0000 7f3c5a000640 1 -- 172.21.15.31:0/2874359733 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3c4c049030 con 0x7f3c640c4c00 2024-09-11T00:31:05.335 INFO:teuthology.orchestra.run.smithi031.stdout:64424509448 2024-09-11T00:31:05.335 INFO:teuthology.orchestra.run.smithi031.stdout:64424509450 2024-09-11T00:31:05.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.336+0000 7f3c63200640 1 -- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c40078040 msgr2=0x7f3c4007a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.336+0000 7f3c63200640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c40078040 0x7f3c4007a500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c64153a70 tx=0x7f3c5c0075f0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f3c63200640 1 -- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c640c4c00 msgr2=0x7f3c640743a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f3c63200640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c640c4c00 0x7f3c640743a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c006010 tx=0x7f3c4c0039e0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f3c63200640 1 -- 172.21.15.31:0/2874359733 shutdown_connections 2024-09-11T00:31:05.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f3c63200640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c40078040 0x7f3c4007a500 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f3c63200640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6406fba0 0x7f3c64152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f3c63200640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c640c62f0 0x7f3c640abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f3c63200640 1 --2- 172.21.15.31:0/2874359733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c640c4c00 0x7f3c640743a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f3c63200640 1 -- 172.21.15.31:0/2874359733 >> 172.21.15.31:0/2874359733 conn(0x7f3c6406caf0 msgr2=0x7f3c640a6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:05.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 -- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f81e0077f70 msgr2=0x7f81e007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f81e0077f70 0x7f81e007a430 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f81f0000f60 tx=0x7f81f0009040 comp rx=0 tx=0).stop 2024-09-11T00:31:05.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 -- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82000c9ef0 msgr2=0x7f820014d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82000c9ef0 0x7f820014d6e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f81f800c6b0 tx=0x7f81f800cb70 comp rx=0 tx=0).stop 2024-09-11T00:31:05.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f3c63200640 1 -- 172.21.15.31:0/2874359733 shutdown_connections 2024-09-11T00:31:05.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 -- 172.21.15.31:0/752317933 shutdown_connections 2024-09-11T00:31:05.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f3c63200640 1 -- 172.21.15.31:0/2874359733 wait complete. 2024-09-11T00:31:05.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f81e0077f70 0x7f81e007a430 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82000c9ef0 0x7f820014d6e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82000c99e0 0x7f820014d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 --2- 172.21.15.31:0/752317933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82000c84f0 0x7f82000c3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 -- 172.21.15.31:0/752317933 >> 172.21.15.31:0/752317933 conn(0x7f82000ad890 msgr2=0x7f82000ca530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:05.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 -- 172.21.15.31:0/752317933 shutdown_connections 2024-09-11T00:31:05.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.337+0000 7f8207200640 1 -- 172.21.15.31:0/752317933 wait complete. 2024-09-11T00:31:05.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.340+0000 7fdc77200640 1 -- 172.21.15.31:0/3523163760 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fdc78153310 con 0x7fdc780c5aa0 2024-09-11T00:31:05.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.340+0000 7f795e800640 1 -- 172.21.15.31:0/2339200057 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f79580c7690 con 0x7f7958070570 2024-09-11T00:31:05.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.340+0000 7fdc6e000640 1 -- 172.21.15.31:0/3523163760 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdc68061fc0 con 0x7fdc780c5aa0 2024-09-11T00:31:05.343 INFO:teuthology.orchestra.run.smithi031.stdout:64424509449 2024-09-11T00:31:05.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.341+0000 7f9b5d000640 1 -- 172.21.15.31:0/1143929635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9b580c31e0 con 0x7f9b580ab730 2024-09-11T00:31:05.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.341+0000 7f794f400640 1 -- 172.21.15.31:0/2339200057 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7948082d00 con 0x7f7958070570 2024-09-11T00:31:05.344 INFO:teuthology.orchestra.run.smithi031.stdout:64424509450 2024-09-11T00:31:05.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.341+0000 7f9b4e000640 1 -- 172.21.15.31:0/1143929635 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9b50063210 con 0x7f9b580ab730 2024-09-11T00:31:05.344 INFO:teuthology.orchestra.run.smithi031.stdout:60129542154 2024-09-11T00:31:05.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.344+0000 7f1fbcc00640 1 -- 172.21.15.31:0/1715408546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1f90005850 con 0x7f1fb80c62f0 2024-09-11T00:31:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.344+0000 7f795e800640 1 -- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7938077fa0 msgr2=0x7f793807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.344+0000 7f795e800640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7938077fa0 0x7f793807a460 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7950002230 tx=0x7f7950009040 comp rx=0 tx=0).stop 2024-09-11T00:31:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.344+0000 7f795e800640 1 -- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7958070570 msgr2=0x7f7958141740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.344+0000 7f795e800640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7958070570 0x7f7958141740 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7948031000 tx=0x7f7948003820 comp rx=0 tx=0).stop 2024-09-11T00:31:05.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 -- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc58078040 msgr2=0x7fdc5807a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc58078040 0x7fdc5807a500 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdc64032000 tx=0x7fdc640046c0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 -- 172.21.15.31:0/3523163760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc780c5aa0 msgr2=0x7fdc78155540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc780c5aa0 0x7fdc78155540 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800c430 tx=0x7fdc6800c8f0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f795e800640 1 -- 172.21.15.31:0/2339200057 shutdown_connections 2024-09-11T00:31:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f795e800640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7938077fa0 0x7f793807a460 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f795e800640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580c2ff0 0x7f7958141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f795e800640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7958070570 0x7f7958141740 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f795e800640 1 --2- 172.21.15.31:0/2339200057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f795806e9c0 0x7f795815fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f795e800640 1 -- 172.21.15.31:0/2339200057 >> 172.21.15.31:0/2339200057 conn(0x7f79580a9750 msgr2=0x7f79580c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 -- 172.21.15.31:0/3523163760 shutdown_connections 2024-09-11T00:31:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc58078040 0x7fdc5807a500 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc780abaf0 0x7fdc781fc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc78153850 0x7fdc781fc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 --2- 172.21.15.31:0/3523163760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc780c5aa0 0x7fdc78155540 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 -- 172.21.15.31:0/3523163760 >> 172.21.15.31:0/3523163760 conn(0x7fdc7806caf0 msgr2=0x7fdc780a8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f795e800640 1 -- 172.21.15.31:0/2339200057 shutdown_connections 2024-09-11T00:31:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 -- 172.21.15.31:0/3523163760 shutdown_connections 2024-09-11T00:31:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f795e800640 1 -- 172.21.15.31:0/2339200057 wait complete. 2024-09-11T00:31:05.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7fdc77200640 1 -- 172.21.15.31:0/3523163760 wait complete. 2024-09-11T00:31:05.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f9b5d000640 1 -- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b34077f70 msgr2=0x7f9b3407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f9b5d000640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b34077f70 0x7f9b3407a430 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9b48034000 tx=0x7f9b48004730 comp rx=0 tx=0).stop 2024-09-11T00:31:05.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.345+0000 7f1faf400640 1 -- 172.21.15.31:0/1715408546 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1fa4083e30 con 0x7f1fb80c62f0 2024-09-11T00:31:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.346+0000 7f9b5d000640 1 -- 172.21.15.31:0/1143929635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b580ab730 msgr2=0x7f9b5814b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.346+0000 7f9b5d000640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b580ab730 0x7f9b5814b950 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9b50009fd0 tx=0x7f9b5000b260 comp rx=0 tx=0).stop 2024-09-11T00:31:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.346+0000 7f9b5d000640 1 -- 172.21.15.31:0/1143929635 shutdown_connections 2024-09-11T00:31:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.346+0000 7f9b5d000640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b34077f70 0x7f9b3407a430 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.346+0000 7f9b5d000640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b580ab730 0x7f9b5814b950 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.346+0000 7f9b5d000640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b580c7460 0x7f9b580c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.351 INFO:teuthology.orchestra.run.smithi031.stdout:64424509450 2024-09-11T00:31:05.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.346+0000 7f9b5d000640 1 --2- 172.21.15.31:0/1143929635 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b580c2d40 0x7f9b581f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.346+0000 7f9b5d000640 1 -- 172.21.15.31:0/1143929635 >> 172.21.15.31:0/1143929635 conn(0x7f9b580a9750 msgr2=0x7f9b58152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:05.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.346+0000 7f9b5d000640 1 -- 172.21.15.31:0/1143929635 shutdown_connections 2024-09-11T00:31:05.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.346+0000 7f9b5d000640 1 -- 172.21.15.31:0/1143929635 wait complete. 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 -- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1f98078070 msgr2=0x7f1f9807a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1f98078070 0x7f1f9807a530 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80046b0 tx=0x7f1fa8007f50 comp rx=0 tx=0).stop 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 -- 172.21.15.31:0/1715408546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb80c62f0 msgr2=0x7f1fb80a7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb80c62f0 0x7f1fb80a7960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4031000 tx=0x7f1fa4004550 comp rx=0 tx=0).stop 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 -- 172.21.15.31:0/1715408546 shutdown_connections 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1f98078070 0x7f1f9807a530 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fb8159ee0 0x7f1fb80aba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fb80c62f0 0x7f1fb80a7960 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 --2- 172.21.15.31:0/1715408546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fb80c4c00 0x7f1fb80a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 -- 172.21.15.31:0/1715408546 >> 172.21.15.31:0/1715408546 conn(0x7f1fb806caf0 msgr2=0x7f1fb806e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 -- 172.21.15.31:0/1715408546 shutdown_connections 2024-09-11T00:31:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.352+0000 7f1fbcc00640 1 -- 172.21.15.31:0/1715408546 wait complete. 2024-09-11T00:31:05.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509448 for osd.3 2024-09-11T00:31:05.363 DEBUG:teuthology.parallel:result is None 2024-09-11T00:31:05.368 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509450 for osd.2 2024-09-11T00:31:05.368 DEBUG:teuthology.parallel:result is None 2024-09-11T00:31:05.373 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509450 for osd.5 2024-09-11T00:31:05.373 DEBUG:teuthology.parallel:result is None 2024-09-11T00:31:05.374 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509449 for osd.7 2024-09-11T00:31:05.374 DEBUG:teuthology.parallel:result is None 2024-09-11T00:31:05.376 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509450 for osd.1 2024-09-11T00:31:05.377 DEBUG:teuthology.parallel:result is None 2024-09-11T00:31:05.377 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542154 for osd.4 2024-09-11T00:31:05.377 DEBUG:teuthology.parallel:result is None 2024-09-11T00:31:05.378 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T00:31:05.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:31:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.575+0000 7fbaae800640 1 -- 172.21.15.31:0/1323590430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbaa80ada20 msgr2=0x7fbaa806e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.575+0000 7fbaae800640 1 --2- 172.21.15.31:0/1323590430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbaa80ada20 0x7fbaa806e140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fba94005fd0 tx=0x7fba9402a5d0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.575+0000 7fbaae800640 1 -- 172.21.15.31:0/1323590430 shutdown_connections 2024-09-11T00:31:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.575+0000 7fbaae800640 1 --2- 172.21.15.31:0/1323590430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbaa80ada20 0x7fbaa806e140 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.575+0000 7fbaae800640 1 --2- 172.21.15.31:0/1323590430 >> v2:172.21.15.204:3300/0 conn(0x7fbaa8150020 0x7fbaa80ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.575+0000 7fbaae800640 1 -- 172.21.15.31:0/1323590430 >> 172.21.15.31:0/1323590430 conn(0x7fbaa8068f10 msgr2=0x7fbaa80cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.575+0000 7fbaae800640 1 -- 172.21.15.31:0/1323590430 shutdown_connections 2024-09-11T00:31:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.575+0000 7fbaae800640 1 -- 172.21.15.31:0/1323590430 wait complete. 2024-09-11T00:31:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.576+0000 7fbaae800640 1 Processor -- start 2024-09-11T00:31:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.577+0000 7fbaae800640 1 -- start start 2024-09-11T00:31:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.577+0000 7fbaae800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbaa80ada20 0x7fbaa81ef270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaae800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaa8150020 0x7fbaa8150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaae800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbaa8156960 0x7fbaa814dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaae800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa8157550 con 0x7fbaa8150020 2024-09-11T00:31:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaae800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa81573d0 con 0x7fbaa80ada20 2024-09-11T00:31:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaae800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa80cc220 con 0x7fbaa8156960 2024-09-11T00:31:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaaca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaa8150020 0x7fbaa8150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaaca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaa8150020 0x7fbaa8150630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37616/0 (socket says 172.21.15.31:37616) 2024-09-11T00:31:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaaca00640 1 -- 172.21.15.31:0/4252732179 learned_addr learned my addr 172.21.15.31:0/4252732179 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaad400640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbaa80ada20 0x7fbaa81ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaade00640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbaa8156960 0x7fbaa814dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaaca00640 1 -- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbaa8156960 msgr2=0x7fbaa814dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaaca00640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbaa8156960 0x7fbaa814dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaaca00640 1 -- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbaa80ada20 msgr2=0x7fbaa81ef270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaaca00640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbaa80ada20 0x7fbaa81ef270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaaca00640 1 -- 172.21.15.31:0/4252732179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa814e980 con 0x7fbaa8150020 2024-09-11T00:31:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.578+0000 7fbaad400640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbaa80ada20 0x7fbaa81ef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.579+0000 7fbaaca00640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaa8150020 0x7fbaa8150630 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fba94009850 tx=0x7fba940035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.579+0000 7fbaa5600640 1 -- 172.21.15.31:0/4252732179 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9403f070 con 0x7fbaa8150020 2024-09-11T00:31:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.579+0000 7fbaa5600640 1 -- 172.21.15.31:0/4252732179 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba94003ce0 con 0x7fbaa8150020 2024-09-11T00:31:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.579+0000 7fbaae800640 1 -- 172.21.15.31:0/4252732179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa8152ec0 con 0x7fbaa8150020 2024-09-11T00:31:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.579+0000 7fbaa5600640 1 -- 172.21.15.31:0/4252732179 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba94035030 con 0x7fbaa8150020 2024-09-11T00:31:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.579+0000 7fbaae800640 1 -- 172.21.15.31:0/4252732179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa814f080 con 0x7fbaa8150020 2024-09-11T00:31:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.580+0000 7fbaae800640 1 -- 172.21.15.31:0/4252732179 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa80ad4e0 con 0x7fbaa8150020 2024-09-11T00:31:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.581+0000 7fbaa5600640 1 -- 172.21.15.31:0/4252732179 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fba94004a00 con 0x7fbaa8150020 2024-09-11T00:31:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.582+0000 7fbaa5600640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba88077f70 0x7fba8807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.582+0000 7fbaa5600640 1 -- 172.21.15.31:0/4252732179 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fba940bafe0 con 0x7fbaa8150020 2024-09-11T00:31:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.583+0000 7fbaad400640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba88077f70 0x7fba8807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:05.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.584+0000 7fbaad400640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba88077f70 0x7fba8807a430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbaa80c4000 tx=0x7fba9c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.586+0000 7fbaa5600640 1 -- 172.21.15.31:0/4252732179 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fba94083750 con 0x7fbaa8150020 2024-09-11T00:31:05.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.777+0000 7fbaae800640 1 -- 172.21.15.31:0/4252732179 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbaa81ef9f0 con 0x7fba88077f70 2024-09-11T00:31:05.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.786+0000 7fbaa5600640 1 -- 172.21.15.31:0/4252732179 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490296 (secure 0 0 0) 0x7fbaa81ef9f0 con 0x7fba88077f70 2024-09-11T00:31:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:05.796 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:31:05.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 -- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba88077f70 msgr2=0x7fba8807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba88077f70 0x7fba8807a430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbaa80c4000 tx=0x7fba9c005f90 comp rx=0 tx=0).stop 2024-09-11T00:31:05.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 -- 172.21.15.31:0/4252732179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaa8150020 msgr2=0x7fbaa8150630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:05.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaa8150020 0x7fbaa8150630 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fba94009850 tx=0x7fba940035f0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 -- 172.21.15.31:0/4252732179 shutdown_connections 2024-09-11T00:31:05.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba88077f70 0x7fba8807a430 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbaa8156960 0x7fbaa814dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaa8150020 0x7fbaa8150630 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 --2- 172.21.15.31:0/4252732179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbaa80ada20 0x7fbaa81ef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:05.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 -- 172.21.15.31:0/4252732179 >> 172.21.15.31:0/4252732179 conn(0x7fbaa8068f10 msgr2=0x7fbaa80cbaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:05.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 -- 172.21.15.31:0/4252732179 shutdown_connections 2024-09-11T00:31:05.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:05.791+0000 7fbaae800640 1 -- 172.21.15.31:0/4252732179 wait complete. 2024-09-11T00:31:05.808 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-11T00:31:04.396383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":191,"num_write_kb":1656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219644,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755076,"statfs":{"total":773094113280,"available":772869197824,"internally_reserved":0,"allocated":3481600,"data_stored":2027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.432387"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719046+0000","last_change":"2024-09-11T00:30:54.983412+0000","last_active":"2024-09-11T00:30:56.719046+0000","last_peered":"2024-09-11T00:30:56.719046+0000","last_clean":"2024-09-11T00:30:56.719046+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:30:56.719046+0000","last_undegraded":"2024-09-11T00:30:56.719046+0000","last_fullsized":"2024-09-11T00:30:56.719046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:51.807846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970097+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:30:55.970097+0000","last_peered":"2024-09-11T00:30:55.970097+0000","last_clean":"2024-09-11T00:30:55.970097+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:30:55.970097+0000","last_undegraded":"2024-09-11T00:30:55.970097+0000","last_fullsized":"2024-09-11T00:30:55.970097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998047+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:30:55.998047+0000","last_peered":"2024-09-11T00:30:55.998047+0000","last_clean":"2024-09-11T00:30:55.998047+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:30:55.998047+0000","last_undegraded":"2024-09-11T00:30:55.998047+0000","last_fullsized":"2024-09-11T00:30:55.998047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000942+0000","last_change":"2024-09-11T00:30:53.984316+0000","last_active":"2024-09-11T00:30:56.000942+0000","last_peered":"2024-09-11T00:30:56.000942+0000","last_clean":"2024-09-11T00:30:56.000942+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:30:56.000942+0000","last_undegraded":"2024-09-11T00:30:56.000942+0000","last_fullsized":"2024-09-11T00:30:56.000942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:14.425265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968662+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:30:55.968662+0000","last_peered":"2024-09-11T00:30:55.968662+0000","last_clean":"2024-09-11T00:30:55.968662+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:30:55.968662+0000","last_undegraded":"2024-09-11T00:30:55.968662+0000","last_fullsized":"2024-09-11T00:30:55.968662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998255+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:30:55.998255+0000","last_peered":"2024-09-11T00:30:55.998255+0000","last_clean":"2024-09-11T00:30:55.998255+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:30:55.998255+0000","last_undegraded":"2024-09-11T00:30:55.998255+0000","last_fullsized":"2024-09-11T00:30:55.998255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998074+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:30:55.998074+0000","last_peered":"2024-09-11T00:30:55.998074+0000","last_clean":"2024-09-11T00:30:55.998074+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:30:55.998074+0000","last_undegraded":"2024-09-11T00:30:55.998074+0000","last_fullsized":"2024-09-11T00:30:55.998074+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172549+0000","last_change":"2024-09-11T00:30:53.984940+0000","last_active":"2024-09-11T00:30:56.172549+0000","last_peered":"2024-09-11T00:30:56.172549+0000","last_clean":"2024-09-11T00:30:56.172549+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:30:56.172549+0000","last_undegraded":"2024-09-11T00:30:56.172549+0000","last_fullsized":"2024-09-11T00:30:56.172549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:19.647165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719205+0000","last_change":"2024-09-11T00:30:54.993570+0000","last_active":"2024-09-11T00:30:56.719205+0000","last_peered":"2024-09-11T00:30:56.719205+0000","last_clean":"2024-09-11T00:30:56.719205+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:30:56.719205+0000","last_undegraded":"2024-09-11T00:30:56.719205+0000","last_fullsized":"2024-09-11T00:30:56.719205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:45.211259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333273+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:30:56.333273+0000","last_peered":"2024-09-11T00:30:56.333273+0000","last_clean":"2024-09-11T00:30:56.333273+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:30:56.333273+0000","last_undegraded":"2024-09-11T00:30:56.333273+0000","last_fullsized":"2024-09-11T00:30:56.333273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719392+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:30:56.719392+0000","last_peered":"2024-09-11T00:30:56.719392+0000","last_clean":"2024-09-11T00:30:56.719392+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:30:56.719392+0000","last_undegraded":"2024-09-11T00:30:56.719392+0000","last_fullsized":"2024-09-11T00:30:56.719392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172696+0000","last_change":"2024-09-11T00:30:53.974699+0000","last_active":"2024-09-11T00:30:56.172696+0000","last_peered":"2024-09-11T00:30:56.172696+0000","last_clean":"2024-09-11T00:30:56.172696+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:30:56.172696+0000","last_undegraded":"2024-09-11T00:30:56.172696+0000","last_fullsized":"2024-09-11T00:30:56.172696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:28.792487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172498+0000","last_change":"2024-09-11T00:30:54.984890+0000","last_active":"2024-09-11T00:30:56.172498+0000","last_peered":"2024-09-11T00:30:56.172498+0000","last_clean":"2024-09-11T00:30:56.172498+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:30:56.172498+0000","last_undegraded":"2024-09-11T00:30:56.172498+0000","last_fullsized":"2024-09-11T00:30:56.172498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:21.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333166+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:30:56.333166+0000","last_peered":"2024-09-11T00:30:56.333166+0000","last_clean":"2024-09-11T00:30:56.333166+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:30:56.333166+0000","last_undegraded":"2024-09-11T00:30:56.333166+0000","last_fullsized":"2024-09-11T00:30:56.333166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968144+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:30:55.968144+0000","last_peered":"2024-09-11T00:30:55.968144+0000","last_clean":"2024-09-11T00:30:55.968144+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:30:55.968144+0000","last_undegraded":"2024-09-11T00:30:55.968144+0000","last_fullsized":"2024-09-11T00:30:55.968144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172339+0000","last_change":"2024-09-11T00:30:53.977743+0000","last_active":"2024-09-11T00:30:56.172339+0000","last_peered":"2024-09-11T00:30:56.172339+0000","last_clean":"2024-09-11T00:30:56.172339+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:30:56.172339+0000","last_undegraded":"2024-09-11T00:30:56.172339+0000","last_fullsized":"2024-09-11T00:30:56.172339+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:20.618615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997709+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:30:55.997709+0000","last_peered":"2024-09-11T00:30:55.997709+0000","last_clean":"2024-09-11T00:30:55.997709+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:30:55.997709+0000","last_undegraded":"2024-09-11T00:30:55.997709+0000","last_fullsized":"2024-09-11T00:30:55.997709+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967362+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:30:55.967362+0000","last_peered":"2024-09-11T00:30:55.967362+0000","last_clean":"2024-09-11T00:30:55.967362+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:30:55.967362+0000","last_undegraded":"2024-09-11T00:30:55.967362+0000","last_fullsized":"2024-09-11T00:30:55.967362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967452+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:30:55.967452+0000","last_peered":"2024-09-11T00:30:55.967452+0000","last_clean":"2024-09-11T00:30:55.967452+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:30:55.967452+0000","last_undegraded":"2024-09-11T00:30:55.967452+0000","last_fullsized":"2024-09-11T00:30:55.967452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967914+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:30:55.967914+0000","last_peered":"2024-09-11T00:30:55.967914+0000","last_clean":"2024-09-11T00:30:55.967914+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:30:55.967914+0000","last_undegraded":"2024-09-11T00:30:55.967914+0000","last_fullsized":"2024-09-11T00:30:55.967914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001419+0000","last_change":"2024-09-11T00:30:54.997397+0000","last_active":"2024-09-11T00:30:56.001419+0000","last_peered":"2024-09-11T00:30:56.001419+0000","last_clean":"2024-09-11T00:30:56.001419+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:30:56.001419+0000","last_undegraded":"2024-09-11T00:30:56.001419+0000","last_fullsized":"2024-09-11T00:30:56.001419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:53.322576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001116+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:30:56.001116+0000","last_peered":"2024-09-11T00:30:56.001116+0000","last_clean":"2024-09-11T00:30:56.001116+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:30:56.001116+0000","last_undegraded":"2024-09-11T00:30:56.001116+0000","last_fullsized":"2024-09-11T00:30:56.001116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970056+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:30:55.970056+0000","last_peered":"2024-09-11T00:30:55.970056+0000","last_clean":"2024-09-11T00:30:55.970056+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:30:55.970056+0000","last_undegraded":"2024-09-11T00:30:55.970056+0000","last_fullsized":"2024-09-11T00:30:55.970056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332817+0000","last_change":"2024-09-11T00:30:53.986858+0000","last_active":"2024-09-11T00:30:56.332817+0000","last_peered":"2024-09-11T00:30:56.332817+0000","last_clean":"2024-09-11T00:30:56.332817+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:30:56.332817+0000","last_undegraded":"2024-09-11T00:30:56.332817+0000","last_fullsized":"2024-09-11T00:30:56.332817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:33.626193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967911+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:30:55.967911+0000","last_peered":"2024-09-11T00:30:55.967911+0000","last_clean":"2024-09-11T00:30:55.967911+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:30:55.967911+0000","last_undegraded":"2024-09-11T00:30:55.967911+0000","last_fullsized":"2024-09-11T00:30:55.967911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968190+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:30:55.968190+0000","last_peered":"2024-09-11T00:30:55.968190+0000","last_clean":"2024-09-11T00:30:55.968190+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:30:55.968190+0000","last_undegraded":"2024-09-11T00:30:55.968190+0000","last_fullsized":"2024-09-11T00:30:55.968190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968850+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:30:55.968850+0000","last_peered":"2024-09-11T00:30:55.968850+0000","last_clean":"2024-09-11T00:30:55.968850+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:30:55.968850+0000","last_undegraded":"2024-09-11T00:30:55.968850+0000","last_fullsized":"2024-09-11T00:30:55.968850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001447+0000","last_change":"2024-09-11T00:30:53.990765+0000","last_active":"2024-09-11T00:30:56.001447+0000","last_peered":"2024-09-11T00:30:56.001447+0000","last_clean":"2024-09-11T00:30:56.001447+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:30:56.001447+0000","last_undegraded":"2024-09-11T00:30:56.001447+0000","last_fullsized":"2024-09-11T00:30:56.001447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:33.720727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719722+0000","last_change":"2024-09-11T00:30:54.984949+0000","last_active":"2024-09-11T00:30:56.719722+0000","last_peered":"2024-09-11T00:30:56.719722+0000","last_clean":"2024-09-11T00:30:56.719722+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:30:56.719722+0000","last_undegraded":"2024-09-11T00:30:56.719722+0000","last_fullsized":"2024-09-11T00:30:56.719722+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:54.346937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967033+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:30:55.967033+0000","last_peered":"2024-09-11T00:30:55.967033+0000","last_clean":"2024-09-11T00:30:55.967033+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:30:55.967033+0000","last_undegraded":"2024-09-11T00:30:55.967033+0000","last_fullsized":"2024-09-11T00:30:55.967033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332357+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:30:56.332357+0000","last_peered":"2024-09-11T00:30:56.332357+0000","last_clean":"2024-09-11T00:30:56.332357+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:30:56.332357+0000","last_undegraded":"2024-09-11T00:30:56.332357+0000","last_fullsized":"2024-09-11T00:30:56.332357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172821+0000","last_change":"2024-09-11T00:30:53.986940+0000","last_active":"2024-09-11T00:30:56.172821+0000","last_peered":"2024-09-11T00:30:56.172821+0000","last_clean":"2024-09-11T00:30:56.172821+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:30:56.172821+0000","last_undegraded":"2024-09-11T00:30:56.172821+0000","last_fullsized":"2024-09-11T00:30:56.172821+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:11.942187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968462+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:30:55.968462+0000","last_peered":"2024-09-11T00:30:55.968462+0000","last_clean":"2024-09-11T00:30:55.968462+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:30:55.968462+0000","last_undegraded":"2024-09-11T00:30:55.968462+0000","last_fullsized":"2024-09-11T00:30:55.968462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968110+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:30:55.968110+0000","last_peered":"2024-09-11T00:30:55.968110+0000","last_clean":"2024-09-11T00:30:55.968110+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:30:55.968110+0000","last_undegraded":"2024-09-11T00:30:55.968110+0000","last_fullsized":"2024-09-11T00:30:55.968110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172483+0000","last_change":"2024-09-11T00:30:53.988539+0000","last_active":"2024-09-11T00:30:56.172483+0000","last_peered":"2024-09-11T00:30:56.172483+0000","last_clean":"2024-09-11T00:30:56.172483+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:30:56.172483+0000","last_undegraded":"2024-09-11T00:30:56.172483+0000","last_fullsized":"2024-09-11T00:30:56.172483+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:19.083664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966824+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:30:55.966824+0000","last_peered":"2024-09-11T00:30:55.966824+0000","last_clean":"2024-09-11T00:30:55.966824+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:30:55.966824+0000","last_undegraded":"2024-09-11T00:30:55.966824+0000","last_fullsized":"2024-09-11T00:30:55.966824+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970081+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:30:55.970081+0000","last_peered":"2024-09-11T00:30:55.970081+0000","last_clean":"2024-09-11T00:30:55.970081+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:30:55.970081+0000","last_undegraded":"2024-09-11T00:30:55.970081+0000","last_fullsized":"2024-09-11T00:30:55.970081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970376+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:30:55.970376+0000","last_peered":"2024-09-11T00:30:55.970376+0000","last_clean":"2024-09-11T00:30:55.970376+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:30:55.970376+0000","last_undegraded":"2024-09-11T00:30:55.970376+0000","last_fullsized":"2024-09-11T00:30:55.970376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968131+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:30:55.968131+0000","last_peered":"2024-09-11T00:30:55.968131+0000","last_clean":"2024-09-11T00:30:55.968131+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:30:55.968131+0000","last_undegraded":"2024-09-11T00:30:55.968131+0000","last_fullsized":"2024-09-11T00:30:55.968131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332511+0000","last_change":"2024-09-11T00:30:54.993118+0000","last_active":"2024-09-11T00:30:56.332511+0000","last_peered":"2024-09-11T00:30:56.332511+0000","last_clean":"2024-09-11T00:30:56.332511+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:30:56.332511+0000","last_undegraded":"2024-09-11T00:30:56.332511+0000","last_fullsized":"2024-09-11T00:30:56.332511+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:07:41.148370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971378+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:30:55.971378+0000","last_peered":"2024-09-11T00:30:55.971378+0000","last_clean":"2024-09-11T00:30:55.971378+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:30:55.971378+0000","last_undegraded":"2024-09-11T00:30:55.971378+0000","last_fullsized":"2024-09-11T00:30:55.971378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968486+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:30:55.968486+0000","last_peered":"2024-09-11T00:30:55.968486+0000","last_clean":"2024-09-11T00:30:55.968486+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:30:55.968486+0000","last_undegraded":"2024-09-11T00:30:55.968486+0000","last_fullsized":"2024-09-11T00:30:55.968486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970903+0000","last_change":"2024-09-11T00:30:53.986843+0000","last_active":"2024-09-11T00:30:55.970903+0000","last_peered":"2024-09-11T00:30:55.970903+0000","last_clean":"2024-09-11T00:30:55.970903+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:30:55.970903+0000","last_undegraded":"2024-09-11T00:30:55.970903+0000","last_fullsized":"2024-09-11T00:30:55.970903+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:44.684154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969549+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:30:55.969549+0000","last_peered":"2024-09-11T00:30:55.969549+0000","last_clean":"2024-09-11T00:30:55.969549+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:30:55.969549+0000","last_undegraded":"2024-09-11T00:30:55.969549+0000","last_fullsized":"2024-09-11T00:30:55.969549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172919+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:30:56.172919+0000","last_peered":"2024-09-11T00:30:56.172919+0000","last_clean":"2024-09-11T00:30:56.172919+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:30:56.172919+0000","last_undegraded":"2024-09-11T00:30:56.172919+0000","last_fullsized":"2024-09-11T00:30:56.172919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997652+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:30:55.997652+0000","last_peered":"2024-09-11T00:30:55.997652+0000","last_clean":"2024-09-11T00:30:55.997652+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:30:55.997652+0000","last_undegraded":"2024-09-11T00:30:55.997652+0000","last_fullsized":"2024-09-11T00:30:55.997652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969390+0000","last_change":"2024-09-11T00:30:53.986896+0000","last_active":"2024-09-11T00:30:55.969390+0000","last_peered":"2024-09-11T00:30:55.969390+0000","last_clean":"2024-09-11T00:30:55.969390+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:30:55.969390+0000","last_undegraded":"2024-09-11T00:30:55.969390+0000","last_fullsized":"2024-09-11T00:30:55.969390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:06.590278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001631+0000","last_change":"2024-09-11T00:30:55.001370+0000","last_active":"2024-09-11T00:30:56.001631+0000","last_peered":"2024-09-11T00:30:56.001631+0000","last_clean":"2024-09-11T00:30:56.001631+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:30:56.001631+0000","last_undegraded":"2024-09-11T00:30:56.001631+0000","last_fullsized":"2024-09-11T00:30:56.001631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:56.592767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333262+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:30:56.333262+0000","last_peered":"2024-09-11T00:30:56.333262+0000","last_clean":"2024-09-11T00:30:56.333262+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:30:56.333262+0000","last_undegraded":"2024-09-11T00:30:56.333262+0000","last_fullsized":"2024-09-11T00:30:56.333262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967594+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:30:55.967594+0000","last_peered":"2024-09-11T00:30:55.967594+0000","last_clean":"2024-09-11T00:30:55.967594+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:30:55.967594+0000","last_undegraded":"2024-09-11T00:30:55.967594+0000","last_fullsized":"2024-09-11T00:30:55.967594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971192+0000","last_change":"2024-09-11T00:30:53.978480+0000","last_active":"2024-09-11T00:30:55.971192+0000","last_peered":"2024-09-11T00:30:55.971192+0000","last_clean":"2024-09-11T00:30:55.971192+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:30:55.971192+0000","last_undegraded":"2024-09-11T00:30:55.971192+0000","last_fullsized":"2024-09-11T00:30:55.971192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:58.900565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001528+0000","last_change":"2024-09-11T00:30:54.997641+0000","last_active":"2024-09-11T00:30:56.001528+0000","last_peered":"2024-09-11T00:30:56.001528+0000","last_clean":"2024-09-11T00:30:56.001528+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:30:56.001528+0000","last_undegraded":"2024-09-11T00:30:56.001528+0000","last_fullsized":"2024-09-11T00:30:56.001528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:18.437490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997311+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:30:55.997311+0000","last_peered":"2024-09-11T00:30:55.997311+0000","last_clean":"2024-09-11T00:30:55.997311+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:30:55.997311+0000","last_undegraded":"2024-09-11T00:30:55.997311+0000","last_fullsized":"2024-09-11T00:30:55.997311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971163+0000","last_change":"2024-09-11T00:30:53.979031+0000","last_active":"2024-09-11T00:30:55.971163+0000","last_peered":"2024-09-11T00:30:55.971163+0000","last_clean":"2024-09-11T00:30:55.971163+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:30:55.971163+0000","last_undegraded":"2024-09-11T00:30:55.971163+0000","last_fullsized":"2024-09-11T00:30:55.971163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:36.604377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967699+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:30:55.967699+0000","last_peered":"2024-09-11T00:30:55.967699+0000","last_clean":"2024-09-11T00:30:55.967699+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:30:55.967699+0000","last_undegraded":"2024-09-11T00:30:55.967699+0000","last_fullsized":"2024-09-11T00:30:55.967699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:39.629122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968089+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:30:55.968089+0000","last_peered":"2024-09-11T00:30:55.968089+0000","last_clean":"2024-09-11T00:30:55.968089+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:30:55.968089+0000","last_undegraded":"2024-09-11T00:30:55.968089+0000","last_fullsized":"2024-09-11T00:30:55.968089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966595+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:30:55.966595+0000","last_peered":"2024-09-11T00:30:55.966595+0000","last_clean":"2024-09-11T00:30:55.966595+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:30:55.966595+0000","last_undegraded":"2024-09-11T00:30:55.966595+0000","last_fullsized":"2024-09-11T00:30:55.966595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"26'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998301+0000","last_change":"2024-09-11T00:30:53.989299+0000","last_active":"2024-09-11T00:30:55.998301+0000","last_peered":"2024-09-11T00:30:55.998301+0000","last_clean":"2024-09-11T00:30:55.998301+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:30:55.998301+0000","last_undegraded":"2024-09-11T00:30:55.998301+0000","last_fullsized":"2024-09-11T00:30:55.998301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:34:49.298915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332781+0000","last_change":"2024-09-11T00:30:54.990026+0000","last_active":"2024-09-11T00:30:56.332781+0000","last_peered":"2024-09-11T00:30:56.332781+0000","last_clean":"2024-09-11T00:30:56.332781+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:30:56.332781+0000","last_undegraded":"2024-09-11T00:30:56.332781+0000","last_fullsized":"2024-09-11T00:30:56.332781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:02:10.800530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969545+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:30:55.969545+0000","last_peered":"2024-09-11T00:30:55.969545+0000","last_clean":"2024-09-11T00:30:55.969545+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:30:55.969545+0000","last_undegraded":"2024-09-11T00:30:55.969545+0000","last_fullsized":"2024-09-11T00:30:55.969545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967491+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:30:55.967491+0000","last_peered":"2024-09-11T00:30:55.967491+0000","last_clean":"2024-09-11T00:30:55.967491+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:30:55.967491+0000","last_undegraded":"2024-09-11T00:30:55.967491+0000","last_fullsized":"2024-09-11T00:30:55.967491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719269+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:30:56.719269+0000","last_peered":"2024-09-11T00:30:56.719269+0000","last_clean":"2024-09-11T00:30:56.719269+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:30:56.719269+0000","last_undegraded":"2024-09-11T00:30:56.719269+0000","last_fullsized":"2024-09-11T00:30:56.719269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968267+0000","last_change":"2024-09-11T00:30:53.982142+0000","last_active":"2024-09-11T00:30:55.968267+0000","last_peered":"2024-09-11T00:30:55.968267+0000","last_clean":"2024-09-11T00:30:55.968267+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:30:55.968267+0000","last_undegraded":"2024-09-11T00:30:55.968267+0000","last_fullsized":"2024-09-11T00:30:55.968267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:41.558944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998013+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:30:55.998013+0000","last_peered":"2024-09-11T00:30:55.998013+0000","last_clean":"2024-09-11T00:30:55.998013+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:30:55.998013+0000","last_undegraded":"2024-09-11T00:30:55.998013+0000","last_fullsized":"2024-09-11T00:30:55.998013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968362+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:30:55.968362+0000","last_peered":"2024-09-11T00:30:55.968362+0000","last_clean":"2024-09-11T00:30:55.968362+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:30:55.968362+0000","last_undegraded":"2024-09-11T00:30:55.968362+0000","last_fullsized":"2024-09-11T00:30:55.968362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971436+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:30:55.971436+0000","last_peered":"2024-09-11T00:30:55.971436+0000","last_clean":"2024-09-11T00:30:55.971436+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:30:55.971436+0000","last_undegraded":"2024-09-11T00:30:55.971436+0000","last_fullsized":"2024-09-11T00:30:55.971436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172455+0000","last_change":"2024-09-11T00:30:53.989292+0000","last_active":"2024-09-11T00:30:56.172455+0000","last_peered":"2024-09-11T00:30:56.172455+0000","last_clean":"2024-09-11T00:30:56.172455+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:30:56.172455+0000","last_undegraded":"2024-09-11T00:30:56.172455+0000","last_fullsized":"2024-09-11T00:30:56.172455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:44.071338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968473+0000","last_change":"2024-09-11T00:30:54.981630+0000","last_active":"2024-09-11T00:30:55.968473+0000","last_peered":"2024-09-11T00:30:55.968473+0000","last_clean":"2024-09-11T00:30:55.968473+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:30:55.968473+0000","last_undegraded":"2024-09-11T00:30:55.968473+0000","last_fullsized":"2024-09-11T00:30:55.968473+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:12.832327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333061+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:30:56.333061+0000","last_peered":"2024-09-11T00:30:56.333061+0000","last_clean":"2024-09-11T00:30:56.333061+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:30:56.333061+0000","last_undegraded":"2024-09-11T00:30:56.333061+0000","last_fullsized":"2024-09-11T00:30:56.333061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997856+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:30:55.997856+0000","last_peered":"2024-09-11T00:30:55.997856+0000","last_clean":"2024-09-11T00:30:55.997856+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:30:55.997856+0000","last_undegraded":"2024-09-11T00:30:55.997856+0000","last_fullsized":"2024-09-11T00:30:55.997856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997848+0000","last_change":"2024-09-11T00:30:53.984013+0000","last_active":"2024-09-11T00:30:55.997848+0000","last_peered":"2024-09-11T00:30:55.997848+0000","last_clean":"2024-09-11T00:30:55.997848+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:30:55.997848+0000","last_undegraded":"2024-09-11T00:30:55.997848+0000","last_fullsized":"2024-09-11T00:30:55.997848+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:49.963623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719176+0000","last_change":"2024-09-11T00:30:54.996860+0000","last_active":"2024-09-11T00:30:56.719176+0000","last_peered":"2024-09-11T00:30:56.719176+0000","last_clean":"2024-09-11T00:30:56.719176+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:30:56.719176+0000","last_undegraded":"2024-09-11T00:30:56.719176+0000","last_fullsized":"2024-09-11T00:30:56.719176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:43.516748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333114+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:30:56.333114+0000","last_peered":"2024-09-11T00:30:56.333114+0000","last_clean":"2024-09-11T00:30:56.333114+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:30:56.333114+0000","last_undegraded":"2024-09-11T00:30:56.333114+0000","last_fullsized":"2024-09-11T00:30:56.333114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970314+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:30:55.970314+0000","last_peered":"2024-09-11T00:30:55.970314+0000","last_clean":"2024-09-11T00:30:55.970314+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:30:55.970314+0000","last_undegraded":"2024-09-11T00:30:55.970314+0000","last_fullsized":"2024-09-11T00:30:55.970314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966317+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:30:55.966317+0000","last_peered":"2024-09-11T00:30:55.966317+0000","last_clean":"2024-09-11T00:30:55.966317+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:30:55.966317+0000","last_undegraded":"2024-09-11T00:30:55.966317+0000","last_fullsized":"2024-09-11T00:30:55.966317+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:19.891052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967870+0000","last_change":"2024-09-11T00:30:54.993736+0000","last_active":"2024-09-11T00:30:55.967870+0000","last_peered":"2024-09-11T00:30:55.967870+0000","last_clean":"2024-09-11T00:30:55.967870+0000","last_became_active":"2024-09-11T00:30:54.993470+0000","last_became_peered":"2024-09-11T00:30:54.993470+0000","last_unstale":"2024-09-11T00:30:55.967870+0000","last_undegraded":"2024-09-11T00:30:55.967870+0000","last_fullsized":"2024-09-11T00:30:55.967870+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:24.712448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173170+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:30:56.173170+0000","last_peered":"2024-09-11T00:30:56.173170+0000","last_clean":"2024-09-11T00:30:56.173170+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:30:56.173170+0000","last_undegraded":"2024-09-11T00:30:56.173170+0000","last_fullsized":"2024-09-11T00:30:56.173170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333218+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:30:56.333218+0000","last_peered":"2024-09-11T00:30:56.333218+0000","last_clean":"2024-09-11T00:30:56.333218+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:30:56.333218+0000","last_undegraded":"2024-09-11T00:30:56.333218+0000","last_fullsized":"2024-09-11T00:30:56.333218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719247+0000","last_change":"2024-09-11T00:30:53.983352+0000","last_active":"2024-09-11T00:30:56.719247+0000","last_peered":"2024-09-11T00:30:56.719247+0000","last_clean":"2024-09-11T00:30:56.719247+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:30:56.719247+0000","last_undegraded":"2024-09-11T00:30:56.719247+0000","last_fullsized":"2024-09-11T00:30:56.719247+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:43.826114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970030+0000","last_change":"2024-09-11T00:30:54.987533+0000","last_active":"2024-09-11T00:30:55.970030+0000","last_peered":"2024-09-11T00:30:55.970030+0000","last_clean":"2024-09-11T00:30:55.970030+0000","last_became_active":"2024-09-11T00:30:54.987253+0000","last_became_peered":"2024-09-11T00:30:54.987253+0000","last_unstale":"2024-09-11T00:30:55.970030+0000","last_undegraded":"2024-09-11T00:30:55.970030+0000","last_fullsized":"2024-09-11T00:30:55.970030+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.923656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966934+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:30:55.966934+0000","last_peered":"2024-09-11T00:30:55.966934+0000","last_clean":"2024-09-11T00:30:55.966934+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:30:55.966934+0000","last_undegraded":"2024-09-11T00:30:55.966934+0000","last_fullsized":"2024-09-11T00:30:55.966934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968528+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:30:55.968528+0000","last_peered":"2024-09-11T00:30:55.968528+0000","last_clean":"2024-09-11T00:30:55.968528+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:30:55.968528+0000","last_undegraded":"2024-09-11T00:30:55.968528+0000","last_fullsized":"2024-09-11T00:30:55.968528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997651+0000","last_change":"2024-09-11T00:30:53.980703+0000","last_active":"2024-09-11T00:30:55.997651+0000","last_peered":"2024-09-11T00:30:55.997651+0000","last_clean":"2024-09-11T00:30:55.997651+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:30:55.997651+0000","last_undegraded":"2024-09-11T00:30:55.997651+0000","last_fullsized":"2024-09-11T00:30:55.997651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:51.050512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967820+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:30:55.967820+0000","last_peered":"2024-09-11T00:30:55.967820+0000","last_clean":"2024-09-11T00:30:55.967820+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:30:55.967820+0000","last_undegraded":"2024-09-11T00:30:55.967820+0000","last_fullsized":"2024-09-11T00:30:55.967820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719139+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:30:56.719139+0000","last_peered":"2024-09-11T00:30:56.719139+0000","last_clean":"2024-09-11T00:30:56.719139+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:30:56.719139+0000","last_undegraded":"2024-09-11T00:30:56.719139+0000","last_fullsized":"2024-09-11T00:30:56.719139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333070+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:30:56.333070+0000","last_peered":"2024-09-11T00:30:56.333070+0000","last_clean":"2024-09-11T00:30:56.333070+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:30:56.333070+0000","last_undegraded":"2024-09-11T00:30:56.333070+0000","last_fullsized":"2024-09-11T00:30:56.333070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997937+0000","last_change":"2024-09-11T00:30:53.987829+0000","last_active":"2024-09-11T00:30:55.997937+0000","last_peered":"2024-09-11T00:30:55.997937+0000","last_clean":"2024-09-11T00:30:55.997937+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:30:55.997937+0000","last_undegraded":"2024-09-11T00:30:55.997937+0000","last_fullsized":"2024-09-11T00:30:55.997937+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:57:43.792089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967657+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:30:55.967657+0000","last_peered":"2024-09-11T00:30:55.967657+0000","last_clean":"2024-09-11T00:30:55.967657+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:30:55.967657+0000","last_undegraded":"2024-09-11T00:30:55.967657+0000","last_fullsized":"2024-09-11T00:30:55.967657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967382+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:30:55.967382+0000","last_peered":"2024-09-11T00:30:55.967382+0000","last_clean":"2024-09-11T00:30:55.967382+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:30:55.967382+0000","last_undegraded":"2024-09-11T00:30:55.967382+0000","last_fullsized":"2024-09-11T00:30:55.967382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967492+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:30:55.967492+0000","last_peered":"2024-09-11T00:30:55.967492+0000","last_clean":"2024-09-11T00:30:55.967492+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:30:55.967492+0000","last_undegraded":"2024-09-11T00:30:55.967492+0000","last_fullsized":"2024-09-11T00:30:55.967492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970951+0000","last_change":"2024-09-11T00:30:53.984057+0000","last_active":"2024-09-11T00:30:55.970951+0000","last_peered":"2024-09-11T00:30:55.970951+0000","last_clean":"2024-09-11T00:30:55.970951+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:30:55.970951+0000","last_undegraded":"2024-09-11T00:30:55.970951+0000","last_fullsized":"2024-09-11T00:30:55.970951+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:10.477916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969373+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:30:55.969373+0000","last_peered":"2024-09-11T00:30:55.969373+0000","last_clean":"2024-09-11T00:30:55.969373+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:30:55.969373+0000","last_undegraded":"2024-09-11T00:30:55.969373+0000","last_fullsized":"2024-09-11T00:30:55.969373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969960+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:30:55.969960+0000","last_peered":"2024-09-11T00:30:55.969960+0000","last_clean":"2024-09-11T00:30:55.969960+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:30:55.969960+0000","last_undegraded":"2024-09-11T00:30:55.969960+0000","last_fullsized":"2024-09-11T00:30:55.969960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001240+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:30:56.001240+0000","last_peered":"2024-09-11T00:30:56.001240+0000","last_clean":"2024-09-11T00:30:56.001240+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:30:56.001240+0000","last_undegraded":"2024-09-11T00:30:56.001240+0000","last_fullsized":"2024-09-11T00:30:56.001240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968074+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:30:55.968074+0000","last_peered":"2024-09-11T00:30:55.968074+0000","last_clean":"2024-09-11T00:30:55.968074+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:30:55.968074+0000","last_undegraded":"2024-09-11T00:30:55.968074+0000","last_fullsized":"2024-09-11T00:30:55.968074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969893+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:30:55.969893+0000","last_peered":"2024-09-11T00:30:55.969893+0000","last_clean":"2024-09-11T00:30:55.969893+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:30:55.969893+0000","last_undegraded":"2024-09-11T00:30:55.969893+0000","last_fullsized":"2024-09-11T00:30:55.969893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332978+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:30:56.332978+0000","last_peered":"2024-09-11T00:30:56.332978+0000","last_clean":"2024-09-11T00:30:56.332978+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:30:56.332978+0000","last_undegraded":"2024-09-11T00:30:56.332978+0000","last_fullsized":"2024-09-11T00:30:56.332978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173134+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:30:56.173134+0000","last_peered":"2024-09-11T00:30:56.173134+0000","last_clean":"2024-09-11T00:30:56.173134+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:30:56.173134+0000","last_undegraded":"2024-09-11T00:30:56.173134+0000","last_fullsized":"2024-09-11T00:30:56.173134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969278+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:30:55.969278+0000","last_peered":"2024-09-11T00:30:55.969278+0000","last_clean":"2024-09-11T00:30:55.969278+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:30:55.969278+0000","last_undegraded":"2024-09-11T00:30:55.969278+0000","last_fullsized":"2024-09-11T00:30:55.969278+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969704+0000","last_change":"2024-09-11T00:30:54.986017+0000","last_active":"2024-09-11T00:30:55.969704+0000","last_peered":"2024-09-11T00:30:55.969704+0000","last_clean":"2024-09-11T00:30:55.969704+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:30:55.969704+0000","last_undegraded":"2024-09-11T00:30:55.969704+0000","last_fullsized":"2024-09-11T00:30:55.969704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:58:12.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967364+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:30:55.967364+0000","last_peered":"2024-09-11T00:30:55.967364+0000","last_clean":"2024-09-11T00:30:55.967364+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:30:55.967364+0000","last_undegraded":"2024-09-11T00:30:55.967364+0000","last_fullsized":"2024-09-11T00:30:55.967364+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001702+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:30:56.001702+0000","last_peered":"2024-09-11T00:30:56.001702+0000","last_clean":"2024-09-11T00:30:56.001702+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:30:56.001702+0000","last_undegraded":"2024-09-11T00:30:56.001702+0000","last_fullsized":"2024-09-11T00:30:56.001702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172588+0000","last_change":"2024-09-11T00:30:53.988462+0000","last_active":"2024-09-11T00:30:56.172588+0000","last_peered":"2024-09-11T00:30:56.172588+0000","last_clean":"2024-09-11T00:30:56.172588+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:30:56.172588+0000","last_undegraded":"2024-09-11T00:30:56.172588+0000","last_fullsized":"2024-09-11T00:30:56.172588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:22.385557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966521+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:30:55.966521+0000","last_peered":"2024-09-11T00:30:55.966521+0000","last_clean":"2024-09-11T00:30:55.966521+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:30:55.966521+0000","last_undegraded":"2024-09-11T00:30:55.966521+0000","last_fullsized":"2024-09-11T00:30:55.966521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998206+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:30:55.998206+0000","last_peered":"2024-09-11T00:30:55.998206+0000","last_clean":"2024-09-11T00:30:55.998206+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:30:55.998206+0000","last_undegraded":"2024-09-11T00:30:55.998206+0000","last_fullsized":"2024-09-11T00:30:55.998206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332956+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:30:56.332956+0000","last_peered":"2024-09-11T00:30:56.332956+0000","last_clean":"2024-09-11T00:30:56.332956+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:30:56.332956+0000","last_undegraded":"2024-09-11T00:30:56.332956+0000","last_fullsized":"2024-09-11T00:30:56.332956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000726+0000","last_change":"2024-09-11T00:30:53.986646+0000","last_active":"2024-09-11T00:30:56.000726+0000","last_peered":"2024-09-11T00:30:56.000726+0000","last_clean":"2024-09-11T00:30:56.000726+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:30:56.000726+0000","last_undegraded":"2024-09-11T00:30:56.000726+0000","last_fullsized":"2024-09-11T00:30:56.000726+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:27.447940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172237+0000","last_change":"2024-09-11T00:30:54.982486+0000","last_active":"2024-09-11T00:30:56.172237+0000","last_peered":"2024-09-11T00:30:56.172237+0000","last_clean":"2024-09-11T00:30:56.172237+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:30:56.172237+0000","last_undegraded":"2024-09-11T00:30:56.172237+0000","last_fullsized":"2024-09-11T00:30:56.172237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:41.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173110+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:30:56.173110+0000","last_peered":"2024-09-11T00:30:56.173110+0000","last_clean":"2024-09-11T00:30:56.173110+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:30:56.173110+0000","last_undegraded":"2024-09-11T00:30:56.173110+0000","last_fullsized":"2024-09-11T00:30:56.173110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719858+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:30:56.719858+0000","last_peered":"2024-09-11T00:30:56.719858+0000","last_clean":"2024-09-11T00:30:56.719858+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:30:56.719858+0000","last_undegraded":"2024-09-11T00:30:56.719858+0000","last_fullsized":"2024-09-11T00:30:56.719858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968520+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:30:55.968520+0000","last_peered":"2024-09-11T00:30:55.968520+0000","last_clean":"2024-09-11T00:30:55.968520+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:30:55.968520+0000","last_undegraded":"2024-09-11T00:30:55.968520+0000","last_fullsized":"2024-09-11T00:30:55.968520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173273+0000","last_change":"2024-09-11T00:30:54.985091+0000","last_active":"2024-09-11T00:30:56.173273+0000","last_peered":"2024-09-11T00:30:56.173273+0000","last_clean":"2024-09-11T00:30:56.173273+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:30:56.173273+0000","last_undegraded":"2024-09-11T00:30:56.173273+0000","last_fullsized":"2024-09-11T00:30:56.173273+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:33.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968811+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:30:55.968811+0000","last_peered":"2024-09-11T00:30:55.968811+0000","last_clean":"2024-09-11T00:30:55.968811+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:30:55.968811+0000","last_undegraded":"2024-09-11T00:30:55.968811+0000","last_fullsized":"2024-09-11T00:30:55.968811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966522+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:30:55.966522+0000","last_peered":"2024-09-11T00:30:55.966522+0000","last_clean":"2024-09-11T00:30:55.966522+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:30:55.966522+0000","last_undegraded":"2024-09-11T00:30:55.966522+0000","last_fullsized":"2024-09-11T00:30:55.966522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971377+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:30:55.971377+0000","last_peered":"2024-09-11T00:30:55.971377+0000","last_clean":"2024-09-11T00:30:55.971377+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:30:55.971377+0000","last_undegraded":"2024-09-11T00:30:55.971377+0000","last_fullsized":"2024-09-11T00:30:55.971377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:11.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967648+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:30:55.967648+0000","last_peered":"2024-09-11T00:30:55.967648+0000","last_clean":"2024-09-11T00:30:55.967648+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:30:55.967648+0000","last_undegraded":"2024-09-11T00:30:55.967648+0000","last_fullsized":"2024-09-11T00:30:55.967648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968196+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:30:55.968196+0000","last_peered":"2024-09-11T00:30:55.968196+0000","last_clean":"2024-09-11T00:30:55.968196+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:30:55.968196+0000","last_undegraded":"2024-09-11T00:30:55.968196+0000","last_fullsized":"2024-09-11T00:30:55.968196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969528+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:30:55.969528+0000","last_peered":"2024-09-11T00:30:55.969528+0000","last_clean":"2024-09-11T00:30:55.969528+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:30:55.969528+0000","last_undegraded":"2024-09-11T00:30:55.969528+0000","last_fullsized":"2024-09-11T00:30:55.969528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967369+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:30:55.967369+0000","last_peered":"2024-09-11T00:30:55.967369+0000","last_clean":"2024-09-11T00:30:55.967369+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:30:55.967369+0000","last_undegraded":"2024-09-11T00:30:55.967369+0000","last_fullsized":"2024-09-11T00:30:55.967369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969812+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:30:55.969812+0000","last_peered":"2024-09-11T00:30:55.969812+0000","last_clean":"2024-09-11T00:30:55.969812+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:30:55.969812+0000","last_undegraded":"2024-09-11T00:30:55.969812+0000","last_fullsized":"2024-09-11T00:30:55.969812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969509+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:30:55.969509+0000","last_peered":"2024-09-11T00:30:55.969509+0000","last_clean":"2024-09-11T00:30:55.969509+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:30:55.969509+0000","last_undegraded":"2024-09-11T00:30:55.969509+0000","last_fullsized":"2024-09-11T00:30:55.969509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173337+0000","last_change":"2024-09-11T00:30:54.998464+0000","last_active":"2024-09-11T00:30:56.173337+0000","last_peered":"2024-09-11T00:30:56.173337+0000","last_clean":"2024-09-11T00:30:56.173337+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:30:56.173337+0000","last_undegraded":"2024-09-11T00:30:56.173337+0000","last_fullsized":"2024-09-11T00:30:56.173337+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:56.469791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998082+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:30:55.998082+0000","last_peered":"2024-09-11T00:30:55.998082+0000","last_clean":"2024-09-11T00:30:55.998082+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:30:55.998082+0000","last_undegraded":"2024-09-11T00:30:55.998082+0000","last_fullsized":"2024-09-11T00:30:55.998082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719278+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:30:56.719278+0000","last_peered":"2024-09-11T00:30:56.719278+0000","last_clean":"2024-09-11T00:30:56.719278+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:30:56.719278+0000","last_undegraded":"2024-09-11T00:30:56.719278+0000","last_fullsized":"2024-09-11T00:30:56.719278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001549+0000","last_change":"2024-09-11T00:30:53.987612+0000","last_active":"2024-09-11T00:30:56.001549+0000","last_peered":"2024-09-11T00:30:56.001549+0000","last_clean":"2024-09-11T00:30:56.001549+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:30:56.001549+0000","last_undegraded":"2024-09-11T00:30:56.001549+0000","last_fullsized":"2024-09-11T00:30:56.001549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:16.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970084+0000","last_change":"2024-09-11T00:30:54.989925+0000","last_active":"2024-09-11T00:30:55.970084+0000","last_peered":"2024-09-11T00:30:55.970084+0000","last_clean":"2024-09-11T00:30:55.970084+0000","last_became_active":"2024-09-11T00:30:54.989401+0000","last_became_peered":"2024-09-11T00:30:54.989401+0000","last_unstale":"2024-09-11T00:30:55.970084+0000","last_undegraded":"2024-09-11T00:30:55.970084+0000","last_fullsized":"2024-09-11T00:30:55.970084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:14.267175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172747+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:30:56.172747+0000","last_peered":"2024-09-11T00:30:56.172747+0000","last_clean":"2024-09-11T00:30:56.172747+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:30:56.172747+0000","last_undegraded":"2024-09-11T00:30:56.172747+0000","last_fullsized":"2024-09-11T00:30:56.172747+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969929+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:30:55.969929+0000","last_peered":"2024-09-11T00:30:55.969929+0000","last_clean":"2024-09-11T00:30:55.969929+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:30:55.969929+0000","last_undegraded":"2024-09-11T00:30:55.969929+0000","last_fullsized":"2024-09-11T00:30:55.969929+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000911+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:30:56.000911+0000","last_peered":"2024-09-11T00:30:56.000911+0000","last_clean":"2024-09-11T00:30:56.000911+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:30:56.000911+0000","last_undegraded":"2024-09-11T00:30:56.000911+0000","last_fullsized":"2024-09-11T00:30:56.000911+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172912+0000","last_change":"2024-09-11T00:30:54.979133+0000","last_active":"2024-09-11T00:30:56.172912+0000","last_peered":"2024-09-11T00:30:56.172912+0000","last_clean":"2024-09-11T00:30:56.172912+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:30:56.172912+0000","last_undegraded":"2024-09-11T00:30:56.172912+0000","last_fullsized":"2024-09-11T00:30:56.172912+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:20.961678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967251+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:30:55.967251+0000","last_peered":"2024-09-11T00:30:55.967251+0000","last_clean":"2024-09-11T00:30:55.967251+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:30:55.967251+0000","last_undegraded":"2024-09-11T00:30:55.967251+0000","last_fullsized":"2024-09-11T00:30:55.967251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997885+0000","last_change":"2024-09-11T00:30:54.988345+0000","last_active":"2024-09-11T00:30:55.997885+0000","last_peered":"2024-09-11T00:30:55.997885+0000","last_clean":"2024-09-11T00:30:55.997885+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:30:55.997885+0000","last_undegraded":"2024-09-11T00:30:55.997885+0000","last_fullsized":"2024-09-11T00:30:55.997885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.112194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998078+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:30:55.998078+0000","last_peered":"2024-09-11T00:30:55.998078+0000","last_clean":"2024-09-11T00:30:55.998078+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:30:55.998078+0000","last_undegraded":"2024-09-11T00:30:55.998078+0000","last_fullsized":"2024-09-11T00:30:55.998078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332489+0000","last_change":"2024-09-11T00:30:54.981185+0000","last_active":"2024-09-11T00:30:56.332489+0000","last_peered":"2024-09-11T00:30:56.332489+0000","last_clean":"2024-09-11T00:30:56.332489+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:30:56.332489+0000","last_undegraded":"2024-09-11T00:30:56.332489+0000","last_fullsized":"2024-09-11T00:30:56.332489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:45.740467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971241+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:30:55.971241+0000","last_peered":"2024-09-11T00:30:55.971241+0000","last_clean":"2024-09-11T00:30:55.971241+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:30:55.971241+0000","last_undegraded":"2024-09-11T00:30:55.971241+0000","last_fullsized":"2024-09-11T00:30:55.971241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332647+0000","last_change":"2024-09-11T00:30:54.992763+0000","last_active":"2024-09-11T00:30:56.332647+0000","last_peered":"2024-09-11T00:30:56.332647+0000","last_clean":"2024-09-11T00:30:56.332647+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:30:56.332647+0000","last_undegraded":"2024-09-11T00:30:56.332647+0000","last_fullsized":"2024-09-11T00:30:56.332647+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:02.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971179+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:30:55.971179+0000","last_peered":"2024-09-11T00:30:55.971179+0000","last_clean":"2024-09-11T00:30:55.971179+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:30:55.971179+0000","last_undegraded":"2024-09-11T00:30:55.971179+0000","last_fullsized":"2024-09-11T00:30:55.971179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966440+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:30:55.966440+0000","last_peered":"2024-09-11T00:30:55.966440+0000","last_clean":"2024-09-11T00:30:55.966440+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:30:55.966440+0000","last_undegraded":"2024-09-11T00:30:55.966440+0000","last_fullsized":"2024-09-11T00:30:55.966440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173066+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:30:56.173066+0000","last_peered":"2024-09-11T00:30:56.173066+0000","last_clean":"2024-09-11T00:30:56.173066+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:30:56.173066+0000","last_undegraded":"2024-09-11T00:30:56.173066+0000","last_fullsized":"2024-09-11T00:30:56.173066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172366+0000","last_change":"2024-09-11T00:30:54.984447+0000","last_active":"2024-09-11T00:30:56.172366+0000","last_peered":"2024-09-11T00:30:56.172366+0000","last_clean":"2024-09-11T00:30:56.172366+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:30:56.172366+0000","last_undegraded":"2024-09-11T00:30:56.172366+0000","last_fullsized":"2024-09-11T00:30:56.172366+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:49.024022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998072+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:30:55.998072+0000","last_peered":"2024-09-11T00:30:55.998072+0000","last_clean":"2024-09-11T00:30:55.998072+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:30:55.998072+0000","last_undegraded":"2024-09-11T00:30:55.998072+0000","last_fullsized":"2024-09-11T00:30:55.998072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001507+0000","last_change":"2024-09-11T00:30:54.996091+0000","last_active":"2024-09-11T00:30:56.001507+0000","last_peered":"2024-09-11T00:30:56.001507+0000","last_clean":"2024-09-11T00:30:56.001507+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:30:56.001507+0000","last_undegraded":"2024-09-11T00:30:56.001507+0000","last_fullsized":"2024-09-11T00:30:56.001507+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:42.987336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966504+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:30:55.966504+0000","last_peered":"2024-09-11T00:30:55.966504+0000","last_clean":"2024-09-11T00:30:55.966504+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:30:55.966504+0000","last_undegraded":"2024-09-11T00:30:55.966504+0000","last_fullsized":"2024-09-11T00:30:55.966504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968347+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:30:55.968347+0000","last_peered":"2024-09-11T00:30:55.968347+0000","last_clean":"2024-09-11T00:30:55.968347+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:30:55.968347+0000","last_undegraded":"2024-09-11T00:30:55.968347+0000","last_fullsized":"2024-09-11T00:30:55.968347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971167+0000","last_change":"2024-09-11T00:30:54.981799+0000","last_active":"2024-09-11T00:30:55.971167+0000","last_peered":"2024-09-11T00:30:55.971167+0000","last_clean":"2024-09-11T00:30:55.971167+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:30:55.971167+0000","last_undegraded":"2024-09-11T00:30:55.971167+0000","last_fullsized":"2024-09-11T00:30:55.971167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:42.670629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719169+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:30:56.719169+0000","last_peered":"2024-09-11T00:30:56.719169+0000","last_clean":"2024-09-11T00:30:56.719169+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:30:56.719169+0000","last_undegraded":"2024-09-11T00:30:56.719169+0000","last_fullsized":"2024-09-11T00:30:56.719169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001453+0000","last_change":"2024-09-11T00:30:54.992697+0000","last_active":"2024-09-11T00:30:56.001453+0000","last_peered":"2024-09-11T00:30:56.001453+0000","last_clean":"2024-09-11T00:30:56.001453+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:30:56.001453+0000","last_undegraded":"2024-09-11T00:30:56.001453+0000","last_fullsized":"2024-09-11T00:30:56.001453+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:54:07.227169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969635+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:30:55.969635+0000","last_peered":"2024-09-11T00:30:55.969635+0000","last_clean":"2024-09-11T00:30:55.969635+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:55.969635+0000","last_undegraded":"2024-09-11T00:30:55.969635+0000","last_fullsized":"2024-09-11T00:30:55.969635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172413+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:30:56.172413+0000","last_peered":"2024-09-11T00:30:56.172413+0000","last_clean":"2024-09-11T00:30:56.172413+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:30:56.172413+0000","last_undegraded":"2024-09-11T00:30:56.172413+0000","last_fullsized":"2024-09-11T00:30:56.172413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332844+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:30:56.332844+0000","last_peered":"2024-09-11T00:30:56.332844+0000","last_clean":"2024-09-11T00:30:56.332844+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:30:56.332844+0000","last_undegraded":"2024-09-11T00:30:56.332844+0000","last_fullsized":"2024-09-11T00:30:56.332844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332895+0000","last_change":"2024-09-11T00:30:54.985827+0000","last_active":"2024-09-11T00:30:56.332895+0000","last_peered":"2024-09-11T00:30:56.332895+0000","last_clean":"2024-09-11T00:30:56.332895+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:30:56.332895+0000","last_undegraded":"2024-09-11T00:30:56.332895+0000","last_fullsized":"2024-09-11T00:30:56.332895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:32.027117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968441+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:30:55.968441+0000","last_peered":"2024-09-11T00:30:55.968441+0000","last_clean":"2024-09-11T00:30:55.968441+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:30:55.968441+0000","last_undegraded":"2024-09-11T00:30:55.968441+0000","last_fullsized":"2024-09-11T00:30:55.968441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172255+0000","last_change":"2024-09-11T00:30:54.989906+0000","last_active":"2024-09-11T00:30:56.172255+0000","last_peered":"2024-09-11T00:30:56.172255+0000","last_clean":"2024-09-11T00:30:56.172255+0000","last_became_active":"2024-09-11T00:30:54.989376+0000","last_became_peered":"2024-09-11T00:30:54.989376+0000","last_unstale":"2024-09-11T00:30:56.172255+0000","last_undegraded":"2024-09-11T00:30:56.172255+0000","last_fullsized":"2024-09-11T00:30:56.172255+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:18:00.138941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001665+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:30:56.001665+0000","last_peered":"2024-09-11T00:30:56.001665+0000","last_clean":"2024-09-11T00:30:56.001665+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:30:56.001665+0000","last_undegraded":"2024-09-11T00:30:56.001665+0000","last_fullsized":"2024-09-11T00:30:56.001665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968974+0000","last_change":"2024-09-11T00:30:54.993156+0000","last_active":"2024-09-11T00:30:55.968974+0000","last_peered":"2024-09-11T00:30:55.968974+0000","last_clean":"2024-09-11T00:30:55.968974+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:30:55.968974+0000","last_undegraded":"2024-09-11T00:30:55.968974+0000","last_fullsized":"2024-09-11T00:30:55.968974+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:25.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968016+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:30:55.968016+0000","last_peered":"2024-09-11T00:30:55.968016+0000","last_clean":"2024-09-11T00:30:55.968016+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:30:55.968016+0000","last_undegraded":"2024-09-11T00:30:55.968016+0000","last_fullsized":"2024-09-11T00:30:55.968016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970335+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:30:55.970335+0000","last_peered":"2024-09-11T00:30:55.970335+0000","last_clean":"2024-09-11T00:30:55.970335+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:30:55.970335+0000","last_undegraded":"2024-09-11T00:30:55.970335+0000","last_fullsized":"2024-09-11T00:30:55.970335+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966434+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:30:55.966434+0000","last_peered":"2024-09-11T00:30:55.966434+0000","last_clean":"2024-09-11T00:30:55.966434+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:30:55.966434+0000","last_undegraded":"2024-09-11T00:30:55.966434+0000","last_fullsized":"2024-09-11T00:30:55.966434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719699+0000","last_change":"2024-09-11T00:30:54.993461+0000","last_active":"2024-09-11T00:30:56.719699+0000","last_peered":"2024-09-11T00:30:56.719699+0000","last_clean":"2024-09-11T00:30:56.719699+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:30:56.719699+0000","last_undegraded":"2024-09-11T00:30:56.719699+0000","last_fullsized":"2024-09-11T00:30:56.719699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:07.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997281+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:30:55.997281+0000","last_peered":"2024-09-11T00:30:55.997281+0000","last_clean":"2024-09-11T00:30:55.997281+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:30:55.997281+0000","last_undegraded":"2024-09-11T00:30:55.997281+0000","last_fullsized":"2024-09-11T00:30:55.997281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719307+0000","last_change":"2024-09-11T00:30:54.986265+0000","last_active":"2024-09-11T00:30:56.719307+0000","last_peered":"2024-09-11T00:30:56.719307+0000","last_clean":"2024-09-11T00:30:56.719307+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:30:56.719307+0000","last_undegraded":"2024-09-11T00:30:56.719307+0000","last_fullsized":"2024-09-11T00:30:56.719307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:49.109328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969766+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:30:55.969766+0000","last_peered":"2024-09-11T00:30:55.969766+0000","last_clean":"2024-09-11T00:30:55.969766+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:30:55.969766+0000","last_undegraded":"2024-09-11T00:30:55.969766+0000","last_fullsized":"2024-09-11T00:30:55.969766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967854+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:30:55.967854+0000","last_peered":"2024-09-11T00:30:55.967854+0000","last_clean":"2024-09-11T00:30:55.967854+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:30:55.967854+0000","last_undegraded":"2024-09-11T00:30:55.967854+0000","last_fullsized":"2024-09-11T00:30:55.967854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971368+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:30:55.971368+0000","last_peered":"2024-09-11T00:30:55.971368+0000","last_clean":"2024-09-11T00:30:55.971368+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:30:55.971368+0000","last_undegraded":"2024-09-11T00:30:55.971368+0000","last_fullsized":"2024-09-11T00:30:55.971368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969452+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:30:55.969452+0000","last_peered":"2024-09-11T00:30:55.969452+0000","last_clean":"2024-09-11T00:30:55.969452+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:30:55.969452+0000","last_undegraded":"2024-09-11T00:30:55.969452+0000","last_fullsized":"2024-09-11T00:30:55.969452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332691+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:30:56.332691+0000","last_peered":"2024-09-11T00:30:56.332691+0000","last_clean":"2024-09-11T00:30:56.332691+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:30:56.332691+0000","last_undegraded":"2024-09-11T00:30:56.332691+0000","last_fullsized":"2024-09-11T00:30:56.332691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970850+0000","last_change":"2024-09-11T00:30:54.994988+0000","last_active":"2024-09-11T00:30:55.970850+0000","last_peered":"2024-09-11T00:30:55.970850+0000","last_clean":"2024-09-11T00:30:55.970850+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:30:55.970850+0000","last_undegraded":"2024-09-11T00:30:55.970850+0000","last_fullsized":"2024-09-11T00:30:55.970850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:31.973641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332924+0000","last_change":"2024-09-11T00:30:45.941635+0000","last_active":"2024-09-11T00:30:56.332924+0000","last_peered":"2024-09-11T00:30:56.332924+0000","last_clean":"2024-09-11T00:30:56.332924+0000","last_became_active":"2024-09-11T00:30:45.941357+0000","last_became_peered":"2024-09-11T00:30:45.941357+0000","last_unstale":"2024-09-11T00:30:56.332924+0000","last_undegraded":"2024-09-11T00:30:56.332924+0000","last_fullsized":"2024-09-11T00:30:56.332924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:12.016406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332421+0000","last_change":"2024-09-11T00:30:54.984767+0000","last_active":"2024-09-11T00:30:56.332421+0000","last_peered":"2024-09-11T00:30:56.332421+0000","last_clean":"2024-09-11T00:30:56.332421+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:30:56.332421+0000","last_undegraded":"2024-09-11T00:30:56.332421+0000","last_fullsized":"2024-09-11T00:30:56.332421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:44.833677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332955+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:30:56.332955+0000","last_peered":"2024-09-11T00:30:56.332955+0000","last_clean":"2024-09-11T00:30:56.332955+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:56.332955+0000","last_undegraded":"2024-09-11T00:30:56.332955+0000","last_fullsized":"2024-09-11T00:30:56.332955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332911+0000","last_change":"2024-09-11T00:30:54.986423+0000","last_active":"2024-09-11T00:30:56.332911+0000","last_peered":"2024-09-11T00:30:56.332911+0000","last_clean":"2024-09-11T00:30:56.332911+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:30:56.332911+0000","last_undegraded":"2024-09-11T00:30:56.332911+0000","last_fullsized":"2024-09-11T00:30:56.332911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:16:50.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719859+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:30:56.719859+0000","last_peered":"2024-09-11T00:30:56.719859+0000","last_clean":"2024-09-11T00:30:56.719859+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:30:56.719859+0000","last_undegraded":"2024-09-11T00:30:56.719859+0000","last_fullsized":"2024-09-11T00:30:56.719859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998062+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:30:55.998062+0000","last_peered":"2024-09-11T00:30:55.998062+0000","last_clean":"2024-09-11T00:30:55.998062+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:30:55.998062+0000","last_undegraded":"2024-09-11T00:30:55.998062+0000","last_fullsized":"2024-09-11T00:30:55.998062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997283+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:30:55.997283+0000","last_peered":"2024-09-11T00:30:55.997283+0000","last_clean":"2024-09-11T00:30:55.997283+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:30:55.997283+0000","last_undegraded":"2024-09-11T00:30:55.997283+0000","last_fullsized":"2024-09-11T00:30:55.997283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173138+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:30:56.173138+0000","last_peered":"2024-09-11T00:30:56.173138+0000","last_clean":"2024-09-11T00:30:56.173138+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:30:56.173138+0000","last_undegraded":"2024-09-11T00:30:56.173138+0000","last_fullsized":"2024-09-11T00:30:56.173138+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333239+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:30:56.333239+0000","last_peered":"2024-09-11T00:30:56.333239+0000","last_clean":"2024-09-11T00:30:56.333239+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:30:56.333239+0000","last_undegraded":"2024-09-11T00:30:56.333239+0000","last_fullsized":"2024-09-11T00:30:56.333239+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970256+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:30:55.970256+0000","last_peered":"2024-09-11T00:30:55.970256+0000","last_clean":"2024-09-11T00:30:55.970256+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:30:55.970256+0000","last_undegraded":"2024-09-11T00:30:55.970256+0000","last_fullsized":"2024-09-11T00:30:55.970256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172539+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:30:56.172539+0000","last_peered":"2024-09-11T00:30:56.172539+0000","last_clean":"2024-09-11T00:30:56.172539+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:30:56.172539+0000","last_undegraded":"2024-09-11T00:30:56.172539+0000","last_fullsized":"2024-09-11T00:30:56.172539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719531+0000","last_change":"2024-09-11T00:30:54.990631+0000","last_active":"2024-09-11T00:30:56.719531+0000","last_peered":"2024-09-11T00:30:56.719531+0000","last_clean":"2024-09-11T00:30:56.719531+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:30:56.719531+0000","last_undegraded":"2024-09-11T00:30:56.719531+0000","last_fullsized":"2024-09-11T00:30:56.719531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:02.422471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333220+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:30:56.333220+0000","last_peered":"2024-09-11T00:30:56.333220+0000","last_clean":"2024-09-11T00:30:56.333220+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:30:56.333220+0000","last_undegraded":"2024-09-11T00:30:56.333220+0000","last_fullsized":"2024-09-11T00:30:56.333220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:54.991047+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:41:48.229261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173027+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:30:56.173027+0000","last_peered":"2024-09-11T00:30:56.173027+0000","last_clean":"2024-09-11T00:30:56.173027+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:30:56.173027+0000","last_undegraded":"2024-09-11T00:30:56.173027+0000","last_fullsized":"2024-09-11T00:30:56.173027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997710+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:30:55.997710+0000","last_peered":"2024-09-11T00:30:55.997710+0000","last_clean":"2024-09-11T00:30:55.997710+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:30:55.997710+0000","last_undegraded":"2024-09-11T00:30:55.997710+0000","last_fullsized":"2024-09-11T00:30:55.997710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997737+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:30:55.997737+0000","last_peered":"2024-09-11T00:30:55.997737+0000","last_clean":"2024-09-11T00:30:55.997737+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:30:55.997737+0000","last_undegraded":"2024-09-11T00:30:55.997737+0000","last_fullsized":"2024-09-11T00:30:55.997737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172698+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:30:56.172698+0000","last_peered":"2024-09-11T00:30:56.172698+0000","last_clean":"2024-09-11T00:30:56.172698+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:30:56.172698+0000","last_undegraded":"2024-09-11T00:30:56.172698+0000","last_fullsized":"2024-09-11T00:30:56.172698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997914+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:30:55.997914+0000","last_peered":"2024-09-11T00:30:55.997914+0000","last_clean":"2024-09-11T00:30:55.997914+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:30:55.997914+0000","last_undegraded":"2024-09-11T00:30:55.997914+0000","last_fullsized":"2024-09-11T00:30:55.997914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172531+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:30:56.172531+0000","last_peered":"2024-09-11T00:30:56.172531+0000","last_clean":"2024-09-11T00:30:56.172531+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:30:56.172531+0000","last_undegraded":"2024-09-11T00:30:56.172531+0000","last_fullsized":"2024-09-11T00:30:56.172531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333081+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:30:56.333081+0000","last_peered":"2024-09-11T00:30:56.333081+0000","last_clean":"2024-09-11T00:30:56.333081+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:30:56.333081+0000","last_undegraded":"2024-09-11T00:30:56.333081+0000","last_fullsized":"2024-09-11T00:30:56.333081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509450,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":286720,"data_stored":106265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":307200,"data_stored":108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:31:05.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:31:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.007+0000 7f2dff600640 1 -- 172.21.15.31:0/2725552678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df80c23a0 msgr2=0x7f2df80c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.007+0000 7f2dff600640 1 --2- 172.21.15.31:0/2725552678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df80c23a0 0x7f2df80c30c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2df80c5500 tx=0x7f2de002a5d0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.007+0000 7f2dff600640 1 -- 172.21.15.31:0/2725552678 shutdown_connections 2024-09-11T00:31:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.007+0000 7f2dff600640 1 --2- 172.21.15.31:0/2725552678 >> v2:172.21.15.204:3300/0 conn(0x7f2df81518b0 0x7f2df80c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.007+0000 7f2dff600640 1 --2- 172.21.15.31:0/2725552678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df80c23a0 0x7f2df80c30c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.007+0000 7f2dff600640 1 -- 172.21.15.31:0/2725552678 >> 172.21.15.31:0/2725552678 conn(0x7f2df80a9750 msgr2=0x7f2df80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.007+0000 7f2dff600640 1 -- 172.21.15.31:0/2725552678 shutdown_connections 2024-09-11T00:31:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.007+0000 7f2dff600640 1 -- 172.21.15.31:0/2725552678 wait complete. 2024-09-11T00:31:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.009+0000 7f2dff600640 1 Processor -- start 2024-09-11T00:31:06.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.009+0000 7f2dff600640 1 -- start start 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dff600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df8072480 0x7f2df8153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dff600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2df80c23a0 0x7f2df8147290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dff600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2df81518b0 0x7f2df81eb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dff600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2df8074200 con 0x7f2df8072480 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dff600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2df8074080 con 0x7f2df80c23a0 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dff600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2df8072860 con 0x7f2df81518b0 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dfcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df8072480 0x7f2df8153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dfcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df8072480 0x7f2df8153b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37634/0 (socket says 172.21.15.31:37634) 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dfcc00640 1 -- 172.21.15.31:0/174156413 learned_addr learned my addr 172.21.15.31:0/174156413 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2defe00640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2df80c23a0 0x7f2df8147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dfcc00640 1 -- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2df81518b0 msgr2=0x7f2df81eb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dfcc00640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2df81518b0 0x7f2df81eb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dfcc00640 1 -- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2df80c23a0 msgr2=0x7f2df8147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dfcc00640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2df80c23a0 0x7f2df8147290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2dfcc00640 1 -- 172.21.15.31:0/174156413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df8152130 con 0x7f2df8072480 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.010+0000 7f2defe00640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2df80c23a0 0x7f2df8147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.011+0000 7f2dfcc00640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df8072480 0x7f2df8153b30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2de800c530 tx=0x7f2de800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.011+0000 7f2df5600640 1 -- 172.21.15.31:0/174156413 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de8017070 con 0x7f2df8072480 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.011+0000 7f2df5600640 1 -- 172.21.15.31:0/174156413 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2de80035f0 con 0x7f2df8072480 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.011+0000 7f2df5600640 1 -- 172.21.15.31:0/174156413 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de8003a40 con 0x7f2df8072480 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.011+0000 7f2dff600640 1 -- 172.21.15.31:0/174156413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df8148b90 con 0x7f2df8072480 2024-09-11T00:31:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.011+0000 7f2dff600640 1 -- 172.21.15.31:0/174156413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df8147ba0 con 0x7f2df8072480 2024-09-11T00:31:06.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.012+0000 7f2dff600640 1 -- 172.21.15.31:0/174156413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2df80c3b90 con 0x7f2df8072480 2024-09-11T00:31:06.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.013+0000 7f2df5600640 1 -- 172.21.15.31:0/174156413 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2de8003be0 con 0x7f2df8072480 2024-09-11T00:31:06.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.014+0000 7f2df5600640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2dd4077fa0 0x7f2dd407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.014+0000 7f2df5600640 1 -- 172.21.15.31:0/174156413 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f2de8099760 con 0x7f2df8072480 2024-09-11T00:31:06.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.015+0000 7f2defe00640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2dd4077fa0 0x7f2dd407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.016+0000 7f2defe00640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2dd4077fa0 0x7f2dd407a460 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2de0034000 tx=0x7f2de0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:06.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.018+0000 7f2df5600640 1 -- 172.21.15.31:0/174156413 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2de8061f20 con 0x7f2df8072480 2024-09-11T00:31:06.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.211+0000 7f2dff600640 1 -- 172.21.15.31:0/174156413 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2df8148d60 con 0x7f2dd4077fa0 2024-09-11T00:31:06.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.219+0000 7f2df5600640 1 -- 172.21.15.31:0/174156413 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490296 (secure 0 0 0) 0x7f2df8148d60 con 0x7f2dd4077fa0 2024-09-11T00:31:06.226 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:06.230 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:31:06.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.224+0000 7f2dff600640 1 -- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2dd4077fa0 msgr2=0x7f2dd407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.224+0000 7f2dff600640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2dd4077fa0 0x7f2dd407a460 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2de0034000 tx=0x7f2de0004730 comp rx=0 tx=0).stop 2024-09-11T00:31:06.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.224+0000 7f2dff600640 1 -- 172.21.15.31:0/174156413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df8072480 msgr2=0x7f2df8153b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.224+0000 7f2dff600640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df8072480 0x7f2df8153b30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2de800c530 tx=0x7f2de800c9f0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.225+0000 7f2dff600640 1 -- 172.21.15.31:0/174156413 shutdown_connections 2024-09-11T00:31:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.225+0000 7f2dff600640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2dd4077fa0 0x7f2dd407a460 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.225+0000 7f2dff600640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2df81518b0 0x7f2df81eb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.225+0000 7f2dff600640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2df80c23a0 0x7f2df8147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.225+0000 7f2dff600640 1 --2- 172.21.15.31:0/174156413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2df8072480 0x7f2df8153b30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.225+0000 7f2dff600640 1 -- 172.21.15.31:0/174156413 >> 172.21.15.31:0/174156413 conn(0x7f2df80a9750 msgr2=0x7f2df814d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.225+0000 7f2dff600640 1 -- 172.21.15.31:0/174156413 shutdown_connections 2024-09-11T00:31:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.225+0000 7f2dff600640 1 -- 172.21.15.31:0/174156413 wait complete. 2024-09-11T00:31:06.242 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-11T00:31:04.396383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":191,"num_write_kb":1656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219644,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755076,"statfs":{"total":773094113280,"available":772869197824,"internally_reserved":0,"allocated":3481600,"data_stored":2027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.432387"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719046+0000","last_change":"2024-09-11T00:30:54.983412+0000","last_active":"2024-09-11T00:30:56.719046+0000","last_peered":"2024-09-11T00:30:56.719046+0000","last_clean":"2024-09-11T00:30:56.719046+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:30:56.719046+0000","last_undegraded":"2024-09-11T00:30:56.719046+0000","last_fullsized":"2024-09-11T00:30:56.719046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:51.807846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970097+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:30:55.970097+0000","last_peered":"2024-09-11T00:30:55.970097+0000","last_clean":"2024-09-11T00:30:55.970097+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:30:55.970097+0000","last_undegraded":"2024-09-11T00:30:55.970097+0000","last_fullsized":"2024-09-11T00:30:55.970097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998047+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:30:55.998047+0000","last_peered":"2024-09-11T00:30:55.998047+0000","last_clean":"2024-09-11T00:30:55.998047+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:30:55.998047+0000","last_undegraded":"2024-09-11T00:30:55.998047+0000","last_fullsized":"2024-09-11T00:30:55.998047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000942+0000","last_change":"2024-09-11T00:30:53.984316+0000","last_active":"2024-09-11T00:30:56.000942+0000","last_peered":"2024-09-11T00:30:56.000942+0000","last_clean":"2024-09-11T00:30:56.000942+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:30:56.000942+0000","last_undegraded":"2024-09-11T00:30:56.000942+0000","last_fullsized":"2024-09-11T00:30:56.000942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:14.425265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968662+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:30:55.968662+0000","last_peered":"2024-09-11T00:30:55.968662+0000","last_clean":"2024-09-11T00:30:55.968662+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:30:55.968662+0000","last_undegraded":"2024-09-11T00:30:55.968662+0000","last_fullsized":"2024-09-11T00:30:55.968662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998255+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:30:55.998255+0000","last_peered":"2024-09-11T00:30:55.998255+0000","last_clean":"2024-09-11T00:30:55.998255+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:30:55.998255+0000","last_undegraded":"2024-09-11T00:30:55.998255+0000","last_fullsized":"2024-09-11T00:30:55.998255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998074+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:30:55.998074+0000","last_peered":"2024-09-11T00:30:55.998074+0000","last_clean":"2024-09-11T00:30:55.998074+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:30:55.998074+0000","last_undegraded":"2024-09-11T00:30:55.998074+0000","last_fullsized":"2024-09-11T00:30:55.998074+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172549+0000","last_change":"2024-09-11T00:30:53.984940+0000","last_active":"2024-09-11T00:30:56.172549+0000","last_peered":"2024-09-11T00:30:56.172549+0000","last_clean":"2024-09-11T00:30:56.172549+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:30:56.172549+0000","last_undegraded":"2024-09-11T00:30:56.172549+0000","last_fullsized":"2024-09-11T00:30:56.172549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:19.647165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719205+0000","last_change":"2024-09-11T00:30:54.993570+0000","last_active":"2024-09-11T00:30:56.719205+0000","last_peered":"2024-09-11T00:30:56.719205+0000","last_clean":"2024-09-11T00:30:56.719205+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:30:56.719205+0000","last_undegraded":"2024-09-11T00:30:56.719205+0000","last_fullsized":"2024-09-11T00:30:56.719205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:45.211259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333273+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:30:56.333273+0000","last_peered":"2024-09-11T00:30:56.333273+0000","last_clean":"2024-09-11T00:30:56.333273+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:30:56.333273+0000","last_undegraded":"2024-09-11T00:30:56.333273+0000","last_fullsized":"2024-09-11T00:30:56.333273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719392+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:30:56.719392+0000","last_peered":"2024-09-11T00:30:56.719392+0000","last_clean":"2024-09-11T00:30:56.719392+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:30:56.719392+0000","last_undegraded":"2024-09-11T00:30:56.719392+0000","last_fullsized":"2024-09-11T00:30:56.719392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172696+0000","last_change":"2024-09-11T00:30:53.974699+0000","last_active":"2024-09-11T00:30:56.172696+0000","last_peered":"2024-09-11T00:30:56.172696+0000","last_clean":"2024-09-11T00:30:56.172696+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:30:56.172696+0000","last_undegraded":"2024-09-11T00:30:56.172696+0000","last_fullsized":"2024-09-11T00:30:56.172696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:28.792487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172498+0000","last_change":"2024-09-11T00:30:54.984890+0000","last_active":"2024-09-11T00:30:56.172498+0000","last_peered":"2024-09-11T00:30:56.172498+0000","last_clean":"2024-09-11T00:30:56.172498+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:30:56.172498+0000","last_undegraded":"2024-09-11T00:30:56.172498+0000","last_fullsized":"2024-09-11T00:30:56.172498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:21.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333166+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:30:56.333166+0000","last_peered":"2024-09-11T00:30:56.333166+0000","last_clean":"2024-09-11T00:30:56.333166+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:30:56.333166+0000","last_undegraded":"2024-09-11T00:30:56.333166+0000","last_fullsized":"2024-09-11T00:30:56.333166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968144+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:30:55.968144+0000","last_peered":"2024-09-11T00:30:55.968144+0000","last_clean":"2024-09-11T00:30:55.968144+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:30:55.968144+0000","last_undegraded":"2024-09-11T00:30:55.968144+0000","last_fullsized":"2024-09-11T00:30:55.968144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172339+0000","last_change":"2024-09-11T00:30:53.977743+0000","last_active":"2024-09-11T00:30:56.172339+0000","last_peered":"2024-09-11T00:30:56.172339+0000","last_clean":"2024-09-11T00:30:56.172339+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:30:56.172339+0000","last_undegraded":"2024-09-11T00:30:56.172339+0000","last_fullsized":"2024-09-11T00:30:56.172339+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:20.618615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997709+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:30:55.997709+0000","last_peered":"2024-09-11T00:30:55.997709+0000","last_clean":"2024-09-11T00:30:55.997709+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:30:55.997709+0000","last_undegraded":"2024-09-11T00:30:55.997709+0000","last_fullsized":"2024-09-11T00:30:55.997709+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967362+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:30:55.967362+0000","last_peered":"2024-09-11T00:30:55.967362+0000","last_clean":"2024-09-11T00:30:55.967362+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:30:55.967362+0000","last_undegraded":"2024-09-11T00:30:55.967362+0000","last_fullsized":"2024-09-11T00:30:55.967362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967452+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:30:55.967452+0000","last_peered":"2024-09-11T00:30:55.967452+0000","last_clean":"2024-09-11T00:30:55.967452+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:30:55.967452+0000","last_undegraded":"2024-09-11T00:30:55.967452+0000","last_fullsized":"2024-09-11T00:30:55.967452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967914+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:30:55.967914+0000","last_peered":"2024-09-11T00:30:55.967914+0000","last_clean":"2024-09-11T00:30:55.967914+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:30:55.967914+0000","last_undegraded":"2024-09-11T00:30:55.967914+0000","last_fullsized":"2024-09-11T00:30:55.967914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001419+0000","last_change":"2024-09-11T00:30:54.997397+0000","last_active":"2024-09-11T00:30:56.001419+0000","last_peered":"2024-09-11T00:30:56.001419+0000","last_clean":"2024-09-11T00:30:56.001419+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:30:56.001419+0000","last_undegraded":"2024-09-11T00:30:56.001419+0000","last_fullsized":"2024-09-11T00:30:56.001419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:53.322576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001116+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:30:56.001116+0000","last_peered":"2024-09-11T00:30:56.001116+0000","last_clean":"2024-09-11T00:30:56.001116+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:30:56.001116+0000","last_undegraded":"2024-09-11T00:30:56.001116+0000","last_fullsized":"2024-09-11T00:30:56.001116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970056+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:30:55.970056+0000","last_peered":"2024-09-11T00:30:55.970056+0000","last_clean":"2024-09-11T00:30:55.970056+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:30:55.970056+0000","last_undegraded":"2024-09-11T00:30:55.970056+0000","last_fullsized":"2024-09-11T00:30:55.970056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332817+0000","last_change":"2024-09-11T00:30:53.986858+0000","last_active":"2024-09-11T00:30:56.332817+0000","last_peered":"2024-09-11T00:30:56.332817+0000","last_clean":"2024-09-11T00:30:56.332817+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:30:56.332817+0000","last_undegraded":"2024-09-11T00:30:56.332817+0000","last_fullsized":"2024-09-11T00:30:56.332817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:33.626193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967911+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:30:55.967911+0000","last_peered":"2024-09-11T00:30:55.967911+0000","last_clean":"2024-09-11T00:30:55.967911+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:30:55.967911+0000","last_undegraded":"2024-09-11T00:30:55.967911+0000","last_fullsized":"2024-09-11T00:30:55.967911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968190+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:30:55.968190+0000","last_peered":"2024-09-11T00:30:55.968190+0000","last_clean":"2024-09-11T00:30:55.968190+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:30:55.968190+0000","last_undegraded":"2024-09-11T00:30:55.968190+0000","last_fullsized":"2024-09-11T00:30:55.968190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968850+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:30:55.968850+0000","last_peered":"2024-09-11T00:30:55.968850+0000","last_clean":"2024-09-11T00:30:55.968850+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:30:55.968850+0000","last_undegraded":"2024-09-11T00:30:55.968850+0000","last_fullsized":"2024-09-11T00:30:55.968850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001447+0000","last_change":"2024-09-11T00:30:53.990765+0000","last_active":"2024-09-11T00:30:56.001447+0000","last_peered":"2024-09-11T00:30:56.001447+0000","last_clean":"2024-09-11T00:30:56.001447+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:30:56.001447+0000","last_undegraded":"2024-09-11T00:30:56.001447+0000","last_fullsized":"2024-09-11T00:30:56.001447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:33.720727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719722+0000","last_change":"2024-09-11T00:30:54.984949+0000","last_active":"2024-09-11T00:30:56.719722+0000","last_peered":"2024-09-11T00:30:56.719722+0000","last_clean":"2024-09-11T00:30:56.719722+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:30:56.719722+0000","last_undegraded":"2024-09-11T00:30:56.719722+0000","last_fullsized":"2024-09-11T00:30:56.719722+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:54.346937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967033+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:30:55.967033+0000","last_peered":"2024-09-11T00:30:55.967033+0000","last_clean":"2024-09-11T00:30:55.967033+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:30:55.967033+0000","last_undegraded":"2024-09-11T00:30:55.967033+0000","last_fullsized":"2024-09-11T00:30:55.967033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332357+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:30:56.332357+0000","last_peered":"2024-09-11T00:30:56.332357+0000","last_clean":"2024-09-11T00:30:56.332357+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:30:56.332357+0000","last_undegraded":"2024-09-11T00:30:56.332357+0000","last_fullsized":"2024-09-11T00:30:56.332357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172821+0000","last_change":"2024-09-11T00:30:53.986940+0000","last_active":"2024-09-11T00:30:56.172821+0000","last_peered":"2024-09-11T00:30:56.172821+0000","last_clean":"2024-09-11T00:30:56.172821+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:30:56.172821+0000","last_undegraded":"2024-09-11T00:30:56.172821+0000","last_fullsized":"2024-09-11T00:30:56.172821+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:11.942187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968462+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:30:55.968462+0000","last_peered":"2024-09-11T00:30:55.968462+0000","last_clean":"2024-09-11T00:30:55.968462+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:30:55.968462+0000","last_undegraded":"2024-09-11T00:30:55.968462+0000","last_fullsized":"2024-09-11T00:30:55.968462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968110+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:30:55.968110+0000","last_peered":"2024-09-11T00:30:55.968110+0000","last_clean":"2024-09-11T00:30:55.968110+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:30:55.968110+0000","last_undegraded":"2024-09-11T00:30:55.968110+0000","last_fullsized":"2024-09-11T00:30:55.968110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172483+0000","last_change":"2024-09-11T00:30:53.988539+0000","last_active":"2024-09-11T00:30:56.172483+0000","last_peered":"2024-09-11T00:30:56.172483+0000","last_clean":"2024-09-11T00:30:56.172483+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:30:56.172483+0000","last_undegraded":"2024-09-11T00:30:56.172483+0000","last_fullsized":"2024-09-11T00:30:56.172483+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:19.083664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966824+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:30:55.966824+0000","last_peered":"2024-09-11T00:30:55.966824+0000","last_clean":"2024-09-11T00:30:55.966824+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:30:55.966824+0000","last_undegraded":"2024-09-11T00:30:55.966824+0000","last_fullsized":"2024-09-11T00:30:55.966824+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970081+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:30:55.970081+0000","last_peered":"2024-09-11T00:30:55.970081+0000","last_clean":"2024-09-11T00:30:55.970081+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:30:55.970081+0000","last_undegraded":"2024-09-11T00:30:55.970081+0000","last_fullsized":"2024-09-11T00:30:55.970081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970376+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:30:55.970376+0000","last_peered":"2024-09-11T00:30:55.970376+0000","last_clean":"2024-09-11T00:30:55.970376+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:30:55.970376+0000","last_undegraded":"2024-09-11T00:30:55.970376+0000","last_fullsized":"2024-09-11T00:30:55.970376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968131+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:30:55.968131+0000","last_peered":"2024-09-11T00:30:55.968131+0000","last_clean":"2024-09-11T00:30:55.968131+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:30:55.968131+0000","last_undegraded":"2024-09-11T00:30:55.968131+0000","last_fullsized":"2024-09-11T00:30:55.968131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332511+0000","last_change":"2024-09-11T00:30:54.993118+0000","last_active":"2024-09-11T00:30:56.332511+0000","last_peered":"2024-09-11T00:30:56.332511+0000","last_clean":"2024-09-11T00:30:56.332511+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:30:56.332511+0000","last_undegraded":"2024-09-11T00:30:56.332511+0000","last_fullsized":"2024-09-11T00:30:56.332511+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:07:41.148370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971378+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:30:55.971378+0000","last_peered":"2024-09-11T00:30:55.971378+0000","last_clean":"2024-09-11T00:30:55.971378+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:30:55.971378+0000","last_undegraded":"2024-09-11T00:30:55.971378+0000","last_fullsized":"2024-09-11T00:30:55.971378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968486+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:30:55.968486+0000","last_peered":"2024-09-11T00:30:55.968486+0000","last_clean":"2024-09-11T00:30:55.968486+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:30:55.968486+0000","last_undegraded":"2024-09-11T00:30:55.968486+0000","last_fullsized":"2024-09-11T00:30:55.968486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970903+0000","last_change":"2024-09-11T00:30:53.986843+0000","last_active":"2024-09-11T00:30:55.970903+0000","last_peered":"2024-09-11T00:30:55.970903+0000","last_clean":"2024-09-11T00:30:55.970903+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:30:55.970903+0000","last_undegraded":"2024-09-11T00:30:55.970903+0000","last_fullsized":"2024-09-11T00:30:55.970903+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:44.684154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969549+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:30:55.969549+0000","last_peered":"2024-09-11T00:30:55.969549+0000","last_clean":"2024-09-11T00:30:55.969549+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:30:55.969549+0000","last_undegraded":"2024-09-11T00:30:55.969549+0000","last_fullsized":"2024-09-11T00:30:55.969549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172919+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:30:56.172919+0000","last_peered":"2024-09-11T00:30:56.172919+0000","last_clean":"2024-09-11T00:30:56.172919+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:30:56.172919+0000","last_undegraded":"2024-09-11T00:30:56.172919+0000","last_fullsized":"2024-09-11T00:30:56.172919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997652+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:30:55.997652+0000","last_peered":"2024-09-11T00:30:55.997652+0000","last_clean":"2024-09-11T00:30:55.997652+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:30:55.997652+0000","last_undegraded":"2024-09-11T00:30:55.997652+0000","last_fullsized":"2024-09-11T00:30:55.997652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969390+0000","last_change":"2024-09-11T00:30:53.986896+0000","last_active":"2024-09-11T00:30:55.969390+0000","last_peered":"2024-09-11T00:30:55.969390+0000","last_clean":"2024-09-11T00:30:55.969390+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:30:55.969390+0000","last_undegraded":"2024-09-11T00:30:55.969390+0000","last_fullsized":"2024-09-11T00:30:55.969390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:06.590278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001631+0000","last_change":"2024-09-11T00:30:55.001370+0000","last_active":"2024-09-11T00:30:56.001631+0000","last_peered":"2024-09-11T00:30:56.001631+0000","last_clean":"2024-09-11T00:30:56.001631+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:30:56.001631+0000","last_undegraded":"2024-09-11T00:30:56.001631+0000","last_fullsized":"2024-09-11T00:30:56.001631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:56.592767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333262+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:30:56.333262+0000","last_peered":"2024-09-11T00:30:56.333262+0000","last_clean":"2024-09-11T00:30:56.333262+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:30:56.333262+0000","last_undegraded":"2024-09-11T00:30:56.333262+0000","last_fullsized":"2024-09-11T00:30:56.333262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967594+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:30:55.967594+0000","last_peered":"2024-09-11T00:30:55.967594+0000","last_clean":"2024-09-11T00:30:55.967594+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:30:55.967594+0000","last_undegraded":"2024-09-11T00:30:55.967594+0000","last_fullsized":"2024-09-11T00:30:55.967594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971192+0000","last_change":"2024-09-11T00:30:53.978480+0000","last_active":"2024-09-11T00:30:55.971192+0000","last_peered":"2024-09-11T00:30:55.971192+0000","last_clean":"2024-09-11T00:30:55.971192+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:30:55.971192+0000","last_undegraded":"2024-09-11T00:30:55.971192+0000","last_fullsized":"2024-09-11T00:30:55.971192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:58.900565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001528+0000","last_change":"2024-09-11T00:30:54.997641+0000","last_active":"2024-09-11T00:30:56.001528+0000","last_peered":"2024-09-11T00:30:56.001528+0000","last_clean":"2024-09-11T00:30:56.001528+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:30:56.001528+0000","last_undegraded":"2024-09-11T00:30:56.001528+0000","last_fullsized":"2024-09-11T00:30:56.001528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:18.437490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997311+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:30:55.997311+0000","last_peered":"2024-09-11T00:30:55.997311+0000","last_clean":"2024-09-11T00:30:55.997311+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:30:55.997311+0000","last_undegraded":"2024-09-11T00:30:55.997311+0000","last_fullsized":"2024-09-11T00:30:55.997311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971163+0000","last_change":"2024-09-11T00:30:53.979031+0000","last_active":"2024-09-11T00:30:55.971163+0000","last_peered":"2024-09-11T00:30:55.971163+0000","last_clean":"2024-09-11T00:30:55.971163+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:30:55.971163+0000","last_undegraded":"2024-09-11T00:30:55.971163+0000","last_fullsized":"2024-09-11T00:30:55.971163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:36.604377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967699+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:30:55.967699+0000","last_peered":"2024-09-11T00:30:55.967699+0000","last_clean":"2024-09-11T00:30:55.967699+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:30:55.967699+0000","last_undegraded":"2024-09-11T00:30:55.967699+0000","last_fullsized":"2024-09-11T00:30:55.967699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:39.629122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968089+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:30:55.968089+0000","last_peered":"2024-09-11T00:30:55.968089+0000","last_clean":"2024-09-11T00:30:55.968089+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:30:55.968089+0000","last_undegraded":"2024-09-11T00:30:55.968089+0000","last_fullsized":"2024-09-11T00:30:55.968089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966595+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:30:55.966595+0000","last_peered":"2024-09-11T00:30:55.966595+0000","last_clean":"2024-09-11T00:30:55.966595+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:30:55.966595+0000","last_undegraded":"2024-09-11T00:30:55.966595+0000","last_fullsized":"2024-09-11T00:30:55.966595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"26'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998301+0000","last_change":"2024-09-11T00:30:53.989299+0000","last_active":"2024-09-11T00:30:55.998301+0000","last_peered":"2024-09-11T00:30:55.998301+0000","last_clean":"2024-09-11T00:30:55.998301+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:30:55.998301+0000","last_undegraded":"2024-09-11T00:30:55.998301+0000","last_fullsized":"2024-09-11T00:30:55.998301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:34:49.298915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332781+0000","last_change":"2024-09-11T00:30:54.990026+0000","last_active":"2024-09-11T00:30:56.332781+0000","last_peered":"2024-09-11T00:30:56.332781+0000","last_clean":"2024-09-11T00:30:56.332781+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:30:56.332781+0000","last_undegraded":"2024-09-11T00:30:56.332781+0000","last_fullsized":"2024-09-11T00:30:56.332781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:02:10.800530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969545+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:30:55.969545+0000","last_peered":"2024-09-11T00:30:55.969545+0000","last_clean":"2024-09-11T00:30:55.969545+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:30:55.969545+0000","last_undegraded":"2024-09-11T00:30:55.969545+0000","last_fullsized":"2024-09-11T00:30:55.969545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967491+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:30:55.967491+0000","last_peered":"2024-09-11T00:30:55.967491+0000","last_clean":"2024-09-11T00:30:55.967491+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:30:55.967491+0000","last_undegraded":"2024-09-11T00:30:55.967491+0000","last_fullsized":"2024-09-11T00:30:55.967491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719269+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:30:56.719269+0000","last_peered":"2024-09-11T00:30:56.719269+0000","last_clean":"2024-09-11T00:30:56.719269+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:30:56.719269+0000","last_undegraded":"2024-09-11T00:30:56.719269+0000","last_fullsized":"2024-09-11T00:30:56.719269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968267+0000","last_change":"2024-09-11T00:30:53.982142+0000","last_active":"2024-09-11T00:30:55.968267+0000","last_peered":"2024-09-11T00:30:55.968267+0000","last_clean":"2024-09-11T00:30:55.968267+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:30:55.968267+0000","last_undegraded":"2024-09-11T00:30:55.968267+0000","last_fullsized":"2024-09-11T00:30:55.968267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:41.558944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998013+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:30:55.998013+0000","last_peered":"2024-09-11T00:30:55.998013+0000","last_clean":"2024-09-11T00:30:55.998013+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:30:55.998013+0000","last_undegraded":"2024-09-11T00:30:55.998013+0000","last_fullsized":"2024-09-11T00:30:55.998013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968362+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:30:55.968362+0000","last_peered":"2024-09-11T00:30:55.968362+0000","last_clean":"2024-09-11T00:30:55.968362+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:30:55.968362+0000","last_undegraded":"2024-09-11T00:30:55.968362+0000","last_fullsized":"2024-09-11T00:30:55.968362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971436+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:30:55.971436+0000","last_peered":"2024-09-11T00:30:55.971436+0000","last_clean":"2024-09-11T00:30:55.971436+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:30:55.971436+0000","last_undegraded":"2024-09-11T00:30:55.971436+0000","last_fullsized":"2024-09-11T00:30:55.971436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172455+0000","last_change":"2024-09-11T00:30:53.989292+0000","last_active":"2024-09-11T00:30:56.172455+0000","last_peered":"2024-09-11T00:30:56.172455+0000","last_clean":"2024-09-11T00:30:56.172455+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:30:56.172455+0000","last_undegraded":"2024-09-11T00:30:56.172455+0000","last_fullsized":"2024-09-11T00:30:56.172455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:44.071338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968473+0000","last_change":"2024-09-11T00:30:54.981630+0000","last_active":"2024-09-11T00:30:55.968473+0000","last_peered":"2024-09-11T00:30:55.968473+0000","last_clean":"2024-09-11T00:30:55.968473+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:30:55.968473+0000","last_undegraded":"2024-09-11T00:30:55.968473+0000","last_fullsized":"2024-09-11T00:30:55.968473+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:12.832327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333061+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:30:56.333061+0000","last_peered":"2024-09-11T00:30:56.333061+0000","last_clean":"2024-09-11T00:30:56.333061+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:30:56.333061+0000","last_undegraded":"2024-09-11T00:30:56.333061+0000","last_fullsized":"2024-09-11T00:30:56.333061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997856+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:30:55.997856+0000","last_peered":"2024-09-11T00:30:55.997856+0000","last_clean":"2024-09-11T00:30:55.997856+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:30:55.997856+0000","last_undegraded":"2024-09-11T00:30:55.997856+0000","last_fullsized":"2024-09-11T00:30:55.997856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997848+0000","last_change":"2024-09-11T00:30:53.984013+0000","last_active":"2024-09-11T00:30:55.997848+0000","last_peered":"2024-09-11T00:30:55.997848+0000","last_clean":"2024-09-11T00:30:55.997848+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:30:55.997848+0000","last_undegraded":"2024-09-11T00:30:55.997848+0000","last_fullsized":"2024-09-11T00:30:55.997848+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:49.963623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719176+0000","last_change":"2024-09-11T00:30:54.996860+0000","last_active":"2024-09-11T00:30:56.719176+0000","last_peered":"2024-09-11T00:30:56.719176+0000","last_clean":"2024-09-11T00:30:56.719176+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:30:56.719176+0000","last_undegraded":"2024-09-11T00:30:56.719176+0000","last_fullsized":"2024-09-11T00:30:56.719176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:43.516748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333114+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:30:56.333114+0000","last_peered":"2024-09-11T00:30:56.333114+0000","last_clean":"2024-09-11T00:30:56.333114+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:30:56.333114+0000","last_undegraded":"2024-09-11T00:30:56.333114+0000","last_fullsized":"2024-09-11T00:30:56.333114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970314+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:30:55.970314+0000","last_peered":"2024-09-11T00:30:55.970314+0000","last_clean":"2024-09-11T00:30:55.970314+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:30:55.970314+0000","last_undegraded":"2024-09-11T00:30:55.970314+0000","last_fullsized":"2024-09-11T00:30:55.970314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966317+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:30:55.966317+0000","last_peered":"2024-09-11T00:30:55.966317+0000","last_clean":"2024-09-11T00:30:55.966317+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:30:55.966317+0000","last_undegraded":"2024-09-11T00:30:55.966317+0000","last_fullsized":"2024-09-11T00:30:55.966317+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:19.891052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967870+0000","last_change":"2024-09-11T00:30:54.993736+0000","last_active":"2024-09-11T00:30:55.967870+0000","last_peered":"2024-09-11T00:30:55.967870+0000","last_clean":"2024-09-11T00:30:55.967870+0000","last_became_active":"2024-09-11T00:30:54.993470+0000","last_became_peered":"2024-09-11T00:30:54.993470+0000","last_unstale":"2024-09-11T00:30:55.967870+0000","last_undegraded":"2024-09-11T00:30:55.967870+0000","last_fullsized":"2024-09-11T00:30:55.967870+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:24.712448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173170+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:30:56.173170+0000","last_peered":"2024-09-11T00:30:56.173170+0000","last_clean":"2024-09-11T00:30:56.173170+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:30:56.173170+0000","last_undegraded":"2024-09-11T00:30:56.173170+0000","last_fullsized":"2024-09-11T00:30:56.173170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333218+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:30:56.333218+0000","last_peered":"2024-09-11T00:30:56.333218+0000","last_clean":"2024-09-11T00:30:56.333218+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:30:56.333218+0000","last_undegraded":"2024-09-11T00:30:56.333218+0000","last_fullsized":"2024-09-11T00:30:56.333218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719247+0000","last_change":"2024-09-11T00:30:53.983352+0000","last_active":"2024-09-11T00:30:56.719247+0000","last_peered":"2024-09-11T00:30:56.719247+0000","last_clean":"2024-09-11T00:30:56.719247+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:30:56.719247+0000","last_undegraded":"2024-09-11T00:30:56.719247+0000","last_fullsized":"2024-09-11T00:30:56.719247+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:43.826114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970030+0000","last_change":"2024-09-11T00:30:54.987533+0000","last_active":"2024-09-11T00:30:55.970030+0000","last_peered":"2024-09-11T00:30:55.970030+0000","last_clean":"2024-09-11T00:30:55.970030+0000","last_became_active":"2024-09-11T00:30:54.987253+0000","last_became_peered":"2024-09-11T00:30:54.987253+0000","last_unstale":"2024-09-11T00:30:55.970030+0000","last_undegraded":"2024-09-11T00:30:55.970030+0000","last_fullsized":"2024-09-11T00:30:55.970030+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.923656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966934+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:30:55.966934+0000","last_peered":"2024-09-11T00:30:55.966934+0000","last_clean":"2024-09-11T00:30:55.966934+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:30:55.966934+0000","last_undegraded":"2024-09-11T00:30:55.966934+0000","last_fullsized":"2024-09-11T00:30:55.966934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968528+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:30:55.968528+0000","last_peered":"2024-09-11T00:30:55.968528+0000","last_clean":"2024-09-11T00:30:55.968528+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:30:55.968528+0000","last_undegraded":"2024-09-11T00:30:55.968528+0000","last_fullsized":"2024-09-11T00:30:55.968528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997651+0000","last_change":"2024-09-11T00:30:53.980703+0000","last_active":"2024-09-11T00:30:55.997651+0000","last_peered":"2024-09-11T00:30:55.997651+0000","last_clean":"2024-09-11T00:30:55.997651+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:30:55.997651+0000","last_undegraded":"2024-09-11T00:30:55.997651+0000","last_fullsized":"2024-09-11T00:30:55.997651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:51.050512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967820+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:30:55.967820+0000","last_peered":"2024-09-11T00:30:55.967820+0000","last_clean":"2024-09-11T00:30:55.967820+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:30:55.967820+0000","last_undegraded":"2024-09-11T00:30:55.967820+0000","last_fullsized":"2024-09-11T00:30:55.967820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719139+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:30:56.719139+0000","last_peered":"2024-09-11T00:30:56.719139+0000","last_clean":"2024-09-11T00:30:56.719139+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:30:56.719139+0000","last_undegraded":"2024-09-11T00:30:56.719139+0000","last_fullsized":"2024-09-11T00:30:56.719139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333070+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:30:56.333070+0000","last_peered":"2024-09-11T00:30:56.333070+0000","last_clean":"2024-09-11T00:30:56.333070+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:30:56.333070+0000","last_undegraded":"2024-09-11T00:30:56.333070+0000","last_fullsized":"2024-09-11T00:30:56.333070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997937+0000","last_change":"2024-09-11T00:30:53.987829+0000","last_active":"2024-09-11T00:30:55.997937+0000","last_peered":"2024-09-11T00:30:55.997937+0000","last_clean":"2024-09-11T00:30:55.997937+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:30:55.997937+0000","last_undegraded":"2024-09-11T00:30:55.997937+0000","last_fullsized":"2024-09-11T00:30:55.997937+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:57:43.792089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967657+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:30:55.967657+0000","last_peered":"2024-09-11T00:30:55.967657+0000","last_clean":"2024-09-11T00:30:55.967657+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:30:55.967657+0000","last_undegraded":"2024-09-11T00:30:55.967657+0000","last_fullsized":"2024-09-11T00:30:55.967657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967382+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:30:55.967382+0000","last_peered":"2024-09-11T00:30:55.967382+0000","last_clean":"2024-09-11T00:30:55.967382+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:30:55.967382+0000","last_undegraded":"2024-09-11T00:30:55.967382+0000","last_fullsized":"2024-09-11T00:30:55.967382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967492+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:30:55.967492+0000","last_peered":"2024-09-11T00:30:55.967492+0000","last_clean":"2024-09-11T00:30:55.967492+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:30:55.967492+0000","last_undegraded":"2024-09-11T00:30:55.967492+0000","last_fullsized":"2024-09-11T00:30:55.967492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970951+0000","last_change":"2024-09-11T00:30:53.984057+0000","last_active":"2024-09-11T00:30:55.970951+0000","last_peered":"2024-09-11T00:30:55.970951+0000","last_clean":"2024-09-11T00:30:55.970951+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:30:55.970951+0000","last_undegraded":"2024-09-11T00:30:55.970951+0000","last_fullsized":"2024-09-11T00:30:55.970951+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:10.477916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969373+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:30:55.969373+0000","last_peered":"2024-09-11T00:30:55.969373+0000","last_clean":"2024-09-11T00:30:55.969373+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:30:55.969373+0000","last_undegraded":"2024-09-11T00:30:55.969373+0000","last_fullsized":"2024-09-11T00:30:55.969373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969960+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:30:55.969960+0000","last_peered":"2024-09-11T00:30:55.969960+0000","last_clean":"2024-09-11T00:30:55.969960+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:30:55.969960+0000","last_undegraded":"2024-09-11T00:30:55.969960+0000","last_fullsized":"2024-09-11T00:30:55.969960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001240+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:30:56.001240+0000","last_peered":"2024-09-11T00:30:56.001240+0000","last_clean":"2024-09-11T00:30:56.001240+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:30:56.001240+0000","last_undegraded":"2024-09-11T00:30:56.001240+0000","last_fullsized":"2024-09-11T00:30:56.001240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968074+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:30:55.968074+0000","last_peered":"2024-09-11T00:30:55.968074+0000","last_clean":"2024-09-11T00:30:55.968074+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:30:55.968074+0000","last_undegraded":"2024-09-11T00:30:55.968074+0000","last_fullsized":"2024-09-11T00:30:55.968074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969893+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:30:55.969893+0000","last_peered":"2024-09-11T00:30:55.969893+0000","last_clean":"2024-09-11T00:30:55.969893+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:30:55.969893+0000","last_undegraded":"2024-09-11T00:30:55.969893+0000","last_fullsized":"2024-09-11T00:30:55.969893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332978+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:30:56.332978+0000","last_peered":"2024-09-11T00:30:56.332978+0000","last_clean":"2024-09-11T00:30:56.332978+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:30:56.332978+0000","last_undegraded":"2024-09-11T00:30:56.332978+0000","last_fullsized":"2024-09-11T00:30:56.332978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173134+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:30:56.173134+0000","last_peered":"2024-09-11T00:30:56.173134+0000","last_clean":"2024-09-11T00:30:56.173134+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:30:56.173134+0000","last_undegraded":"2024-09-11T00:30:56.173134+0000","last_fullsized":"2024-09-11T00:30:56.173134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969278+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:30:55.969278+0000","last_peered":"2024-09-11T00:30:55.969278+0000","last_clean":"2024-09-11T00:30:55.969278+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:30:55.969278+0000","last_undegraded":"2024-09-11T00:30:55.969278+0000","last_fullsized":"2024-09-11T00:30:55.969278+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969704+0000","last_change":"2024-09-11T00:30:54.986017+0000","last_active":"2024-09-11T00:30:55.969704+0000","last_peered":"2024-09-11T00:30:55.969704+0000","last_clean":"2024-09-11T00:30:55.969704+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:30:55.969704+0000","last_undegraded":"2024-09-11T00:30:55.969704+0000","last_fullsized":"2024-09-11T00:30:55.969704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:58:12.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967364+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:30:55.967364+0000","last_peered":"2024-09-11T00:30:55.967364+0000","last_clean":"2024-09-11T00:30:55.967364+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:30:55.967364+0000","last_undegraded":"2024-09-11T00:30:55.967364+0000","last_fullsized":"2024-09-11T00:30:55.967364+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001702+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:30:56.001702+0000","last_peered":"2024-09-11T00:30:56.001702+0000","last_clean":"2024-09-11T00:30:56.001702+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:30:56.001702+0000","last_undegraded":"2024-09-11T00:30:56.001702+0000","last_fullsized":"2024-09-11T00:30:56.001702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172588+0000","last_change":"2024-09-11T00:30:53.988462+0000","last_active":"2024-09-11T00:30:56.172588+0000","last_peered":"2024-09-11T00:30:56.172588+0000","last_clean":"2024-09-11T00:30:56.172588+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:30:56.172588+0000","last_undegraded":"2024-09-11T00:30:56.172588+0000","last_fullsized":"2024-09-11T00:30:56.172588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:22.385557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966521+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:30:55.966521+0000","last_peered":"2024-09-11T00:30:55.966521+0000","last_clean":"2024-09-11T00:30:55.966521+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:30:55.966521+0000","last_undegraded":"2024-09-11T00:30:55.966521+0000","last_fullsized":"2024-09-11T00:30:55.966521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998206+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:30:55.998206+0000","last_peered":"2024-09-11T00:30:55.998206+0000","last_clean":"2024-09-11T00:30:55.998206+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:30:55.998206+0000","last_undegraded":"2024-09-11T00:30:55.998206+0000","last_fullsized":"2024-09-11T00:30:55.998206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332956+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:30:56.332956+0000","last_peered":"2024-09-11T00:30:56.332956+0000","last_clean":"2024-09-11T00:30:56.332956+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:30:56.332956+0000","last_undegraded":"2024-09-11T00:30:56.332956+0000","last_fullsized":"2024-09-11T00:30:56.332956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000726+0000","last_change":"2024-09-11T00:30:53.986646+0000","last_active":"2024-09-11T00:30:56.000726+0000","last_peered":"2024-09-11T00:30:56.000726+0000","last_clean":"2024-09-11T00:30:56.000726+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:30:56.000726+0000","last_undegraded":"2024-09-11T00:30:56.000726+0000","last_fullsized":"2024-09-11T00:30:56.000726+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:27.447940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172237+0000","last_change":"2024-09-11T00:30:54.982486+0000","last_active":"2024-09-11T00:30:56.172237+0000","last_peered":"2024-09-11T00:30:56.172237+0000","last_clean":"2024-09-11T00:30:56.172237+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:30:56.172237+0000","last_undegraded":"2024-09-11T00:30:56.172237+0000","last_fullsized":"2024-09-11T00:30:56.172237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:41.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173110+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:30:56.173110+0000","last_peered":"2024-09-11T00:30:56.173110+0000","last_clean":"2024-09-11T00:30:56.173110+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:30:56.173110+0000","last_undegraded":"2024-09-11T00:30:56.173110+0000","last_fullsized":"2024-09-11T00:30:56.173110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719858+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:30:56.719858+0000","last_peered":"2024-09-11T00:30:56.719858+0000","last_clean":"2024-09-11T00:30:56.719858+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:30:56.719858+0000","last_undegraded":"2024-09-11T00:30:56.719858+0000","last_fullsized":"2024-09-11T00:30:56.719858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968520+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:30:55.968520+0000","last_peered":"2024-09-11T00:30:55.968520+0000","last_clean":"2024-09-11T00:30:55.968520+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:30:55.968520+0000","last_undegraded":"2024-09-11T00:30:55.968520+0000","last_fullsized":"2024-09-11T00:30:55.968520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173273+0000","last_change":"2024-09-11T00:30:54.985091+0000","last_active":"2024-09-11T00:30:56.173273+0000","last_peered":"2024-09-11T00:30:56.173273+0000","last_clean":"2024-09-11T00:30:56.173273+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:30:56.173273+0000","last_undegraded":"2024-09-11T00:30:56.173273+0000","last_fullsized":"2024-09-11T00:30:56.173273+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:33.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968811+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:30:55.968811+0000","last_peered":"2024-09-11T00:30:55.968811+0000","last_clean":"2024-09-11T00:30:55.968811+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:30:55.968811+0000","last_undegraded":"2024-09-11T00:30:55.968811+0000","last_fullsized":"2024-09-11T00:30:55.968811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966522+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:30:55.966522+0000","last_peered":"2024-09-11T00:30:55.966522+0000","last_clean":"2024-09-11T00:30:55.966522+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:30:55.966522+0000","last_undegraded":"2024-09-11T00:30:55.966522+0000","last_fullsized":"2024-09-11T00:30:55.966522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971377+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:30:55.971377+0000","last_peered":"2024-09-11T00:30:55.971377+0000","last_clean":"2024-09-11T00:30:55.971377+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:30:55.971377+0000","last_undegraded":"2024-09-11T00:30:55.971377+0000","last_fullsized":"2024-09-11T00:30:55.971377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:11.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967648+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:30:55.967648+0000","last_peered":"2024-09-11T00:30:55.967648+0000","last_clean":"2024-09-11T00:30:55.967648+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:30:55.967648+0000","last_undegraded":"2024-09-11T00:30:55.967648+0000","last_fullsized":"2024-09-11T00:30:55.967648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968196+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:30:55.968196+0000","last_peered":"2024-09-11T00:30:55.968196+0000","last_clean":"2024-09-11T00:30:55.968196+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:30:55.968196+0000","last_undegraded":"2024-09-11T00:30:55.968196+0000","last_fullsized":"2024-09-11T00:30:55.968196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969528+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:30:55.969528+0000","last_peered":"2024-09-11T00:30:55.969528+0000","last_clean":"2024-09-11T00:30:55.969528+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:30:55.969528+0000","last_undegraded":"2024-09-11T00:30:55.969528+0000","last_fullsized":"2024-09-11T00:30:55.969528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967369+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:30:55.967369+0000","last_peered":"2024-09-11T00:30:55.967369+0000","last_clean":"2024-09-11T00:30:55.967369+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:30:55.967369+0000","last_undegraded":"2024-09-11T00:30:55.967369+0000","last_fullsized":"2024-09-11T00:30:55.967369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969812+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:30:55.969812+0000","last_peered":"2024-09-11T00:30:55.969812+0000","last_clean":"2024-09-11T00:30:55.969812+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:30:55.969812+0000","last_undegraded":"2024-09-11T00:30:55.969812+0000","last_fullsized":"2024-09-11T00:30:55.969812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969509+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:30:55.969509+0000","last_peered":"2024-09-11T00:30:55.969509+0000","last_clean":"2024-09-11T00:30:55.969509+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:30:55.969509+0000","last_undegraded":"2024-09-11T00:30:55.969509+0000","last_fullsized":"2024-09-11T00:30:55.969509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173337+0000","last_change":"2024-09-11T00:30:54.998464+0000","last_active":"2024-09-11T00:30:56.173337+0000","last_peered":"2024-09-11T00:30:56.173337+0000","last_clean":"2024-09-11T00:30:56.173337+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:30:56.173337+0000","last_undegraded":"2024-09-11T00:30:56.173337+0000","last_fullsized":"2024-09-11T00:30:56.173337+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:56.469791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998082+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:30:55.998082+0000","last_peered":"2024-09-11T00:30:55.998082+0000","last_clean":"2024-09-11T00:30:55.998082+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:30:55.998082+0000","last_undegraded":"2024-09-11T00:30:55.998082+0000","last_fullsized":"2024-09-11T00:30:55.998082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719278+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:30:56.719278+0000","last_peered":"2024-09-11T00:30:56.719278+0000","last_clean":"2024-09-11T00:30:56.719278+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:30:56.719278+0000","last_undegraded":"2024-09-11T00:30:56.719278+0000","last_fullsized":"2024-09-11T00:30:56.719278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001549+0000","last_change":"2024-09-11T00:30:53.987612+0000","last_active":"2024-09-11T00:30:56.001549+0000","last_peered":"2024-09-11T00:30:56.001549+0000","last_clean":"2024-09-11T00:30:56.001549+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:30:56.001549+0000","last_undegraded":"2024-09-11T00:30:56.001549+0000","last_fullsized":"2024-09-11T00:30:56.001549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:16.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970084+0000","last_change":"2024-09-11T00:30:54.989925+0000","last_active":"2024-09-11T00:30:55.970084+0000","last_peered":"2024-09-11T00:30:55.970084+0000","last_clean":"2024-09-11T00:30:55.970084+0000","last_became_active":"2024-09-11T00:30:54.989401+0000","last_became_peered":"2024-09-11T00:30:54.989401+0000","last_unstale":"2024-09-11T00:30:55.970084+0000","last_undegraded":"2024-09-11T00:30:55.970084+0000","last_fullsized":"2024-09-11T00:30:55.970084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:14.267175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172747+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:30:56.172747+0000","last_peered":"2024-09-11T00:30:56.172747+0000","last_clean":"2024-09-11T00:30:56.172747+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:30:56.172747+0000","last_undegraded":"2024-09-11T00:30:56.172747+0000","last_fullsized":"2024-09-11T00:30:56.172747+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969929+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:30:55.969929+0000","last_peered":"2024-09-11T00:30:55.969929+0000","last_clean":"2024-09-11T00:30:55.969929+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:30:55.969929+0000","last_undegraded":"2024-09-11T00:30:55.969929+0000","last_fullsized":"2024-09-11T00:30:55.969929+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000911+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:30:56.000911+0000","last_peered":"2024-09-11T00:30:56.000911+0000","last_clean":"2024-09-11T00:30:56.000911+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:30:56.000911+0000","last_undegraded":"2024-09-11T00:30:56.000911+0000","last_fullsized":"2024-09-11T00:30:56.000911+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172912+0000","last_change":"2024-09-11T00:30:54.979133+0000","last_active":"2024-09-11T00:30:56.172912+0000","last_peered":"2024-09-11T00:30:56.172912+0000","last_clean":"2024-09-11T00:30:56.172912+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:30:56.172912+0000","last_undegraded":"2024-09-11T00:30:56.172912+0000","last_fullsized":"2024-09-11T00:30:56.172912+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:20.961678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967251+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:30:55.967251+0000","last_peered":"2024-09-11T00:30:55.967251+0000","last_clean":"2024-09-11T00:30:55.967251+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:30:55.967251+0000","last_undegraded":"2024-09-11T00:30:55.967251+0000","last_fullsized":"2024-09-11T00:30:55.967251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997885+0000","last_change":"2024-09-11T00:30:54.988345+0000","last_active":"2024-09-11T00:30:55.997885+0000","last_peered":"2024-09-11T00:30:55.997885+0000","last_clean":"2024-09-11T00:30:55.997885+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:30:55.997885+0000","last_undegraded":"2024-09-11T00:30:55.997885+0000","last_fullsized":"2024-09-11T00:30:55.997885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.112194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998078+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:30:55.998078+0000","last_peered":"2024-09-11T00:30:55.998078+0000","last_clean":"2024-09-11T00:30:55.998078+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:30:55.998078+0000","last_undegraded":"2024-09-11T00:30:55.998078+0000","last_fullsized":"2024-09-11T00:30:55.998078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332489+0000","last_change":"2024-09-11T00:30:54.981185+0000","last_active":"2024-09-11T00:30:56.332489+0000","last_peered":"2024-09-11T00:30:56.332489+0000","last_clean":"2024-09-11T00:30:56.332489+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:30:56.332489+0000","last_undegraded":"2024-09-11T00:30:56.332489+0000","last_fullsized":"2024-09-11T00:30:56.332489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:45.740467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971241+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:30:55.971241+0000","last_peered":"2024-09-11T00:30:55.971241+0000","last_clean":"2024-09-11T00:30:55.971241+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:30:55.971241+0000","last_undegraded":"2024-09-11T00:30:55.971241+0000","last_fullsized":"2024-09-11T00:30:55.971241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332647+0000","last_change":"2024-09-11T00:30:54.992763+0000","last_active":"2024-09-11T00:30:56.332647+0000","last_peered":"2024-09-11T00:30:56.332647+0000","last_clean":"2024-09-11T00:30:56.332647+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:30:56.332647+0000","last_undegraded":"2024-09-11T00:30:56.332647+0000","last_fullsized":"2024-09-11T00:30:56.332647+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:02.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971179+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:30:55.971179+0000","last_peered":"2024-09-11T00:30:55.971179+0000","last_clean":"2024-09-11T00:30:55.971179+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:30:55.971179+0000","last_undegraded":"2024-09-11T00:30:55.971179+0000","last_fullsized":"2024-09-11T00:30:55.971179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966440+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:30:55.966440+0000","last_peered":"2024-09-11T00:30:55.966440+0000","last_clean":"2024-09-11T00:30:55.966440+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:30:55.966440+0000","last_undegraded":"2024-09-11T00:30:55.966440+0000","last_fullsized":"2024-09-11T00:30:55.966440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173066+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:30:56.173066+0000","last_peered":"2024-09-11T00:30:56.173066+0000","last_clean":"2024-09-11T00:30:56.173066+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:30:56.173066+0000","last_undegraded":"2024-09-11T00:30:56.173066+0000","last_fullsized":"2024-09-11T00:30:56.173066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172366+0000","last_change":"2024-09-11T00:30:54.984447+0000","last_active":"2024-09-11T00:30:56.172366+0000","last_peered":"2024-09-11T00:30:56.172366+0000","last_clean":"2024-09-11T00:30:56.172366+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:30:56.172366+0000","last_undegraded":"2024-09-11T00:30:56.172366+0000","last_fullsized":"2024-09-11T00:30:56.172366+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:49.024022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998072+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:30:55.998072+0000","last_peered":"2024-09-11T00:30:55.998072+0000","last_clean":"2024-09-11T00:30:55.998072+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:30:55.998072+0000","last_undegraded":"2024-09-11T00:30:55.998072+0000","last_fullsized":"2024-09-11T00:30:55.998072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001507+0000","last_change":"2024-09-11T00:30:54.996091+0000","last_active":"2024-09-11T00:30:56.001507+0000","last_peered":"2024-09-11T00:30:56.001507+0000","last_clean":"2024-09-11T00:30:56.001507+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:30:56.001507+0000","last_undegraded":"2024-09-11T00:30:56.001507+0000","last_fullsized":"2024-09-11T00:30:56.001507+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:42.987336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966504+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:30:55.966504+0000","last_peered":"2024-09-11T00:30:55.966504+0000","last_clean":"2024-09-11T00:30:55.966504+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:30:55.966504+0000","last_undegraded":"2024-09-11T00:30:55.966504+0000","last_fullsized":"2024-09-11T00:30:55.966504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968347+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:30:55.968347+0000","last_peered":"2024-09-11T00:30:55.968347+0000","last_clean":"2024-09-11T00:30:55.968347+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:30:55.968347+0000","last_undegraded":"2024-09-11T00:30:55.968347+0000","last_fullsized":"2024-09-11T00:30:55.968347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971167+0000","last_change":"2024-09-11T00:30:54.981799+0000","last_active":"2024-09-11T00:30:55.971167+0000","last_peered":"2024-09-11T00:30:55.971167+0000","last_clean":"2024-09-11T00:30:55.971167+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:30:55.971167+0000","last_undegraded":"2024-09-11T00:30:55.971167+0000","last_fullsized":"2024-09-11T00:30:55.971167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:42.670629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719169+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:30:56.719169+0000","last_peered":"2024-09-11T00:30:56.719169+0000","last_clean":"2024-09-11T00:30:56.719169+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:30:56.719169+0000","last_undegraded":"2024-09-11T00:30:56.719169+0000","last_fullsized":"2024-09-11T00:30:56.719169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001453+0000","last_change":"2024-09-11T00:30:54.992697+0000","last_active":"2024-09-11T00:30:56.001453+0000","last_peered":"2024-09-11T00:30:56.001453+0000","last_clean":"2024-09-11T00:30:56.001453+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:30:56.001453+0000","last_undegraded":"2024-09-11T00:30:56.001453+0000","last_fullsized":"2024-09-11T00:30:56.001453+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:54:07.227169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969635+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:30:55.969635+0000","last_peered":"2024-09-11T00:30:55.969635+0000","last_clean":"2024-09-11T00:30:55.969635+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:55.969635+0000","last_undegraded":"2024-09-11T00:30:55.969635+0000","last_fullsized":"2024-09-11T00:30:55.969635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172413+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:30:56.172413+0000","last_peered":"2024-09-11T00:30:56.172413+0000","last_clean":"2024-09-11T00:30:56.172413+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:30:56.172413+0000","last_undegraded":"2024-09-11T00:30:56.172413+0000","last_fullsized":"2024-09-11T00:30:56.172413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332844+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:30:56.332844+0000","last_peered":"2024-09-11T00:30:56.332844+0000","last_clean":"2024-09-11T00:30:56.332844+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:30:56.332844+0000","last_undegraded":"2024-09-11T00:30:56.332844+0000","last_fullsized":"2024-09-11T00:30:56.332844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332895+0000","last_change":"2024-09-11T00:30:54.985827+0000","last_active":"2024-09-11T00:30:56.332895+0000","last_peered":"2024-09-11T00:30:56.332895+0000","last_clean":"2024-09-11T00:30:56.332895+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:30:56.332895+0000","last_undegraded":"2024-09-11T00:30:56.332895+0000","last_fullsized":"2024-09-11T00:30:56.332895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:32.027117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968441+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:30:55.968441+0000","last_peered":"2024-09-11T00:30:55.968441+0000","last_clean":"2024-09-11T00:30:55.968441+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:30:55.968441+0000","last_undegraded":"2024-09-11T00:30:55.968441+0000","last_fullsized":"2024-09-11T00:30:55.968441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172255+0000","last_change":"2024-09-11T00:30:54.989906+0000","last_active":"2024-09-11T00:30:56.172255+0000","last_peered":"2024-09-11T00:30:56.172255+0000","last_clean":"2024-09-11T00:30:56.172255+0000","last_became_active":"2024-09-11T00:30:54.989376+0000","last_became_peered":"2024-09-11T00:30:54.989376+0000","last_unstale":"2024-09-11T00:30:56.172255+0000","last_undegraded":"2024-09-11T00:30:56.172255+0000","last_fullsized":"2024-09-11T00:30:56.172255+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:18:00.138941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001665+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:30:56.001665+0000","last_peered":"2024-09-11T00:30:56.001665+0000","last_clean":"2024-09-11T00:30:56.001665+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:30:56.001665+0000","last_undegraded":"2024-09-11T00:30:56.001665+0000","last_fullsized":"2024-09-11T00:30:56.001665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968974+0000","last_change":"2024-09-11T00:30:54.993156+0000","last_active":"2024-09-11T00:30:55.968974+0000","last_peered":"2024-09-11T00:30:55.968974+0000","last_clean":"2024-09-11T00:30:55.968974+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:30:55.968974+0000","last_undegraded":"2024-09-11T00:30:55.968974+0000","last_fullsized":"2024-09-11T00:30:55.968974+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:25.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968016+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:30:55.968016+0000","last_peered":"2024-09-11T00:30:55.968016+0000","last_clean":"2024-09-11T00:30:55.968016+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:30:55.968016+0000","last_undegraded":"2024-09-11T00:30:55.968016+0000","last_fullsized":"2024-09-11T00:30:55.968016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970335+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:30:55.970335+0000","last_peered":"2024-09-11T00:30:55.970335+0000","last_clean":"2024-09-11T00:30:55.970335+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:30:55.970335+0000","last_undegraded":"2024-09-11T00:30:55.970335+0000","last_fullsized":"2024-09-11T00:30:55.970335+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966434+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:30:55.966434+0000","last_peered":"2024-09-11T00:30:55.966434+0000","last_clean":"2024-09-11T00:30:55.966434+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:30:55.966434+0000","last_undegraded":"2024-09-11T00:30:55.966434+0000","last_fullsized":"2024-09-11T00:30:55.966434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719699+0000","last_change":"2024-09-11T00:30:54.993461+0000","last_active":"2024-09-11T00:30:56.719699+0000","last_peered":"2024-09-11T00:30:56.719699+0000","last_clean":"2024-09-11T00:30:56.719699+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:30:56.719699+0000","last_undegraded":"2024-09-11T00:30:56.719699+0000","last_fullsized":"2024-09-11T00:30:56.719699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:07.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997281+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:30:55.997281+0000","last_peered":"2024-09-11T00:30:55.997281+0000","last_clean":"2024-09-11T00:30:55.997281+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:30:55.997281+0000","last_undegraded":"2024-09-11T00:30:55.997281+0000","last_fullsized":"2024-09-11T00:30:55.997281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719307+0000","last_change":"2024-09-11T00:30:54.986265+0000","last_active":"2024-09-11T00:30:56.719307+0000","last_peered":"2024-09-11T00:30:56.719307+0000","last_clean":"2024-09-11T00:30:56.719307+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:30:56.719307+0000","last_undegraded":"2024-09-11T00:30:56.719307+0000","last_fullsized":"2024-09-11T00:30:56.719307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:49.109328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969766+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:30:55.969766+0000","last_peered":"2024-09-11T00:30:55.969766+0000","last_clean":"2024-09-11T00:30:55.969766+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:30:55.969766+0000","last_undegraded":"2024-09-11T00:30:55.969766+0000","last_fullsized":"2024-09-11T00:30:55.969766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967854+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:30:55.967854+0000","last_peered":"2024-09-11T00:30:55.967854+0000","last_clean":"2024-09-11T00:30:55.967854+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:30:55.967854+0000","last_undegraded":"2024-09-11T00:30:55.967854+0000","last_fullsized":"2024-09-11T00:30:55.967854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971368+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:30:55.971368+0000","last_peered":"2024-09-11T00:30:55.971368+0000","last_clean":"2024-09-11T00:30:55.971368+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:30:55.971368+0000","last_undegraded":"2024-09-11T00:30:55.971368+0000","last_fullsized":"2024-09-11T00:30:55.971368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969452+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:30:55.969452+0000","last_peered":"2024-09-11T00:30:55.969452+0000","last_clean":"2024-09-11T00:30:55.969452+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:30:55.969452+0000","last_undegraded":"2024-09-11T00:30:55.969452+0000","last_fullsized":"2024-09-11T00:30:55.969452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332691+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:30:56.332691+0000","last_peered":"2024-09-11T00:30:56.332691+0000","last_clean":"2024-09-11T00:30:56.332691+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:30:56.332691+0000","last_undegraded":"2024-09-11T00:30:56.332691+0000","last_fullsized":"2024-09-11T00:30:56.332691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970850+0000","last_change":"2024-09-11T00:30:54.994988+0000","last_active":"2024-09-11T00:30:55.970850+0000","last_peered":"2024-09-11T00:30:55.970850+0000","last_clean":"2024-09-11T00:30:55.970850+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:30:55.970850+0000","last_undegraded":"2024-09-11T00:30:55.970850+0000","last_fullsized":"2024-09-11T00:30:55.970850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:31.973641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332924+0000","last_change":"2024-09-11T00:30:45.941635+0000","last_active":"2024-09-11T00:30:56.332924+0000","last_peered":"2024-09-11T00:30:56.332924+0000","last_clean":"2024-09-11T00:30:56.332924+0000","last_became_active":"2024-09-11T00:30:45.941357+0000","last_became_peered":"2024-09-11T00:30:45.941357+0000","last_unstale":"2024-09-11T00:30:56.332924+0000","last_undegraded":"2024-09-11T00:30:56.332924+0000","last_fullsized":"2024-09-11T00:30:56.332924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:12.016406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332421+0000","last_change":"2024-09-11T00:30:54.984767+0000","last_active":"2024-09-11T00:30:56.332421+0000","last_peered":"2024-09-11T00:30:56.332421+0000","last_clean":"2024-09-11T00:30:56.332421+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:30:56.332421+0000","last_undegraded":"2024-09-11T00:30:56.332421+0000","last_fullsized":"2024-09-11T00:30:56.332421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:44.833677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332955+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:30:56.332955+0000","last_peered":"2024-09-11T00:30:56.332955+0000","last_clean":"2024-09-11T00:30:56.332955+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:56.332955+0000","last_undegraded":"2024-09-11T00:30:56.332955+0000","last_fullsized":"2024-09-11T00:30:56.332955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332911+0000","last_change":"2024-09-11T00:30:54.986423+0000","last_active":"2024-09-11T00:30:56.332911+0000","last_peered":"2024-09-11T00:30:56.332911+0000","last_clean":"2024-09-11T00:30:56.332911+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:30:56.332911+0000","last_undegraded":"2024-09-11T00:30:56.332911+0000","last_fullsized":"2024-09-11T00:30:56.332911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:16:50.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719859+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:30:56.719859+0000","last_peered":"2024-09-11T00:30:56.719859+0000","last_clean":"2024-09-11T00:30:56.719859+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:30:56.719859+0000","last_undegraded":"2024-09-11T00:30:56.719859+0000","last_fullsized":"2024-09-11T00:30:56.719859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998062+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:30:55.998062+0000","last_peered":"2024-09-11T00:30:55.998062+0000","last_clean":"2024-09-11T00:30:55.998062+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:30:55.998062+0000","last_undegraded":"2024-09-11T00:30:55.998062+0000","last_fullsized":"2024-09-11T00:30:55.998062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997283+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:30:55.997283+0000","last_peered":"2024-09-11T00:30:55.997283+0000","last_clean":"2024-09-11T00:30:55.997283+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:30:55.997283+0000","last_undegraded":"2024-09-11T00:30:55.997283+0000","last_fullsized":"2024-09-11T00:30:55.997283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173138+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:30:56.173138+0000","last_peered":"2024-09-11T00:30:56.173138+0000","last_clean":"2024-09-11T00:30:56.173138+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:30:56.173138+0000","last_undegraded":"2024-09-11T00:30:56.173138+0000","last_fullsized":"2024-09-11T00:30:56.173138+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333239+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:30:56.333239+0000","last_peered":"2024-09-11T00:30:56.333239+0000","last_clean":"2024-09-11T00:30:56.333239+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:30:56.333239+0000","last_undegraded":"2024-09-11T00:30:56.333239+0000","last_fullsized":"2024-09-11T00:30:56.333239+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970256+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:30:55.970256+0000","last_peered":"2024-09-11T00:30:55.970256+0000","last_clean":"2024-09-11T00:30:55.970256+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:30:55.970256+0000","last_undegraded":"2024-09-11T00:30:55.970256+0000","last_fullsized":"2024-09-11T00:30:55.970256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172539+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:30:56.172539+0000","last_peered":"2024-09-11T00:30:56.172539+0000","last_clean":"2024-09-11T00:30:56.172539+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:30:56.172539+0000","last_undegraded":"2024-09-11T00:30:56.172539+0000","last_fullsized":"2024-09-11T00:30:56.172539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719531+0000","last_change":"2024-09-11T00:30:54.990631+0000","last_active":"2024-09-11T00:30:56.719531+0000","last_peered":"2024-09-11T00:30:56.719531+0000","last_clean":"2024-09-11T00:30:56.719531+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:30:56.719531+0000","last_undegraded":"2024-09-11T00:30:56.719531+0000","last_fullsized":"2024-09-11T00:30:56.719531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:02.422471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333220+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:30:56.333220+0000","last_peered":"2024-09-11T00:30:56.333220+0000","last_clean":"2024-09-11T00:30:56.333220+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:30:56.333220+0000","last_undegraded":"2024-09-11T00:30:56.333220+0000","last_fullsized":"2024-09-11T00:30:56.333220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:54.991047+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:41:48.229261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173027+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:30:56.173027+0000","last_peered":"2024-09-11T00:30:56.173027+0000","last_clean":"2024-09-11T00:30:56.173027+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:30:56.173027+0000","last_undegraded":"2024-09-11T00:30:56.173027+0000","last_fullsized":"2024-09-11T00:30:56.173027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997710+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:30:55.997710+0000","last_peered":"2024-09-11T00:30:55.997710+0000","last_clean":"2024-09-11T00:30:55.997710+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:30:55.997710+0000","last_undegraded":"2024-09-11T00:30:55.997710+0000","last_fullsized":"2024-09-11T00:30:55.997710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997737+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:30:55.997737+0000","last_peered":"2024-09-11T00:30:55.997737+0000","last_clean":"2024-09-11T00:30:55.997737+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:30:55.997737+0000","last_undegraded":"2024-09-11T00:30:55.997737+0000","last_fullsized":"2024-09-11T00:30:55.997737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172698+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:30:56.172698+0000","last_peered":"2024-09-11T00:30:56.172698+0000","last_clean":"2024-09-11T00:30:56.172698+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:30:56.172698+0000","last_undegraded":"2024-09-11T00:30:56.172698+0000","last_fullsized":"2024-09-11T00:30:56.172698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997914+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:30:55.997914+0000","last_peered":"2024-09-11T00:30:55.997914+0000","last_clean":"2024-09-11T00:30:55.997914+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:30:55.997914+0000","last_undegraded":"2024-09-11T00:30:55.997914+0000","last_fullsized":"2024-09-11T00:30:55.997914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172531+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:30:56.172531+0000","last_peered":"2024-09-11T00:30:56.172531+0000","last_clean":"2024-09-11T00:30:56.172531+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:30:56.172531+0000","last_undegraded":"2024-09-11T00:30:56.172531+0000","last_fullsized":"2024-09-11T00:30:56.172531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333081+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:30:56.333081+0000","last_peered":"2024-09-11T00:30:56.333081+0000","last_clean":"2024-09-11T00:30:56.333081+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:30:56.333081+0000","last_undegraded":"2024-09-11T00:30:56.333081+0000","last_fullsized":"2024-09-11T00:30:56.333081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509450,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":286720,"data_stored":106265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":307200,"data_stored":108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:31:06.249 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T00:31:06.249 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T00:31:06.249 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T00:31:06.250 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T00:31:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.441+0000 7fc153200640 1 -- 172.21.15.31:0/2483463315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14c0aad60 msgr2=0x7fc14c157cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.441+0000 7fc153200640 1 --2- 172.21.15.31:0/2483463315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14c0aad60 0x7fc14c157cd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0aa340 tx=0x7fc13402a4d0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.441+0000 7fc153200640 1 -- 172.21.15.31:0/2483463315 shutdown_connections 2024-09-11T00:31:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.441+0000 7fc153200640 1 --2- 172.21.15.31:0/2483463315 >> v2:172.21.15.204:3300/0 conn(0x7fc14c0c2d90 0x7fc14c0c3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.441+0000 7fc153200640 1 --2- 172.21.15.31:0/2483463315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14c0aad60 0x7fc14c157cd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.441+0000 7fc153200640 1 -- 172.21.15.31:0/2483463315 >> 172.21.15.31:0/2483463315 conn(0x7fc14c0a96e0 msgr2=0x7fc14c0a9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.441+0000 7fc153200640 1 -- 172.21.15.31:0/2483463315 shutdown_connections 2024-09-11T00:31:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.441+0000 7fc153200640 1 -- 172.21.15.31:0/2483463315 wait complete. 2024-09-11T00:31:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.443+0000 7fc153200640 1 Processor -- start 2024-09-11T00:31:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.443+0000 7fc153200640 1 -- start start 2024-09-11T00:31:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc153200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc14c0aad60 0x7fc14c072040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc153200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14c0c2d90 0x7fc14c074b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc153200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc14c0c3a10 0x7fc14c0c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc153200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc14c0ab5d0 con 0x7fc14c0c2d90 2024-09-11T00:31:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc153200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc14c1546e0 con 0x7fc14c0c3a10 2024-09-11T00:31:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc153200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc14c0c6fa0 con 0x7fc14c0aad60 2024-09-11T00:31:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc143400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14c0c2d90 0x7fc14c074b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc143400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14c0c2d90 0x7fc14c074b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37654/0 (socket says 172.21.15.31:37654) 2024-09-11T00:31:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc143400640 1 -- 172.21.15.31:0/4248714268 learned_addr learned my addr 172.21.15.31:0/4248714268 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc14be00640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc14c0aad60 0x7fc14c072040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc151200640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc14c0c3a10 0x7fc14c0c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc143400640 1 -- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc14c0aad60 msgr2=0x7fc14c072040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc143400640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc14c0aad60 0x7fc14c072040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc143400640 1 -- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc14c0c3a10 msgr2=0x7fc14c0c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc143400640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc14c0c3a10 0x7fc14c0c48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.444+0000 7fc143400640 1 -- 172.21.15.31:0/4248714268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14c0726d0 con 0x7fc14c0c2d90 2024-09-11T00:31:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.445+0000 7fc143400640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14c0c2d90 0x7fc14c074b90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc1340099c0 tx=0x7fc1340039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.445+0000 7fc148c00640 1 -- 172.21.15.31:0/4248714268 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13403f070 con 0x7fc14c0c2d90 2024-09-11T00:31:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.445+0000 7fc148c00640 1 -- 172.21.15.31:0/4248714268 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc134004890 con 0x7fc14c0c2d90 2024-09-11T00:31:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.445+0000 7fc148c00640 1 -- 172.21.15.31:0/4248714268 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134035030 con 0x7fc14c0c2d90 2024-09-11T00:31:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.445+0000 7fc153200640 1 -- 172.21.15.31:0/4248714268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14c0729b0 con 0x7fc14c0c2d90 2024-09-11T00:31:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.445+0000 7fc153200640 1 -- 172.21.15.31:0/4248714268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc14c074670 con 0x7fc14c0c2d90 2024-09-11T00:31:06.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.446+0000 7fc153200640 1 -- 172.21.15.31:0/4248714268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc14c0c8840 con 0x7fc14c0c2d90 2024-09-11T00:31:06.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.447+0000 7fc148c00640 1 -- 172.21.15.31:0/4248714268 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc134004ce0 con 0x7fc14c0c2d90 2024-09-11T00:31:06.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.448+0000 7fc148c00640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc128077f70 0x7fc12807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.448+0000 7fc148c00640 1 -- 172.21.15.31:0/4248714268 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fc1340bb100 con 0x7fc14c0c2d90 2024-09-11T00:31:06.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.449+0000 7fc14be00640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc128077f70 0x7fc12807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.449+0000 7fc14be00640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc128077f70 0x7fc12807a430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc13c006fd0 tx=0x7fc13c009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:06.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.452+0000 7fc148c00640 1 -- 172.21.15.31:0/4248714268 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc14c0c8840 con 0x7fc14c0c2d90 2024-09-11T00:31:06.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.686+0000 7fc153200640 1 -- 172.21.15.31:0/4248714268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc14c0c3df0 con 0x7fc14c0c2d90 2024-09-11T00:31:06.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.687+0000 7fc148c00640 1 -- 172.21.15.31:0/4248714268 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fc1340832f0 con 0x7fc14c0c2d90 2024-09-11T00:31:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:06.690 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T00:31:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.689+0000 7fc153200640 1 -- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc128077f70 msgr2=0x7fc12807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.689+0000 7fc153200640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc128077f70 0x7fc12807a430 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc13c006fd0 tx=0x7fc13c009060 comp rx=0 tx=0).stop 2024-09-11T00:31:06.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.689+0000 7fc153200640 1 -- 172.21.15.31:0/4248714268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14c0c2d90 msgr2=0x7fc14c074b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.689+0000 7fc153200640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14c0c2d90 0x7fc14c074b90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc1340099c0 tx=0x7fc1340039e0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.689+0000 7fc153200640 1 -- 172.21.15.31:0/4248714268 shutdown_connections 2024-09-11T00:31:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.689+0000 7fc153200640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc128077f70 0x7fc12807a430 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.689+0000 7fc153200640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc14c0c3a10 0x7fc14c0c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.689+0000 7fc153200640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14c0c2d90 0x7fc14c074b90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.689+0000 7fc153200640 1 --2- 172.21.15.31:0/4248714268 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc14c0aad60 0x7fc14c072040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.689+0000 7fc153200640 1 -- 172.21.15.31:0/4248714268 >> 172.21.15.31:0/4248714268 conn(0x7fc14c0a96e0 msgr2=0x7fc14c0c0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.690+0000 7fc153200640 1 -- 172.21.15.31:0/4248714268 shutdown_connections 2024-09-11T00:31:06.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.690+0000 7fc153200640 1 -- 172.21.15.31:0/4248714268 wait complete. 2024-09-11T00:31:06.706 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T00:31:06.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:06.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.864+0000 7fd009800640 1 -- 172.21.15.31:0/1153181829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd004072480 msgr2=0x7fd0041511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.864+0000 7fd009800640 1 --2- 172.21.15.31:0/1153181829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd004072480 0x7fd0041511a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd0040add10 tx=0x7fcff402a5f0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.864+0000 7fd009800640 1 -- 172.21.15.31:0/1153181829 shutdown_connections 2024-09-11T00:31:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.864+0000 7fd009800640 1 --2- 172.21.15.31:0/1153181829 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd004153f20 0x7fd0040c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.864+0000 7fd009800640 1 --2- 172.21.15.31:0/1153181829 >> v2:172.21.15.204:3300/0 conn(0x7fd0040c23a0 0x7fd0041516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.864+0000 7fd009800640 1 --2- 172.21.15.31:0/1153181829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd004072480 0x7fd0041511a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.864+0000 7fd009800640 1 -- 172.21.15.31:0/1153181829 >> 172.21.15.31:0/1153181829 conn(0x7fd0040a9750 msgr2=0x7fd0040a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.865+0000 7fd009800640 1 -- 172.21.15.31:0/1153181829 shutdown_connections 2024-09-11T00:31:06.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.865+0000 7fd009800640 1 -- 172.21.15.31:0/1153181829 wait complete. 2024-09-11T00:31:06.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.866+0000 7fd009800640 1 Processor -- start 2024-09-11T00:31:06.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.866+0000 7fd009800640 1 -- start start 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.867+0000 7fd009800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd004072480 0x7fd0041e1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.867+0000 7fd009800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0040c23a0 0x7fd004156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.867+0000 7fd009800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd004153f20 0x7fd004155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.867+0000 7fd009800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd004074200 con 0x7fd004153f20 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.867+0000 7fd009800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd004074080 con 0x7fd0040c23a0 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.867+0000 7fd009800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd004072860 con 0x7fd004072480 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fd003400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd004153f20 0x7fd004155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fd002a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd004072480 0x7fd0041e1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fd003400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd004153f20 0x7fd004155280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37666/0 (socket says 172.21.15.31:37666) 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fd003400640 1 -- 172.21.15.31:0/1921889405 learned_addr learned my addr 172.21.15.31:0/1921889405 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:06.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fcffa000640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0040c23a0 0x7fd004156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fd003400640 1 -- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd004072480 msgr2=0x7fd0041e1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fd003400640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd004072480 0x7fd0041e1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fd003400640 1 -- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0040c23a0 msgr2=0x7fd004156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:06.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fd003400640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0040c23a0 0x7fd004156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:06.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fd003400640 1 -- 172.21.15.31:0/1921889405 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd004157cc0 con 0x7fd004153f20 2024-09-11T00:31:06.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.868+0000 7fd003400640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd004153f20 0x7fd004155280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00e960 tx=0x7fcffc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:06.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.869+0000 7fcffb400640 1 -- 172.21.15.31:0/1921889405 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc004d20 con 0x7fd004153f20 2024-09-11T00:31:06.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.869+0000 7fcffb400640 1 -- 172.21.15.31:0/1921889405 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcffc00cac0 con 0x7fd004153f20 2024-09-11T00:31:06.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.869+0000 7fd009800640 1 -- 172.21.15.31:0/1921889405 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0040c7740 con 0x7fd004153f20 2024-09-11T00:31:06.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.869+0000 7fcffb400640 1 -- 172.21.15.31:0/1921889405 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc005020 con 0x7fd004153f20 2024-09-11T00:31:06.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.869+0000 7fd009800640 1 -- 172.21.15.31:0/1921889405 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00414b8c0 con 0x7fd004153f20 2024-09-11T00:31:06.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.870+0000 7fd009800640 1 -- 172.21.15.31:0/1921889405 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0040c7410 con 0x7fd004153f20 2024-09-11T00:31:06.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.871+0000 7fcffb400640 1 -- 172.21.15.31:0/1921889405 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fcffc00f040 con 0x7fd004153f20 2024-09-11T00:31:06.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.872+0000 7fcffb400640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcfe0077fa0 0x7fcfe007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:06.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.873+0000 7fcffb400640 1 -- 172.21.15.31:0/1921889405 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fcffc09a540 con 0x7fd004153f20 2024-09-11T00:31:06.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.873+0000 7fd002a00640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcfe0077fa0 0x7fcfe007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:06.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.873+0000 7fd002a00640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcfe0077fa0 0x7fcfe007a460 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcff4033000 tx=0x7fcff4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:06.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:06.877+0000 7fcffb400640 1 -- 172.21.15.31:0/1921889405 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcffc016030 con 0x7fd004153f20 2024-09-11T00:31:07.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.111+0000 7fd009800640 1 -- 172.21.15.31:0/1921889405 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0041511a0 con 0x7fd004153f20 2024-09-11T00:31:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.111+0000 7fcffb400640 1 -- 172.21.15.31:0/1921889405 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fcffc06b420 con 0x7fd004153f20 2024-09-11T00:31:07.113 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:07.113 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:07.114 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 -- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcfe0077fa0 msgr2=0x7fcfe007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcfe0077fa0 0x7fcfe007a460 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcff4033000 tx=0x7fcff4004730 comp rx=0 tx=0).stop 2024-09-11T00:31:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 -- 172.21.15.31:0/1921889405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd004153f20 msgr2=0x7fd004155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd004153f20 0x7fd004155280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00e960 tx=0x7fcffc00ee20 comp rx=0 tx=0).stop 2024-09-11T00:31:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 -- 172.21.15.31:0/1921889405 shutdown_connections 2024-09-11T00:31:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcfe0077fa0 0x7fcfe007a460 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd004153f20 0x7fd004155280 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0040c23a0 0x7fd004156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 --2- 172.21.15.31:0/1921889405 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd004072480 0x7fd0041e1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 -- 172.21.15.31:0/1921889405 >> 172.21.15.31:0/1921889405 conn(0x7fd0040a9750 msgr2=0x7fd00414c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:07.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 -- 172.21.15.31:0/1921889405 shutdown_connections 2024-09-11T00:31:07.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.114+0000 7fd009800640 1 -- 172.21.15.31:0/1921889405 wait complete. 2024-09-11T00:31:07.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:07.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.291+0000 7f5066e00640 1 -- 172.21.15.31:0/1523540448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50600ac2c0 msgr2=0x7f50600ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.291+0000 7f5066e00640 1 --2- 172.21.15.31:0/1523540448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50600ac2c0 0x7f50600ac6a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5054009830 tx=0x7f505402a3f0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.291+0000 7f5066e00640 1 -- 172.21.15.31:0/1523540448 shutdown_connections 2024-09-11T00:31:07.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.291+0000 7f5066e00640 1 --2- 172.21.15.31:0/1523540448 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50600acbe0 0x7f50600c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.291+0000 7f5066e00640 1 --2- 172.21.15.31:0/1523540448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50600ac2c0 0x7f50600ac6a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.291+0000 7f5066e00640 1 -- 172.21.15.31:0/1523540448 >> 172.21.15.31:0/1523540448 conn(0x7f506006caf0 msgr2=0x7f506006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:07.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.291+0000 7f5066e00640 1 -- 172.21.15.31:0/1523540448 shutdown_connections 2024-09-11T00:31:07.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.291+0000 7f5066e00640 1 -- 172.21.15.31:0/1523540448 wait complete. 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.292+0000 7f5066e00640 1 Processor -- start 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.293+0000 7f5066e00640 1 -- start start 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.293+0000 7f5066e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50600a6e20 0x7f506014a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.293+0000 7f5066e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50600ac2c0 0x7f5060155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.293+0000 7f5066e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50600acbe0 0x7f5060146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.293+0000 7f5066e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5060151950 con 0x7f50600ac2c0 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.293+0000 7f5066e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f50601517d0 con 0x7f50600acbe0 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.293+0000 7f5066e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f50600c66f0 con 0x7f50600a6e20 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50600ac2c0 0x7f5060155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50600ac2c0 0x7f5060155300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37690/0 (socket says 172.21.15.31:37690) 2024-09-11T00:31:07.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065000640 1 -- 172.21.15.31:0/3242122113 learned_addr learned my addr 172.21.15.31:0/3242122113 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065a00640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50600a6e20 0x7f506014a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065000640 1 -- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50600a6e20 msgr2=0x7f506014a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5066400640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50600acbe0 0x7f5060146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065000640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50600a6e20 0x7f506014a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065000640 1 -- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50600acbe0 msgr2=0x7f5060146e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065000640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50600acbe0 0x7f5060146e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065000640 1 -- 172.21.15.31:0/3242122113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506006f360 con 0x7f50600ac2c0 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065a00640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50600a6e20 0x7f506014a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5066400640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50600acbe0 0x7f5060146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f5065000640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50600ac2c0 0x7f5060155300 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f504c00c4c0 tx=0x7f504c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.294+0000 7f505d600640 1 -- 172.21.15.31:0/3242122113 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c016020 con 0x7f50600ac2c0 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.295+0000 7f505d600640 1 -- 172.21.15.31:0/3242122113 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f504c012940 con 0x7f50600ac2c0 2024-09-11T00:31:07.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.295+0000 7f5066e00640 1 -- 172.21.15.31:0/3242122113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50601efbc0 con 0x7f50600ac2c0 2024-09-11T00:31:07.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.295+0000 7f505d600640 1 -- 172.21.15.31:0/3242122113 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c012d90 con 0x7f50600ac2c0 2024-09-11T00:31:07.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.295+0000 7f5066e00640 1 -- 172.21.15.31:0/3242122113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50601efea0 con 0x7f50600ac2c0 2024-09-11T00:31:07.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.296+0000 7f5066e00640 1 -- 172.21.15.31:0/3242122113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5060070390 con 0x7f50600ac2c0 2024-09-11T00:31:07.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.297+0000 7f505d600640 1 -- 172.21.15.31:0/3242122113 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f504c01a3b0 con 0x7f50600ac2c0 2024-09-11T00:31:07.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.298+0000 7f505d600640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5040077f70 0x7f504007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:07.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.298+0000 7f505d600640 1 -- 172.21.15.31:0/3242122113 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f504c099fe0 con 0x7f50600ac2c0 2024-09-11T00:31:07.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.298+0000 7f5065a00640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5040077f70 0x7f504007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:07.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.299+0000 7f5065a00640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5040077f70 0x7f504007a430 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5054034000 tx=0x7f5054004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:07.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.302+0000 7f505d600640 1 -- 172.21.15.31:0/3242122113 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f504c020030 con 0x7f50600ac2c0 2024-09-11T00:31:07.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.538+0000 7f5066e00640 1 -- 172.21.15.31:0/3242122113 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50600c6b50 con 0x7f50600ac2c0 2024-09-11T00:31:07.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.538+0000 7f505d600640 1 -- 172.21.15.31:0/3242122113 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f504c0627e0 con 0x7f50600ac2c0 2024-09-11T00:31:07.540 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:07.540 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:07.540 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:07.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.540+0000 7f5066e00640 1 -- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5040077f70 msgr2=0x7f504007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.540+0000 7f5066e00640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5040077f70 0x7f504007a430 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5054034000 tx=0x7f5054004730 comp rx=0 tx=0).stop 2024-09-11T00:31:07.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.540+0000 7f5066e00640 1 -- 172.21.15.31:0/3242122113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50600ac2c0 msgr2=0x7f5060155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.540+0000 7f5066e00640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50600ac2c0 0x7f5060155300 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f504c00c4c0 tx=0x7f504c00c980 comp rx=0 tx=0).stop 2024-09-11T00:31:07.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.540+0000 7f5066e00640 1 -- 172.21.15.31:0/3242122113 shutdown_connections 2024-09-11T00:31:07.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.540+0000 7f5066e00640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5040077f70 0x7f504007a430 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.540+0000 7f5066e00640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f50600acbe0 0x7f5060146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.540+0000 7f5066e00640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50600ac2c0 0x7f5060155300 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.540+0000 7f5066e00640 1 --2- 172.21.15.31:0/3242122113 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f50600a6e20 0x7f506014a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.540+0000 7f5066e00640 1 -- 172.21.15.31:0/3242122113 >> 172.21.15.31:0/3242122113 conn(0x7f506006caf0 msgr2=0x7f5060153550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:07.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.541+0000 7f5066e00640 1 -- 172.21.15.31:0/3242122113 shutdown_connections 2024-09-11T00:31:07.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.541+0000 7f5066e00640 1 -- 172.21.15.31:0/3242122113 wait complete. 2024-09-11T00:31:07.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:31:07.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.716+0000 7f0512600640 1 -- 172.21.15.31:0/2486333963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f050c0cc220 msgr2=0x7f050c06e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.716+0000 7f0512600640 1 --2- 172.21.15.31:0/2486333963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f050c0cc220 0x7f050c06e680 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f050c153d20 tx=0x7f050402aa10 comp rx=0 tx=0).stop 2024-09-11T00:31:07.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.717+0000 7f0512600640 1 -- 172.21.15.31:0/2486333963 shutdown_connections 2024-09-11T00:31:07.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.717+0000 7f0512600640 1 --2- 172.21.15.31:0/2486333963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f050c0cc220 0x7f050c06e680 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.717+0000 7f0512600640 1 --2- 172.21.15.31:0/2486333963 >> v2:172.21.15.204:3300/0 conn(0x7f050c156960 0x7f050c06e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.717+0000 7f0512600640 1 -- 172.21.15.31:0/2486333963 >> 172.21.15.31:0/2486333963 conn(0x7f050c068f10 msgr2=0x7f050c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:07.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.717+0000 7f0512600640 1 -- 172.21.15.31:0/2486333963 shutdown_connections 2024-09-11T00:31:07.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.717+0000 7f0512600640 1 -- 172.21.15.31:0/2486333963 wait complete. 2024-09-11T00:31:07.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.718+0000 7f0512600640 1 Processor -- start 2024-09-11T00:31:07.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.718+0000 7f0512600640 1 -- start start 2024-09-11T00:31:07.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.719+0000 7f0512600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f050c0cc220 0x7f050c14eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:07.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.719+0000 7f0512600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f050c150020 0x7f050c14f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.719+0000 7f0512600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f050c156960 0x7f050c0bf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.719+0000 7f0512600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f050c0c6ca0 con 0x7f050c150020 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.719+0000 7f0512600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f050c0c6b20 con 0x7f050c0cc220 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.719+0000 7f0512600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f050c1573d0 con 0x7f050c156960 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.719+0000 7f0503e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f050c150020 0x7f050c14f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.719+0000 7f0503e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f050c150020 0x7f050c14f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37714/0 (socket says 172.21.15.31:37714) 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.719+0000 7f0503e00640 1 -- 172.21.15.31:0/628824540 learned_addr learned my addr 172.21.15.31:0/628824540 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0511c00640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f050c156960 0x7f050c0bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0511200640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f050c0cc220 0x7f050c14eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0503e00640 1 -- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f050c156960 msgr2=0x7f050c0bf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0503e00640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f050c156960 0x7f050c0bf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0503e00640 1 -- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f050c0cc220 msgr2=0x7f050c14eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0503e00640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f050c0cc220 0x7f050c14eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0503e00640 1 -- 172.21.15.31:0/628824540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050c1ecb80 con 0x7f050c150020 2024-09-11T00:31:07.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0511200640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f050c0cc220 0x7f050c14eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:07.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0503e00640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f050c150020 0x7f050c14f410 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f04f400e7c0 tx=0x7f04f400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0509600640 1 -- 172.21.15.31:0/628824540 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f400ccc0 con 0x7f050c150020 2024-09-11T00:31:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0509600640 1 -- 172.21.15.31:0/628824540 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04f4010430 con 0x7f050c150020 2024-09-11T00:31:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0509600640 1 -- 172.21.15.31:0/628824540 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4010860 con 0x7f050c150020 2024-09-11T00:31:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.720+0000 7f0512600640 1 -- 172.21.15.31:0/628824540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050c0afab0 con 0x7f050c150020 2024-09-11T00:31:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.721+0000 7f0512600640 1 -- 172.21.15.31:0/628824540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f050c1ecdf0 con 0x7f050c150020 2024-09-11T00:31:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.722+0000 7f0512600640 1 -- 172.21.15.31:0/628824540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f050c0ad4e0 con 0x7f050c150020 2024-09-11T00:31:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.723+0000 7f0509600640 1 -- 172.21.15.31:0/628824540 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f04f4010a00 con 0x7f050c150020 2024-09-11T00:31:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.724+0000 7f0509600640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f04e8077f70 0x7f04e807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.724+0000 7f0509600640 1 -- 172.21.15.31:0/628824540 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f04f409aab0 con 0x7f050c150020 2024-09-11T00:31:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.724+0000 7f0511200640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f04e8077f70 0x7f04e807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:07.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.725+0000 7f0511200640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f04e8077f70 0x7f04e807a430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f04fc001740 tx=0x7f04fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:07.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.728+0000 7f0509600640 1 -- 172.21.15.31:0/628824540 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f04f4063270 con 0x7f050c150020 2024-09-11T00:31:07.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.921+0000 7f0512600640 1 -- 172.21.15.31:0/628824540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f050c0b1820 con 0x7f050c150020 2024-09-11T00:31:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.922+0000 7f0509600640 1 -- 172.21.15.31:0/628824540 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20383 (secure 0 0 0) 0x7f04f4062b70 con 0x7f050c150020 2024-09-11T00:31:07.924 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:07.925 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:55.962415+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.924+0000 7f0512600640 1 -- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f04e8077f70 msgr2=0x7f04e807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.924+0000 7f0512600640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f04e8077f70 0x7f04e807a430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f04fc001740 tx=0x7f04fc009040 comp rx=0 tx=0).stop 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.925+0000 7f0512600640 1 -- 172.21.15.31:0/628824540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f050c150020 msgr2=0x7f050c14f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.925+0000 7f0512600640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f050c150020 0x7f050c14f410 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f04f400e7c0 tx=0x7f04f400ec80 comp rx=0 tx=0).stop 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.925+0000 7f0512600640 1 -- 172.21.15.31:0/628824540 shutdown_connections 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.925+0000 7f0512600640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f04e8077f70 0x7f04e807a430 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.925+0000 7f0512600640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f050c156960 0x7f050c0bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.925+0000 7f0512600640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f050c150020 0x7f050c14f410 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.925+0000 7f0512600640 1 --2- 172.21.15.31:0/628824540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f050c0cc220 0x7f050c14eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.925+0000 7f0512600640 1 -- 172.21.15.31:0/628824540 >> 172.21.15.31:0/628824540 conn(0x7f050c068f10 msgr2=0x7f050c1549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:07.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.925+0000 7f0512600640 1 -- 172.21.15.31:0/628824540 shutdown_connections 2024-09-11T00:31:07.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:07.925+0000 7f0512600640 1 -- 172.21.15.31:0/628824540 wait complete. 2024-09-11T00:31:07.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:08.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.104+0000 7f2fefc00640 1 -- 172.21.15.31:0/1075140806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fcc003a50 msgr2=0x7f2fcc003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.104+0000 7f2fefc00640 1 --2- 172.21.15.31:0/1075140806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fcc003a50 0x7f2fcc003e90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009b50 tx=0x7f2fcc005830 comp rx=0 tx=0).stop 2024-09-11T00:31:08.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.104+0000 7f2fefc00640 1 -- 172.21.15.31:0/1075140806 shutdown_connections 2024-09-11T00:31:08.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.104+0000 7f2fefc00640 1 --2- 172.21.15.31:0/1075140806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fcc003a50 0x7f2fcc003e90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.104+0000 7f2fefc00640 1 -- 172.21.15.31:0/1075140806 >> 172.21.15.31:0/1075140806 conn(0x7f2fe80a9750 msgr2=0x7f2fe80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:08.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.104+0000 7f2fefc00640 1 -- 172.21.15.31:0/1075140806 shutdown_connections 2024-09-11T00:31:08.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.104+0000 7f2fefc00640 1 -- 172.21.15.31:0/1075140806 wait complete. 2024-09-11T00:31:08.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.106+0000 7f2fefc00640 1 Processor -- start 2024-09-11T00:31:08.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.106+0000 7f2fefc00640 1 -- start start 2024-09-11T00:31:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.106+0000 7f2fefc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2fcc003a50 0x7f2fe806e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fefc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fe806f790 0x7f2fe8148710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fefc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe806e740 0x7f2fe8148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fefc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe8074200 con 0x7f2fe806e740 2024-09-11T00:31:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fefc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe8074080 con 0x7f2fe806f790 2024-09-11T00:31:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fefc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2fe8072860 con 0x7f2fcc003a50 2024-09-11T00:31:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fedc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe806e740 0x7f2fe8148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fedc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe806e740 0x7f2fe8148c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46092/0 (socket says 172.21.15.31:46092) 2024-09-11T00:31:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fedc00640 1 -- 172.21.15.31:0/1313072588 learned_addr learned my addr 172.21.15.31:0/1313072588 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:08.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fed200640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2fcc003a50 0x7f2fe806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fdfe00640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fe806f790 0x7f2fe8148710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fedc00640 1 -- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2fcc003a50 msgr2=0x7f2fe806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fedc00640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2fcc003a50 0x7f2fe806e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fedc00640 1 -- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fe806f790 msgr2=0x7f2fe8148710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fedc00640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fe806f790 0x7f2fe8148710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fedc00640 1 -- 172.21.15.31:0/1313072588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe806fcc0 con 0x7f2fe806e740 2024-09-11T00:31:08.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.107+0000 7f2fedc00640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe806e740 0x7f2fe8148c50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80c1e40 tx=0x7f2fe0006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.108+0000 7f2fe5600640 1 -- 172.21.15.31:0/1313072588 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0015070 con 0x7f2fe806e740 2024-09-11T00:31:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.108+0000 7f2fe5600640 1 -- 172.21.15.31:0/1313072588 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fe000fa20 con 0x7f2fe806e740 2024-09-11T00:31:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.108+0000 7f2fefc00640 1 -- 172.21.15.31:0/1313072588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fe81ec980 con 0x7f2fe806e740 2024-09-11T00:31:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.108+0000 7f2fe5600640 1 -- 172.21.15.31:0/1313072588 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000fe30 con 0x7f2fe806e740 2024-09-11T00:31:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.108+0000 7f2fefc00640 1 -- 172.21.15.31:0/1313072588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fe81df920 con 0x7f2fe806e740 2024-09-11T00:31:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.109+0000 7f2fefc00640 1 -- 172.21.15.31:0/1313072588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fe81511a0 con 0x7f2fe806e740 2024-09-11T00:31:08.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.110+0000 7f2fe5600640 1 -- 172.21.15.31:0/1313072588 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2fe0011430 con 0x7f2fe806e740 2024-09-11T00:31:08.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.111+0000 7f2fe5600640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2fc4077f70 0x7f2fc407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.111+0000 7f2fe5600640 1 -- 172.21.15.31:0/1313072588 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f2fe009ac30 con 0x7f2fe806e740 2024-09-11T00:31:08.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.112+0000 7f2fed200640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2fc4077f70 0x7f2fc407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.112+0000 7f2fed200640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2fc4077f70 0x7f2fc407a430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8035000 tx=0x7f2fd80046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:08.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.115+0000 7f2fe5600640 1 -- 172.21.15.31:0/1313072588 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2fe0062e40 con 0x7f2fe806e740 2024-09-11T00:31:08.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.346+0000 7f2fefc00640 1 -- 172.21.15.31:0/1313072588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fe8151530 con 0x7f2fe806e740 2024-09-11T00:31:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.347+0000 7f2fe5600640 1 -- 172.21.15.31:0/1313072588 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f2fe0067870 con 0x7f2fe806e740 2024-09-11T00:31:08.349 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:08.349 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 -- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2fc4077f70 msgr2=0x7f2fc407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2fc4077f70 0x7f2fc407a430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8035000 tx=0x7f2fd80046c0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 -- 172.21.15.31:0/1313072588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe806e740 msgr2=0x7f2fe8148c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe806e740 0x7f2fe8148c50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80c1e40 tx=0x7f2fe0006720 comp rx=0 tx=0).stop 2024-09-11T00:31:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 -- 172.21.15.31:0/1313072588 shutdown_connections 2024-09-11T00:31:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2fc4077f70 0x7f2fc407a430 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe806e740 0x7f2fe8148c50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2fe806f790 0x7f2fe8148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 --2- 172.21.15.31:0/1313072588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2fcc003a50 0x7f2fe806e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 -- 172.21.15.31:0/1313072588 >> 172.21.15.31:0/1313072588 conn(0x7f2fe80a9750 msgr2=0x7f2fe81e2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:08.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 -- 172.21.15.31:0/1313072588 shutdown_connections 2024-09-11T00:31:08.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.349+0000 7f2fefc00640 1 -- 172.21.15.31:0/1313072588 wait complete. 2024-09-11T00:31:08.769 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-11T00:30:45.342170+0000', 'modified': '2024-09-11T00:30:49.364654+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4390}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4390': {'gid': 4390, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.204:6835/1801149506', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6834', 'nonce': 1801149506}, {'type': 'v1', 'addr': '172.21.15.204:6835', 'nonce': 1801149506}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'a', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4390, 'qdb_cluster': [4390]} 2024-09-11T00:31:08.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-11T00:31:08.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-11T00:31:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.934+0000 7fda51e00640 1 -- 172.21.15.31:0/1583195595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda30004000 msgr2=0x7fda30004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.934+0000 7fda51e00640 1 --2- 172.21.15.31:0/1583195595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda30004000 0x7fda30004800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda34009b80 tx=0x7fda30005830 comp rx=0 tx=0).stop 2024-09-11T00:31:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.934+0000 7fda51400640 1 -- 172.21.15.31:0/1583195595 reap_dead start 2024-09-11T00:31:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.934+0000 7fda51e00640 1 -- 172.21.15.31:0/1583195595 shutdown_connections 2024-09-11T00:31:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.934+0000 7fda51e00640 1 -- 172.21.15.31:0/1583195595 >> 172.21.15.31:0/1583195595 conn(0x7fda4c068f10 msgr2=0x7fda4c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.935+0000 7fda51e00640 1 -- 172.21.15.31:0/1583195595 shutdown_connections 2024-09-11T00:31:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.935+0000 7fda51e00640 1 -- 172.21.15.31:0/1583195595 wait complete. 2024-09-11T00:31:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.936+0000 7fda51e00640 1 Processor -- start 2024-09-11T00:31:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.936+0000 7fda51e00640 1 -- start start 2024-09-11T00:31:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda51e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda30004000 0x7fda4c150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda51e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda4c14e530 0x7fda4c1eee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda51e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4c1ef3a0 0x7fda4c0ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda51e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c0c6ca0 con 0x7fda4c1ef3a0 2024-09-11T00:31:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda51e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c0c6b20 con 0x7fda4c14e530 2024-09-11T00:31:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda51e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fda4c1573d0 con 0x7fda30004000 2024-09-11T00:31:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda51400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4c1ef3a0 0x7fda4c0ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda51400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4c1ef3a0 0x7fda4c0ae830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46112/0 (socket says 172.21.15.31:46112) 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda51400640 1 -- 172.21.15.31:0/3654791370 learned_addr learned my addr 172.21.15.31:0/3654791370 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda43e00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda4c14e530 0x7fda4c1eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.937+0000 7fda50a00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda30004000 0x7fda4c150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda51400640 1 -- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda30004000 msgr2=0x7fda4c150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda51400640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda30004000 0x7fda4c150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda51400640 1 -- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda4c14e530 msgr2=0x7fda4c1eee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda51400640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda4c14e530 0x7fda4c1eee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda51400640 1 -- 172.21.15.31:0/3654791370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c152b50 con 0x7fda4c1ef3a0 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda50a00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda30004000 0x7fda4c150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda51400640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4c1ef3a0 0x7fda4c0ae830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0c7e30 tx=0x7fda440021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda49600640 1 -- 172.21.15.31:0/3654791370 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda440019e0 con 0x7fda4c1ef3a0 2024-09-11T00:31:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda49600640 1 -- 172.21.15.31:0/3654791370 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda44016020 con 0x7fda4c1ef3a0 2024-09-11T00:31:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda49600640 1 -- 172.21.15.31:0/3654791370 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4400fa40 con 0x7fda4c1ef3a0 2024-09-11T00:31:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda51e00640 1 -- 172.21.15.31:0/3654791370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda4c150ab0 con 0x7fda4c1ef3a0 2024-09-11T00:31:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.938+0000 7fda51e00640 1 -- 172.21.15.31:0/3654791370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4c152df0 con 0x7fda4c1ef3a0 2024-09-11T00:31:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.941+0000 7fda49600640 1 -- 172.21.15.31:0/3654791370 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fda4400fbe0 con 0x7fda4c1ef3a0 2024-09-11T00:31:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.942+0000 7fda49600640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda28077f70 0x7fda2807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.942+0000 7fda49600640 1 -- 172.21.15.31:0/3654791370 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fda4409a690 con 0x7fda4c1ef3a0 2024-09-11T00:31:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.942+0000 7fda50a00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda28077f70 0x7fda2807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.942+0000 7fda561fc740 1 Processor -- start 2024-09-11T00:31:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.942+0000 7fda561fc740 1 -- start start 2024-09-11T00:31:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda50a00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda28077f70 0x7fda2807a430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0c5f20 tx=0x7fda3c0067a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda561fc740 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b57fa0 0x565210b335f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda561fc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b72740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda561fc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b72c80 0x565210b36350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda561fc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x565210abb430 con 0x565210b72c80 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda561fc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x565210aab5d0 con 0x565210b58380 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda561fc740 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x565210a9aa00 con 0x565210b57fa0 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda51400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b72c80 0x565210b36350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda51400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b72c80 0x565210b36350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46126/0 (socket says 172.21.15.31:46126) 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda51400640 1 -- 172.21.15.31:0/1404882362 learned_addr learned my addr 172.21.15.31:0/1404882362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda50a00640 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b57fa0 0x565210b335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda43e00640 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b72740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda51400640 1 -- 172.21.15.31:0/1404882362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b57fa0 msgr2=0x565210b335f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda51400640 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b57fa0 0x565210b335f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda51400640 1 -- 172.21.15.31:0/1404882362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 msgr2=0x565210b72740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda51400640 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b72740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.943+0000 7fda51400640 1 -- 172.21.15.31:0/1404882362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565210a63520 con 0x565210b72c80 2024-09-11T00:31:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.944+0000 7fda43e00640 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b72740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.944+0000 7fda50a00640 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b57fa0 0x565210b335f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.944+0000 7fda51400640 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b72c80 0x565210b36350 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda44062780 tx=0x7fda44061f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e8462b930c9e77d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.944+0000 7fda41200640 1 -- 172.21.15.31:0/1404882362 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44015070 con 0x565210b72c80 2024-09-11T00:31:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.944+0000 7fda41200640 1 -- 172.21.15.31:0/1404882362 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda4404c070 con 0x565210b72c80 2024-09-11T00:31:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.944+0000 7fda41200640 1 -- 172.21.15.31:0/1404882362 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44061c50 con 0x565210b72c80 2024-09-11T00:31:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.945+0000 7fda561fc740 1 -- 172.21.15.31:0/1404882362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b72c80 msgr2=0x565210b36350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.945+0000 7fda561fc740 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b72c80 0x565210b36350 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda44062780 tx=0x7fda44061f60 comp rx=0 tx=0).stop 2024-09-11T00:31:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.945+0000 7fda561fc740 1 -- 172.21.15.31:0/1404882362 shutdown_connections 2024-09-11T00:31:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.945+0000 7fda561fc740 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b72c80 0x565210b36350 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.945+0000 7fda561fc740 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b72740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.945+0000 7fda561fc740 1 --2- 172.21.15.31:0/1404882362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b57fa0 0x565210b335f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.945+0000 7fda561fc740 1 -- 172.21.15.31:0/1404882362 >> 172.21.15.31:0/1404882362 conn(0x565210b340c0 msgr2=0x565210b378b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:08.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.945+0000 7fda561fc740 1 -- 172.21.15.31:0/1404882362 shutdown_connections 2024-09-11T00:31:08.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.945+0000 7fda561fc740 1 -- 172.21.15.31:0/1404882362 wait complete. 2024-09-11T00:31:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.945+0000 7fda561fc740 1 -- start start 2024-09-11T00:31:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.946+0000 7fda561fc740 1 Processor -- start 2024-09-11T00:31:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.946+0000 7fda561fc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b593c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.946+0000 7fda561fc740 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 0x565210b6af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda561fc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b5afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda561fc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x565210a41330 con 0x565210b6c680 2024-09-11T00:31:08.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda561fc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x565210a9e810 con 0x565210b58380 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda561fc740 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x56521080b620 con 0x565210b72c80 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda561fc740 10 client.0 _ll_register_callbacks cb 0x565210a46420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda43e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 0x565210b6af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda50a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda51400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b5afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda43e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 0x565210b6af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:56906/0 (socket says 172.21.15.31:56906) 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda43e00640 1 -- 172.21.15.31:0/63402284 learned_addr learned my addr 172.21.15.31:0/63402284 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda51400640 1 -- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 msgr2=0x565210b6af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda51400640 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 0x565210b6af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda51400640 1 -- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 msgr2=0x565210b593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:08.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda51400640 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda51400640 1 -- 172.21.15.31:0/63402284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652107fc850 con 0x565210b6c680 2024-09-11T00:31:08.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.947+0000 7fda50a00640 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:08.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda51400640 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b5afa0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fda440624f0 tx=0x7fda44061e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fff48e6821ac141 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:08.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda39600640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-11T00:31:08.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda39600640 1 -- 172.21.15.31:0/63402284 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44015040 con 0x565210b6c680 2024-09-11T00:31:08.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda39600640 1 -- 172.21.15.31:0/63402284 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fda44063d30 con 0x565210b6c680 2024-09-11T00:31:08.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda561fc740 1 -- 172.21.15.31:0/63402284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda4404c070 con 0x565210b6c680 2024-09-11T00:31:08.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda39600640 1 -- 172.21.15.31:0/63402284 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fda440627f0 con 0x565210b6c680 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda561fc740 10 client.4746 fetch_fsmap learned FSMap version 15 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda561fc740 1 -- 172.21.15.31:0/63402284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x5652108736b0 con 0x565210b6c680 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda39600640 1 -- 172.21.15.31:0/63402284 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda4404c070 con 0x565210b6c680 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda39600640 1 -- 172.21.15.31:0/63402284 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 15) ==== 4136+0+0 (secure 0 0 0) 0x7fda4405e140 con 0x565210b6c680 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda561fc740 10 client.4746 fetch_fsmap finished waiting for FSMap version 15 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda561fc740 10 client.4746 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda561fc740 20 client.4746 populate_metadata read hostname 'smithi031' 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda561fc740 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210b869a0 0x565210b86d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda561fc740 4 client.4746 mds_command: new command op to 4390 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.948+0000 7fda561fc740 1 -- 172.21.15.31:0/63402284 --> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565210aad520 con 0x565210b869a0 2024-09-11T00:31:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.949+0000 7fda50a00640 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210b869a0 0x565210b86d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.949+0000 7fda50a00640 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210b869a0 0x565210b86d80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21cb7ebe66ae82e2 server_cookie=d6e2267ad7137d76 in_seq=0 out_seq=0 2024-09-11T00:31:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.949+0000 7fda39600640 10 client.4746 ms_handle_connect on v2:172.21.15.204:6834/1801149506 2024-09-11T00:31:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.950+0000 7fda39600640 1 -- 172.21.15.31:0/63402284 <== mds.0 v2:172.21.15.204:6834/1801149506 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x565210aad520 con 0x565210b869a0 2024-09-11T00:31:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda39600640 10 client.4746 handle_command_reply: tid=0 2024-09-11T00:31:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 client.4746 shutdown 2024-09-11T00:31:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 -- 172.21.15.31:0/63402284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 msgr2=0x565210b5afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b5afa0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fda440624f0 tx=0x7fda44061e90 comp rx=0 tx=0).stop 2024-09-11T00:31:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 -- 172.21.15.31:0/63402284 shutdown_connections 2024-09-11T00:31:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210b869a0 0x565210b86d80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b5afa0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 0x565210b6af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda39600640 0 client.4746 ms_handle_reset on v2:172.21.15.204:6834/1801149506 2024-09-11T00:31:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 --2- 172.21.15.31:0/63402284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 -- 172.21.15.31:0/63402284 >> 172.21.15.31:0/63402284 conn(0x565210b57fa0 msgr2=0x565210b35970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 -- 172.21.15.31:0/63402284 shutdown_connections 2024-09-11T00:31:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 1 -- 172.21.15.31:0/63402284 wait complete. 2024-09-11T00:31:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.951+0000 7fda561fc740 20 client.4746 trim_cache size 0 max 16384 2024-09-11T00:31:08.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.971+0000 7fda561fc740 1 Processor -- start 2024-09-11T00:31:08.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.971+0000 7fda561fc740 1 -- start start 2024-09-11T00:31:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda561fc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b58380 0x565210b6fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda561fc740 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b6c680 0x565210b58c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda561fc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b72c80 0x565210b59140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda561fc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c1fc2a0 con 0x565210b58380 2024-09-11T00:31:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda561fc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c1f9c90 con 0x565210b72c80 2024-09-11T00:31:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda561fc740 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fda4c1fa2b0 con 0x565210b6c680 2024-09-11T00:31:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda50a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b58380 0x565210b6fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda50a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b58380 0x565210b6fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46152/0 (socket says 172.21.15.31:46152) 2024-09-11T00:31:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda50a00640 1 -- 172.21.15.31:0/2649050906 learned_addr learned my addr 172.21.15.31:0/2649050906 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda43e00640 1 --2- 172.21.15.31:0/2649050906 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b6c680 0x565210b58c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda51400640 1 --2- 172.21.15.31:0/2649050906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b72c80 0x565210b59140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda50a00640 1 -- 172.21.15.31:0/2649050906 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b6c680 msgr2=0x565210b58c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda50a00640 1 --2- 172.21.15.31:0/2649050906 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b6c680 0x565210b58c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda50a00640 1 -- 172.21.15.31:0/2649050906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b72c80 msgr2=0x565210b59140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda50a00640 1 --2- 172.21.15.31:0/2649050906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b72c80 0x565210b59140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.972+0000 7fda50a00640 1 -- 172.21.15.31:0/2649050906 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565210b6f730 con 0x565210b58380 2024-09-11T00:31:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.973+0000 7fda50a00640 1 --2- 172.21.15.31:0/2649050906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b58380 0x565210b6fa70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x565210b6b830 tx=0x7fda3c00c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c9fe385f7dd097 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.973+0000 7fda3be00640 1 -- 172.21.15.31:0/2649050906 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c005310 con 0x565210b58380 2024-09-11T00:31:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.973+0000 7fda3be00640 1 -- 172.21.15.31:0/2649050906 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda3c0054b0 con 0x565210b58380 2024-09-11T00:31:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.973+0000 7fda3be00640 1 -- 172.21.15.31:0/2649050906 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c029610 con 0x565210b58380 2024-09-11T00:31:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.973+0000 7fda561fc740 1 -- 172.21.15.31:0/2649050906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b58380 msgr2=0x565210b6fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.973+0000 7fda561fc740 1 --2- 172.21.15.31:0/2649050906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b58380 0x565210b6fa70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x565210b6b830 tx=0x7fda3c00c000 comp rx=0 tx=0).stop 2024-09-11T00:31:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.974+0000 7fda561fc740 1 -- 172.21.15.31:0/2649050906 shutdown_connections 2024-09-11T00:31:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.974+0000 7fda561fc740 1 --2- 172.21.15.31:0/2649050906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b72c80 0x565210b59140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.974+0000 7fda561fc740 1 --2- 172.21.15.31:0/2649050906 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b6c680 0x565210b58c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.974+0000 7fda561fc740 1 --2- 172.21.15.31:0/2649050906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b58380 0x565210b6fa70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.974+0000 7fda561fc740 1 -- 172.21.15.31:0/2649050906 >> 172.21.15.31:0/2649050906 conn(0x565210b57fa0 msgr2=0x565210b6bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.974+0000 7fda561fc740 1 -- 172.21.15.31:0/2649050906 shutdown_connections 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.974+0000 7fda561fc740 1 -- 172.21.15.31:0/2649050906 wait complete. 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.974+0000 7fda561fc740 1 -- start start 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.974+0000 7fda561fc740 1 Processor -- start 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.975+0000 7fda561fc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b59a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.975+0000 7fda561fc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b6fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.975+0000 7fda561fc740 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 0x565210bedb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.975+0000 7fda561fc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x565210a41330 con 0x565210b6c680 2024-09-11T00:31:08.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.975+0000 7fda561fc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56521080b620 con 0x565210b58380 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.975+0000 7fda561fc740 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x565210b6a7f0 con 0x565210b72c80 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.975+0000 7fda561fc740 10 client.0 _ll_register_callbacks cb 0x7fda44062650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda43e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b6fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda43e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b6fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46162/0 (socket says 172.21.15.31:46162) 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda43e00640 1 -- 172.21.15.31:0/2302872805 learned_addr learned my addr 172.21.15.31:0/2302872805 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda50a00640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b59a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda43e00640 1 -- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 msgr2=0x565210bedb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda51400640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 0x565210bedb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda43e00640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 0x565210bedb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda43e00640 1 -- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 msgr2=0x565210b59a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda43e00640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b59a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:08.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda43e00640 1 -- 172.21.15.31:0/2302872805 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565210b34780 con 0x565210b6c680 2024-09-11T00:31:08.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda50a00640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b59a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:08.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda43e00640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b6fb50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fda34009b50 tx=0x7fda34004130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16f8dc85fd05f3f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:08.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda39600640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-11T00:31:08.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.976+0000 7fda39600640 1 -- 172.21.15.31:0/2302872805 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34043020 con 0x565210b6c680 2024-09-11T00:31:08.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda39600640 1 -- 172.21.15.31:0/2302872805 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fda3402b800 con 0x565210b6c680 2024-09-11T00:31:08.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda561fc740 1 -- 172.21.15.31:0/2302872805 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda3c0054b0 con 0x565210b6c680 2024-09-11T00:31:08.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda39600640 1 -- 172.21.15.31:0/2302872805 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fda3404d050 con 0x565210b6c680 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda561fc740 10 client.4752 fetch_fsmap learned FSMap version 15 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda561fc740 1 -- 172.21.15.31:0/2302872805 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x565210b69b30 con 0x565210b6c680 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda39600640 1 -- 172.21.15.31:0/2302872805 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda3c0054b0 con 0x565210b6c680 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda39600640 1 -- 172.21.15.31:0/2302872805 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 15) ==== 4136+0+0 (secure 0 0 0) 0x7fda34050020 con 0x565210b6c680 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda561fc740 10 client.4752 fetch_fsmap finished waiting for FSMap version 15 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda561fc740 10 client.4752 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda561fc740 20 client.4752 populate_metadata read hostname 'smithi031' 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda561fc740 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210bf78a0 0x565210bf7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda561fc740 4 client.4752 mds_command: new command op to 4390 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.977+0000 7fda561fc740 1 -- 172.21.15.31:0/2302872805 --> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] -- command(tid 0: {"prefix": "status"}) -- 0x565210aa8fd0 con 0x565210bf78a0 2024-09-11T00:31:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.978+0000 7fda50a00640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210bf78a0 0x565210bf7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:08.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.978+0000 7fda50a00640 1 -- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210bf78a0 msgr2=0x565210bf7c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-09-11T00:31:08.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.978+0000 7fda50a00640 1 -- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210bf78a0 msgr2=0x565210bf7c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T00:31:08.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.978+0000 7fda50a00640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210bf78a0 0x565210bf7c80 crc :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T00:31:08.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:08.978+0000 7fda50a00640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210bf78a0 0x565210bf7c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T00:31:09.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:31:09.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:31:09.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.180+0000 7fda50a00640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210bf78a0 0x565210bf7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:09.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.181+0000 7fda50a00640 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210bf78a0 0x565210bf7c80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44c92af053208687 server_cookie=7f6e98dcee3311d7 in_seq=0 out_seq=0 2024-09-11T00:31:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.181+0000 7fda39600640 10 client.4752 ms_handle_connect on v2:172.21.15.204:6834/1801149506 2024-09-11T00:31:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.181+0000 7fda39600640 1 -- 172.21.15.31:0/2302872805 <== mds.0 v2:172.21.15.204:6834/1801149506 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x565210aa8fd0 con 0x565210bf78a0 2024-09-11T00:31:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.181+0000 7fda39600640 10 client.4752 handle_command_reply: tid=0 2024-09-11T00:31:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 client.4752 shutdown 2024-09-11T00:31:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 -- 172.21.15.31:0/2302872805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 msgr2=0x565210b6fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b6fb50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fda34009b50 tx=0x7fda34004130 comp rx=0 tx=0).stop 2024-09-11T00:31:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 -- 172.21.15.31:0/2302872805 shutdown_connections 2024-09-11T00:31:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:6834/1801149506,v1:172.21.15.204:6835/1801149506] conn(0x565210bf78a0 0x565210bf7c80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x565210b72c80 0x565210bedb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x565210b6c680 0x565210b6fb50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 --2- 172.21.15.31:0/2302872805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x565210b58380 0x565210b59a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda39600640 0 client.4752 ms_handle_reset on v2:172.21.15.204:6834/1801149506 2024-09-11T00:31:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 -- 172.21.15.31:0/2302872805 >> 172.21.15.31:0/2302872805 conn(0x565210b57fa0 msgr2=0x565210b6aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 -- 172.21.15.31:0/2302872805 shutdown_connections 2024-09-11T00:31:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 1 -- 172.21.15.31:0/2302872805 wait complete. 2024-09-11T00:31:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.182+0000 7fda561fc740 20 client.4752 trim_cache size 0 max 16384 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "4a2c9b4f-c931-4bcb-847e-5521a1bafcf6", 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 4390, 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "a", 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 23.284132912, 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 9, 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 26, 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 22, 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 29.308214599999999 2024-09-11T00:31:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:31:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 -- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda28077f70 msgr2=0x7fda2807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda28077f70 0x7fda2807a430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0c5f20 tx=0x7fda3c0067a0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 -- 172.21.15.31:0/3654791370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4c1ef3a0 msgr2=0x7fda4c0ae830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4c1ef3a0 0x7fda4c0ae830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0c7e30 tx=0x7fda440021f0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 -- 172.21.15.31:0/3654791370 shutdown_connections 2024-09-11T00:31:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda28077f70 0x7fda2807a430 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4c1ef3a0 0x7fda4c0ae830 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda4c14e530 0x7fda4c1eee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 --2- 172.21.15.31:0/3654791370 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda30004000 0x7fda4c150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 -- 172.21.15.31:0/3654791370 >> 172.21.15.31:0/3654791370 conn(0x7fda4c068f10 msgr2=0x7fda4c0acbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 -- 172.21.15.31:0/3654791370 shutdown_connections 2024-09-11T00:31:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.183+0000 7fda51e00640 1 -- 172.21.15.31:0/3654791370 wait complete. 2024-09-11T00:31:09.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:09.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.359+0000 7f63a9c00640 1 -- 172.21.15.31:0/2977177514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63a40c23a0 msgr2=0x7f63a40c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.359+0000 7f63a9c00640 1 --2- 172.21.15.31:0/2977177514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63a40c23a0 0x7f63a40c30c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6390009830 tx=0x7f639002a5a0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.360+0000 7f63a9c00640 1 -- 172.21.15.31:0/2977177514 shutdown_connections 2024-09-11T00:31:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.360+0000 7f63a9c00640 1 --2- 172.21.15.31:0/2977177514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63a40c23a0 0x7f63a40c30c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.360+0000 7f63a9c00640 1 -- 172.21.15.31:0/2977177514 >> 172.21.15.31:0/2977177514 conn(0x7f63a40a9750 msgr2=0x7f63a40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.360+0000 7f63a9c00640 1 -- 172.21.15.31:0/2977177514 shutdown_connections 2024-09-11T00:31:09.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.360+0000 7f63a9c00640 1 -- 172.21.15.31:0/2977177514 wait complete. 2024-09-11T00:31:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.361+0000 7f63a9c00640 1 Processor -- start 2024-09-11T00:31:09.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.361+0000 7f63a9c00640 1 -- start start 2024-09-11T00:31:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a9c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63a4072480 0x7f63a41dd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a9c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63a40c23a0 0x7f63a41de190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a9c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63a41518b0 0x7f63a41de6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a9c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f63a4074080 con 0x7f63a41518b0 2024-09-11T00:31:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a9c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f63a40c3d50 con 0x7f63a40c23a0 2024-09-11T00:31:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a9c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f63a4074200 con 0x7f63a4072480 2024-09-11T00:31:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63a41518b0 0x7f63a41de6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63a41518b0 0x7f63a41de6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46182/0 (socket says 172.21.15.31:46182) 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a3400640 1 -- 172.21.15.31:0/3072967672 learned_addr learned my addr 172.21.15.31:0/3072967672 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a2a00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63a4072480 0x7f63a41dd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.362+0000 7f63a2000640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63a40c23a0 0x7f63a41de190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f63a3400640 1 -- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63a4072480 msgr2=0x7f63a41dd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f63a3400640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63a4072480 0x7f63a41dd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f63a3400640 1 -- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63a40c23a0 msgr2=0x7f63a41de190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f63a3400640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63a40c23a0 0x7f63a41de190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f63a3400640 1 -- 172.21.15.31:0/3072967672 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a406ebb0 con 0x7f63a41518b0 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f63a2a00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63a4072480 0x7f63a41dd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f63a3400640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63a41518b0 0x7f63a41de6d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f639c00c7b0 tx=0x7f639c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f639aa00640 1 -- 172.21.15.31:0/3072967672 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c004b60 con 0x7f63a41518b0 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f639aa00640 1 -- 172.21.15.31:0/3072967672 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f639c004df0 con 0x7f63a41518b0 2024-09-11T00:31:09.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f639aa00640 1 -- 172.21.15.31:0/3072967672 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c00aa90 con 0x7f63a41518b0 2024-09-11T00:31:09.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f63a9c00640 1 -- 172.21.15.31:0/3072967672 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63a40c57a0 con 0x7f63a41518b0 2024-09-11T00:31:09.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.363+0000 7f63a9c00640 1 -- 172.21.15.31:0/3072967672 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63a406feb0 con 0x7f63a41518b0 2024-09-11T00:31:09.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.364+0000 7f63a9c00640 1 -- 172.21.15.31:0/3072967672 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63a40a8f70 con 0x7f63a41518b0 2024-09-11T00:31:09.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.366+0000 7f639aa00640 1 -- 172.21.15.31:0/3072967672 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f639c00ac30 con 0x7f63a41518b0 2024-09-11T00:31:09.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.366+0000 7f639aa00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6384077f70 0x7f638407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:09.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.367+0000 7f639aa00640 1 -- 172.21.15.31:0/3072967672 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f639c09a400 con 0x7f63a41518b0 2024-09-11T00:31:09.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.367+0000 7f63a2a00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6384077f70 0x7f638407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:09.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.367+0000 7f63a2a00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6384077f70 0x7f638407a430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6394000f60 tx=0x7f6394009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:09.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.371+0000 7f639aa00640 1 -- 172.21.15.31:0/3072967672 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f63a40a8f70 con 0x7f63a41518b0 2024-09-11T00:31:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.599+0000 7f63a9c00640 1 -- 172.21.15.31:0/3072967672 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63a4153f20 con 0x7f63a41518b0 2024-09-11T00:31:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.600+0000 7f639aa00640 1 -- 172.21.15.31:0/3072967672 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f639c020070 con 0x7f63a41518b0 2024-09-11T00:31:09.601 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:09.602 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:09.602 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:09.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 -- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6384077f70 msgr2=0x7f638407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6384077f70 0x7f638407a430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6394000f60 tx=0x7f6394009040 comp rx=0 tx=0).stop 2024-09-11T00:31:09.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 -- 172.21.15.31:0/3072967672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63a41518b0 msgr2=0x7f63a41de6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63a41518b0 0x7f63a41de6d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f639c00c7b0 tx=0x7f639c00cc70 comp rx=0 tx=0).stop 2024-09-11T00:31:09.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 -- 172.21.15.31:0/3072967672 shutdown_connections 2024-09-11T00:31:09.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6384077f70 0x7f638407a430 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63a41518b0 0x7f63a41de6d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63a40c23a0 0x7f63a41de190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 --2- 172.21.15.31:0/3072967672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63a4072480 0x7f63a41dd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 -- 172.21.15.31:0/3072967672 >> 172.21.15.31:0/3072967672 conn(0x7f63a40a9750 msgr2=0x7f63a40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:09.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 -- 172.21.15.31:0/3072967672 shutdown_connections 2024-09-11T00:31:09.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.602+0000 7f63a9c00640 1 -- 172.21.15.31:0/3072967672 wait complete. 2024-09-11T00:31:09.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:31:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.778+0000 7f5605200640 1 -- 172.21.15.31:0/3323792816 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56000c23a0 msgr2=0x7f56001516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.778+0000 7f5605200640 1 --2- 172.21.15.31:0/3323792816 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56000c23a0 0x7f56001516e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009830 tx=0x7f55e802a5a0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.778+0000 7f5605200640 1 -- 172.21.15.31:0/3323792816 shutdown_connections 2024-09-11T00:31:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.778+0000 7f5605200640 1 --2- 172.21.15.31:0/3323792816 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56000c23a0 0x7f56001516e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.778+0000 7f5605200640 1 --2- 172.21.15.31:0/3323792816 >> v2:172.21.15.204:3300/0 conn(0x7f5600072480 0x7f56001511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.778+0000 7f5605200640 1 -- 172.21.15.31:0/3323792816 >> 172.21.15.31:0/3323792816 conn(0x7f56000a9750 msgr2=0x7f56000a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.778+0000 7f5605200640 1 -- 172.21.15.31:0/3323792816 shutdown_connections 2024-09-11T00:31:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.778+0000 7f5605200640 1 -- 172.21.15.31:0/3323792816 wait complete. 2024-09-11T00:31:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.779+0000 7f5605200640 1 Processor -- start 2024-09-11T00:31:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.779+0000 7f5605200640 1 -- start start 2024-09-11T00:31:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.780+0000 7f5605200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5600072480 0x7f5600147700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.780+0000 7f5605200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56000c23a0 0x7f56000c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.780+0000 7f5605200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5600153f20 0x7f5600156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.780+0000 7f5605200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5600074200 con 0x7f5600072480 2024-09-11T00:31:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.780+0000 7f5605200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5600074080 con 0x7f5600153f20 2024-09-11T00:31:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.780+0000 7f5605200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5600072860 con 0x7f56000c23a0 2024-09-11T00:31:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.780+0000 7f55fde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5600072480 0x7f5600147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.780+0000 7f55fde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5600072480 0x7f5600147700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46194/0 (socket says 172.21.15.31:46194) 2024-09-11T00:31:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.780+0000 7f55fde00640 1 -- 172.21.15.31:0/668075443 learned_addr learned my addr 172.21.15.31:0/668075443 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55f5400640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56000c23a0 0x7f56000c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55fe800640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5600153f20 0x7f5600156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55fde00640 1 -- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56000c23a0 msgr2=0x7f56000c0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55fde00640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56000c23a0 0x7f56000c0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55fde00640 1 -- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5600153f20 msgr2=0x7f5600156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55fde00640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5600153f20 0x7f5600156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55fde00640 1 -- 172.21.15.31:0/668075443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f560006ecb0 con 0x7f5600072480 2024-09-11T00:31:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55f5400640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56000c23a0 0x7f56000c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55fde00640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5600072480 0x7f5600147700 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f55f000ea60 tx=0x7f55f000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55f6a00640 1 -- 172.21.15.31:0/668075443 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f000cde0 con 0x7f5600072480 2024-09-11T00:31:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55f6a00640 1 -- 172.21.15.31:0/668075443 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55f0004020 con 0x7f5600072480 2024-09-11T00:31:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f55f6a00640 1 -- 172.21.15.31:0/668075443 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0004450 con 0x7f5600072480 2024-09-11T00:31:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.781+0000 7f5605200640 1 -- 172.21.15.31:0/668075443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f560006ef50 con 0x7f5600072480 2024-09-11T00:31:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.782+0000 7f5605200640 1 -- 172.21.15.31:0/668075443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f560014cdf0 con 0x7f5600072480 2024-09-11T00:31:09.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.783+0000 7f5605200640 1 -- 172.21.15.31:0/668075443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f560006fb10 con 0x7f5600072480 2024-09-11T00:31:09.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.784+0000 7f55f6a00640 1 -- 172.21.15.31:0/668075443 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f55f00035f0 con 0x7f5600072480 2024-09-11T00:31:09.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.785+0000 7f55f6a00640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55dc077f70 0x7f55dc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:09.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.785+0000 7f55f6a00640 1 -- 172.21.15.31:0/668075443 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f55f009aac0 con 0x7f5600072480 2024-09-11T00:31:09.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.785+0000 7f55f5400640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55dc077f70 0x7f55dc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:09.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.786+0000 7f55f5400640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55dc077f70 0x7f55dc07a430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f55e8034000 tx=0x7f55e8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.789+0000 7f55f6a00640 1 -- 172.21.15.31:0/668075443 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f560006fb10 con 0x7f5600072480 2024-09-11T00:31:09.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.987+0000 7f5605200640 1 -- 172.21.15.31:0/668075443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5600073630 con 0x7f5600072480 2024-09-11T00:31:09.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.989+0000 7f55f6a00640 1 -- 172.21.15.31:0/668075443 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20383 (secure 0 0 0) 0x7f5600073630 con 0x7f5600072480 2024-09-11T00:31:09.991 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:09.992 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:55.962415+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:31:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 -- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55dc077f70 msgr2=0x7f55dc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55dc077f70 0x7f55dc07a430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f55e8034000 tx=0x7f55e8004730 comp rx=0 tx=0).stop 2024-09-11T00:31:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 -- 172.21.15.31:0/668075443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5600072480 msgr2=0x7f5600147700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5600072480 0x7f5600147700 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f55f000ea60 tx=0x7f55f000ef20 comp rx=0 tx=0).stop 2024-09-11T00:31:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 -- 172.21.15.31:0/668075443 shutdown_connections 2024-09-11T00:31:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55dc077f70 0x7f55dc07a430 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5600153f20 0x7f5600156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56000c23a0 0x7f56000c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 --2- 172.21.15.31:0/668075443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5600072480 0x7f5600147700 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 -- 172.21.15.31:0/668075443 >> 172.21.15.31:0/668075443 conn(0x7f56000a9750 msgr2=0x7f560014a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 -- 172.21.15.31:0/668075443 shutdown_connections 2024-09-11T00:31:09.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:09.991+0000 7f5605200640 1 -- 172.21.15.31:0/668075443 wait complete. 2024-09-11T00:31:10.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.168+0000 7fa197800640 1 -- 172.21.15.31:0/3939098421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1900c23a0 msgr2=0x7fa1900c27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.168+0000 7fa197800640 1 --2- 172.21.15.31:0/3939098421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1900c23a0 0x7fa1900c27d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa1900c89e0 tx=0x7fa18802a820 comp rx=0 tx=0).stop 2024-09-11T00:31:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.168+0000 7fa197800640 1 -- 172.21.15.31:0/3939098421 shutdown_connections 2024-09-11T00:31:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.168+0000 7fa197800640 1 --2- 172.21.15.31:0/3939098421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1900c23a0 0x7fa1900c27d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:10.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.168+0000 7fa197800640 1 --2- 172.21.15.31:0/3939098421 >> v2:172.21.15.204:3300/0 conn(0x7fa190156300 0x7fa1900c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.168+0000 7fa197800640 1 -- 172.21.15.31:0/3939098421 >> 172.21.15.31:0/3939098421 conn(0x7fa1900a9750 msgr2=0x7fa1900a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.168+0000 7fa197800640 1 -- 172.21.15.31:0/3939098421 shutdown_connections 2024-09-11T00:31:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.168+0000 7fa197800640 1 -- 172.21.15.31:0/3939098421 wait complete. 2024-09-11T00:31:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.170+0000 7fa197800640 1 Processor -- start 2024-09-11T00:31:10.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.170+0000 7fa197800640 1 -- start start 2024-09-11T00:31:10.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa197800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa19006fa70 0x7fa1900a9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa197800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900c23a0 0x7fa1901df7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa197800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa190156300 0x7fa190151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa197800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa1900c77a0 con 0x7fa19006fa70 2024-09-11T00:31:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa197800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa1900c7620 con 0x7fa190156300 2024-09-11T00:31:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa197800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa190156df0 con 0x7fa1900c23a0 2024-09-11T00:31:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa194e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa19006fa70 0x7fa1900a9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa194e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa19006fa70 0x7fa1900a9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46216/0 (socket says 172.21.15.31:46216) 2024-09-11T00:31:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa194e00640 1 -- 172.21.15.31:0/3152434030 learned_addr learned my addr 172.21.15.31:0/3152434030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa187e00640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900c23a0 0x7fa1901df7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:10.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa195800640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa190156300 0x7fa190151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa194e00640 1 -- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900c23a0 msgr2=0x7fa1901df7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa194e00640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900c23a0 0x7fa1901df7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa194e00640 1 -- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa190156300 msgr2=0x7fa190151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa194e00640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa190156300 0x7fa190151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa194e00640 1 -- 172.21.15.31:0/3152434030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1901e1060 con 0x7fa19006fa70 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa195800640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa190156300 0x7fa190151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa187e00640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900c23a0 0x7fa1901df7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.171+0000 7fa194e00640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa19006fa70 0x7fa1900a9050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa18000d740 tx=0x7fa18000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.172+0000 7fa18d600640 1 -- 172.21.15.31:0/3152434030 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa180017070 con 0x7fa19006fa70 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.172+0000 7fa18d600640 1 -- 172.21.15.31:0/3152434030 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa180009d20 con 0x7fa19006fa70 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.172+0000 7fa18d600640 1 -- 172.21.15.31:0/3152434030 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1800106c0 con 0x7fa19006fa70 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.172+0000 7fa197800640 1 -- 172.21.15.31:0/3152434030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1901dfd30 con 0x7fa19006fa70 2024-09-11T00:31:10.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.172+0000 7fa197800640 1 -- 172.21.15.31:0/3152434030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa190073800 con 0x7fa19006fa70 2024-09-11T00:31:10.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.173+0000 7fa197800640 1 -- 172.21.15.31:0/3152434030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa19014d120 con 0x7fa19006fa70 2024-09-11T00:31:10.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.174+0000 7fa18d600640 1 -- 172.21.15.31:0/3152434030 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa180010860 con 0x7fa19006fa70 2024-09-11T00:31:10.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.175+0000 7fa18d600640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa16c077f70 0x7fa16c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:10.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.175+0000 7fa18d600640 1 -- 172.21.15.31:0/3152434030 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fa18009a4b0 con 0x7fa19006fa70 2024-09-11T00:31:10.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.175+0000 7fa187e00640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa16c077f70 0x7fa16c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:10.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.176+0000 7fa187e00640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa16c077f70 0x7fa16c07a430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa190074fb0 tx=0x7fa178009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.179+0000 7fa18d600640 1 -- 172.21.15.31:0/3152434030 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa1901dfd30 con 0x7fa19006fa70 2024-09-11T00:31:10.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.416+0000 7fa197800640 1 -- 172.21.15.31:0/3152434030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1900c7240 con 0x7fa19006fa70 2024-09-11T00:31:10.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.416+0000 7fa18d600640 1 -- 172.21.15.31:0/3152434030 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fa180062c70 con 0x7fa19006fa70 2024-09-11T00:31:10.419 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:10.419 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:10.420 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 -- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa16c077f70 msgr2=0x7fa16c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa16c077f70 0x7fa16c07a430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa190074fb0 tx=0x7fa178009040 comp rx=0 tx=0).stop 2024-09-11T00:31:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 -- 172.21.15.31:0/3152434030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa19006fa70 msgr2=0x7fa1900a9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa19006fa70 0x7fa1900a9050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa18000d740 tx=0x7fa18000dc00 comp rx=0 tx=0).stop 2024-09-11T00:31:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 -- 172.21.15.31:0/3152434030 shutdown_connections 2024-09-11T00:31:10.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa16c077f70 0x7fa16c07a430 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa190156300 0x7fa190151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1900c23a0 0x7fa1901df7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 --2- 172.21.15.31:0/3152434030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa19006fa70 0x7fa1900a9050 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 -- 172.21.15.31:0/3152434030 >> 172.21.15.31:0/3152434030 conn(0x7fa1900a9750 msgr2=0x7fa19014bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 -- 172.21.15.31:0/3152434030 shutdown_connections 2024-09-11T00:31:10.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:10.419+0000 7fa197800640 1 -- 172.21.15.31:0/3152434030 wait complete. 2024-09-11T00:31:10.840 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-11T00:30:54.411975+0000', 'modified': '2024-09-11T00:30:58.435146+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4374}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4374': {'gid': 4374, 'name': 'b', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.204:6837/3441932736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6836', 'nonce': 3441932736}, {'type': 'v1', 'addr': '172.21.15.204:6837', 'nonce': 3441932736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'b', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4374, 'qdb_cluster': [4374]} 2024-09-11T00:31:10.840 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-11T00:31:10.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-11T00:31:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.006+0000 7fde6c600640 1 -- 172.21.15.31:0/4114245324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde64156ad0 msgr2=0x7fde64153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.006+0000 7fde6c600640 1 --2- 172.21.15.31:0/4114245324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde64156ad0 0x7fde64153480 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fde50009860 tx=0x7fde5002a5a0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.006+0000 7fde6c600640 1 -- 172.21.15.31:0/4114245324 shutdown_connections 2024-09-11T00:31:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.006+0000 7fde6c600640 1 --2- 172.21.15.31:0/4114245324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde64156ad0 0x7fde64153480 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.006+0000 7fde6c600640 1 -- 172.21.15.31:0/4114245324 >> 172.21.15.31:0/4114245324 conn(0x7fde640a9750 msgr2=0x7fde640a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.006+0000 7fde6c600640 1 -- 172.21.15.31:0/4114245324 shutdown_connections 2024-09-11T00:31:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.006+0000 7fde6c600640 1 -- 172.21.15.31:0/4114245324 wait complete. 2024-09-11T00:31:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.008+0000 7fde6c600640 1 Processor -- start 2024-09-11T00:31:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.008+0000 7fde6c600640 1 -- start start 2024-09-11T00:31:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde6c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde640c5b10 0x7fde640721f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde6c600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde641539c0 0x7fde64155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde6c600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde64156ad0 0x7fde641ebd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde6c600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fde640c7320 con 0x7fde640c5b10 2024-09-11T00:31:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde6c600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fde640c3950 con 0x7fde641539c0 2024-09-11T00:31:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde6c600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fde64159d80 con 0x7fde64156ad0 2024-09-11T00:31:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde69c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde640c5b10 0x7fde640721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde69c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde640c5b10 0x7fde640721f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46218/0 (socket says 172.21.15.31:46218) 2024-09-11T00:31:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde69c00640 1 -- 172.21.15.31:0/1232751003 learned_addr learned my addr 172.21.15.31:0/1232751003 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde69200640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde641539c0 0x7fde64155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.009+0000 7fde6a600640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde64156ad0 0x7fde641ebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde69c00640 1 -- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde64156ad0 msgr2=0x7fde641ebd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde69c00640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde64156ad0 0x7fde641ebd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde69c00640 1 -- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde641539c0 msgr2=0x7fde64155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde69c00640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde641539c0 0x7fde64155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde69c00640 1 -- 172.21.15.31:0/1232751003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde64072c30 con 0x7fde640c5b10 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde69c00640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde640c5b10 0x7fde640721f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fde5800c490 tx=0x7fde5800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde61600640 1 -- 172.21.15.31:0/1232751003 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde580052d0 con 0x7fde640c5b10 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde61600640 1 -- 172.21.15.31:0/1232751003 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde58005490 con 0x7fde640c5b10 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde6c600640 1 -- 172.21.15.31:0/1232751003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde64072990 con 0x7fde640c5b10 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde61600640 1 -- 172.21.15.31:0/1232751003 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde58012890 con 0x7fde640c5b10 2024-09-11T00:31:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.010+0000 7fde6c600640 1 -- 172.21.15.31:0/1232751003 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde640c6a50 con 0x7fde640c5b10 2024-09-11T00:31:11.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.012+0000 7fde61600640 1 -- 172.21.15.31:0/1232751003 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fde58012a30 con 0x7fde640c5b10 2024-09-11T00:31:11.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.013+0000 7fde61600640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde44077f70 0x7fde4407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.014+0000 7fde61600640 1 -- 172.21.15.31:0/1232751003 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fde58099b30 con 0x7fde640c5b10 2024-09-11T00:31:11.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.014+0000 7fde69200640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde44077f70 0x7fde4407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.014+0000 7fde6f443740 1 Processor -- start 2024-09-11T00:31:11.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.014+0000 7fde6f443740 1 -- start start 2024-09-11T00:31:11.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde69200640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde44077f70 0x7fde4407a430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fde50034000 tx=0x7fde50004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde6f443740 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e2db0 0x5591597be400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde6f443740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e3190 0x5591597fd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde6f443740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597fda90 0x5591597c1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde6f443740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559159746430 con 0x5591597e3190 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde6f443740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5591597365d0 con 0x5591597fda90 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde6f443740 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x559159725a00 con 0x5591597e2db0 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde69200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e3190 0x5591597fd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde69200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e3190 0x5591597fd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46222/0 (socket says 172.21.15.31:46222) 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde69200640 1 -- 172.21.15.31:0/2752817330 learned_addr learned my addr 172.21.15.31:0/2752817330 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.015+0000 7fde69c00640 1 --2- 172.21.15.31:0/2752817330 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e2db0 0x5591597be400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde6a600640 1 --2- 172.21.15.31:0/2752817330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597fda90 0x5591597c1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde69200640 1 -- 172.21.15.31:0/2752817330 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e2db0 msgr2=0x5591597be400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:11.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde69200640 1 --2- 172.21.15.31:0/2752817330 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e2db0 0x5591597be400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde69200640 1 -- 172.21.15.31:0/2752817330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597fda90 msgr2=0x5591597c1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:11.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde69200640 1 --2- 172.21.15.31:0/2752817330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597fda90 0x5591597c1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde69200640 1 -- 172.21.15.31:0/2752817330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591596ee520 con 0x5591597e3190 2024-09-11T00:31:11.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde6a600640 1 --2- 172.21.15.31:0/2752817330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597fda90 0x5591597c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:11.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde69200640 1 --2- 172.21.15.31:0/2752817330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e3190 0x5591597fd550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fde50000c00 tx=0x7fde5003e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13862e594e81880d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:11.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde56000640 1 -- 172.21.15.31:0/2752817330 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5004d070 con 0x5591597e3190 2024-09-11T00:31:11.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde56000640 1 -- 172.21.15.31:0/2752817330 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde50003bb0 con 0x5591597e3190 2024-09-11T00:31:11.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.016+0000 7fde56000640 1 -- 172.21.15.31:0/2752817330 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde50049450 con 0x5591597e3190 2024-09-11T00:31:11.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.017+0000 7fde6f443740 1 -- 172.21.15.31:0/2752817330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e3190 msgr2=0x5591597fd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.017+0000 7fde6f443740 1 --2- 172.21.15.31:0/2752817330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e3190 0x5591597fd550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fde50000c00 tx=0x7fde5003e070 comp rx=0 tx=0).stop 2024-09-11T00:31:11.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.017+0000 7fde6f443740 1 -- 172.21.15.31:0/2752817330 shutdown_connections 2024-09-11T00:31:11.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.017+0000 7fde6f443740 1 --2- 172.21.15.31:0/2752817330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597fda90 0x5591597c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.017+0000 7fde6f443740 1 --2- 172.21.15.31:0/2752817330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e3190 0x5591597fd550 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.017+0000 7fde6f443740 1 --2- 172.21.15.31:0/2752817330 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e2db0 0x5591597be400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.017+0000 7fde6f443740 1 -- 172.21.15.31:0/2752817330 >> 172.21.15.31:0/2752817330 conn(0x5591597beed0 msgr2=0x5591597c26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.017+0000 7fde6f443740 1 -- 172.21.15.31:0/2752817330 shutdown_connections 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.017+0000 7fde6f443740 1 -- 172.21.15.31:0/2752817330 wait complete. 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.018+0000 7fde6f443740 1 -- start start 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.018+0000 7fde6f443740 1 Processor -- start 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde6f443740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597f5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde6f443740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597fa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde6f443740 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e1f70 0x5591597e12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde6f443740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5591596cc330 con 0x5591597fda90 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde6f443740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559159729810 con 0x5591597e3190 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde6f443740 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x559159496620 con 0x5591597e1f70 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde6f443740 10 client.0 _ll_register_callbacks cb 0x5591596d1420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-11T00:31:11.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde69200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597fa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde69200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597fa2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46228/0 (socket says 172.21.15.31:46228) 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde69200640 1 -- 172.21.15.31:0/144977621 learned_addr learned my addr 172.21.15.31:0/144977621 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde69c00640 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde6a600640 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e1f70 0x5591597e12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde69200640 1 -- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e1f70 msgr2=0x5591597e12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde69200640 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e1f70 0x5591597e12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde69200640 1 -- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 msgr2=0x5591597f5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde69200640 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597f5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde69200640 1 -- 172.21.15.31:0/144977621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559159487850 con 0x5591597fda90 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.019+0000 7fde69c00640 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597f5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.020+0000 7fde69200640 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597fa2c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fde50005b60 tx=0x7fde50053ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6af40157c7755e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:11.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.020+0000 7fde4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.020+0000 7fde4aa00640 1 -- 172.21.15.31:0/144977621 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5004d070 con 0x5591597fda90 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.020+0000 7fde6f443740 1 -- 172.21.15.31:0/144977621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde50003bb0 con 0x5591597fda90 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.020+0000 7fde4aa00640 1 -- 172.21.15.31:0/144977621 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fde50049650 con 0x5591597fda90 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.020+0000 7fde4aa00640 1 -- 172.21.15.31:0/144977621 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fde50063050 con 0x5591597fda90 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.020+0000 7fde6f443740 10 client.4785 fetch_fsmap learned FSMap version 15 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.020+0000 7fde6f443740 1 -- 172.21.15.31:0/144977621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x5591594fe6b0 con 0x5591597fda90 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.020+0000 7fde4aa00640 1 -- 172.21.15.31:0/144977621 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde50003bb0 con 0x5591597fda90 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.021+0000 7fde4aa00640 1 -- 172.21.15.31:0/144977621 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 15) ==== 4136+0+0 (secure 0 0 0) 0x7fde50066020 con 0x5591597fda90 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.021+0000 7fde6f443740 10 client.4785 fetch_fsmap finished waiting for FSMap version 15 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.021+0000 7fde6f443740 10 client.4785 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-11T00:31:11.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.021+0000 7fde6f443740 20 client.4785 populate_metadata read hostname 'smithi031' 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.021+0000 7fde6f443740 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x5591598120c0 0x55915981cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.021+0000 7fde6f443740 4 client.4785 mds_command: new command op to 4374 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.021+0000 7fde6f443740 1 -- 172.21.15.31:0/144977621 --> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559159738520 con 0x5591598120c0 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.021+0000 7fde69c00640 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x5591598120c0 0x55915981cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.022+0000 7fde69c00640 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x5591598120c0 0x55915981cf10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25f49595213f264d server_cookie=dcbbe6acf55ea7ea in_seq=0 out_seq=0 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.022+0000 7fde4aa00640 10 client.4785 ms_handle_connect on v2:172.21.15.204:6836/3441932736 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde4aa00640 1 -- 172.21.15.31:0/144977621 <== mds.0 v2:172.21.15.204:6836/3441932736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559159738520 con 0x5591598120c0 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde4aa00640 10 client.4785 handle_command_reply: tid=0 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 client.4785 shutdown 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 -- 172.21.15.31:0/144977621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 msgr2=0x5591597fa2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597fa2c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fde50005b60 tx=0x7fde50053ba0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 -- 172.21.15.31:0/144977621 shutdown_connections 2024-09-11T00:31:11.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x5591598120c0 0x55915981cf10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e1f70 0x5591597e12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597fa2c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde4aa00640 0 client.4785 ms_handle_reset on v2:172.21.15.204:6836/3441932736 2024-09-11T00:31:11.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 --2- 172.21.15.31:0/144977621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597f5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 -- 172.21.15.31:0/144977621 >> 172.21.15.31:0/144977621 conn(0x5591597e2db0 msgr2=0x5591597c0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:11.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 -- 172.21.15.31:0/144977621 shutdown_connections 2024-09-11T00:31:11.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.023+0000 7fde6f443740 1 -- 172.21.15.31:0/144977621 wait complete. 2024-09-11T00:31:11.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.024+0000 7fde6f443740 20 client.4785 trim_cache size 0 max 16384 2024-09-11T00:31:11.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.043+0000 7fde6f443740 1 Processor -- start 2024-09-11T00:31:11.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.043+0000 7fde6f443740 1 -- start start 2024-09-11T00:31:11.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.043+0000 7fde6f443740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e1f70 0x55915980e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.043+0000 7fde6f443740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597fb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.043+0000 7fde6f443740 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597fda90 0x5591597fb640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.043+0000 7fde6f443740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fde641faaf0 con 0x5591597e1f70 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.043+0000 7fde6f443740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fde641fbb80 con 0x5591597e3190 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.043+0000 7fde6f443740 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fde641f9550 con 0x5591597fda90 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e1f70 0x55915980e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e1f70 0x55915980e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46238/0 (socket says 172.21.15.31:46238) 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69c00640 1 -- 172.21.15.31:0/2681665503 learned_addr learned my addr 172.21.15.31:0/2681665503 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde6a600640 1 --2- 172.21.15.31:0/2681665503 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597fda90 0x5591597fb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69200640 1 --2- 172.21.15.31:0/2681665503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597fb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69c00640 1 -- 172.21.15.31:0/2681665503 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597fda90 msgr2=0x5591597fb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:11.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69c00640 1 --2- 172.21.15.31:0/2681665503 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597fda90 0x5591597fb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69c00640 1 -- 172.21.15.31:0/2681665503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 msgr2=0x5591597fb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:11.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69c00640 1 --2- 172.21.15.31:0/2681665503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597fb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69c00640 1 -- 172.21.15.31:0/2681665503 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591597e4300 con 0x5591597e1f70 2024-09-11T00:31:11.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69200640 1 --2- 172.21.15.31:0/2681665503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597fb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:11.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde69c00640 1 --2- 172.21.15.31:0/2681665503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e1f70 0x55915980e2d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde58097d20 tx=0x7fde58065040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04a2238c4f48608 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde55600640 1 -- 172.21.15.31:0/2681665503 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde58095070 con 0x5591597e1f70 2024-09-11T00:31:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde55600640 1 -- 172.21.15.31:0/2681665503 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde58012d50 con 0x5591597e1f70 2024-09-11T00:31:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.044+0000 7fde55600640 1 -- 172.21.15.31:0/2681665503 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5807b070 con 0x5591597e1f70 2024-09-11T00:31:11.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.045+0000 7fde6f443740 1 -- 172.21.15.31:0/2681665503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e1f70 msgr2=0x55915980e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.045+0000 7fde6f443740 1 --2- 172.21.15.31:0/2681665503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e1f70 0x55915980e2d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde58097d20 tx=0x7fde58065040 comp rx=0 tx=0).stop 2024-09-11T00:31:11.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.045+0000 7fde6f443740 1 -- 172.21.15.31:0/2681665503 shutdown_connections 2024-09-11T00:31:11.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.045+0000 7fde6f443740 1 --2- 172.21.15.31:0/2681665503 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597fda90 0x5591597fb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.045+0000 7fde6f443740 1 --2- 172.21.15.31:0/2681665503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597fb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.045+0000 7fde6f443740 1 --2- 172.21.15.31:0/2681665503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597e1f70 0x55915980e2d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.045+0000 7fde6f443740 1 -- 172.21.15.31:0/2681665503 >> 172.21.15.31:0/2681665503 conn(0x5591597e2db0 msgr2=0x5591597c0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:11.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.045+0000 7fde6f443740 1 -- 172.21.15.31:0/2681665503 shutdown_connections 2024-09-11T00:31:11.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.045+0000 7fde6f443740 1 -- 172.21.15.31:0/2681665503 wait complete. 2024-09-11T00:31:11.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.046+0000 7fde6f443740 1 -- start start 2024-09-11T00:31:11.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.046+0000 7fde6f443740 1 Processor -- start 2024-09-11T00:31:11.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6f443740 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e1f70 0x5591597be870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6f443740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597fb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6f443740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6f443740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5591597e1d80 con 0x5591597fda90 2024-09-11T00:31:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6f443740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fde641f9ba0 con 0x5591597e3190 2024-09-11T00:31:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6f443740 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x559159811b30 con 0x5591597e1f70 2024-09-11T00:31:11.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6f443740 10 client.0 _ll_register_callbacks cb 0x7fde500537b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-11T00:31:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597f6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46240/0 (socket says 172.21.15.31:46240) 2024-09-11T00:31:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6a600640 1 -- 172.21.15.31:0/511281221 learned_addr learned my addr 172.21.15.31:0/511281221 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde69c00640 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e1f70 0x5591597be870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde69200640 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597fb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6a600640 1 -- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e1f70 msgr2=0x5591597be870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6a600640 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e1f70 0x5591597be870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6a600640 1 -- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 msgr2=0x5591597fb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T00:31:11.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6a600640 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597fb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.047+0000 7fde6a600640 1 -- 172.21.15.31:0/511281221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591597e2720 con 0x5591597fda90 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.048+0000 7fde6a600640 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597f6dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fde5c017ec0 tx=0x7fde5c015780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d0876c7196bf7c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.048+0000 7fde4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.048+0000 7fde4aa00640 1 -- 172.21.15.31:0/511281221 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c01f3f0 con 0x5591597fda90 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.048+0000 7fde4aa00640 1 -- 172.21.15.31:0/511281221 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fde5c01f590 con 0x5591597fda90 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.048+0000 7fde6f443740 1 -- 172.21.15.31:0/511281221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5591597bf630 con 0x5591597fda90 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.048+0000 7fde4aa00640 1 -- 172.21.15.31:0/511281221 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fde5c029050 con 0x5591597fda90 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.048+0000 7fde6f443740 10 client.4791 fetch_fsmap learned FSMap version 15 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.048+0000 7fde6f443740 1 -- 172.21.15.31:0/511281221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x5591597fb770 con 0x5591597fda90 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde4aa00640 1 -- 172.21.15.31:0/511281221 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x5591597bf630 con 0x5591597fda90 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde4aa00640 1 -- 172.21.15.31:0/511281221 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 15) ==== 4136+0+0 (secure 0 0 0) 0x7fde5c02c020 con 0x5591597fda90 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde6f443740 10 client.4791 fetch_fsmap finished waiting for FSMap version 15 2024-09-11T00:31:11.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde6f443740 10 client.4791 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-11T00:31:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde6f443740 20 client.4791 populate_metadata read hostname 'smithi031' 2024-09-11T00:31:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde6f443740 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x55915987f490 0x55915987f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde6f443740 4 client.4791 mds_command: new command op to 4374 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-11T00:31:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde6f443740 1 -- 172.21.15.31:0/511281221 --> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] -- command(tid 0: {"prefix": "status"}) -- 0x559159812810 con 0x55915987f490 2024-09-11T00:31:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde69c00640 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x55915987f490 0x55915987f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde69c00640 1 -- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x55915987f490 msgr2=0x55915987f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-09-11T00:31:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde69c00640 1 -- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x55915987f490 msgr2=0x55915987f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T00:31:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde69c00640 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x55915987f490 0x55915987f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T00:31:11.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.049+0000 7fde69c00640 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x55915987f490 0x55915987f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T00:31:11.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.250+0000 7fde69c00640 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x55915987f490 0x55915987f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde69c00640 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x55915987f490 0x55915987f870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1a3c6bf080de334 server_cookie=8efff351cf81a9ee in_seq=0 out_seq=0 2024-09-11T00:31:11.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde4aa00640 10 client.4791 ms_handle_connect on v2:172.21.15.204:6836/3441932736 2024-09-11T00:31:11.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde4aa00640 1 -- 172.21.15.31:0/511281221 <== mds.0 v2:172.21.15.204:6836/3441932736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x559159812810 con 0x55915987f490 2024-09-11T00:31:11.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde4aa00640 10 client.4791 handle_command_reply: tid=0 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde6f443740 1 client.4791 shutdown 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde6f443740 1 -- 172.21.15.31:0/511281221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 msgr2=0x5591597f6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde6f443740 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597f6dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fde5c017ec0 tx=0x7fde5c015780 comp rx=0 tx=0).stop 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde6f443740 1 -- 172.21.15.31:0/511281221 shutdown_connections 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde6f443740 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:6836/3441932736,v1:172.21.15.204:6837/3441932736] conn(0x55915987f490 0x55915987f870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde6f443740 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5591597fda90 0x5591597f6dd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde6f443740 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591597e3190 0x5591597fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde6f443740 1 --2- 172.21.15.31:0/511281221 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x5591597e1f70 0x5591597be870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde4aa00640 0 client.4791 ms_handle_reset on v2:172.21.15.204:6836/3441932736 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.251+0000 7fde6f443740 1 -- 172.21.15.31:0/511281221 >> 172.21.15.31:0/511281221 conn(0x5591597e2db0 msgr2=0x55915980e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.252+0000 7fde6f443740 1 -- 172.21.15.31:0/511281221 shutdown_connections 2024-09-11T00:31:11.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.252+0000 7fde6f443740 1 -- 172.21.15.31:0/511281221 wait complete. 2024-09-11T00:31:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.252+0000 7fde6f443740 20 client.4791 trim_cache size 0 max 16384 2024-09-11T00:31:11.256 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:31:11.256 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "4a2c9b4f-c931-4bcb-847e-5521a1bafcf6", 2024-09-11T00:31:11.256 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-09-11T00:31:11.256 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 4374, 2024-09-11T00:31:11.256 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-09-11T00:31:11.256 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-09-11T00:31:11.256 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "b", 2024-09-11T00:31:11.256 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 16.272644540999998, 2024-09-11T00:31:11.257 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 15, 2024-09-11T00:31:11.257 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 26, 2024-09-11T00:31:11.257 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 26, 2024-09-11T00:31:11.257 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 31.375989605000001 2024-09-11T00:31:11.257 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.252+0000 7fde6c600640 1 -- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde44077f70 msgr2=0x7fde4407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.252+0000 7fde6c600640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde44077f70 0x7fde4407a430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fde50034000 tx=0x7fde50004730 comp rx=0 tx=0).stop 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.252+0000 7fde6c600640 1 -- 172.21.15.31:0/1232751003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde640c5b10 msgr2=0x7fde640721f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.252+0000 7fde6c600640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde640c5b10 0x7fde640721f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fde5800c490 tx=0x7fde5800c950 comp rx=0 tx=0).stop 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.253+0000 7fde6c600640 1 -- 172.21.15.31:0/1232751003 shutdown_connections 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.253+0000 7fde6c600640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde44077f70 0x7fde4407a430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.253+0000 7fde6c600640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde64156ad0 0x7fde641ebd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.253+0000 7fde6c600640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde641539c0 0x7fde64155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.253+0000 7fde6c600640 1 --2- 172.21.15.31:0/1232751003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde640c5b10 0x7fde640721f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.253+0000 7fde6c600640 1 -- 172.21.15.31:0/1232751003 >> 172.21.15.31:0/1232751003 conn(0x7fde640a9750 msgr2=0x7fde641504d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:11.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.253+0000 7fde6c600640 1 -- 172.21.15.31:0/1232751003 shutdown_connections 2024-09-11T00:31:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.253+0000 7fde6c600640 1 -- 172.21.15.31:0/1232751003 wait complete. 2024-09-11T00:31:11.273 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-11T00:31:11.308 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-11T00:31:11.308 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-09-11T00:31:11.309 INFO:tasks.ceph_fuse:client.0 config is {'cephfs_name': 'a'} 2024-09-11T00:31:11.309 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-11T00:31:11.310 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:11.310 INFO:tasks.ceph_fuse:client.1 config is {'cephfs_name': 'b'} 2024-09-11T00:31:11.310 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-11T00:31:11.310 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-11T00:31:11.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:11.310 DEBUG:teuthology.orchestra.run.smithi204:> ip netns list 2024-09-11T00:31:11.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:11.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip link delete ceph-brx 2024-09-11T00:31:11.455 INFO:teuthology.orchestra.run.smithi204.stderr:Cannot find device "ceph-brx" 2024-09-11T00:31:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T00:31:11.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:11.458 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-11T00:31:11.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:11.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-09-11T00:31:11.594 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2024-09-11T00:31:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T00:31:11.597 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-11T00:31:11.597 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-11T00:31:11.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-11T00:31:11.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.754+0000 7f5790200640 1 -- 172.21.15.31:0/4095403306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57880c5ef0 msgr2=0x7f5788153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.754+0000 7f5790200640 1 --2- 172.21.15.31:0/4095403306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57880c5ef0 0x7f5788153480 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5774009830 tx=0x7f577402a510 comp rx=0 tx=0).stop 2024-09-11T00:31:11.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.754+0000 7f5790200640 1 -- 172.21.15.31:0/4095403306 shutdown_connections 2024-09-11T00:31:11.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.754+0000 7f5790200640 1 --2- 172.21.15.31:0/4095403306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57880c5ef0 0x7f5788153480 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.754+0000 7f5790200640 1 --2- 172.21.15.31:0/4095403306 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57880c5b10 0x7f5788156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.754+0000 7f5790200640 1 -- 172.21.15.31:0/4095403306 >> 172.21.15.31:0/4095403306 conn(0x7f57880a9750 msgr2=0x7f57880a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:11.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.754+0000 7f5790200640 1 -- 172.21.15.31:0/4095403306 shutdown_connections 2024-09-11T00:31:11.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.754+0000 7f5790200640 1 -- 172.21.15.31:0/4095403306 wait complete. 2024-09-11T00:31:11.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.756+0000 7f5790200640 1 Processor -- start 2024-09-11T00:31:11.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.756+0000 7f5790200640 1 -- start start 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f5790200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57880c5b10 0x7f5788150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f5790200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57880c5ef0 0x7f57881eca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f5790200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57880c6310 0x7f57881e1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f5790200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f57881539c0 con 0x7f57880c5b10 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f5790200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f57880c3950 con 0x7f57880c5ef0 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f5790200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5788153b40 con 0x7f57880c6310 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57880c5b10 0x7f5788150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57880c5b10 0x7f5788150530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46252/0 (socket says 172.21.15.31:46252) 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578d800640 1 -- 172.21.15.31:0/3229582788 learned_addr learned my addr 172.21.15.31:0/3229582788 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578ce00640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57880c5ef0 0x7f57881eca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578e200640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57880c6310 0x7f57881e1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578d800640 1 -- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57880c6310 msgr2=0x7f57881e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578d800640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57880c6310 0x7f57881e1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578d800640 1 -- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57880c5ef0 msgr2=0x7f57881eca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578d800640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57880c5ef0 0x7f57881eca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578d800640 1 -- 172.21.15.31:0/3229582788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57881521c0 con 0x7f57880c5b10 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.757+0000 7f578e200640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57880c6310 0x7f57881e1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.758+0000 7f578d800640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57880c5b10 0x7f5788150530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f577c00d950 tx=0x7f577c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.758+0000 7f5785600640 1 -- 172.21.15.31:0/3229582788 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c017070 con 0x7f57880c5b10 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.758+0000 7f5785600640 1 -- 172.21.15.31:0/3229582788 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f577c009d20 con 0x7f57880c5b10 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.758+0000 7f5785600640 1 -- 172.21.15.31:0/3229582788 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c0106c0 con 0x7f57880c5b10 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.758+0000 7f5790200640 1 -- 172.21.15.31:0/3229582788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57880a6e90 con 0x7f57880c5b10 2024-09-11T00:31:11.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.758+0000 7f5790200640 1 -- 172.21.15.31:0/3229582788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5788155e70 con 0x7f57880c5b10 2024-09-11T00:31:11.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.759+0000 7f5790200640 1 -- 172.21.15.31:0/3229582788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5788156ad0 con 0x7f57880c5b10 2024-09-11T00:31:11.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.760+0000 7f5785600640 1 -- 172.21.15.31:0/3229582788 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f577c010860 con 0x7f57880c5b10 2024-09-11T00:31:11.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.761+0000 7f5785600640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5768077f70 0x7f576807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:11.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.761+0000 7f5785600640 1 -- 172.21.15.31:0/3229582788 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f577c09a4e0 con 0x7f57880c5b10 2024-09-11T00:31:11.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.762+0000 7f578ce00640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5768077f70 0x7f576807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:11.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.763+0000 7f578ce00640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5768077f70 0x7f576807a430 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5774034000 tx=0x7f5774004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:11.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.765+0000 7f5785600640 1 -- 172.21.15.31:0/3229582788 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f577c062ca0 con 0x7f57880c5b10 2024-09-11T00:31:12.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.997+0000 7f5790200640 1 -- 172.21.15.31:0/3229582788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f57880c1e70 con 0x7f57880c5b10 2024-09-11T00:31:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:11.998+0000 7f5785600640 1 -- 172.21.15.31:0/3229582788 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+116 (secure 0 0 0) 0x7f577c0625a0 con 0x7f57880c5b10 2024-09-11T00:31:12.001 INFO:teuthology.orchestra.run.smithi031.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2024-09-11T00:31:12.001 INFO:teuthology.orchestra.run.smithi031.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2024-09-11T00:31:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 -- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5768077f70 msgr2=0x7f576807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5768077f70 0x7f576807a430 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5774034000 tx=0x7f5774004730 comp rx=0 tx=0).stop 2024-09-11T00:31:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 -- 172.21.15.31:0/3229582788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57880c5b10 msgr2=0x7f5788150530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57880c5b10 0x7f5788150530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f577c00d950 tx=0x7f577c00de10 comp rx=0 tx=0).stop 2024-09-11T00:31:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 -- 172.21.15.31:0/3229582788 shutdown_connections 2024-09-11T00:31:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5768077f70 0x7f576807a430 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f57880c6310 0x7f57881e1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57880c5ef0 0x7f57881eca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 --2- 172.21.15.31:0/3229582788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57880c5b10 0x7f5788150530 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 -- 172.21.15.31:0/3229582788 >> 172.21.15.31:0/3229582788 conn(0x7f57880a9750 msgr2=0x7f578806ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 -- 172.21.15.31:0/3229582788 shutdown_connections 2024-09-11T00:31:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12.000+0000 7f5790200640 1 -- 172.21.15.31:0/3229582788 wait complete. 2024-09-11T00:31:12.019 INFO:tasks.cephfs.mount:Mounting Ceph FS a; just confirmed its presence on cluster 2024-09-11T00:31:12.019 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-11T00:31:12.019 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-09-11T00:31:12.019 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-11T00:31:12.019 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:12.019 INFO:tasks.cephfs.mount:self.cephfs_name = a 2024-09-11T00:31:12.019 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-11T00:31:12.019 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-11T00:31:12.019 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-11T00:31:12.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:12.020 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-09-11T00:31:12.055 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-11T00:31:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-11T00:31:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-09-11T00:31:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-11T00:31:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-09-11T00:31:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-11T00:31:12.056 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-11T00:31:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-09-11T00:31:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-09-11T00:31:12.056 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-11T00:31:12.056 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42644sec preferred_lft 42644sec 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-09-11T00:31:12.057 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-09-11T00:31:12.058 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-11T00:31:12.058 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-11T00:31:12.058 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-11T00:31:12.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add name ceph-brx type bridge 2024-09-11T00:31:12.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr flush dev ceph-brx 2024-09-11T00:31:12.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set ceph-brx up 2024-09-11T00:31:12.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-11T00:31:12.058 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-11T00:31:12.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-11T00:31:12.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:12.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:12.457 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-11T00:31:12.513 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-09-11T00:31:12.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:12.517 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-09-11T00:31:12.560 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-09-11T00:31:12.560 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-11T00:31:12.560 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-11T00:31:12.560 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-11T00:31:12.560 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-11T00:31:12.561 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-11T00:31:12.561 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-11T00:31:12.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-11T00:31:12.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-11T00:31:12.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-11T00:31:12.561 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-11T00:31:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-11T00:31:13.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:13.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:13.215 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-11T00:31:13.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:13.252 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2024-09-11T00:31:13.327 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-11T00:31:13.327 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-11T00:31:13.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-11T00:31:13.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-11T00:31:13.328 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-11T00:31:13.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-11T00:31:13.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:13.507 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-11T00:31:13.507 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-11T00:31:13.507 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-11T00:31:13.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-11T00:31:13.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-11T00:31:13.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-11T00:31:13.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-11T00:31:13.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-11T00:31:13.508 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-11T00:31:13.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-11T00:31:13.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:13.750 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-11T00:31:13.750 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-09-11T00:31:13.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-09-11T00:31:13.750 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.0 master ceph-brx 2024-09-11T00:31:13.750 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-09-11T00:31:13.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-11T00:31:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:13.917 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'cephfs_name': 'a'} 2024-09-11T00:31:13.917 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-11T00:31:13.917 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:13.952 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-11T00:31:13.952 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-11T00:31:13.953 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:14.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo modprobe fuse 2024-09-11T00:31:14.125 DEBUG:teuthology.orchestra.run.smithi031:> cat /proc/self/mounts | awk '{print $2}' 2024-09-11T00:31:14.161 INFO:teuthology.orchestra.run.smithi031.stdout:/proc 2024-09-11T00:31:14.161 INFO:teuthology.orchestra.run.smithi031.stdout:/sys 2024-09-11T00:31:14.161 INFO:teuthology.orchestra.run.smithi031.stdout:/dev 2024-09-11T00:31:14.161 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/security 2024-09-11T00:31:14.162 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/shm 2024-09-11T00:31:14.162 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/pts 2024-09-11T00:31:14.162 INFO:teuthology.orchestra.run.smithi031.stdout:/run 2024-09-11T00:31:14.162 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/cgroup 2024-09-11T00:31:14.162 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/pstore 2024-09-11T00:31:14.162 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/bpf 2024-09-11T00:31:14.162 INFO:teuthology.orchestra.run.smithi031.stdout:/ 2024-09-11T00:31:14.162 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/selinux 2024-09-11T00:31:14.162 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-09-11T00:31:14.163 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/mqueue 2024-09-11T00:31:14.163 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/hugepages 2024-09-11T00:31:14.163 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/debug 2024-09-11T00:31:14.163 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/tracing 2024-09-11T00:31:14.163 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/fuse/connections 2024-09-11T00:31:14.163 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/config 2024-09-11T00:31:14.164 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysctl.service 2024-09-11T00:31:14.164 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-11T00:31:14.164 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-11T00:31:14.164 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/nfs/rpc_pipefs 2024-09-11T00:31:14.164 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/1000 2024-09-11T00:31:14.164 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph 2024-09-11T00:31:14.164 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-0 2024-09-11T00:31:14.164 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-1 2024-09-11T00:31:14.165 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-2 2024-09-11T00:31:14.165 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-3 2024-09-11T00:31:14.165 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns 2024-09-11T00:31:14.165 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-11T00:31:14.165 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-11T00:31:14.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:14.166 DEBUG:teuthology.orchestra.run.smithi031:> ls /sys/fs/fuse/connections 2024-09-11T00:31:14.242 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-11T00:31:14.242 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_fs=a) 2024-09-11T00:31:14.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo modprobe fuse 2024-09-11T00:31:14.343 DEBUG:teuthology.orchestra.run.smithi031:> cat /proc/self/mounts | awk '{print $2}' 2024-09-11T00:31:14.380 INFO:teuthology.orchestra.run.smithi031.stdout:/proc 2024-09-11T00:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout:/sys 2024-09-11T00:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout:/dev 2024-09-11T00:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/security 2024-09-11T00:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/shm 2024-09-11T00:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/pts 2024-09-11T00:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout:/run 2024-09-11T00:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/cgroup 2024-09-11T00:31:14.381 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/pstore 2024-09-11T00:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/bpf 2024-09-11T00:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout:/ 2024-09-11T00:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/selinux 2024-09-11T00:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-09-11T00:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/mqueue 2024-09-11T00:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/hugepages 2024-09-11T00:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/debug 2024-09-11T00:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/tracing 2024-09-11T00:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/fuse/connections 2024-09-11T00:31:14.382 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/config 2024-09-11T00:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysctl.service 2024-09-11T00:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-11T00:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-11T00:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/nfs/rpc_pipefs 2024-09-11T00:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/1000 2024-09-11T00:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph 2024-09-11T00:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-0 2024-09-11T00:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-1 2024-09-11T00:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-2 2024-09-11T00:31:14.383 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-3 2024-09-11T00:31:14.384 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns 2024-09-11T00:31:14.384 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-11T00:31:14.384 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-11T00:31:14.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:14.385 DEBUG:teuthology.orchestra.run.smithi031:> ls /sys/fs/fuse/connections 2024-09-11T00:31:14.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi031.stderr:2024-09-11T00:31:14.464+0000 7f8a0924b480 -1 init, newargv = 0x5571c3aae1f0 newargc=15 2024-09-11T00:31:14.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi031.stderr:ceph-fuse[45156]: starting ceph client 2024-09-11T00:31:14.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi031.stderr:ceph-fuse[45156]: starting fuse 2024-09-11T00:31:15.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo modprobe fuse 2024-09-11T00:31:15.628 DEBUG:teuthology.orchestra.run.smithi031:> cat /proc/self/mounts | awk '{print $2}' 2024-09-11T00:31:15.666 INFO:teuthology.orchestra.run.smithi031.stdout:/proc 2024-09-11T00:31:15.666 INFO:teuthology.orchestra.run.smithi031.stdout:/sys 2024-09-11T00:31:15.666 INFO:teuthology.orchestra.run.smithi031.stdout:/dev 2024-09-11T00:31:15.666 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/security 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/shm 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/pts 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/run 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/cgroup 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/pstore 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/bpf 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/ 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/selinux 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/proc/sys/fs/binfmt_misc 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/mqueue 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/hugepages 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/debug 2024-09-11T00:31:15.667 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/tracing 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/fs/fuse/connections 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/sys/kernel/config 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-sysctl.service 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/nfs/rpc_pipefs 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/run/user/1000 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-0 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-1 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-2 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/var/lib/ceph/osd/ceph-3 2024-09-11T00:31:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns 2024-09-11T00:31:15.669 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-11T00:31:15.669 INFO:teuthology.orchestra.run.smithi031.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-11T00:31:15.669 INFO:teuthology.orchestra.run.smithi031.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:15.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:15.670 DEBUG:teuthology.orchestra.run.smithi031:> ls /sys/fs/fuse/connections 2024-09-11T00:31:15.747 INFO:teuthology.orchestra.run.smithi031.stdout:42 2024-09-11T00:31:15.747 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-09-11T00:31:15.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo stdin-killer -- python3 -c ' 2024-09-11T00:31:15.748 DEBUG:teuthology.orchestra.run.smithi031:> import glob 2024-09-11T00:31:15.748 DEBUG:teuthology.orchestra.run.smithi031:> import re 2024-09-11T00:31:15.748 DEBUG:teuthology.orchestra.run.smithi031:> import os 2024-09-11T00:31:15.748 DEBUG:teuthology.orchestra.run.smithi031:> import subprocess 2024-09-11T00:31:15.748 DEBUG:teuthology.orchestra.run.smithi031:> 2024-09-11T00:31:15.748 DEBUG:teuthology.orchestra.run.smithi031:> def _find_admin_socket(client_name): 2024-09-11T00:31:15.748 DEBUG:teuthology.orchestra.run.smithi031:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-11T00:31:15.748 DEBUG:teuthology.orchestra.run.smithi031:> files = glob.glob(asok_path) 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> # Given a non-glob path, it better be there 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> if "*" not in asok_path: 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> assert(len(files) == 1) 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> return files[0] 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> for f in files: 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> contents = proc_f.read() 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> if mountpoint in contents: 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> return f 2024-09-11T00:31:15.749 DEBUG:teuthology.orchestra.run.smithi031:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-11T00:31:15.750 DEBUG:teuthology.orchestra.run.smithi031:> 2024-09-11T00:31:15.750 DEBUG:teuthology.orchestra.run.smithi031:> print(_find_admin_socket("client.0")) 2024-09-11T00:31:15.750 DEBUG:teuthology.orchestra.run.smithi031:> ' 2024-09-11T00:31:15.913 INFO:teuthology.orchestra.run.smithi031.stdout:/var/run/ceph/ceph-client.0.45156.asok 2024-09-11T00:31:15.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:15.927 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45156.asok 2024-09-11T00:31:15.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:15.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45156.asok status 2024-09-11T00:31:16.113 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:31:16.113 INFO:teuthology.orchestra.run.smithi031.stdout: "metadata": { 2024-09-11T00:31:16.113 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", 2024-09-11T00:31:16.113 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", 2024-09-11T00:31:16.113 INFO:teuthology.orchestra.run.smithi031.stdout: "entity_id": "0", 2024-09-11T00:31:16.113 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": "smithi031", 2024-09-11T00:31:16.113 INFO:teuthology.orchestra.run.smithi031.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "pid": "45156", 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "root": "/" 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "dentry_count": 0, 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "dentry_pinned_count": 0, 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 4806, 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "inst": { 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": { 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "client", 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "num": 4806 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": { 2024-09-11T00:31:16.114 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "192.168.0.1:0", 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1419332941 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": { 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "192.168.0.1:0", 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1419332941 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: "inst_str": "client.4806 192.168.0.1:0/1419332941", 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: "addr_str": "192.168.0.1:0/1419332941", 2024-09-11T00:31:16.115 INFO:teuthology.orchestra.run.smithi031.stdout: "inode_count": 1, 2024-09-11T00:31:16.116 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_epoch": 9, 2024-09-11T00:31:16.116 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_epoch": 26, 2024-09-11T00:31:16.116 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_epoch_barrier": 0, 2024-09-11T00:31:16.116 INFO:teuthology.orchestra.run.smithi031.stdout: "blocklisted": false, 2024-09-11T00:31:16.116 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "a" 2024-09-11T00:31:16.116 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:31:16.125 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-11T00:31:16.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-11T00:31:16.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.283+0000 7f0269e00640 1 -- 172.21.15.31:0/1827569366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02641518b0 msgr2=0x7f02640c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:16.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.283+0000 7f0269e00640 1 --2- 172.21.15.31:0/1827569366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02641518b0 0x7f02640c3600 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f02640add10 tx=0x7f025c02ab40 comp rx=0 tx=0).stop 2024-09-11T00:31:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.283+0000 7f0269e00640 1 -- 172.21.15.31:0/1827569366 shutdown_connections 2024-09-11T00:31:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.283+0000 7f0269e00640 1 --2- 172.21.15.31:0/1827569366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02641518b0 0x7f02640c3600 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.283+0000 7f0269e00640 1 --2- 172.21.15.31:0/1827569366 >> v2:172.21.15.204:3300/0 conn(0x7f02640c23a0 0x7f02640c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.283+0000 7f0269e00640 1 -- 172.21.15.31:0/1827569366 >> 172.21.15.31:0/1827569366 conn(0x7f02640a9750 msgr2=0x7f02640a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.283+0000 7f0269e00640 1 -- 172.21.15.31:0/1827569366 shutdown_connections 2024-09-11T00:31:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.283+0000 7f0269e00640 1 -- 172.21.15.31:0/1827569366 wait complete. 2024-09-11T00:31:16.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.285+0000 7f0269e00640 1 Processor -- start 2024-09-11T00:31:16.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.285+0000 7f0269e00640 1 -- start start 2024-09-11T00:31:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f0269e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0264072480 0x7f02641dffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f0269e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02640c23a0 0x7f026406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f0269e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02641518b0 0x7f02640c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f0269e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0264074200 con 0x7f02640c23a0 2024-09-11T00:31:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f0269e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0264074080 con 0x7f02641518b0 2024-09-11T00:31:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f0269e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0264072860 con 0x7f0264072480 2024-09-11T00:31:16.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f025a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02640c23a0 0x7f026406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f025a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02640c23a0 0x7f026406ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46272/0 (socket says 172.21.15.31:46272) 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f025a000640 1 -- 172.21.15.31:0/3379192054 learned_addr learned my addr 172.21.15.31:0/3379192054 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f0262a00640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0264072480 0x7f02641dffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f0263400640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02641518b0 0x7f02640c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f025a000640 1 -- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0264072480 msgr2=0x7f02641dffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f025a000640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0264072480 0x7f02641dffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f025a000640 1 -- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02641518b0 msgr2=0x7f02640c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.286+0000 7f025a000640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02641518b0 0x7f02640c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.287+0000 7f025a000640 1 -- 172.21.15.31:0/3379192054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02641ebce0 con 0x7f02640c23a0 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.287+0000 7f025a000640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02640c23a0 0x7f026406ee40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f024c00d770 tx=0x7f024c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:16.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.287+0000 7f025be00640 1 -- 172.21.15.31:0/3379192054 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c017070 con 0x7f02640c23a0 2024-09-11T00:31:16.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.287+0000 7f025be00640 1 -- 172.21.15.31:0/3379192054 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f024c009cc0 con 0x7f02640c23a0 2024-09-11T00:31:16.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.287+0000 7f025be00640 1 -- 172.21.15.31:0/3379192054 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024c005280 con 0x7f02640c23a0 2024-09-11T00:31:16.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.287+0000 7f0269e00640 1 -- 172.21.15.31:0/3379192054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02641ebf80 con 0x7f02640c23a0 2024-09-11T00:31:16.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.287+0000 7f0269e00640 1 -- 172.21.15.31:0/3379192054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02641ec540 con 0x7f02640c23a0 2024-09-11T00:31:16.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.288+0000 7f0269e00640 1 -- 172.21.15.31:0/3379192054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0264153f20 con 0x7f02640c23a0 2024-09-11T00:31:16.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.289+0000 7f025be00640 1 -- 172.21.15.31:0/3379192054 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f024c005420 con 0x7f02640c23a0 2024-09-11T00:31:16.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.290+0000 7f025be00640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0240077f70 0x7f024007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:16.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.290+0000 7f025be00640 1 -- 172.21.15.31:0/3379192054 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f024c09a1f0 con 0x7f02640c23a0 2024-09-11T00:31:16.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.291+0000 7f0262a00640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0240077f70 0x7f024007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:16.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.292+0000 7f0262a00640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0240077f70 0x7f024007a430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f02640703e0 tx=0x7f0254009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:16.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.295+0000 7f025be00640 1 -- 172.21.15.31:0/3379192054 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f024c0629b0 con 0x7f02640c23a0 2024-09-11T00:31:16.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.527+0000 7f0269e00640 1 -- 172.21.15.31:0/3379192054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f026414af30 con 0x7f02640c23a0 2024-09-11T00:31:16.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.527+0000 7f025be00640 1 -- 172.21.15.31:0/3379192054 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+116 (secure 0 0 0) 0x7f024c06b0d0 con 0x7f02640c23a0 2024-09-11T00:31:16.530 INFO:teuthology.orchestra.run.smithi031.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2024-09-11T00:31:16.530 INFO:teuthology.orchestra.run.smithi031.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2024-09-11T00:31:16.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.529+0000 7f0269e00640 1 -- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0240077f70 msgr2=0x7f024007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.529+0000 7f0269e00640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0240077f70 0x7f024007a430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f02640703e0 tx=0x7f0254009040 comp rx=0 tx=0).stop 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.529+0000 7f0269e00640 1 -- 172.21.15.31:0/3379192054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02640c23a0 msgr2=0x7f026406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.529+0000 7f0269e00640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02640c23a0 0x7f026406ee40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f024c00d770 tx=0x7f024c00dc30 comp rx=0 tx=0).stop 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.529+0000 7f0269e00640 1 -- 172.21.15.31:0/3379192054 shutdown_connections 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.529+0000 7f0269e00640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0240077f70 0x7f024007a430 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.529+0000 7f0269e00640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02641518b0 0x7f02640c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.530+0000 7f0269e00640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02640c23a0 0x7f026406ee40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.530+0000 7f0269e00640 1 --2- 172.21.15.31:0/3379192054 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0264072480 0x7f02641dffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.530+0000 7f0269e00640 1 -- 172.21.15.31:0/3379192054 >> 172.21.15.31:0/3379192054 conn(0x7f02640a9750 msgr2=0x7f026414ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.530+0000 7f0269e00640 1 -- 172.21.15.31:0/3379192054 shutdown_connections 2024-09-11T00:31:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:16.530+0000 7f0269e00640 1 -- 172.21.15.31:0/3379192054 wait complete. 2024-09-11T00:31:16.548 INFO:tasks.cephfs.mount:Mounting Ceph FS b; just confirmed its presence on cluster 2024-09-11T00:31:16.548 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-11T00:31:16.548 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi204.front.sepia.ceph.com 2024-09-11T00:31:16.548 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-11T00:31:16.548 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-11T00:31:16.548 INFO:tasks.cephfs.mount:self.cephfs_name = b 2024-09-11T00:31:16.548 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-11T00:31:16.548 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-11T00:31:16.548 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-11T00:31:16.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:16.549 DEBUG:teuthology.orchestra.run.smithi204:> ip addr 2024-09-11T00:31:16.589 INFO:teuthology.orchestra.run.smithi204.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-11T00:31:16.589 INFO:teuthology.orchestra.run.smithi204.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-11T00:31:16.589 INFO:teuthology.orchestra.run.smithi204.stdout: inet 127.0.0.1/8 scope host lo 2024-09-11T00:31:16.589 INFO:teuthology.orchestra.run.smithi204.stdout: valid_lft forever preferred_lft forever 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout: inet6 ::1/128 scope host 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout: valid_lft forever preferred_lft forever 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout: link/ether 0c:c4:7a:d9:36:2a brd ff:ff:ff:ff:ff:ff 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout: altname enp7s0f0 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout: link/ether 0c:c4:7a:d9:36:2b brd ff:ff:ff:ff:ff:ff 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout: altname enp7s0f1 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout: link/ether 0c:c4:7a:88:77:aa brd ff:ff:ff:ff:ff:ff 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-11T00:31:16.590 INFO:teuthology.orchestra.run.smithi204.stdout: link/ether 0c:c4:7a:88:77:ab brd ff:ff:ff:ff:ff:ff 2024-09-11T00:31:16.591 INFO:teuthology.orchestra.run.smithi204.stdout: inet 172.21.15.204/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-11T00:31:16.591 INFO:teuthology.orchestra.run.smithi204.stdout: valid_lft 42664sec preferred_lft 42664sec 2024-09-11T00:31:16.591 INFO:teuthology.orchestra.run.smithi204.stdout: inet6 fe80::ec4:7aff:fe88:77ab/64 scope link noprefixroute 2024-09-11T00:31:16.591 INFO:teuthology.orchestra.run.smithi204.stdout: valid_lft forever preferred_lft forever 2024-09-11T00:31:16.591 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-11T00:31:16.591 DEBUG:teuthology.orchestra.run.smithi204:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-11T00:31:16.591 DEBUG:teuthology.orchestra.run.smithi204:> set -e 2024-09-11T00:31:16.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip link add name ceph-brx type bridge 2024-09-11T00:31:16.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip addr flush dev ceph-brx 2024-09-11T00:31:16.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip link set ceph-brx up 2024-09-11T00:31:16.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-11T00:31:16.592 DEBUG:teuthology.orchestra.run.smithi204:> ') 2024-09-11T00:31:16.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-11T00:31:17.005 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:17.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:17.014 DEBUG:teuthology.orchestra.run.smithi204:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-11T00:31:17.071 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-09-11T00:31:17.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:17.074 DEBUG:teuthology.orchestra.run.smithi204:> route 2024-09-11T00:31:17.123 INFO:teuthology.orchestra.run.smithi204.stdout:Kernel IP routing table 2024-09-11T00:31:17.124 INFO:teuthology.orchestra.run.smithi204.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-11T00:31:17.124 INFO:teuthology.orchestra.run.smithi204.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-11T00:31:17.124 INFO:teuthology.orchestra.run.smithi204.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-11T00:31:17.124 INFO:teuthology.orchestra.run.smithi204.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-11T00:31:17.124 DEBUG:teuthology.orchestra.run.smithi204:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-11T00:31:17.124 DEBUG:teuthology.orchestra.run.smithi204:> set -e 2024-09-11T00:31:17.124 DEBUG:teuthology.orchestra.run.smithi204:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-11T00:31:17.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-11T00:31:17.125 DEBUG:teuthology.orchestra.run.smithi204:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-11T00:31:17.125 DEBUG:teuthology.orchestra.run.smithi204:> ') 2024-09-11T00:31:17.202 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-11T00:31:17.685 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:17.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:17.695 DEBUG:teuthology.orchestra.run.smithi204:> ip netns list 2024-09-11T00:31:17.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:17.737 DEBUG:teuthology.orchestra.run.smithi204:> ip netns list-id 2024-09-11T00:31:17.817 DEBUG:teuthology.orchestra.run.smithi204:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-11T00:31:17.817 DEBUG:teuthology.orchestra.run.smithi204:> set -e 2024-09-11T00:31:17.817 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-11T00:31:17.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-11T00:31:17.818 DEBUG:teuthology.orchestra.run.smithi204:> ') 2024-09-11T00:31:17.946 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-11T00:31:18.005 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:18.011 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-11T00:31:18.011 DEBUG:teuthology.orchestra.run.smithi204:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-11T00:31:18.011 DEBUG:teuthology.orchestra.run.smithi204:> set -e 2024-09-11T00:31:18.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-11T00:31:18.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-11T00:31:18.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-11T00:31:18.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-11T00:31:18.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-11T00:31:18.011 DEBUG:teuthology.orchestra.run.smithi204:> ') 2024-09-11T00:31:18.091 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-11T00:31:18.273 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:18.281 DEBUG:teuthology.orchestra.run.smithi204:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-11T00:31:18.281 DEBUG:teuthology.orchestra.run.smithi204:> set -e 2024-09-11T00:31:18.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip link set brx.0 up 2024-09-11T00:31:18.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip link set dev brx.0 master ceph-brx 2024-09-11T00:31:18.281 DEBUG:teuthology.orchestra.run.smithi204:> ') 2024-09-11T00:31:18.360 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-11T00:31:18.441 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:18.449 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {'cephfs_name': 'b'} 2024-09-11T00:31:18.450 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-11T00:31:18.450 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-11T00:31:18.485 INFO:teuthology.orchestra.run.smithi204.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-11T00:31:18.486 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-11T00:31:18.486 DEBUG:teuthology.orchestra.run.smithi204:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-11T00:31:18.561 DEBUG:teuthology.orchestra.run.smithi204:> sudo modprobe fuse 2024-09-11T00:31:18.660 DEBUG:teuthology.orchestra.run.smithi204:> cat /proc/self/mounts | awk '{print $2}' 2024-09-11T00:31:18.696 INFO:teuthology.orchestra.run.smithi204.stdout:/proc 2024-09-11T00:31:18.696 INFO:teuthology.orchestra.run.smithi204.stdout:/sys 2024-09-11T00:31:18.696 INFO:teuthology.orchestra.run.smithi204.stdout:/dev 2024-09-11T00:31:18.696 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/kernel/security 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/shm 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/pts 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/run 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/fs/cgroup 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/fs/pstore 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/fs/bpf 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/ 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/fs/selinux 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/proc/sys/fs/binfmt_misc 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/mqueue 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/hugepages 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/kernel/debug 2024-09-11T00:31:18.697 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/kernel/tracing 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/fs/fuse/connections 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/kernel/config 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/run/credentials/systemd-sysctl.service 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/nfs/rpc_pipefs 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/run/user/1000 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/ceph 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/ceph/osd/ceph-4 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/ceph/osd/ceph-5 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/ceph/osd/ceph-6 2024-09-11T00:31:18.698 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/ceph/osd/ceph-7 2024-09-11T00:31:18.699 INFO:teuthology.orchestra.run.smithi204.stdout:/run/netns 2024-09-11T00:31:18.699 INFO:teuthology.orchestra.run.smithi204.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-11T00:31:18.699 INFO:teuthology.orchestra.run.smithi204.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-11T00:31:18.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:18.700 DEBUG:teuthology.orchestra.run.smithi204:> ls /sys/fs/fuse/connections 2024-09-11T00:31:18.774 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-11T00:31:18.775 DEBUG:teuthology.orchestra.run.smithi204:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_fs=b) 2024-09-11T00:31:18.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo modprobe fuse 2024-09-11T00:31:18.876 DEBUG:teuthology.orchestra.run.smithi204:> cat /proc/self/mounts | awk '{print $2}' 2024-09-11T00:31:18.914 INFO:teuthology.orchestra.run.smithi204.stdout:/proc 2024-09-11T00:31:18.915 INFO:teuthology.orchestra.run.smithi204.stdout:/sys 2024-09-11T00:31:18.915 INFO:teuthology.orchestra.run.smithi204.stdout:/dev 2024-09-11T00:31:18.915 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/kernel/security 2024-09-11T00:31:18.915 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/shm 2024-09-11T00:31:18.915 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/pts 2024-09-11T00:31:18.915 INFO:teuthology.orchestra.run.smithi204.stdout:/run 2024-09-11T00:31:18.915 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/fs/cgroup 2024-09-11T00:31:18.915 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/fs/pstore 2024-09-11T00:31:18.915 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/fs/bpf 2024-09-11T00:31:18.915 INFO:teuthology.orchestra.run.smithi204.stdout:/ 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/fs/selinux 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/proc/sys/fs/binfmt_misc 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/mqueue 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/hugepages 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/kernel/debug 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/kernel/tracing 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/fs/fuse/connections 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/sys/kernel/config 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/run/credentials/systemd-sysctl.service 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-11T00:31:18.916 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/nfs/rpc_pipefs 2024-09-11T00:31:18.917 INFO:teuthology.orchestra.run.smithi204.stdout:/run/user/1000 2024-09-11T00:31:18.917 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/ceph 2024-09-11T00:31:18.917 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/ceph/osd/ceph-4 2024-09-11T00:31:18.917 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/ceph/osd/ceph-5 2024-09-11T00:31:18.917 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/ceph/osd/ceph-6 2024-09-11T00:31:18.917 INFO:teuthology.orchestra.run.smithi204.stdout:/var/lib/ceph/osd/ceph-7 2024-09-11T00:31:18.917 INFO:teuthology.orchestra.run.smithi204.stdout:/run/netns 2024-09-11T00:31:18.917 INFO:teuthology.orchestra.run.smithi204.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-11T00:31:18.917 INFO:teuthology.orchestra.run.smithi204.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-11T00:31:18.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:18.918 DEBUG:teuthology.orchestra.run.smithi204:> ls /sys/fs/fuse/connections 2024-09-11T00:31:18.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi204.stderr:2024-09-11T00:31:18.976+0000 7f291e973480 -1 init, newargv = 0x55fe5cf90fb0 newargc=15 2024-09-11T00:31:18.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi204.stderr:ceph-fuse[41119]: starting ceph client 2024-09-11T00:31:18.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi204.stderr:ceph-fuse[41119]: starting fuse 2024-09-11T00:31:19.001 INFO:teuthology.orchestra.run.smithi204.stdout:43 2024-09-11T00:31:19.001 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-09-11T00:31:19.001 DEBUG:teuthology.orchestra.run.smithi204:> sudo stdin-killer -- python3 -c ' 2024-09-11T00:31:19.001 DEBUG:teuthology.orchestra.run.smithi204:> import glob 2024-09-11T00:31:19.001 DEBUG:teuthology.orchestra.run.smithi204:> import re 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> import os 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> import subprocess 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> def _find_admin_socket(client_name): 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> files = glob.glob(asok_path) 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> # Given a non-glob path, it better be there 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> if "*" not in asok_path: 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> assert(len(files) == 1) 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> return files[0] 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> 2024-09-11T00:31:19.002 DEBUG:teuthology.orchestra.run.smithi204:> for f in files: 2024-09-11T00:31:19.003 DEBUG:teuthology.orchestra.run.smithi204:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-11T00:31:19.003 DEBUG:teuthology.orchestra.run.smithi204:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-11T00:31:19.003 DEBUG:teuthology.orchestra.run.smithi204:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-11T00:31:19.003 DEBUG:teuthology.orchestra.run.smithi204:> contents = proc_f.read() 2024-09-11T00:31:19.003 DEBUG:teuthology.orchestra.run.smithi204:> if mountpoint in contents: 2024-09-11T00:31:19.003 DEBUG:teuthology.orchestra.run.smithi204:> return f 2024-09-11T00:31:19.003 DEBUG:teuthology.orchestra.run.smithi204:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-11T00:31:19.003 DEBUG:teuthology.orchestra.run.smithi204:> 2024-09-11T00:31:19.003 DEBUG:teuthology.orchestra.run.smithi204:> print(_find_admin_socket("client.1")) 2024-09-11T00:31:19.003 DEBUG:teuthology.orchestra.run.smithi204:> ' 2024-09-11T00:31:19.166 INFO:teuthology.orchestra.run.smithi204.stdout:/var/run/ceph/ceph-client.1.41119.asok 2024-09-11T00:31:19.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-11T00:31:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-11T00:31:19.180 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.41119.asok 2024-09-11T00:31:19.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:19.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.41119.asok status 2024-09-11T00:31:19.364 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "metadata": { 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_sha1": "8f219c476cb4f26fc5719c661c0323719a45cb30", 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_version": "ceph version 19.3.0-4854-g8f219c47 (8f219c476cb4f26fc5719c661c0323719a45cb30) squid (dev)", 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "entity_id": "1", 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "hostname": "smithi204", 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "pid": "41119", 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "root": "/" 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "dentry_count": 0, 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "dentry_pinned_count": 0, 2024-09-11T00:31:19.365 INFO:teuthology.orchestra.run.smithi204.stdout: "id": 4804, 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: "inst": { 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: "name": { 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: "type": "client", 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: "num": 4804 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: "addr": { 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: "type": "v1", 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: "addr": "192.168.0.1:0", 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: "nonce": 633280907 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: "addr": { 2024-09-11T00:31:19.366 INFO:teuthology.orchestra.run.smithi204.stdout: "type": "v1", 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: "addr": "192.168.0.1:0", 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: "nonce": 633280907 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: "inst_str": "client.4804 192.168.0.1:0/633280907", 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: "addr_str": "192.168.0.1:0/633280907", 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: "inode_count": 1, 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_epoch": 15, 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_epoch": 26, 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_epoch_barrier": 0, 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: "blocklisted": false, 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout: "fs_name": "b" 2024-09-11T00:31:19.367 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-11T00:31:19.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:19.376 DEBUG:teuthology.orchestra.run.smithi031:> stat --file-system '--printf=%T 2024-09-11T00:31:19.376 DEBUG:teuthology.orchestra.run.smithi031:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:19.417 INFO:teuthology.orchestra.run.smithi031.stdout:fuseblk 2024-09-11T00:31:19.418 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:19.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:19.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:19.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:19.521 DEBUG:teuthology.orchestra.run.smithi204:> stat --file-system '--printf=%T 2024-09-11T00:31:19.521 DEBUG:teuthology.orchestra.run.smithi204:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-11T00:31:19.557 INFO:teuthology.orchestra.run.smithi204.stdout:fuseblk 2024-09-11T00:31:19.557 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-11T00:31:19.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:31:19.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-11T00:31:19.655 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-09-11T00:31:19.697 INFO:tasks.mds_thrash:mds thrasher using random seed: 1726014679 2024-09-11T00:31:19.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:31:19.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.866+0000 7f292c600640 1 -- 172.21.15.31:0/2714730487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29241518b0 msgr2=0x7f29240c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:19.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.866+0000 7f292c600640 1 --2- 172.21.15.31:0/2714730487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29241518b0 0x7f29240c3600 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29240add10 tx=0x7f291c02ab40 comp rx=0 tx=0).stop 2024-09-11T00:31:19.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.866+0000 7f292c600640 1 -- 172.21.15.31:0/2714730487 shutdown_connections 2024-09-11T00:31:19.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.866+0000 7f292c600640 1 --2- 172.21.15.31:0/2714730487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29241518b0 0x7f29240c3600 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:19.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.866+0000 7f292c600640 1 --2- 172.21.15.31:0/2714730487 >> v2:172.21.15.204:3300/0 conn(0x7f29240c23a0 0x7f29240c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:19.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.866+0000 7f292c600640 1 -- 172.21.15.31:0/2714730487 >> 172.21.15.31:0/2714730487 conn(0x7f29240a9750 msgr2=0x7f29240a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:19.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.866+0000 7f292c600640 1 -- 172.21.15.31:0/2714730487 shutdown_connections 2024-09-11T00:31:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.866+0000 7f292c600640 1 -- 172.21.15.31:0/2714730487 wait complete. 2024-09-11T00:31:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.867+0000 7f292c600640 1 Processor -- start 2024-09-11T00:31:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.868+0000 7f292c600640 1 -- start start 2024-09-11T00:31:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.868+0000 7f292c600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2924072480 0x7f2924148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.868+0000 7f292c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29240c23a0 0x7f29240c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.868+0000 7f292c600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29241518b0 0x7f292414f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f292c600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2924074200 con 0x7f29240c23a0 2024-09-11T00:31:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f292c600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2924074080 con 0x7f2924072480 2024-09-11T00:31:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f292c600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2924072860 con 0x7f29241518b0 2024-09-11T00:31:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f2929200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29240c23a0 0x7f29240c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f2929200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29240c23a0 0x7f29240c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47716/0 (socket says 172.21.15.31:47716) 2024-09-11T00:31:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f2929200640 1 -- 172.21.15.31:0/504767324 learned_addr learned my addr 172.21.15.31:0/504767324 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f2929c00640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2924072480 0x7f2924148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:19.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f292a600640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29241518b0 0x7f292414f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:19.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f2929200640 1 -- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29241518b0 msgr2=0x7f292414f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:19.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f2929200640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29241518b0 0x7f292414f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:19.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f2929200640 1 -- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2924072480 msgr2=0x7f2924148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:19.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f2929200640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2924072480 0x7f2924148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:19.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.869+0000 7f2929200640 1 -- 172.21.15.31:0/504767324 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29240c70e0 con 0x7f29240c23a0 2024-09-11T00:31:19.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.870+0000 7f2929200640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29240c23a0 0x7f29240c6ba0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f291000e9c0 tx=0x7f291000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:19.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.870+0000 7f2921600640 1 -- 172.21.15.31:0/504767324 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291000cd80 con 0x7f29240c23a0 2024-09-11T00:31:19.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.870+0000 7f2921600640 1 -- 172.21.15.31:0/504767324 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2910005020 con 0x7f29240c23a0 2024-09-11T00:31:19.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.870+0000 7f2921600640 1 -- 172.21.15.31:0/504767324 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910005470 con 0x7f29240c23a0 2024-09-11T00:31:19.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.870+0000 7f292c600640 1 -- 172.21.15.31:0/504767324 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29241ec960 con 0x7f29240c23a0 2024-09-11T00:31:19.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.870+0000 7f292c600640 1 -- 172.21.15.31:0/504767324 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2924157d80 con 0x7f29240c23a0 2024-09-11T00:31:19.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.871+0000 7f292c600640 1 -- 172.21.15.31:0/504767324 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2924153f20 con 0x7f29240c23a0 2024-09-11T00:31:19.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.872+0000 7f2921600640 1 -- 172.21.15.31:0/504767324 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f29100047a0 con 0x7f29240c23a0 2024-09-11T00:31:19.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.873+0000 7f2921600640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2904077f70 0x7f290407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:19.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.873+0000 7f2921600640 1 -- 172.21.15.31:0/504767324 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f291009aaa0 con 0x7f29240c23a0 2024-09-11T00:31:19.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.874+0000 7f2929c00640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2904077f70 0x7f290407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.874+0000 7f2929c00640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2904077f70 0x7f290407a430 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2918007fd0 tx=0x7f2918009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:19.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:19.878+0000 7f2921600640 1 -- 172.21.15.31:0/504767324 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2910062b80 con 0x7f29240c23a0 2024-09-11T00:31:20.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.078+0000 7f292c600640 1 -- 172.21.15.31:0/504767324 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f292414e7f0 con 0x7f29240c23a0 2024-09-11T00:31:20.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.079+0000 7f2921600640 1 -- 172.21.15.31:0/504767324 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20383 (secure 0 0 0) 0x7f29100675b0 con 0x7f29240c23a0 2024-09-11T00:31:20.082 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:20.082 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:30:55.962415+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:31:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 -- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2904077f70 msgr2=0x7f290407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2904077f70 0x7f290407a430 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2918007fd0 tx=0x7f2918009040 comp rx=0 tx=0).stop 2024-09-11T00:31:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 -- 172.21.15.31:0/504767324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29240c23a0 msgr2=0x7f29240c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29240c23a0 0x7f29240c6ba0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f291000e9c0 tx=0x7f291000ee80 comp rx=0 tx=0).stop 2024-09-11T00:31:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 -- 172.21.15.31:0/504767324 shutdown_connections 2024-09-11T00:31:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2904077f70 0x7f290407a430 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f29241518b0 0x7f292414f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29240c23a0 0x7f29240c6ba0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 --2- 172.21.15.31:0/504767324 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2924072480 0x7f2924148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 -- 172.21.15.31:0/504767324 >> 172.21.15.31:0/504767324 conn(0x7f29240a9750 msgr2=0x7f292414c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 -- 172.21.15.31:0/504767324 shutdown_connections 2024-09-11T00:31:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.081+0000 7f292c600640 1 -- 172.21.15.31:0/504767324 wait complete. 2024-09-11T00:31:20.101 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T00:30:34.418004+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-09-11T00:30:43.375426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-09-11T00:30:44.303848+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-09-11T00:30:52.399156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-09-11T00:30:53.364342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-11T00:31:20.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-11T00:31:20.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.262+0000 7f1179000640 1 -- 172.21.15.31:0/4120625391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11740aab00 msgr2=0x7f1174150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.262+0000 7f1179000640 1 --2- 172.21.15.31:0/4120625391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11740aab00 0x7f1174150de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1164005fd0 tx=0x7f116402a540 comp rx=0 tx=0).stop 2024-09-11T00:31:20.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.263+0000 7f1179000640 1 -- 172.21.15.31:0/4120625391 shutdown_connections 2024-09-11T00:31:20.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.263+0000 7f1179000640 1 --2- 172.21.15.31:0/4120625391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11740c23a0 0x7f1174151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.263+0000 7f1179000640 1 --2- 172.21.15.31:0/4120625391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11740aab00 0x7f1174150de0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.263+0000 7f1179000640 1 -- 172.21.15.31:0/4120625391 >> 172.21.15.31:0/4120625391 conn(0x7f117406caf0 msgr2=0x7f117406ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.263+0000 7f1179000640 1 -- 172.21.15.31:0/4120625391 shutdown_connections 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.263+0000 7f1179000640 1 -- 172.21.15.31:0/4120625391 wait complete. 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.264+0000 7f1179000640 1 Processor -- start 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.264+0000 7f1179000640 1 -- start start 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.265+0000 7f1179000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11740aab00 0x7f11740c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.265+0000 7f1179000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11740ad5a0 0x7f1174145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.265+0000 7f1179000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11740c23a0 0x7f1174146150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.265+0000 7f1179000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f11740aaee0 con 0x7f11740aab00 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.265+0000 7f1179000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f11740704c0 con 0x7f11740c23a0 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.265+0000 7f1179000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f11740ab060 con 0x7f11740ad5a0 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.265+0000 7f1173400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11740aab00 0x7f11740c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:20.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.265+0000 7f1173400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11740aab00 0x7f11740c44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47728/0 (socket says 172.21.15.31:47728) 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.265+0000 7f1173400640 1 -- 172.21.15.31:0/406236222 learned_addr learned my addr 172.21.15.31:0/406236222 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.266+0000 7f116aa00640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11740ad5a0 0x7f1174145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.266+0000 7f1173e00640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11740c23a0 0x7f1174146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.266+0000 7f1173400640 1 -- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11740ad5a0 msgr2=0x7f1174145c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.266+0000 7f1173400640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11740ad5a0 0x7f1174145c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.266+0000 7f1173400640 1 -- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11740c23a0 msgr2=0x7f1174146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.266+0000 7f1173400640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11740c23a0 0x7f1174146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.266+0000 7f1173400640 1 -- 172.21.15.31:0/406236222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1174153600 con 0x7f11740aab00 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.266+0000 7f1173e00640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11740c23a0 0x7f1174146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.266+0000 7f1173400640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11740aab00 0x7f11740c44e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f11640099f0 tx=0x7f11640038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.266+0000 7f116be00640 1 -- 172.21.15.31:0/406236222 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116403f070 con 0x7f11740aab00 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.267+0000 7f116be00640 1 -- 172.21.15.31:0/406236222 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11640047a0 con 0x7f11740aab00 2024-09-11T00:31:20.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.267+0000 7f116be00640 1 -- 172.21.15.31:0/406236222 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164035030 con 0x7f11740aab00 2024-09-11T00:31:20.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.267+0000 7f1179000640 1 -- 172.21.15.31:0/406236222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1174146fe0 con 0x7f11740aab00 2024-09-11T00:31:20.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.267+0000 7f1179000640 1 -- 172.21.15.31:0/406236222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11741538a0 con 0x7f11740aab00 2024-09-11T00:31:20.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.268+0000 7f1179000640 1 -- 172.21.15.31:0/406236222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1174150de0 con 0x7f11740aab00 2024-09-11T00:31:20.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.269+0000 7f116be00640 1 -- 172.21.15.31:0/406236222 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1164004bf0 con 0x7f11740aab00 2024-09-11T00:31:20.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.270+0000 7f116be00640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1150077f70 0x7f115007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:20.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.270+0000 7f116be00640 1 -- 172.21.15.31:0/406236222 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f11640bb430 con 0x7f11740aab00 2024-09-11T00:31:20.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.270+0000 7f116aa00640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1150077f70 0x7f115007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:20.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.271+0000 7f116aa00640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1150077f70 0x7f115007a430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f11740a71e0 tx=0x7f115c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:20.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.274+0000 7f116be00640 1 -- 172.21.15.31:0/406236222 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f11640835a0 con 0x7f11740aab00 2024-09-11T00:31:20.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.466+0000 7f1179000640 1 -- 172.21.15.31:0/406236222 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1174146690 con 0x7f11740aab00 2024-09-11T00:31:20.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.467+0000 7f116be00640 1 -- 172.21.15.31:0/406236222 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f116408c310 con 0x7f11740aab00 2024-09-11T00:31:20.470 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-09-11T00:31:20.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.469+0000 7f1179000640 1 -- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1150077f70 msgr2=0x7f115007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.469+0000 7f1179000640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1150077f70 0x7f115007a430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f11740a71e0 tx=0x7f115c005f90 comp rx=0 tx=0).stop 2024-09-11T00:31:20.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.469+0000 7f1179000640 1 -- 172.21.15.31:0/406236222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11740aab00 msgr2=0x7f11740c44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.469+0000 7f1179000640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11740aab00 0x7f11740c44e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f11640099f0 tx=0x7f11640038a0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.469+0000 7f1179000640 1 -- 172.21.15.31:0/406236222 shutdown_connections 2024-09-11T00:31:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.469+0000 7f1179000640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1150077f70 0x7f115007a430 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.469+0000 7f1179000640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f11740c23a0 0x7f1174146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.469+0000 7f1179000640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f11740ad5a0 0x7f1174145c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.469+0000 7f1179000640 1 --2- 172.21.15.31:0/406236222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f11740aab00 0x7f11740c44e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.469+0000 7f1179000640 1 -- 172.21.15.31:0/406236222 >> 172.21.15.31:0/406236222 conn(0x7f117406caf0 msgr2=0x7f11740a8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.470+0000 7f1179000640 1 -- 172.21.15.31:0/406236222 shutdown_connections 2024-09-11T00:31:20.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.470+0000 7f1179000640 1 -- 172.21.15.31:0/406236222 wait complete. 2024-09-11T00:31:20.488 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-09-11T00:31:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.647+0000 7f1beb800640 1 -- 172.21.15.31:0/1332613568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1be40c3bd0 msgr2=0x7f1be40aa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.647+0000 7f1beb800640 1 --2- 172.21.15.31:0/1332613568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1be40c3bd0 0x7f1be40aa9e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1be40a6f20 tx=0x7f1bdc02a840 comp rx=0 tx=0).stop 2024-09-11T00:31:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.647+0000 7f1beb800640 1 -- 172.21.15.31:0/1332613568 shutdown_connections 2024-09-11T00:31:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.647+0000 7f1beb800640 1 --2- 172.21.15.31:0/1332613568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1be40c3bd0 0x7f1be40aa9e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.647+0000 7f1beb800640 1 --2- 172.21.15.31:0/1332613568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1be4150740 0x7f1be4152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.647+0000 7f1beb800640 1 -- 172.21.15.31:0/1332613568 >> 172.21.15.31:0/1332613568 conn(0x7f1be406d560 msgr2=0x7f1be40c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:20.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.647+0000 7f1beb800640 1 -- 172.21.15.31:0/1332613568 shutdown_connections 2024-09-11T00:31:20.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.647+0000 7f1beb800640 1 -- 172.21.15.31:0/1332613568 wait complete. 2024-09-11T00:31:20.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.649+0000 7f1beb800640 1 Processor -- start 2024-09-11T00:31:20.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.649+0000 7f1beb800640 1 -- start start 2024-09-11T00:31:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1beb800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1be40c3500 0x7f1be4149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1beb800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1be40c3bd0 0x7f1be4147be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:20.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1beb800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1be4150740 0x7f1be4148120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1beb800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1be40c2f00 con 0x7f1be4150740 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1beb800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1be40c2c20 con 0x7f1be40c3500 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1beb800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1be40c31e0 con 0x7f1be40c3bd0 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1be9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1be4150740 0x7f1be4148120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1be9800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1be4150740 0x7f1be4148120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47738/0 (socket says 172.21.15.31:47738) 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1be9800640 1 -- 172.21.15.31:0/1357394809 learned_addr learned my addr 172.21.15.31:0/1357394809 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1be8e00640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1be40c3500 0x7f1be4149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1bdbe00640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1be40c3bd0 0x7f1be4147be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1be9800640 1 -- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1be40c3bd0 msgr2=0x7f1be4147be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1be9800640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1be40c3bd0 0x7f1be4147be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1be9800640 1 -- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1be40c3500 msgr2=0x7f1be4149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1be9800640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1be40c3500 0x7f1be4149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.650+0000 7f1be9800640 1 -- 172.21.15.31:0/1357394809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be4151320 con 0x7f1be4150740 2024-09-11T00:31:20.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.651+0000 7f1be8e00640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1be40c3500 0x7f1be4149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:20.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.651+0000 7f1be9800640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1be4150740 0x7f1be4148120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc035040 tx=0x7f1bdc033e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:20.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.651+0000 7f1be1600640 1 -- 172.21.15.31:0/1357394809 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc042020 con 0x7f1be4150740 2024-09-11T00:31:20.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.651+0000 7f1be1600640 1 -- 172.21.15.31:0/1357394809 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bdc004330 con 0x7f1be4150740 2024-09-11T00:31:20.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.651+0000 7f1be1600640 1 -- 172.21.15.31:0/1357394809 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc037040 con 0x7f1be4150740 2024-09-11T00:31:20.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.651+0000 7f1beb800640 1 -- 172.21.15.31:0/1357394809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be4151040 con 0x7f1be4150740 2024-09-11T00:31:20.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.651+0000 7f1beb800640 1 -- 172.21.15.31:0/1357394809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be4154ac0 con 0x7f1be4150740 2024-09-11T00:31:20.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.652+0000 7f1beb800640 1 -- 172.21.15.31:0/1357394809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be40702d0 con 0x7f1be4150740 2024-09-11T00:31:20.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.654+0000 7f1be1600640 1 -- 172.21.15.31:0/1357394809 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1bdc031070 con 0x7f1be4150740 2024-09-11T00:31:20.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.655+0000 7f1be1600640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1bc0077f70 0x7f1bc007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:20.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.655+0000 7f1be1600640 1 -- 172.21.15.31:0/1357394809 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f1bdc0bb0d0 con 0x7f1be4150740 2024-09-11T00:31:20.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.655+0000 7f1be8e00640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1bc0077f70 0x7f1bc007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:20.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.656+0000 7f1be8e00640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1bc0077f70 0x7f1bc007a430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400ab40 tx=0x7f1bd400a520 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:20.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.659+0000 7f1be1600640 1 -- 172.21.15.31:0/1357394809 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1be40702d0 con 0x7f1be4150740 2024-09-11T00:31:20.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.852+0000 7f1beb800640 1 -- 172.21.15.31:0/1357394809 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f1be40ab050 con 0x7f1be4150740 2024-09-11T00:31:20.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.852+0000 7f1be1600640 1 -- 172.21.15.31:0/1357394809 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v26) ==== 99+0+11 (secure 0 0 0) 0x7f1bdc04c030 con 0x7f1be4150740 2024-09-11T00:31:20.855 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 -- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1bc0077f70 msgr2=0x7f1bc007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1bc0077f70 0x7f1bc007a430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400ab40 tx=0x7f1bd400a520 comp rx=0 tx=0).stop 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 -- 172.21.15.31:0/1357394809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1be4150740 msgr2=0x7f1be4148120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1be4150740 0x7f1be4148120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc035040 tx=0x7f1bdc033e80 comp rx=0 tx=0).stop 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 -- 172.21.15.31:0/1357394809 shutdown_connections 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1bc0077f70 0x7f1bc007a430 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1be4150740 0x7f1be4148120 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1be40c3bd0 0x7f1be4147be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 --2- 172.21.15.31:0/1357394809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1be40c3500 0x7f1be4149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 -- 172.21.15.31:0/1357394809 >> 172.21.15.31:0/1357394809 conn(0x7f1be406d560 msgr2=0x7f1be41563b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 -- 172.21.15.31:0/1357394809 shutdown_connections 2024-09-11T00:31:20.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:20.855+0000 7f1beb800640 1 -- 172.21.15.31:0/1357394809 wait complete. 2024-09-11T00:31:20.874 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-09-11T00:31:21.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.034+0000 7efee3200640 1 -- 172.21.15.31:0/1548129526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efee40b13b0 msgr2=0x7efee40c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.034+0000 7efee3200640 1 --2- 172.21.15.31:0/1548129526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efee40b13b0 0x7efee40c8160 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efedc009be0 tx=0x7efedc02a7c0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.034+0000 7efee3200640 1 -- 172.21.15.31:0/1548129526 shutdown_connections 2024-09-11T00:31:21.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.034+0000 7efee3200640 1 --2- 172.21.15.31:0/1548129526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efee40b13b0 0x7efee40c8160 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.034+0000 7efee3200640 1 --2- 172.21.15.31:0/1548129526 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efee40cbce0 0x7efee41569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.034+0000 7efee3200640 1 -- 172.21.15.31:0/1548129526 >> 172.21.15.31:0/1548129526 conn(0x7efee40688e0 msgr2=0x7efee4068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:21.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.034+0000 7efee3200640 1 -- 172.21.15.31:0/1548129526 shutdown_connections 2024-09-11T00:31:21.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.034+0000 7efee3200640 1 -- 172.21.15.31:0/1548129526 wait complete. 2024-09-11T00:31:21.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.035+0000 7efee3200640 1 Processor -- start 2024-09-11T00:31:21.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.036+0000 7efee3200640 1 -- start start 2024-09-11T00:31:21.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.036+0000 7efee3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efee40b13b0 0x7efee41505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.036+0000 7efee3200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efee40c9df0 0x7efee41439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.036+0000 7efee3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee40cbce0 0x7efee4151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.036+0000 7efee3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efee40cc240 con 0x7efee40b13b0 2024-09-11T00:31:21.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.036+0000 7efee3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efee40cc0c0 con 0x7efee40cbce0 2024-09-11T00:31:21.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.036+0000 7efee3200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efee40c86a0 con 0x7efee40c9df0 2024-09-11T00:31:21.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efee40b13b0 0x7efee41505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efee40b13b0 0x7efee41505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47752/0 (socket says 172.21.15.31:47752) 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee1e00640 1 -- 172.21.15.31:0/1153775864 learned_addr learned my addr 172.21.15.31:0/1153775864 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efed9400640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efee40c9df0 0x7efee41439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee2800640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee40cbce0 0x7efee4151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee1e00640 1 -- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efee40c9df0 msgr2=0x7efee41439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee1e00640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efee40c9df0 0x7efee41439f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee1e00640 1 -- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee40cbce0 msgr2=0x7efee4151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee1e00640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee40cbce0 0x7efee4151220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee1e00640 1 -- 172.21.15.31:0/1153775864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee41ebc80 con 0x7efee40b13b0 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee2800640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee40cbce0 0x7efee4151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:21.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.037+0000 7efee1e00640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efee40b13b0 0x7efee41505b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efed400d740 tx=0x7efed400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.038+0000 7efedaa00640 1 -- 172.21.15.31:0/1153775864 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed4017070 con 0x7efee40b13b0 2024-09-11T00:31:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.038+0000 7efedaa00640 1 -- 172.21.15.31:0/1153775864 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efed4009d20 con 0x7efee40b13b0 2024-09-11T00:31:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.038+0000 7efee3200640 1 -- 172.21.15.31:0/1153775864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee41efc90 con 0x7efee40b13b0 2024-09-11T00:31:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.038+0000 7efedaa00640 1 -- 172.21.15.31:0/1153775864 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed40106c0 con 0x7efee40b13b0 2024-09-11T00:31:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.038+0000 7efee3200640 1 -- 172.21.15.31:0/1153775864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee41eff70 con 0x7efee40b13b0 2024-09-11T00:31:21.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.039+0000 7efee3200640 1 -- 172.21.15.31:0/1153775864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee40c8b20 con 0x7efee40b13b0 2024-09-11T00:31:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.040+0000 7efedaa00640 1 -- 172.21.15.31:0/1153775864 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7efed4010860 con 0x7efee40b13b0 2024-09-11T00:31:21.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.041+0000 7efedaa00640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efec0077f70 0x7efec007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.041+0000 7efedaa00640 1 -- 172.21.15.31:0/1153775864 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7efed409a440 con 0x7efee40b13b0 2024-09-11T00:31:21.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.042+0000 7efed9400640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efec0077f70 0x7efec007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:21.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.042+0000 7efed9400640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efec0077f70 0x7efec007a430 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efee41e1230 tx=0x7efecc007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:21.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.046+0000 7efedaa00640 1 -- 172.21.15.31:0/1153775864 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efed40625b0 con 0x7efee40b13b0 2024-09-11T00:31:21.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.240+0000 7efee3200640 1 -- 172.21.15.31:0/1153775864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7efee415a660 con 0x7efee40b13b0 2024-09-11T00:31:21.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.240+0000 7efedaa00640 1 -- 172.21.15.31:0/1153775864 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v26) ==== 95+0+11 (secure 0 0 0) 0x7efed4066fe0 con 0x7efee40b13b0 2024-09-11T00:31:21.243 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-09-11T00:31:21.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 -- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efec0077f70 msgr2=0x7efec007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efec0077f70 0x7efec007a430 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efee41e1230 tx=0x7efecc007e90 comp rx=0 tx=0).stop 2024-09-11T00:31:21.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 -- 172.21.15.31:0/1153775864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efee40b13b0 msgr2=0x7efee41505b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efee40b13b0 0x7efee41505b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efed400d740 tx=0x7efed400dc00 comp rx=0 tx=0).stop 2024-09-11T00:31:21.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 -- 172.21.15.31:0/1153775864 shutdown_connections 2024-09-11T00:31:21.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efec0077f70 0x7efec007a430 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee40cbce0 0x7efee4151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efee40c9df0 0x7efee41439f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 --2- 172.21.15.31:0/1153775864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efee40b13b0 0x7efee41505b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 -- 172.21.15.31:0/1153775864 >> 172.21.15.31:0/1153775864 conn(0x7efee40688e0 msgr2=0x7efee4156940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:21.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 -- 172.21.15.31:0/1153775864 shutdown_connections 2024-09-11T00:31:21.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.242+0000 7efee3200640 1 -- 172.21.15.31:0/1153775864 wait complete. 2024-09-11T00:31:21.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-09-11T00:31:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.423+0000 7f575be00640 1 -- 172.21.15.31:0/3032428254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57400039e0 msgr2=0x7f5740003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.423+0000 7f575be00640 1 --2- 172.21.15.31:0/3032428254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57400039e0 0x7f5740003dc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f574c001ec0 tx=0x7f5740005830 comp rx=0 tx=0).stop 2024-09-11T00:31:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.423+0000 7f575b400640 1 -- 172.21.15.31:0/3032428254 reap_dead start 2024-09-11T00:31:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.423+0000 7f575be00640 1 -- 172.21.15.31:0/3032428254 shutdown_connections 2024-09-11T00:31:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.423+0000 7f575be00640 1 -- 172.21.15.31:0/3032428254 >> 172.21.15.31:0/3032428254 conn(0x7f575c0688e0 msgr2=0x7f575c068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.423+0000 7f575be00640 1 -- 172.21.15.31:0/3032428254 shutdown_connections 2024-09-11T00:31:21.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.423+0000 7f575be00640 1 -- 172.21.15.31:0/3032428254 wait complete. 2024-09-11T00:31:21.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.424+0000 7f575be00640 1 Processor -- start 2024-09-11T00:31:21.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.425+0000 7f575be00640 1 -- start start 2024-09-11T00:31:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.425+0000 7f575be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57400039e0 0x7f575c1ef4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.425+0000 7f575be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f575c0abff0 0x7f575c0ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.425+0000 7f575be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f575c14c3f0 0x7f575c0ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.425+0000 7f575be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f575c0cc240 con 0x7f575c14c3f0 2024-09-11T00:31:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.425+0000 7f575be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f575c0cc0c0 con 0x7f57400039e0 2024-09-11T00:31:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.425+0000 7f575be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f575c1569b0 con 0x7f575c0abff0 2024-09-11T00:31:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.426+0000 7f575b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f575c14c3f0 0x7f575c0ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.426+0000 7f575b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f575c14c3f0 0x7f575c0ac650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47766/0 (socket says 172.21.15.31:47766) 2024-09-11T00:31:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.426+0000 7f575b400640 1 -- 172.21.15.31:0/2315163337 learned_addr learned my addr 172.21.15.31:0/2315163337 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.426+0000 7f575b400640 1 -- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f575c0abff0 msgr2=0x7f575c0ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.426+0000 7f575b400640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f575c0abff0 0x7f575c0ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.426+0000 7f575b400640 1 -- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57400039e0 msgr2=0x7f575c1ef4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.426+0000 7f575b400640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57400039e0 0x7f575c1ef4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.426+0000 7f575b400640 1 -- 172.21.15.31:0/2315163337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575c0ae130 con 0x7f575c14c3f0 2024-09-11T00:31:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.426+0000 7f575b400640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f575c14c3f0 0x7f575c0ac650 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f575c0c7a10 tx=0x7f57540035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.427+0000 7f5753400640 1 -- 172.21.15.31:0/2315163337 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754015070 con 0x7f575c14c3f0 2024-09-11T00:31:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.427+0000 7f5753400640 1 -- 172.21.15.31:0/2315163337 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5754011650 con 0x7f575c14c3f0 2024-09-11T00:31:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.427+0000 7f5753400640 1 -- 172.21.15.31:0/2315163337 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754011aa0 con 0x7f575c14c3f0 2024-09-11T00:31:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.427+0000 7f575be00640 1 -- 172.21.15.31:0/2315163337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f575c1efa10 con 0x7f575c14c3f0 2024-09-11T00:31:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.427+0000 7f575be00640 1 -- 172.21.15.31:0/2315163337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f575c15a240 con 0x7f575c14c3f0 2024-09-11T00:31:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.428+0000 7f575be00640 1 -- 172.21.15.31:0/2315163337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f575c069bb0 con 0x7f575c14c3f0 2024-09-11T00:31:21.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.429+0000 7f5753400640 1 -- 172.21.15.31:0/2315163337 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5754011c40 con 0x7f575c14c3f0 2024-09-11T00:31:21.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.430+0000 7f5753400640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f573c077f70 0x7f573c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.430+0000 7f5753400640 1 -- 172.21.15.31:0/2315163337 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f575409adb0 con 0x7f575c14c3f0 2024-09-11T00:31:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.430+0000 7f575aa00640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f573c077f70 0x7f573c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.431+0000 7f575aa00640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f573c077f70 0x7f573c07a430 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f574c033000 tx=0x7f574c001670 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.434+0000 7f5753400640 1 -- 172.21.15.31:0/2315163337 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f575c069bb0 con 0x7f575c14c3f0 2024-09-11T00:31:21.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.627+0000 7f575be00640 1 -- 172.21.15.31:0/2315163337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7f575c0b18f0 con 0x7f575c14c3f0 2024-09-11T00:31:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.627+0000 7f5753400640 1 -- 172.21.15.31:0/2315163337 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v26) ==== 99+0+11 (secure 0 0 0) 0x7f575409f070 con 0x7f575c14c3f0 2024-09-11T00:31:21.630 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 -- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f573c077f70 msgr2=0x7f573c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f573c077f70 0x7f573c07a430 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f574c033000 tx=0x7f574c001670 comp rx=0 tx=0).stop 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 -- 172.21.15.31:0/2315163337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f575c14c3f0 msgr2=0x7f575c0ac650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f575c14c3f0 0x7f575c0ac650 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f575c0c7a10 tx=0x7f57540035f0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 -- 172.21.15.31:0/2315163337 shutdown_connections 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f573c077f70 0x7f573c07a430 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f575c14c3f0 0x7f575c0ac650 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f575c0abff0 0x7f575c0ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 --2- 172.21.15.31:0/2315163337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57400039e0 0x7f575c1ef4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 -- 172.21.15.31:0/2315163337 >> 172.21.15.31:0/2315163337 conn(0x7f575c0688e0 msgr2=0x7f575c0aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 -- 172.21.15.31:0/2315163337 shutdown_connections 2024-09-11T00:31:21.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.630+0000 7f575be00640 1 -- 172.21.15.31:0/2315163337 wait complete. 2024-09-11T00:31:21.648 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-09-11T00:31:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.809+0000 7fa431e00640 1 -- 172.21.15.31:0/1770560004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa42c0c9940 msgr2=0x7fa42c06d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.809+0000 7fa431e00640 1 --2- 172.21.15.31:0/1770560004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa42c0c9940 0x7fa42c06d2b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa418009830 tx=0x7fa41802a4a0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.809+0000 7fa431e00640 1 -- 172.21.15.31:0/1770560004 shutdown_connections 2024-09-11T00:31:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.809+0000 7fa431e00640 1 --2- 172.21.15.31:0/1770560004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa42c0c9940 0x7fa42c06d2b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.809+0000 7fa431e00640 1 --2- 172.21.15.31:0/1770560004 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa42c0b10e0 0x7fa42c0c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.809+0000 7fa431e00640 1 -- 172.21.15.31:0/1770560004 >> 172.21.15.31:0/1770560004 conn(0x7fa42c0ad5f0 msgr2=0x7fa42c0ada00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.809+0000 7fa431e00640 1 -- 172.21.15.31:0/1770560004 shutdown_connections 2024-09-11T00:31:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.809+0000 7fa431e00640 1 -- 172.21.15.31:0/1770560004 wait complete. 2024-09-11T00:31:21.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.810+0000 7fa431e00640 1 Processor -- start 2024-09-11T00:31:21.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.810+0000 7fa431e00640 1 -- start start 2024-09-11T00:31:21.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.811+0000 7fa431e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa42c0b10e0 0x7fa42c0c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.811+0000 7fa431e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa42c0c9940 0x7fa42c0cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.811+0000 7fa431e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa42c0c9e50 0x7fa42c0c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.811+0000 7fa431e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c06d970 con 0x7fa42c0b10e0 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.811+0000 7fa431e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c06d7f0 con 0x7fa42c0c9e50 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.811+0000 7fa431e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa42c160870 con 0x7fa42c0c9940 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.811+0000 7fa430a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa42c0b10e0 0x7fa42c0c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa430a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa42c0b10e0 0x7fa42c0c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47772/0 (socket says 172.21.15.31:47772) 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa430a00640 1 -- 172.21.15.31:0/307347164 learned_addr learned my addr 172.21.15.31:0/307347164 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa431400640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa42c0c9e50 0x7fa42c0c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa42be00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa42c0c9940 0x7fa42c0cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa430a00640 1 -- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa42c0c9940 msgr2=0x7fa42c0cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa430a00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa42c0c9940 0x7fa42c0cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa430a00640 1 -- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa42c0c9e50 msgr2=0x7fa42c0c2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:21.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa430a00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa42c0c9e50 0x7fa42c0c2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:21.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa430a00640 1 -- 172.21.15.31:0/307347164 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c0b0c20 con 0x7fa42c0b10e0 2024-09-11T00:31:21.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa431400640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa42c0c9e50 0x7fa42c0c2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:21.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.812+0000 7fa430a00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa42c0b10e0 0x7fa42c0c5a90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa42000ea20 tx=0x7fa42000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:21.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.813+0000 7fa428c00640 1 -- 172.21.15.31:0/307347164 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42000cd80 con 0x7fa42c0b10e0 2024-09-11T00:31:21.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.813+0000 7fa428c00640 1 -- 172.21.15.31:0/307347164 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa420010430 con 0x7fa42c0b10e0 2024-09-11T00:31:21.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.813+0000 7fa431e00640 1 -- 172.21.15.31:0/307347164 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c1ec3f0 con 0x7fa42c0b10e0 2024-09-11T00:31:21.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.813+0000 7fa428c00640 1 -- 172.21.15.31:0/307347164 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420010840 con 0x7fa42c0b10e0 2024-09-11T00:31:21.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.813+0000 7fa431e00640 1 -- 172.21.15.31:0/307347164 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c157ca0 con 0x7fa42c0b10e0 2024-09-11T00:31:21.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.814+0000 7fa431e00640 1 -- 172.21.15.31:0/307347164 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42c0c56d0 con 0x7fa42c0b10e0 2024-09-11T00:31:21.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.815+0000 7fa428c00640 1 -- 172.21.15.31:0/307347164 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa4200109e0 con 0x7fa42c0b10e0 2024-09-11T00:31:21.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.816+0000 7fa428c00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa40c077f70 0x7fa40c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:21.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.816+0000 7fa428c00640 1 -- 172.21.15.31:0/307347164 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fa42009ac30 con 0x7fa42c0b10e0 2024-09-11T00:31:21.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.816+0000 7fa42be00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa40c077f70 0x7fa40c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:21.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.817+0000 7fa42be00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa40c077f70 0x7fa40c07a430 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa418034000 tx=0x7fa418004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:21.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:21.820+0000 7fa428c00640 1 -- 172.21.15.31:0/307347164 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa4200633f0 con 0x7fa42c0b10e0 2024-09-11T00:31:22.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.014+0000 7fa431e00640 1 -- 172.21.15.31:0/307347164 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7fa42c0c67a0 con 0x7fa42c0b10e0 2024-09-11T00:31:22.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.015+0000 7fa428c00640 1 -- 172.21.15.31:0/307347164 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v26) ==== 95+0+11 (secure 0 0 0) 0x7fa42009a020 con 0x7fa42c0b10e0 2024-09-11T00:31:22.018 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-09-11T00:31:22.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 -- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa40c077f70 msgr2=0x7fa40c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:22.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa40c077f70 0x7fa40c07a430 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa418034000 tx=0x7fa418004730 comp rx=0 tx=0).stop 2024-09-11T00:31:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 -- 172.21.15.31:0/307347164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa42c0b10e0 msgr2=0x7fa42c0c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa42c0b10e0 0x7fa42c0c5a90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa42000ea20 tx=0x7fa42000eee0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 -- 172.21.15.31:0/307347164 shutdown_connections 2024-09-11T00:31:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa40c077f70 0x7fa40c07a430 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa42c0c9e50 0x7fa42c0c2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa42c0c9940 0x7fa42c0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 --2- 172.21.15.31:0/307347164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa42c0b10e0 0x7fa42c0c5a90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 -- 172.21.15.31:0/307347164 >> 172.21.15.31:0/307347164 conn(0x7fa42c0ad5f0 msgr2=0x7fa42c15fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 -- 172.21.15.31:0/307347164 shutdown_connections 2024-09-11T00:31:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.017+0000 7fa431e00640 1 -- 172.21.15.31:0/307347164 wait complete. 2024-09-11T00:31:22.035 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-09-11T00:31:22.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:22.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.193+0000 7fedda400640 1 -- 172.21.15.31:0/2301445402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedd4153f20 msgr2=0x7fedd41516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:22.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.193+0000 7fedda400640 1 --2- 172.21.15.31:0/2301445402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedd4153f20 0x7fedd41516e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fedd40c4400 tx=0x7fedcc02aaa0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.193+0000 7fedda400640 1 -- 172.21.15.31:0/2301445402 shutdown_connections 2024-09-11T00:31:22.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.193+0000 7fedda400640 1 --2- 172.21.15.31:0/2301445402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedd4153f20 0x7fedd41516e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.193+0000 7fedda400640 1 --2- 172.21.15.31:0/2301445402 >> v2:172.21.15.204:3300/0 conn(0x7fedd4072480 0x7fedd41511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.193+0000 7fedda400640 1 -- 172.21.15.31:0/2301445402 >> 172.21.15.31:0/2301445402 conn(0x7fedd40a9750 msgr2=0x7fedd40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.193+0000 7fedda400640 1 -- 172.21.15.31:0/2301445402 shutdown_connections 2024-09-11T00:31:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.193+0000 7fedda400640 1 -- 172.21.15.31:0/2301445402 wait complete. 2024-09-11T00:31:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.195+0000 7fedda400640 1 Processor -- start 2024-09-11T00:31:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.195+0000 7fedda400640 1 -- start start 2024-09-11T00:31:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedda400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedd4072480 0x7fedd40c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedda400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedd40c23a0 0x7fedd40c3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedda400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedd4153f20 0x7fedd40c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedda400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fedd4074200 con 0x7fedd4072480 2024-09-11T00:31:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedda400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fedd4074080 con 0x7fedd40c23a0 2024-09-11T00:31:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedda400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fedd4072860 con 0x7fedd4153f20 2024-09-11T00:31:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedd3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedd4072480 0x7fedd40c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedd3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedd4072480 0x7fedd40c4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47794/0 (socket says 172.21.15.31:47794) 2024-09-11T00:31:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedd3400640 1 -- 172.21.15.31:0/436477158 learned_addr learned my addr 172.21.15.31:0/436477158 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedd3e00640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedd4153f20 0x7fedd40c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedcaa00640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedd40c23a0 0x7fedd40c3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedd3400640 1 -- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedd4153f20 msgr2=0x7fedd40c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedd3400640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedd4153f20 0x7fedd40c5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedd3400640 1 -- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedd40c23a0 msgr2=0x7fedd40c3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedd3400640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedd40c23a0 0x7fedd40c3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.196+0000 7fedd3400640 1 -- 172.21.15.31:0/436477158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd40ad4d0 con 0x7fedd4072480 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.197+0000 7fedd3400640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedd4072480 0x7fedd40c4c40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fedc400b410 tx=0x7fedc400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.197+0000 7fedcbe00640 1 -- 172.21.15.31:0/436477158 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4015020 con 0x7fedd4072480 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.197+0000 7fedcbe00640 1 -- 172.21.15.31:0/436477158 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fedc4011940 con 0x7fedd4072480 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.197+0000 7fedcbe00640 1 -- 172.21.15.31:0/436477158 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4011d90 con 0x7fedd4072480 2024-09-11T00:31:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.197+0000 7fedda400640 1 -- 172.21.15.31:0/436477158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd414d1e0 con 0x7fedd4072480 2024-09-11T00:31:22.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.197+0000 7fedda400640 1 -- 172.21.15.31:0/436477158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd40c7370 con 0x7fedd4072480 2024-09-11T00:31:22.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.198+0000 7fedda400640 1 -- 172.21.15.31:0/436477158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedd414cfd0 con 0x7fedd4072480 2024-09-11T00:31:22.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.199+0000 7fedcbe00640 1 -- 172.21.15.31:0/436477158 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fedc401a400 con 0x7fedd4072480 2024-09-11T00:31:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.200+0000 7fedcbe00640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fedb0077f70 0x7fedb007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.200+0000 7fedcbe00640 1 -- 172.21.15.31:0/436477158 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fedc409a320 con 0x7fedd4072480 2024-09-11T00:31:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.201+0000 7fedcaa00640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fedb0077f70 0x7fedb007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.202+0000 7fedcaa00640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fedb0077f70 0x7fedb007a430 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fedd40a9e80 tx=0x7fedbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.205+0000 7fedcbe00640 1 -- 172.21.15.31:0/436477158 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fedc401f020 con 0x7fedd4072480 2024-09-11T00:31:22.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.439+0000 7fedda400640 1 -- 172.21.15.31:0/436477158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedd41511a0 con 0x7fedd4072480 2024-09-11T00:31:22.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.439+0000 7fedcbe00640 1 -- 172.21.15.31:0/436477158 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fedc4062a50 con 0x7fedd4072480 2024-09-11T00:31:22.442 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:22.443 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:22.443 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:22.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 -- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fedb0077f70 msgr2=0x7fedb007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:22.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fedb0077f70 0x7fedb007a430 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fedd40a9e80 tx=0x7fedbc009040 comp rx=0 tx=0).stop 2024-09-11T00:31:22.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 -- 172.21.15.31:0/436477158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedd4072480 msgr2=0x7fedd40c4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:22.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedd4072480 0x7fedd40c4c40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fedc400b410 tx=0x7fedc400b8d0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 -- 172.21.15.31:0/436477158 shutdown_connections 2024-09-11T00:31:22.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fedb0077f70 0x7fedb007a430 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fedd4153f20 0x7fedd40c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedd40c23a0 0x7fedd40c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 --2- 172.21.15.31:0/436477158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fedd4072480 0x7fedd40c4c40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 -- 172.21.15.31:0/436477158 >> 172.21.15.31:0/436477158 conn(0x7fedd40a9750 msgr2=0x7fedd41536e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:22.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 -- 172.21.15.31:0/436477158 shutdown_connections 2024-09-11T00:31:22.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.442+0000 7fedda400640 1 -- 172.21.15.31:0/436477158 wait complete. 2024-09-11T00:31:22.461 INFO:tasks.mds_thrash:Ready to start thrashing 2024-09-11T00:31:22.462 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-09-11T00:31:22.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:31:22.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.620+0000 7f631ce00640 1 -- 172.21.15.31:0/2787089804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62fc003aa0 msgr2=0x7f62fc003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:22.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.620+0000 7f631ce00640 1 --2- 172.21.15.31:0/2787089804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62fc003aa0 0x7f62fc003ec0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6308006f90 tx=0x7f62fc005830 comp rx=0 tx=0).stop 2024-09-11T00:31:22.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.620+0000 7f6316800640 1 -- 172.21.15.31:0/2787089804 reap_dead start 2024-09-11T00:31:22.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.620+0000 7f631ce00640 1 -- 172.21.15.31:0/2787089804 shutdown_connections 2024-09-11T00:31:22.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.620+0000 7f631ce00640 1 -- 172.21.15.31:0/2787089804 >> 172.21.15.31:0/2787089804 conn(0x7f63180a9750 msgr2=0x7f63180a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:22.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.620+0000 7f631ce00640 1 -- 172.21.15.31:0/2787089804 shutdown_connections 2024-09-11T00:31:22.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.620+0000 7f631ce00640 1 -- 172.21.15.31:0/2787089804 wait complete. 2024-09-11T00:31:22.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.622+0000 7f631ce00640 1 Processor -- start 2024-09-11T00:31:22.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.622+0000 7f631ce00640 1 -- start start 2024-09-11T00:31:22.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f631ce00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f62fc003aa0 0x7f631814b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:22.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f631ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f631814bc80 0x7f63180c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:22.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f631ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63180a8810 0x7f631814d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f631ce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6318156f20 con 0x7f631814bc80 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f631ce00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f63180c7320 con 0x7f63180a8810 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f631ce00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f63180c3950 con 0x7f62fc003aa0 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f630d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f631814bc80 0x7f63180c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f630d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f631814bc80 0x7f63180c0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47800/0 (socket says 172.21.15.31:47800) 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f630d400640 1 -- 172.21.15.31:0/2300161245 learned_addr learned my addr 172.21.15.31:0/2300161245 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f6316800640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63180a8810 0x7f631814d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.623+0000 7f6315e00640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f62fc003aa0 0x7f631814b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f630d400640 1 -- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f62fc003aa0 msgr2=0x7f631814b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f630d400640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f62fc003aa0 0x7f631814b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f630d400640 1 -- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63180a8810 msgr2=0x7f631814d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:22.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f630d400640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63180a8810 0x7f631814d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:22.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f630d400640 1 -- 172.21.15.31:0/2300161245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63181efa70 con 0x7f631814bc80 2024-09-11T00:31:22.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f630d400640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f631814bc80 0x7f63180c0dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6300011ae0 tx=0x7f6300011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:22.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f630ea00640 1 -- 172.21.15.31:0/2300161245 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630000fc40 con 0x7f631814bc80 2024-09-11T00:31:22.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f630ea00640 1 -- 172.21.15.31:0/2300161245 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f630000fde0 con 0x7f631814bc80 2024-09-11T00:31:22.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f630ea00640 1 -- 172.21.15.31:0/2300161245 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300004280 con 0x7f631814bc80 2024-09-11T00:31:22.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f631ce00640 1 -- 172.21.15.31:0/2300161245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63181ef4f0 con 0x7f631814bc80 2024-09-11T00:31:22.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.624+0000 7f631ce00640 1 -- 172.21.15.31:0/2300161245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63180a8d60 con 0x7f631814bc80 2024-09-11T00:31:22.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.625+0000 7f631ce00640 1 -- 172.21.15.31:0/2300161245 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63180a79f0 con 0x7f631814bc80 2024-09-11T00:31:22.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.626+0000 7f630ea00640 1 -- 172.21.15.31:0/2300161245 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f63000035f0 con 0x7f631814bc80 2024-09-11T00:31:22.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.627+0000 7f630ea00640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f62f4077f70 0x7f62f407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:22.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.628+0000 7f630ea00640 1 -- 172.21.15.31:0/2300161245 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f630009ab00 con 0x7f631814bc80 2024-09-11T00:31:22.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.628+0000 7f6315e00640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f62f4077f70 0x7f62f407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:22.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.629+0000 7f6315e00640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f62f4077f70 0x7f62f407a430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6308033000 tx=0x7f6318071e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:22.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.632+0000 7f630ea00640 1 -- 172.21.15.31:0/2300161245 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f63000632c0 con 0x7f631814bc80 2024-09-11T00:31:22.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.824+0000 7f631ce00640 1 -- 172.21.15.31:0/2300161245 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6318153480 con 0x7f62f4077f70 2024-09-11T00:31:23.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.834+0000 7f630ea00640 1 -- 172.21.15.31:0/2300161245 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490314 (secure 0 0 0) 0x7f6318153480 con 0x7f62f4077f70 2024-09-11T00:31:23.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:23.510 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:31:23.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 -- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f62f4077f70 msgr2=0x7f62f407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:23.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f62f4077f70 0x7f62f407a430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6308033000 tx=0x7f6318071e90 comp rx=0 tx=0).stop 2024-09-11T00:31:23.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 -- 172.21.15.31:0/2300161245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f631814bc80 msgr2=0x7f63180c0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:23.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f631814bc80 0x7f63180c0dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6300011ae0 tx=0x7f6300011ee0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 -- 172.21.15.31:0/2300161245 shutdown_connections 2024-09-11T00:31:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f62f4077f70 0x7f62f407a430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63180a8810 0x7f631814d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f631814bc80 0x7f63180c0dd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 --2- 172.21.15.31:0/2300161245 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f62fc003aa0 0x7f631814b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 -- 172.21.15.31:0/2300161245 >> 172.21.15.31:0/2300161245 conn(0x7f63180a9750 msgr2=0x7f631806f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 -- 172.21.15.31:0/2300161245 shutdown_connections 2024-09-11T00:31:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:22.839+0000 7f631ce00640 1 -- 172.21.15.31:0/2300161245 wait complete. 2024-09-11T00:31:23.515 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-11T00:31:22.401600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595629,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":193,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219624,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755096,"statfs":{"total":773094113280,"available":772869218304,"internally_reserved":0,"allocated":3481600,"data_stored":2028210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":441,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003685"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719046+0000","last_change":"2024-09-11T00:30:54.983412+0000","last_active":"2024-09-11T00:30:56.719046+0000","last_peered":"2024-09-11T00:30:56.719046+0000","last_clean":"2024-09-11T00:30:56.719046+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:30:56.719046+0000","last_undegraded":"2024-09-11T00:30:56.719046+0000","last_fullsized":"2024-09-11T00:30:56.719046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:51.807846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970097+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:30:55.970097+0000","last_peered":"2024-09-11T00:30:55.970097+0000","last_clean":"2024-09-11T00:30:55.970097+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:30:55.970097+0000","last_undegraded":"2024-09-11T00:30:55.970097+0000","last_fullsized":"2024-09-11T00:30:55.970097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998047+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:30:55.998047+0000","last_peered":"2024-09-11T00:30:55.998047+0000","last_clean":"2024-09-11T00:30:55.998047+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:30:55.998047+0000","last_undegraded":"2024-09-11T00:30:55.998047+0000","last_fullsized":"2024-09-11T00:30:55.998047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000942+0000","last_change":"2024-09-11T00:30:53.984316+0000","last_active":"2024-09-11T00:30:56.000942+0000","last_peered":"2024-09-11T00:30:56.000942+0000","last_clean":"2024-09-11T00:30:56.000942+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:30:56.000942+0000","last_undegraded":"2024-09-11T00:30:56.000942+0000","last_fullsized":"2024-09-11T00:30:56.000942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:14.425265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968662+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:30:55.968662+0000","last_peered":"2024-09-11T00:30:55.968662+0000","last_clean":"2024-09-11T00:30:55.968662+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:30:55.968662+0000","last_undegraded":"2024-09-11T00:30:55.968662+0000","last_fullsized":"2024-09-11T00:30:55.968662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998255+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:30:55.998255+0000","last_peered":"2024-09-11T00:30:55.998255+0000","last_clean":"2024-09-11T00:30:55.998255+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:30:55.998255+0000","last_undegraded":"2024-09-11T00:30:55.998255+0000","last_fullsized":"2024-09-11T00:30:55.998255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998074+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:30:55.998074+0000","last_peered":"2024-09-11T00:30:55.998074+0000","last_clean":"2024-09-11T00:30:55.998074+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:30:55.998074+0000","last_undegraded":"2024-09-11T00:30:55.998074+0000","last_fullsized":"2024-09-11T00:30:55.998074+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172549+0000","last_change":"2024-09-11T00:30:53.984940+0000","last_active":"2024-09-11T00:30:56.172549+0000","last_peered":"2024-09-11T00:30:56.172549+0000","last_clean":"2024-09-11T00:30:56.172549+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:30:56.172549+0000","last_undegraded":"2024-09-11T00:30:56.172549+0000","last_fullsized":"2024-09-11T00:30:56.172549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:19.647165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719205+0000","last_change":"2024-09-11T00:30:54.993570+0000","last_active":"2024-09-11T00:30:56.719205+0000","last_peered":"2024-09-11T00:30:56.719205+0000","last_clean":"2024-09-11T00:30:56.719205+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:30:56.719205+0000","last_undegraded":"2024-09-11T00:30:56.719205+0000","last_fullsized":"2024-09-11T00:30:56.719205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:45.211259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333273+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:30:56.333273+0000","last_peered":"2024-09-11T00:30:56.333273+0000","last_clean":"2024-09-11T00:30:56.333273+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:30:56.333273+0000","last_undegraded":"2024-09-11T00:30:56.333273+0000","last_fullsized":"2024-09-11T00:30:56.333273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719392+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:30:56.719392+0000","last_peered":"2024-09-11T00:30:56.719392+0000","last_clean":"2024-09-11T00:30:56.719392+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:30:56.719392+0000","last_undegraded":"2024-09-11T00:30:56.719392+0000","last_fullsized":"2024-09-11T00:30:56.719392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172696+0000","last_change":"2024-09-11T00:30:53.974699+0000","last_active":"2024-09-11T00:30:56.172696+0000","last_peered":"2024-09-11T00:30:56.172696+0000","last_clean":"2024-09-11T00:30:56.172696+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:30:56.172696+0000","last_undegraded":"2024-09-11T00:30:56.172696+0000","last_fullsized":"2024-09-11T00:30:56.172696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:28.792487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172498+0000","last_change":"2024-09-11T00:30:54.984890+0000","last_active":"2024-09-11T00:30:56.172498+0000","last_peered":"2024-09-11T00:30:56.172498+0000","last_clean":"2024-09-11T00:30:56.172498+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:30:56.172498+0000","last_undegraded":"2024-09-11T00:30:56.172498+0000","last_fullsized":"2024-09-11T00:30:56.172498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:21.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333166+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:30:56.333166+0000","last_peered":"2024-09-11T00:30:56.333166+0000","last_clean":"2024-09-11T00:30:56.333166+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:30:56.333166+0000","last_undegraded":"2024-09-11T00:30:56.333166+0000","last_fullsized":"2024-09-11T00:30:56.333166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968144+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:30:55.968144+0000","last_peered":"2024-09-11T00:30:55.968144+0000","last_clean":"2024-09-11T00:30:55.968144+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:30:55.968144+0000","last_undegraded":"2024-09-11T00:30:55.968144+0000","last_fullsized":"2024-09-11T00:30:55.968144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172339+0000","last_change":"2024-09-11T00:30:53.977743+0000","last_active":"2024-09-11T00:30:56.172339+0000","last_peered":"2024-09-11T00:30:56.172339+0000","last_clean":"2024-09-11T00:30:56.172339+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:30:56.172339+0000","last_undegraded":"2024-09-11T00:30:56.172339+0000","last_fullsized":"2024-09-11T00:30:56.172339+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:20.618615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997709+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:30:55.997709+0000","last_peered":"2024-09-11T00:30:55.997709+0000","last_clean":"2024-09-11T00:30:55.997709+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:30:55.997709+0000","last_undegraded":"2024-09-11T00:30:55.997709+0000","last_fullsized":"2024-09-11T00:30:55.997709+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967362+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:30:55.967362+0000","last_peered":"2024-09-11T00:30:55.967362+0000","last_clean":"2024-09-11T00:30:55.967362+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:30:55.967362+0000","last_undegraded":"2024-09-11T00:30:55.967362+0000","last_fullsized":"2024-09-11T00:30:55.967362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967452+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:30:55.967452+0000","last_peered":"2024-09-11T00:30:55.967452+0000","last_clean":"2024-09-11T00:30:55.967452+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:30:55.967452+0000","last_undegraded":"2024-09-11T00:30:55.967452+0000","last_fullsized":"2024-09-11T00:30:55.967452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967914+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:30:55.967914+0000","last_peered":"2024-09-11T00:30:55.967914+0000","last_clean":"2024-09-11T00:30:55.967914+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:30:55.967914+0000","last_undegraded":"2024-09-11T00:30:55.967914+0000","last_fullsized":"2024-09-11T00:30:55.967914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001419+0000","last_change":"2024-09-11T00:30:54.997397+0000","last_active":"2024-09-11T00:30:56.001419+0000","last_peered":"2024-09-11T00:30:56.001419+0000","last_clean":"2024-09-11T00:30:56.001419+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:30:56.001419+0000","last_undegraded":"2024-09-11T00:30:56.001419+0000","last_fullsized":"2024-09-11T00:30:56.001419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:53.322576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001116+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:30:56.001116+0000","last_peered":"2024-09-11T00:30:56.001116+0000","last_clean":"2024-09-11T00:30:56.001116+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:30:56.001116+0000","last_undegraded":"2024-09-11T00:30:56.001116+0000","last_fullsized":"2024-09-11T00:30:56.001116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970056+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:30:55.970056+0000","last_peered":"2024-09-11T00:30:55.970056+0000","last_clean":"2024-09-11T00:30:55.970056+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:30:55.970056+0000","last_undegraded":"2024-09-11T00:30:55.970056+0000","last_fullsized":"2024-09-11T00:30:55.970056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332817+0000","last_change":"2024-09-11T00:30:53.986858+0000","last_active":"2024-09-11T00:30:56.332817+0000","last_peered":"2024-09-11T00:30:56.332817+0000","last_clean":"2024-09-11T00:30:56.332817+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:30:56.332817+0000","last_undegraded":"2024-09-11T00:30:56.332817+0000","last_fullsized":"2024-09-11T00:30:56.332817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:33.626193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967911+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:30:55.967911+0000","last_peered":"2024-09-11T00:30:55.967911+0000","last_clean":"2024-09-11T00:30:55.967911+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:30:55.967911+0000","last_undegraded":"2024-09-11T00:30:55.967911+0000","last_fullsized":"2024-09-11T00:30:55.967911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968190+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:30:55.968190+0000","last_peered":"2024-09-11T00:30:55.968190+0000","last_clean":"2024-09-11T00:30:55.968190+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:30:55.968190+0000","last_undegraded":"2024-09-11T00:30:55.968190+0000","last_fullsized":"2024-09-11T00:30:55.968190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968850+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:30:55.968850+0000","last_peered":"2024-09-11T00:30:55.968850+0000","last_clean":"2024-09-11T00:30:55.968850+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:30:55.968850+0000","last_undegraded":"2024-09-11T00:30:55.968850+0000","last_fullsized":"2024-09-11T00:30:55.968850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001447+0000","last_change":"2024-09-11T00:30:53.990765+0000","last_active":"2024-09-11T00:30:56.001447+0000","last_peered":"2024-09-11T00:30:56.001447+0000","last_clean":"2024-09-11T00:30:56.001447+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:30:56.001447+0000","last_undegraded":"2024-09-11T00:30:56.001447+0000","last_fullsized":"2024-09-11T00:30:56.001447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:33.720727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719722+0000","last_change":"2024-09-11T00:30:54.984949+0000","last_active":"2024-09-11T00:30:56.719722+0000","last_peered":"2024-09-11T00:30:56.719722+0000","last_clean":"2024-09-11T00:30:56.719722+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:30:56.719722+0000","last_undegraded":"2024-09-11T00:30:56.719722+0000","last_fullsized":"2024-09-11T00:30:56.719722+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:54.346937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967033+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:30:55.967033+0000","last_peered":"2024-09-11T00:30:55.967033+0000","last_clean":"2024-09-11T00:30:55.967033+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:30:55.967033+0000","last_undegraded":"2024-09-11T00:30:55.967033+0000","last_fullsized":"2024-09-11T00:30:55.967033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332357+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:30:56.332357+0000","last_peered":"2024-09-11T00:30:56.332357+0000","last_clean":"2024-09-11T00:30:56.332357+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:30:56.332357+0000","last_undegraded":"2024-09-11T00:30:56.332357+0000","last_fullsized":"2024-09-11T00:30:56.332357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172821+0000","last_change":"2024-09-11T00:30:53.986940+0000","last_active":"2024-09-11T00:30:56.172821+0000","last_peered":"2024-09-11T00:30:56.172821+0000","last_clean":"2024-09-11T00:30:56.172821+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:30:56.172821+0000","last_undegraded":"2024-09-11T00:30:56.172821+0000","last_fullsized":"2024-09-11T00:30:56.172821+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:11.942187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968462+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:30:55.968462+0000","last_peered":"2024-09-11T00:30:55.968462+0000","last_clean":"2024-09-11T00:30:55.968462+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:30:55.968462+0000","last_undegraded":"2024-09-11T00:30:55.968462+0000","last_fullsized":"2024-09-11T00:30:55.968462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968110+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:30:55.968110+0000","last_peered":"2024-09-11T00:30:55.968110+0000","last_clean":"2024-09-11T00:30:55.968110+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:30:55.968110+0000","last_undegraded":"2024-09-11T00:30:55.968110+0000","last_fullsized":"2024-09-11T00:30:55.968110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172483+0000","last_change":"2024-09-11T00:30:53.988539+0000","last_active":"2024-09-11T00:30:56.172483+0000","last_peered":"2024-09-11T00:30:56.172483+0000","last_clean":"2024-09-11T00:30:56.172483+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:30:56.172483+0000","last_undegraded":"2024-09-11T00:30:56.172483+0000","last_fullsized":"2024-09-11T00:30:56.172483+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:19.083664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966824+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:30:55.966824+0000","last_peered":"2024-09-11T00:30:55.966824+0000","last_clean":"2024-09-11T00:30:55.966824+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:30:55.966824+0000","last_undegraded":"2024-09-11T00:30:55.966824+0000","last_fullsized":"2024-09-11T00:30:55.966824+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970081+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:30:55.970081+0000","last_peered":"2024-09-11T00:30:55.970081+0000","last_clean":"2024-09-11T00:30:55.970081+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:30:55.970081+0000","last_undegraded":"2024-09-11T00:30:55.970081+0000","last_fullsized":"2024-09-11T00:30:55.970081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970376+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:30:55.970376+0000","last_peered":"2024-09-11T00:30:55.970376+0000","last_clean":"2024-09-11T00:30:55.970376+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:30:55.970376+0000","last_undegraded":"2024-09-11T00:30:55.970376+0000","last_fullsized":"2024-09-11T00:30:55.970376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968131+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:30:55.968131+0000","last_peered":"2024-09-11T00:30:55.968131+0000","last_clean":"2024-09-11T00:30:55.968131+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:30:55.968131+0000","last_undegraded":"2024-09-11T00:30:55.968131+0000","last_fullsized":"2024-09-11T00:30:55.968131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332511+0000","last_change":"2024-09-11T00:30:54.993118+0000","last_active":"2024-09-11T00:30:56.332511+0000","last_peered":"2024-09-11T00:30:56.332511+0000","last_clean":"2024-09-11T00:30:56.332511+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:30:56.332511+0000","last_undegraded":"2024-09-11T00:30:56.332511+0000","last_fullsized":"2024-09-11T00:30:56.332511+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:07:41.148370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971378+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:30:55.971378+0000","last_peered":"2024-09-11T00:30:55.971378+0000","last_clean":"2024-09-11T00:30:55.971378+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:30:55.971378+0000","last_undegraded":"2024-09-11T00:30:55.971378+0000","last_fullsized":"2024-09-11T00:30:55.971378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968486+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:30:55.968486+0000","last_peered":"2024-09-11T00:30:55.968486+0000","last_clean":"2024-09-11T00:30:55.968486+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:30:55.968486+0000","last_undegraded":"2024-09-11T00:30:55.968486+0000","last_fullsized":"2024-09-11T00:30:55.968486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970903+0000","last_change":"2024-09-11T00:30:53.986843+0000","last_active":"2024-09-11T00:30:55.970903+0000","last_peered":"2024-09-11T00:30:55.970903+0000","last_clean":"2024-09-11T00:30:55.970903+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:30:55.970903+0000","last_undegraded":"2024-09-11T00:30:55.970903+0000","last_fullsized":"2024-09-11T00:30:55.970903+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:44.684154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969549+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:30:55.969549+0000","last_peered":"2024-09-11T00:30:55.969549+0000","last_clean":"2024-09-11T00:30:55.969549+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:30:55.969549+0000","last_undegraded":"2024-09-11T00:30:55.969549+0000","last_fullsized":"2024-09-11T00:30:55.969549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172919+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:30:56.172919+0000","last_peered":"2024-09-11T00:30:56.172919+0000","last_clean":"2024-09-11T00:30:56.172919+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:30:56.172919+0000","last_undegraded":"2024-09-11T00:30:56.172919+0000","last_fullsized":"2024-09-11T00:30:56.172919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997652+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:30:55.997652+0000","last_peered":"2024-09-11T00:30:55.997652+0000","last_clean":"2024-09-11T00:30:55.997652+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:30:55.997652+0000","last_undegraded":"2024-09-11T00:30:55.997652+0000","last_fullsized":"2024-09-11T00:30:55.997652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969390+0000","last_change":"2024-09-11T00:30:53.986896+0000","last_active":"2024-09-11T00:30:55.969390+0000","last_peered":"2024-09-11T00:30:55.969390+0000","last_clean":"2024-09-11T00:30:55.969390+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:30:55.969390+0000","last_undegraded":"2024-09-11T00:30:55.969390+0000","last_fullsized":"2024-09-11T00:30:55.969390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:06.590278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001631+0000","last_change":"2024-09-11T00:30:55.001370+0000","last_active":"2024-09-11T00:30:56.001631+0000","last_peered":"2024-09-11T00:30:56.001631+0000","last_clean":"2024-09-11T00:30:56.001631+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:30:56.001631+0000","last_undegraded":"2024-09-11T00:30:56.001631+0000","last_fullsized":"2024-09-11T00:30:56.001631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:56.592767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333262+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:30:56.333262+0000","last_peered":"2024-09-11T00:30:56.333262+0000","last_clean":"2024-09-11T00:30:56.333262+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:30:56.333262+0000","last_undegraded":"2024-09-11T00:30:56.333262+0000","last_fullsized":"2024-09-11T00:30:56.333262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967594+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:30:55.967594+0000","last_peered":"2024-09-11T00:30:55.967594+0000","last_clean":"2024-09-11T00:30:55.967594+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:30:55.967594+0000","last_undegraded":"2024-09-11T00:30:55.967594+0000","last_fullsized":"2024-09-11T00:30:55.967594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971192+0000","last_change":"2024-09-11T00:30:53.978480+0000","last_active":"2024-09-11T00:30:55.971192+0000","last_peered":"2024-09-11T00:30:55.971192+0000","last_clean":"2024-09-11T00:30:55.971192+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:30:55.971192+0000","last_undegraded":"2024-09-11T00:30:55.971192+0000","last_fullsized":"2024-09-11T00:30:55.971192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:58.900565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001528+0000","last_change":"2024-09-11T00:30:54.997641+0000","last_active":"2024-09-11T00:30:56.001528+0000","last_peered":"2024-09-11T00:30:56.001528+0000","last_clean":"2024-09-11T00:30:56.001528+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:30:56.001528+0000","last_undegraded":"2024-09-11T00:30:56.001528+0000","last_fullsized":"2024-09-11T00:30:56.001528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:18.437490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997311+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:30:55.997311+0000","last_peered":"2024-09-11T00:30:55.997311+0000","last_clean":"2024-09-11T00:30:55.997311+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:30:55.997311+0000","last_undegraded":"2024-09-11T00:30:55.997311+0000","last_fullsized":"2024-09-11T00:30:55.997311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971163+0000","last_change":"2024-09-11T00:30:53.979031+0000","last_active":"2024-09-11T00:30:55.971163+0000","last_peered":"2024-09-11T00:30:55.971163+0000","last_clean":"2024-09-11T00:30:55.971163+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:30:55.971163+0000","last_undegraded":"2024-09-11T00:30:55.971163+0000","last_fullsized":"2024-09-11T00:30:55.971163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:36.604377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967699+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:30:55.967699+0000","last_peered":"2024-09-11T00:30:55.967699+0000","last_clean":"2024-09-11T00:30:55.967699+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:30:55.967699+0000","last_undegraded":"2024-09-11T00:30:55.967699+0000","last_fullsized":"2024-09-11T00:30:55.967699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:39.629122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968089+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:30:55.968089+0000","last_peered":"2024-09-11T00:30:55.968089+0000","last_clean":"2024-09-11T00:30:55.968089+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:30:55.968089+0000","last_undegraded":"2024-09-11T00:30:55.968089+0000","last_fullsized":"2024-09-11T00:30:55.968089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966595+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:30:55.966595+0000","last_peered":"2024-09-11T00:30:55.966595+0000","last_clean":"2024-09-11T00:30:55.966595+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:30:55.966595+0000","last_undegraded":"2024-09-11T00:30:55.966595+0000","last_fullsized":"2024-09-11T00:30:55.966595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"26'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998301+0000","last_change":"2024-09-11T00:30:53.989299+0000","last_active":"2024-09-11T00:30:55.998301+0000","last_peered":"2024-09-11T00:30:55.998301+0000","last_clean":"2024-09-11T00:30:55.998301+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:30:55.998301+0000","last_undegraded":"2024-09-11T00:30:55.998301+0000","last_fullsized":"2024-09-11T00:30:55.998301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:34:49.298915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332781+0000","last_change":"2024-09-11T00:30:54.990026+0000","last_active":"2024-09-11T00:30:56.332781+0000","last_peered":"2024-09-11T00:30:56.332781+0000","last_clean":"2024-09-11T00:30:56.332781+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:30:56.332781+0000","last_undegraded":"2024-09-11T00:30:56.332781+0000","last_fullsized":"2024-09-11T00:30:56.332781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:02:10.800530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969545+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:30:55.969545+0000","last_peered":"2024-09-11T00:30:55.969545+0000","last_clean":"2024-09-11T00:30:55.969545+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:30:55.969545+0000","last_undegraded":"2024-09-11T00:30:55.969545+0000","last_fullsized":"2024-09-11T00:30:55.969545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967491+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:30:55.967491+0000","last_peered":"2024-09-11T00:30:55.967491+0000","last_clean":"2024-09-11T00:30:55.967491+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:30:55.967491+0000","last_undegraded":"2024-09-11T00:30:55.967491+0000","last_fullsized":"2024-09-11T00:30:55.967491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719269+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:30:56.719269+0000","last_peered":"2024-09-11T00:30:56.719269+0000","last_clean":"2024-09-11T00:30:56.719269+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:30:56.719269+0000","last_undegraded":"2024-09-11T00:30:56.719269+0000","last_fullsized":"2024-09-11T00:30:56.719269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968267+0000","last_change":"2024-09-11T00:30:53.982142+0000","last_active":"2024-09-11T00:30:55.968267+0000","last_peered":"2024-09-11T00:30:55.968267+0000","last_clean":"2024-09-11T00:30:55.968267+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:30:55.968267+0000","last_undegraded":"2024-09-11T00:30:55.968267+0000","last_fullsized":"2024-09-11T00:30:55.968267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:41.558944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998013+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:30:55.998013+0000","last_peered":"2024-09-11T00:30:55.998013+0000","last_clean":"2024-09-11T00:30:55.998013+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:30:55.998013+0000","last_undegraded":"2024-09-11T00:30:55.998013+0000","last_fullsized":"2024-09-11T00:30:55.998013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968362+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:30:55.968362+0000","last_peered":"2024-09-11T00:30:55.968362+0000","last_clean":"2024-09-11T00:30:55.968362+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:30:55.968362+0000","last_undegraded":"2024-09-11T00:30:55.968362+0000","last_fullsized":"2024-09-11T00:30:55.968362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971436+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:30:55.971436+0000","last_peered":"2024-09-11T00:30:55.971436+0000","last_clean":"2024-09-11T00:30:55.971436+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:30:55.971436+0000","last_undegraded":"2024-09-11T00:30:55.971436+0000","last_fullsized":"2024-09-11T00:30:55.971436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172455+0000","last_change":"2024-09-11T00:30:53.989292+0000","last_active":"2024-09-11T00:30:56.172455+0000","last_peered":"2024-09-11T00:30:56.172455+0000","last_clean":"2024-09-11T00:30:56.172455+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:30:56.172455+0000","last_undegraded":"2024-09-11T00:30:56.172455+0000","last_fullsized":"2024-09-11T00:30:56.172455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:44.071338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968473+0000","last_change":"2024-09-11T00:30:54.981630+0000","last_active":"2024-09-11T00:30:55.968473+0000","last_peered":"2024-09-11T00:30:55.968473+0000","last_clean":"2024-09-11T00:30:55.968473+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:30:55.968473+0000","last_undegraded":"2024-09-11T00:30:55.968473+0000","last_fullsized":"2024-09-11T00:30:55.968473+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:12.832327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333061+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:30:56.333061+0000","last_peered":"2024-09-11T00:30:56.333061+0000","last_clean":"2024-09-11T00:30:56.333061+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:30:56.333061+0000","last_undegraded":"2024-09-11T00:30:56.333061+0000","last_fullsized":"2024-09-11T00:30:56.333061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997856+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:30:55.997856+0000","last_peered":"2024-09-11T00:30:55.997856+0000","last_clean":"2024-09-11T00:30:55.997856+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:30:55.997856+0000","last_undegraded":"2024-09-11T00:30:55.997856+0000","last_fullsized":"2024-09-11T00:30:55.997856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997848+0000","last_change":"2024-09-11T00:30:53.984013+0000","last_active":"2024-09-11T00:30:55.997848+0000","last_peered":"2024-09-11T00:30:55.997848+0000","last_clean":"2024-09-11T00:30:55.997848+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:30:55.997848+0000","last_undegraded":"2024-09-11T00:30:55.997848+0000","last_fullsized":"2024-09-11T00:30:55.997848+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:49.963623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719176+0000","last_change":"2024-09-11T00:30:54.996860+0000","last_active":"2024-09-11T00:30:56.719176+0000","last_peered":"2024-09-11T00:30:56.719176+0000","last_clean":"2024-09-11T00:30:56.719176+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:30:56.719176+0000","last_undegraded":"2024-09-11T00:30:56.719176+0000","last_fullsized":"2024-09-11T00:30:56.719176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:43.516748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333114+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:30:56.333114+0000","last_peered":"2024-09-11T00:30:56.333114+0000","last_clean":"2024-09-11T00:30:56.333114+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:30:56.333114+0000","last_undegraded":"2024-09-11T00:30:56.333114+0000","last_fullsized":"2024-09-11T00:30:56.333114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970314+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:30:55.970314+0000","last_peered":"2024-09-11T00:30:55.970314+0000","last_clean":"2024-09-11T00:30:55.970314+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:30:55.970314+0000","last_undegraded":"2024-09-11T00:30:55.970314+0000","last_fullsized":"2024-09-11T00:30:55.970314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966317+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:30:55.966317+0000","last_peered":"2024-09-11T00:30:55.966317+0000","last_clean":"2024-09-11T00:30:55.966317+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:30:55.966317+0000","last_undegraded":"2024-09-11T00:30:55.966317+0000","last_fullsized":"2024-09-11T00:30:55.966317+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:19.891052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967870+0000","last_change":"2024-09-11T00:30:54.993736+0000","last_active":"2024-09-11T00:30:55.967870+0000","last_peered":"2024-09-11T00:30:55.967870+0000","last_clean":"2024-09-11T00:30:55.967870+0000","last_became_active":"2024-09-11T00:30:54.993470+0000","last_became_peered":"2024-09-11T00:30:54.993470+0000","last_unstale":"2024-09-11T00:30:55.967870+0000","last_undegraded":"2024-09-11T00:30:55.967870+0000","last_fullsized":"2024-09-11T00:30:55.967870+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:24.712448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173170+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:30:56.173170+0000","last_peered":"2024-09-11T00:30:56.173170+0000","last_clean":"2024-09-11T00:30:56.173170+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:30:56.173170+0000","last_undegraded":"2024-09-11T00:30:56.173170+0000","last_fullsized":"2024-09-11T00:30:56.173170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333218+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:30:56.333218+0000","last_peered":"2024-09-11T00:30:56.333218+0000","last_clean":"2024-09-11T00:30:56.333218+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:30:56.333218+0000","last_undegraded":"2024-09-11T00:30:56.333218+0000","last_fullsized":"2024-09-11T00:30:56.333218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719247+0000","last_change":"2024-09-11T00:30:53.983352+0000","last_active":"2024-09-11T00:30:56.719247+0000","last_peered":"2024-09-11T00:30:56.719247+0000","last_clean":"2024-09-11T00:30:56.719247+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:30:56.719247+0000","last_undegraded":"2024-09-11T00:30:56.719247+0000","last_fullsized":"2024-09-11T00:30:56.719247+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:43.826114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970030+0000","last_change":"2024-09-11T00:30:54.987533+0000","last_active":"2024-09-11T00:30:55.970030+0000","last_peered":"2024-09-11T00:30:55.970030+0000","last_clean":"2024-09-11T00:30:55.970030+0000","last_became_active":"2024-09-11T00:30:54.987253+0000","last_became_peered":"2024-09-11T00:30:54.987253+0000","last_unstale":"2024-09-11T00:30:55.970030+0000","last_undegraded":"2024-09-11T00:30:55.970030+0000","last_fullsized":"2024-09-11T00:30:55.970030+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.923656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966934+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:30:55.966934+0000","last_peered":"2024-09-11T00:30:55.966934+0000","last_clean":"2024-09-11T00:30:55.966934+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:30:55.966934+0000","last_undegraded":"2024-09-11T00:30:55.966934+0000","last_fullsized":"2024-09-11T00:30:55.966934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968528+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:30:55.968528+0000","last_peered":"2024-09-11T00:30:55.968528+0000","last_clean":"2024-09-11T00:30:55.968528+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:30:55.968528+0000","last_undegraded":"2024-09-11T00:30:55.968528+0000","last_fullsized":"2024-09-11T00:30:55.968528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997651+0000","last_change":"2024-09-11T00:30:53.980703+0000","last_active":"2024-09-11T00:30:55.997651+0000","last_peered":"2024-09-11T00:30:55.997651+0000","last_clean":"2024-09-11T00:30:55.997651+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:30:55.997651+0000","last_undegraded":"2024-09-11T00:30:55.997651+0000","last_fullsized":"2024-09-11T00:30:55.997651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:51.050512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967820+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:30:55.967820+0000","last_peered":"2024-09-11T00:30:55.967820+0000","last_clean":"2024-09-11T00:30:55.967820+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:30:55.967820+0000","last_undegraded":"2024-09-11T00:30:55.967820+0000","last_fullsized":"2024-09-11T00:30:55.967820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719139+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:30:56.719139+0000","last_peered":"2024-09-11T00:30:56.719139+0000","last_clean":"2024-09-11T00:30:56.719139+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:30:56.719139+0000","last_undegraded":"2024-09-11T00:30:56.719139+0000","last_fullsized":"2024-09-11T00:30:56.719139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333070+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:30:56.333070+0000","last_peered":"2024-09-11T00:30:56.333070+0000","last_clean":"2024-09-11T00:30:56.333070+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:30:56.333070+0000","last_undegraded":"2024-09-11T00:30:56.333070+0000","last_fullsized":"2024-09-11T00:30:56.333070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997937+0000","last_change":"2024-09-11T00:30:53.987829+0000","last_active":"2024-09-11T00:30:55.997937+0000","last_peered":"2024-09-11T00:30:55.997937+0000","last_clean":"2024-09-11T00:30:55.997937+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:30:55.997937+0000","last_undegraded":"2024-09-11T00:30:55.997937+0000","last_fullsized":"2024-09-11T00:30:55.997937+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:57:43.792089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967657+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:30:55.967657+0000","last_peered":"2024-09-11T00:30:55.967657+0000","last_clean":"2024-09-11T00:30:55.967657+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:30:55.967657+0000","last_undegraded":"2024-09-11T00:30:55.967657+0000","last_fullsized":"2024-09-11T00:30:55.967657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967382+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:30:55.967382+0000","last_peered":"2024-09-11T00:30:55.967382+0000","last_clean":"2024-09-11T00:30:55.967382+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:30:55.967382+0000","last_undegraded":"2024-09-11T00:30:55.967382+0000","last_fullsized":"2024-09-11T00:30:55.967382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967492+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:30:55.967492+0000","last_peered":"2024-09-11T00:30:55.967492+0000","last_clean":"2024-09-11T00:30:55.967492+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:30:55.967492+0000","last_undegraded":"2024-09-11T00:30:55.967492+0000","last_fullsized":"2024-09-11T00:30:55.967492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970951+0000","last_change":"2024-09-11T00:30:53.984057+0000","last_active":"2024-09-11T00:30:55.970951+0000","last_peered":"2024-09-11T00:30:55.970951+0000","last_clean":"2024-09-11T00:30:55.970951+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:30:55.970951+0000","last_undegraded":"2024-09-11T00:30:55.970951+0000","last_fullsized":"2024-09-11T00:30:55.970951+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:10.477916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969373+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:30:55.969373+0000","last_peered":"2024-09-11T00:30:55.969373+0000","last_clean":"2024-09-11T00:30:55.969373+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:30:55.969373+0000","last_undegraded":"2024-09-11T00:30:55.969373+0000","last_fullsized":"2024-09-11T00:30:55.969373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969960+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:30:55.969960+0000","last_peered":"2024-09-11T00:30:55.969960+0000","last_clean":"2024-09-11T00:30:55.969960+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:30:55.969960+0000","last_undegraded":"2024-09-11T00:30:55.969960+0000","last_fullsized":"2024-09-11T00:30:55.969960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001240+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:30:56.001240+0000","last_peered":"2024-09-11T00:30:56.001240+0000","last_clean":"2024-09-11T00:30:56.001240+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:30:56.001240+0000","last_undegraded":"2024-09-11T00:30:56.001240+0000","last_fullsized":"2024-09-11T00:30:56.001240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968074+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:30:55.968074+0000","last_peered":"2024-09-11T00:30:55.968074+0000","last_clean":"2024-09-11T00:30:55.968074+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:30:55.968074+0000","last_undegraded":"2024-09-11T00:30:55.968074+0000","last_fullsized":"2024-09-11T00:30:55.968074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969893+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:30:55.969893+0000","last_peered":"2024-09-11T00:30:55.969893+0000","last_clean":"2024-09-11T00:30:55.969893+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:30:55.969893+0000","last_undegraded":"2024-09-11T00:30:55.969893+0000","last_fullsized":"2024-09-11T00:30:55.969893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332978+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:30:56.332978+0000","last_peered":"2024-09-11T00:30:56.332978+0000","last_clean":"2024-09-11T00:30:56.332978+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:30:56.332978+0000","last_undegraded":"2024-09-11T00:30:56.332978+0000","last_fullsized":"2024-09-11T00:30:56.332978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:31:14.474646+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:31:14.474646+0000","last_peered":"2024-09-11T00:31:14.474646+0000","last_clean":"2024-09-11T00:31:14.474646+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:31:14.474646+0000","last_undegraded":"2024-09-11T00:31:14.474646+0000","last_fullsized":"2024-09-11T00:31:14.474646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969278+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:30:55.969278+0000","last_peered":"2024-09-11T00:30:55.969278+0000","last_clean":"2024-09-11T00:30:55.969278+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:30:55.969278+0000","last_undegraded":"2024-09-11T00:30:55.969278+0000","last_fullsized":"2024-09-11T00:30:55.969278+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969704+0000","last_change":"2024-09-11T00:30:54.986017+0000","last_active":"2024-09-11T00:30:55.969704+0000","last_peered":"2024-09-11T00:30:55.969704+0000","last_clean":"2024-09-11T00:30:55.969704+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:30:55.969704+0000","last_undegraded":"2024-09-11T00:30:55.969704+0000","last_fullsized":"2024-09-11T00:30:55.969704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:58:12.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967364+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:30:55.967364+0000","last_peered":"2024-09-11T00:30:55.967364+0000","last_clean":"2024-09-11T00:30:55.967364+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:30:55.967364+0000","last_undegraded":"2024-09-11T00:30:55.967364+0000","last_fullsized":"2024-09-11T00:30:55.967364+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001702+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:30:56.001702+0000","last_peered":"2024-09-11T00:30:56.001702+0000","last_clean":"2024-09-11T00:30:56.001702+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:30:56.001702+0000","last_undegraded":"2024-09-11T00:30:56.001702+0000","last_fullsized":"2024-09-11T00:30:56.001702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172588+0000","last_change":"2024-09-11T00:30:53.988462+0000","last_active":"2024-09-11T00:30:56.172588+0000","last_peered":"2024-09-11T00:30:56.172588+0000","last_clean":"2024-09-11T00:30:56.172588+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:30:56.172588+0000","last_undegraded":"2024-09-11T00:30:56.172588+0000","last_fullsized":"2024-09-11T00:30:56.172588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:22.385557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966521+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:30:55.966521+0000","last_peered":"2024-09-11T00:30:55.966521+0000","last_clean":"2024-09-11T00:30:55.966521+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:30:55.966521+0000","last_undegraded":"2024-09-11T00:30:55.966521+0000","last_fullsized":"2024-09-11T00:30:55.966521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998206+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:30:55.998206+0000","last_peered":"2024-09-11T00:30:55.998206+0000","last_clean":"2024-09-11T00:30:55.998206+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:30:55.998206+0000","last_undegraded":"2024-09-11T00:30:55.998206+0000","last_fullsized":"2024-09-11T00:30:55.998206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332956+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:30:56.332956+0000","last_peered":"2024-09-11T00:30:56.332956+0000","last_clean":"2024-09-11T00:30:56.332956+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:30:56.332956+0000","last_undegraded":"2024-09-11T00:30:56.332956+0000","last_fullsized":"2024-09-11T00:30:56.332956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000726+0000","last_change":"2024-09-11T00:30:53.986646+0000","last_active":"2024-09-11T00:30:56.000726+0000","last_peered":"2024-09-11T00:30:56.000726+0000","last_clean":"2024-09-11T00:30:56.000726+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:30:56.000726+0000","last_undegraded":"2024-09-11T00:30:56.000726+0000","last_fullsized":"2024-09-11T00:30:56.000726+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:27.447940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172237+0000","last_change":"2024-09-11T00:30:54.982486+0000","last_active":"2024-09-11T00:30:56.172237+0000","last_peered":"2024-09-11T00:30:56.172237+0000","last_clean":"2024-09-11T00:30:56.172237+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:30:56.172237+0000","last_undegraded":"2024-09-11T00:30:56.172237+0000","last_fullsized":"2024-09-11T00:30:56.172237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:41.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173110+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:30:56.173110+0000","last_peered":"2024-09-11T00:30:56.173110+0000","last_clean":"2024-09-11T00:30:56.173110+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:30:56.173110+0000","last_undegraded":"2024-09-11T00:30:56.173110+0000","last_fullsized":"2024-09-11T00:30:56.173110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719858+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:30:56.719858+0000","last_peered":"2024-09-11T00:30:56.719858+0000","last_clean":"2024-09-11T00:30:56.719858+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:30:56.719858+0000","last_undegraded":"2024-09-11T00:30:56.719858+0000","last_fullsized":"2024-09-11T00:30:56.719858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968520+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:30:55.968520+0000","last_peered":"2024-09-11T00:30:55.968520+0000","last_clean":"2024-09-11T00:30:55.968520+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:30:55.968520+0000","last_undegraded":"2024-09-11T00:30:55.968520+0000","last_fullsized":"2024-09-11T00:30:55.968520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173273+0000","last_change":"2024-09-11T00:30:54.985091+0000","last_active":"2024-09-11T00:30:56.173273+0000","last_peered":"2024-09-11T00:30:56.173273+0000","last_clean":"2024-09-11T00:30:56.173273+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:30:56.173273+0000","last_undegraded":"2024-09-11T00:30:56.173273+0000","last_fullsized":"2024-09-11T00:30:56.173273+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:33.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968811+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:30:55.968811+0000","last_peered":"2024-09-11T00:30:55.968811+0000","last_clean":"2024-09-11T00:30:55.968811+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:30:55.968811+0000","last_undegraded":"2024-09-11T00:30:55.968811+0000","last_fullsized":"2024-09-11T00:30:55.968811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966522+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:30:55.966522+0000","last_peered":"2024-09-11T00:30:55.966522+0000","last_clean":"2024-09-11T00:30:55.966522+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:30:55.966522+0000","last_undegraded":"2024-09-11T00:30:55.966522+0000","last_fullsized":"2024-09-11T00:30:55.966522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971377+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:30:55.971377+0000","last_peered":"2024-09-11T00:30:55.971377+0000","last_clean":"2024-09-11T00:30:55.971377+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:30:55.971377+0000","last_undegraded":"2024-09-11T00:30:55.971377+0000","last_fullsized":"2024-09-11T00:30:55.971377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:11.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967648+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:30:55.967648+0000","last_peered":"2024-09-11T00:30:55.967648+0000","last_clean":"2024-09-11T00:30:55.967648+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:30:55.967648+0000","last_undegraded":"2024-09-11T00:30:55.967648+0000","last_fullsized":"2024-09-11T00:30:55.967648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968196+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:30:55.968196+0000","last_peered":"2024-09-11T00:30:55.968196+0000","last_clean":"2024-09-11T00:30:55.968196+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:30:55.968196+0000","last_undegraded":"2024-09-11T00:30:55.968196+0000","last_fullsized":"2024-09-11T00:30:55.968196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969528+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:30:55.969528+0000","last_peered":"2024-09-11T00:30:55.969528+0000","last_clean":"2024-09-11T00:30:55.969528+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:30:55.969528+0000","last_undegraded":"2024-09-11T00:30:55.969528+0000","last_fullsized":"2024-09-11T00:30:55.969528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967369+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:30:55.967369+0000","last_peered":"2024-09-11T00:30:55.967369+0000","last_clean":"2024-09-11T00:30:55.967369+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:30:55.967369+0000","last_undegraded":"2024-09-11T00:30:55.967369+0000","last_fullsized":"2024-09-11T00:30:55.967369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969812+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:30:55.969812+0000","last_peered":"2024-09-11T00:30:55.969812+0000","last_clean":"2024-09-11T00:30:55.969812+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:30:55.969812+0000","last_undegraded":"2024-09-11T00:30:55.969812+0000","last_fullsized":"2024-09-11T00:30:55.969812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969509+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:30:55.969509+0000","last_peered":"2024-09-11T00:30:55.969509+0000","last_clean":"2024-09-11T00:30:55.969509+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:30:55.969509+0000","last_undegraded":"2024-09-11T00:30:55.969509+0000","last_fullsized":"2024-09-11T00:30:55.969509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173337+0000","last_change":"2024-09-11T00:30:54.998464+0000","last_active":"2024-09-11T00:30:56.173337+0000","last_peered":"2024-09-11T00:30:56.173337+0000","last_clean":"2024-09-11T00:30:56.173337+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:30:56.173337+0000","last_undegraded":"2024-09-11T00:30:56.173337+0000","last_fullsized":"2024-09-11T00:30:56.173337+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:56.469791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998082+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:30:55.998082+0000","last_peered":"2024-09-11T00:30:55.998082+0000","last_clean":"2024-09-11T00:30:55.998082+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:30:55.998082+0000","last_undegraded":"2024-09-11T00:30:55.998082+0000","last_fullsized":"2024-09-11T00:30:55.998082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719278+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:30:56.719278+0000","last_peered":"2024-09-11T00:30:56.719278+0000","last_clean":"2024-09-11T00:30:56.719278+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:30:56.719278+0000","last_undegraded":"2024-09-11T00:30:56.719278+0000","last_fullsized":"2024-09-11T00:30:56.719278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001549+0000","last_change":"2024-09-11T00:30:53.987612+0000","last_active":"2024-09-11T00:30:56.001549+0000","last_peered":"2024-09-11T00:30:56.001549+0000","last_clean":"2024-09-11T00:30:56.001549+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:30:56.001549+0000","last_undegraded":"2024-09-11T00:30:56.001549+0000","last_fullsized":"2024-09-11T00:30:56.001549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:16.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970084+0000","last_change":"2024-09-11T00:30:54.989925+0000","last_active":"2024-09-11T00:30:55.970084+0000","last_peered":"2024-09-11T00:30:55.970084+0000","last_clean":"2024-09-11T00:30:55.970084+0000","last_became_active":"2024-09-11T00:30:54.989401+0000","last_became_peered":"2024-09-11T00:30:54.989401+0000","last_unstale":"2024-09-11T00:30:55.970084+0000","last_undegraded":"2024-09-11T00:30:55.970084+0000","last_fullsized":"2024-09-11T00:30:55.970084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:14.267175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172747+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:30:56.172747+0000","last_peered":"2024-09-11T00:30:56.172747+0000","last_clean":"2024-09-11T00:30:56.172747+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:30:56.172747+0000","last_undegraded":"2024-09-11T00:30:56.172747+0000","last_fullsized":"2024-09-11T00:30:56.172747+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969929+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:30:55.969929+0000","last_peered":"2024-09-11T00:30:55.969929+0000","last_clean":"2024-09-11T00:30:55.969929+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:30:55.969929+0000","last_undegraded":"2024-09-11T00:30:55.969929+0000","last_fullsized":"2024-09-11T00:30:55.969929+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000911+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:30:56.000911+0000","last_peered":"2024-09-11T00:30:56.000911+0000","last_clean":"2024-09-11T00:30:56.000911+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:30:56.000911+0000","last_undegraded":"2024-09-11T00:30:56.000911+0000","last_fullsized":"2024-09-11T00:30:56.000911+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172912+0000","last_change":"2024-09-11T00:30:54.979133+0000","last_active":"2024-09-11T00:30:56.172912+0000","last_peered":"2024-09-11T00:30:56.172912+0000","last_clean":"2024-09-11T00:30:56.172912+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:30:56.172912+0000","last_undegraded":"2024-09-11T00:30:56.172912+0000","last_fullsized":"2024-09-11T00:30:56.172912+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:20.961678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967251+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:30:55.967251+0000","last_peered":"2024-09-11T00:30:55.967251+0000","last_clean":"2024-09-11T00:30:55.967251+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:30:55.967251+0000","last_undegraded":"2024-09-11T00:30:55.967251+0000","last_fullsized":"2024-09-11T00:30:55.967251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997885+0000","last_change":"2024-09-11T00:30:54.988345+0000","last_active":"2024-09-11T00:30:55.997885+0000","last_peered":"2024-09-11T00:30:55.997885+0000","last_clean":"2024-09-11T00:30:55.997885+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:30:55.997885+0000","last_undegraded":"2024-09-11T00:30:55.997885+0000","last_fullsized":"2024-09-11T00:30:55.997885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.112194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998078+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:30:55.998078+0000","last_peered":"2024-09-11T00:30:55.998078+0000","last_clean":"2024-09-11T00:30:55.998078+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:30:55.998078+0000","last_undegraded":"2024-09-11T00:30:55.998078+0000","last_fullsized":"2024-09-11T00:30:55.998078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332489+0000","last_change":"2024-09-11T00:30:54.981185+0000","last_active":"2024-09-11T00:30:56.332489+0000","last_peered":"2024-09-11T00:30:56.332489+0000","last_clean":"2024-09-11T00:30:56.332489+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:30:56.332489+0000","last_undegraded":"2024-09-11T00:30:56.332489+0000","last_fullsized":"2024-09-11T00:30:56.332489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:45.740467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971241+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:30:55.971241+0000","last_peered":"2024-09-11T00:30:55.971241+0000","last_clean":"2024-09-11T00:30:55.971241+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:30:55.971241+0000","last_undegraded":"2024-09-11T00:30:55.971241+0000","last_fullsized":"2024-09-11T00:30:55.971241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332647+0000","last_change":"2024-09-11T00:30:54.992763+0000","last_active":"2024-09-11T00:30:56.332647+0000","last_peered":"2024-09-11T00:30:56.332647+0000","last_clean":"2024-09-11T00:30:56.332647+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:30:56.332647+0000","last_undegraded":"2024-09-11T00:30:56.332647+0000","last_fullsized":"2024-09-11T00:30:56.332647+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:02.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971179+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:30:55.971179+0000","last_peered":"2024-09-11T00:30:55.971179+0000","last_clean":"2024-09-11T00:30:55.971179+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:30:55.971179+0000","last_undegraded":"2024-09-11T00:30:55.971179+0000","last_fullsized":"2024-09-11T00:30:55.971179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966440+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:30:55.966440+0000","last_peered":"2024-09-11T00:30:55.966440+0000","last_clean":"2024-09-11T00:30:55.966440+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:30:55.966440+0000","last_undegraded":"2024-09-11T00:30:55.966440+0000","last_fullsized":"2024-09-11T00:30:55.966440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173066+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:30:56.173066+0000","last_peered":"2024-09-11T00:30:56.173066+0000","last_clean":"2024-09-11T00:30:56.173066+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:30:56.173066+0000","last_undegraded":"2024-09-11T00:30:56.173066+0000","last_fullsized":"2024-09-11T00:30:56.173066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172366+0000","last_change":"2024-09-11T00:30:54.984447+0000","last_active":"2024-09-11T00:30:56.172366+0000","last_peered":"2024-09-11T00:30:56.172366+0000","last_clean":"2024-09-11T00:30:56.172366+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:30:56.172366+0000","last_undegraded":"2024-09-11T00:30:56.172366+0000","last_fullsized":"2024-09-11T00:30:56.172366+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:49.024022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998072+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:30:55.998072+0000","last_peered":"2024-09-11T00:30:55.998072+0000","last_clean":"2024-09-11T00:30:55.998072+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:30:55.998072+0000","last_undegraded":"2024-09-11T00:30:55.998072+0000","last_fullsized":"2024-09-11T00:30:55.998072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001507+0000","last_change":"2024-09-11T00:30:54.996091+0000","last_active":"2024-09-11T00:30:56.001507+0000","last_peered":"2024-09-11T00:30:56.001507+0000","last_clean":"2024-09-11T00:30:56.001507+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:30:56.001507+0000","last_undegraded":"2024-09-11T00:30:56.001507+0000","last_fullsized":"2024-09-11T00:30:56.001507+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:42.987336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966504+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:30:55.966504+0000","last_peered":"2024-09-11T00:30:55.966504+0000","last_clean":"2024-09-11T00:30:55.966504+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:30:55.966504+0000","last_undegraded":"2024-09-11T00:30:55.966504+0000","last_fullsized":"2024-09-11T00:30:55.966504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968347+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:30:55.968347+0000","last_peered":"2024-09-11T00:30:55.968347+0000","last_clean":"2024-09-11T00:30:55.968347+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:30:55.968347+0000","last_undegraded":"2024-09-11T00:30:55.968347+0000","last_fullsized":"2024-09-11T00:30:55.968347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971167+0000","last_change":"2024-09-11T00:30:54.981799+0000","last_active":"2024-09-11T00:30:55.971167+0000","last_peered":"2024-09-11T00:30:55.971167+0000","last_clean":"2024-09-11T00:30:55.971167+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:30:55.971167+0000","last_undegraded":"2024-09-11T00:30:55.971167+0000","last_fullsized":"2024-09-11T00:30:55.971167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:42.670629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719169+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:30:56.719169+0000","last_peered":"2024-09-11T00:30:56.719169+0000","last_clean":"2024-09-11T00:30:56.719169+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:30:56.719169+0000","last_undegraded":"2024-09-11T00:30:56.719169+0000","last_fullsized":"2024-09-11T00:30:56.719169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001453+0000","last_change":"2024-09-11T00:30:54.992697+0000","last_active":"2024-09-11T00:30:56.001453+0000","last_peered":"2024-09-11T00:30:56.001453+0000","last_clean":"2024-09-11T00:30:56.001453+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:30:56.001453+0000","last_undegraded":"2024-09-11T00:30:56.001453+0000","last_fullsized":"2024-09-11T00:30:56.001453+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:54:07.227169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969635+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:30:55.969635+0000","last_peered":"2024-09-11T00:30:55.969635+0000","last_clean":"2024-09-11T00:30:55.969635+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:55.969635+0000","last_undegraded":"2024-09-11T00:30:55.969635+0000","last_fullsized":"2024-09-11T00:30:55.969635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172413+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:30:56.172413+0000","last_peered":"2024-09-11T00:30:56.172413+0000","last_clean":"2024-09-11T00:30:56.172413+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:30:56.172413+0000","last_undegraded":"2024-09-11T00:30:56.172413+0000","last_fullsized":"2024-09-11T00:30:56.172413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332844+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:30:56.332844+0000","last_peered":"2024-09-11T00:30:56.332844+0000","last_clean":"2024-09-11T00:30:56.332844+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:30:56.332844+0000","last_undegraded":"2024-09-11T00:30:56.332844+0000","last_fullsized":"2024-09-11T00:30:56.332844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332895+0000","last_change":"2024-09-11T00:30:54.985827+0000","last_active":"2024-09-11T00:30:56.332895+0000","last_peered":"2024-09-11T00:30:56.332895+0000","last_clean":"2024-09-11T00:30:56.332895+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:30:56.332895+0000","last_undegraded":"2024-09-11T00:30:56.332895+0000","last_fullsized":"2024-09-11T00:30:56.332895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:32.027117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968441+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:30:55.968441+0000","last_peered":"2024-09-11T00:30:55.968441+0000","last_clean":"2024-09-11T00:30:55.968441+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:30:55.968441+0000","last_undegraded":"2024-09-11T00:30:55.968441+0000","last_fullsized":"2024-09-11T00:30:55.968441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172255+0000","last_change":"2024-09-11T00:30:54.989906+0000","last_active":"2024-09-11T00:30:56.172255+0000","last_peered":"2024-09-11T00:30:56.172255+0000","last_clean":"2024-09-11T00:30:56.172255+0000","last_became_active":"2024-09-11T00:30:54.989376+0000","last_became_peered":"2024-09-11T00:30:54.989376+0000","last_unstale":"2024-09-11T00:30:56.172255+0000","last_undegraded":"2024-09-11T00:30:56.172255+0000","last_fullsized":"2024-09-11T00:30:56.172255+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:18:00.138941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001665+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:30:56.001665+0000","last_peered":"2024-09-11T00:30:56.001665+0000","last_clean":"2024-09-11T00:30:56.001665+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:30:56.001665+0000","last_undegraded":"2024-09-11T00:30:56.001665+0000","last_fullsized":"2024-09-11T00:30:56.001665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968974+0000","last_change":"2024-09-11T00:30:54.993156+0000","last_active":"2024-09-11T00:30:55.968974+0000","last_peered":"2024-09-11T00:30:55.968974+0000","last_clean":"2024-09-11T00:30:55.968974+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:30:55.968974+0000","last_undegraded":"2024-09-11T00:30:55.968974+0000","last_fullsized":"2024-09-11T00:30:55.968974+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:25.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968016+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:30:55.968016+0000","last_peered":"2024-09-11T00:30:55.968016+0000","last_clean":"2024-09-11T00:30:55.968016+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:30:55.968016+0000","last_undegraded":"2024-09-11T00:30:55.968016+0000","last_fullsized":"2024-09-11T00:30:55.968016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970335+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:30:55.970335+0000","last_peered":"2024-09-11T00:30:55.970335+0000","last_clean":"2024-09-11T00:30:55.970335+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:30:55.970335+0000","last_undegraded":"2024-09-11T00:30:55.970335+0000","last_fullsized":"2024-09-11T00:30:55.970335+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966434+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:30:55.966434+0000","last_peered":"2024-09-11T00:30:55.966434+0000","last_clean":"2024-09-11T00:30:55.966434+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:30:55.966434+0000","last_undegraded":"2024-09-11T00:30:55.966434+0000","last_fullsized":"2024-09-11T00:30:55.966434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719699+0000","last_change":"2024-09-11T00:30:54.993461+0000","last_active":"2024-09-11T00:30:56.719699+0000","last_peered":"2024-09-11T00:30:56.719699+0000","last_clean":"2024-09-11T00:30:56.719699+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:30:56.719699+0000","last_undegraded":"2024-09-11T00:30:56.719699+0000","last_fullsized":"2024-09-11T00:30:56.719699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:07.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997281+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:30:55.997281+0000","last_peered":"2024-09-11T00:30:55.997281+0000","last_clean":"2024-09-11T00:30:55.997281+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:30:55.997281+0000","last_undegraded":"2024-09-11T00:30:55.997281+0000","last_fullsized":"2024-09-11T00:30:55.997281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719307+0000","last_change":"2024-09-11T00:30:54.986265+0000","last_active":"2024-09-11T00:30:56.719307+0000","last_peered":"2024-09-11T00:30:56.719307+0000","last_clean":"2024-09-11T00:30:56.719307+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:30:56.719307+0000","last_undegraded":"2024-09-11T00:30:56.719307+0000","last_fullsized":"2024-09-11T00:30:56.719307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:49.109328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969766+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:30:55.969766+0000","last_peered":"2024-09-11T00:30:55.969766+0000","last_clean":"2024-09-11T00:30:55.969766+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:30:55.969766+0000","last_undegraded":"2024-09-11T00:30:55.969766+0000","last_fullsized":"2024-09-11T00:30:55.969766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967854+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:30:55.967854+0000","last_peered":"2024-09-11T00:30:55.967854+0000","last_clean":"2024-09-11T00:30:55.967854+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:30:55.967854+0000","last_undegraded":"2024-09-11T00:30:55.967854+0000","last_fullsized":"2024-09-11T00:30:55.967854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971368+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:30:55.971368+0000","last_peered":"2024-09-11T00:30:55.971368+0000","last_clean":"2024-09-11T00:30:55.971368+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:30:55.971368+0000","last_undegraded":"2024-09-11T00:30:55.971368+0000","last_fullsized":"2024-09-11T00:30:55.971368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969452+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:30:55.969452+0000","last_peered":"2024-09-11T00:30:55.969452+0000","last_clean":"2024-09-11T00:30:55.969452+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:30:55.969452+0000","last_undegraded":"2024-09-11T00:30:55.969452+0000","last_fullsized":"2024-09-11T00:30:55.969452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332691+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:30:56.332691+0000","last_peered":"2024-09-11T00:30:56.332691+0000","last_clean":"2024-09-11T00:30:56.332691+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:30:56.332691+0000","last_undegraded":"2024-09-11T00:30:56.332691+0000","last_fullsized":"2024-09-11T00:30:56.332691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970850+0000","last_change":"2024-09-11T00:30:54.994988+0000","last_active":"2024-09-11T00:30:55.970850+0000","last_peered":"2024-09-11T00:30:55.970850+0000","last_clean":"2024-09-11T00:30:55.970850+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:30:55.970850+0000","last_undegraded":"2024-09-11T00:30:55.970850+0000","last_fullsized":"2024-09-11T00:30:55.970850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:31.973641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332924+0000","last_change":"2024-09-11T00:30:45.941635+0000","last_active":"2024-09-11T00:30:56.332924+0000","last_peered":"2024-09-11T00:30:56.332924+0000","last_clean":"2024-09-11T00:30:56.332924+0000","last_became_active":"2024-09-11T00:30:45.941357+0000","last_became_peered":"2024-09-11T00:30:45.941357+0000","last_unstale":"2024-09-11T00:30:56.332924+0000","last_undegraded":"2024-09-11T00:30:56.332924+0000","last_fullsized":"2024-09-11T00:30:56.332924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:12.016406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332421+0000","last_change":"2024-09-11T00:30:54.984767+0000","last_active":"2024-09-11T00:30:56.332421+0000","last_peered":"2024-09-11T00:30:56.332421+0000","last_clean":"2024-09-11T00:30:56.332421+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:30:56.332421+0000","last_undegraded":"2024-09-11T00:30:56.332421+0000","last_fullsized":"2024-09-11T00:30:56.332421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:44.833677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332955+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:30:56.332955+0000","last_peered":"2024-09-11T00:30:56.332955+0000","last_clean":"2024-09-11T00:30:56.332955+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:56.332955+0000","last_undegraded":"2024-09-11T00:30:56.332955+0000","last_fullsized":"2024-09-11T00:30:56.332955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332911+0000","last_change":"2024-09-11T00:30:54.986423+0000","last_active":"2024-09-11T00:30:56.332911+0000","last_peered":"2024-09-11T00:30:56.332911+0000","last_clean":"2024-09-11T00:30:56.332911+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:30:56.332911+0000","last_undegraded":"2024-09-11T00:30:56.332911+0000","last_fullsized":"2024-09-11T00:30:56.332911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:16:50.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719859+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:30:56.719859+0000","last_peered":"2024-09-11T00:30:56.719859+0000","last_clean":"2024-09-11T00:30:56.719859+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:30:56.719859+0000","last_undegraded":"2024-09-11T00:30:56.719859+0000","last_fullsized":"2024-09-11T00:30:56.719859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998062+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:30:55.998062+0000","last_peered":"2024-09-11T00:30:55.998062+0000","last_clean":"2024-09-11T00:30:55.998062+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:30:55.998062+0000","last_undegraded":"2024-09-11T00:30:55.998062+0000","last_fullsized":"2024-09-11T00:30:55.998062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997283+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:30:55.997283+0000","last_peered":"2024-09-11T00:30:55.997283+0000","last_clean":"2024-09-11T00:30:55.997283+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:30:55.997283+0000","last_undegraded":"2024-09-11T00:30:55.997283+0000","last_fullsized":"2024-09-11T00:30:55.997283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173138+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:30:56.173138+0000","last_peered":"2024-09-11T00:30:56.173138+0000","last_clean":"2024-09-11T00:30:56.173138+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:30:56.173138+0000","last_undegraded":"2024-09-11T00:30:56.173138+0000","last_fullsized":"2024-09-11T00:30:56.173138+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333239+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:30:56.333239+0000","last_peered":"2024-09-11T00:30:56.333239+0000","last_clean":"2024-09-11T00:30:56.333239+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:30:56.333239+0000","last_undegraded":"2024-09-11T00:30:56.333239+0000","last_fullsized":"2024-09-11T00:30:56.333239+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970256+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:30:55.970256+0000","last_peered":"2024-09-11T00:30:55.970256+0000","last_clean":"2024-09-11T00:30:55.970256+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:30:55.970256+0000","last_undegraded":"2024-09-11T00:30:55.970256+0000","last_fullsized":"2024-09-11T00:30:55.970256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172539+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:30:56.172539+0000","last_peered":"2024-09-11T00:30:56.172539+0000","last_clean":"2024-09-11T00:30:56.172539+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:30:56.172539+0000","last_undegraded":"2024-09-11T00:30:56.172539+0000","last_fullsized":"2024-09-11T00:30:56.172539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719531+0000","last_change":"2024-09-11T00:30:54.990631+0000","last_active":"2024-09-11T00:30:56.719531+0000","last_peered":"2024-09-11T00:30:56.719531+0000","last_clean":"2024-09-11T00:30:56.719531+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:30:56.719531+0000","last_undegraded":"2024-09-11T00:30:56.719531+0000","last_fullsized":"2024-09-11T00:30:56.719531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:02.422471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333220+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:30:56.333220+0000","last_peered":"2024-09-11T00:30:56.333220+0000","last_clean":"2024-09-11T00:30:56.333220+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:30:56.333220+0000","last_undegraded":"2024-09-11T00:30:56.333220+0000","last_fullsized":"2024-09-11T00:30:56.333220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:54.991047+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:41:48.229261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173027+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:30:56.173027+0000","last_peered":"2024-09-11T00:30:56.173027+0000","last_clean":"2024-09-11T00:30:56.173027+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:30:56.173027+0000","last_undegraded":"2024-09-11T00:30:56.173027+0000","last_fullsized":"2024-09-11T00:30:56.173027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997710+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:30:55.997710+0000","last_peered":"2024-09-11T00:30:55.997710+0000","last_clean":"2024-09-11T00:30:55.997710+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:30:55.997710+0000","last_undegraded":"2024-09-11T00:30:55.997710+0000","last_fullsized":"2024-09-11T00:30:55.997710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997737+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:30:55.997737+0000","last_peered":"2024-09-11T00:30:55.997737+0000","last_clean":"2024-09-11T00:30:55.997737+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:30:55.997737+0000","last_undegraded":"2024-09-11T00:30:55.997737+0000","last_fullsized":"2024-09-11T00:30:55.997737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172698+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:30:56.172698+0000","last_peered":"2024-09-11T00:30:56.172698+0000","last_clean":"2024-09-11T00:30:56.172698+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:30:56.172698+0000","last_undegraded":"2024-09-11T00:30:56.172698+0000","last_fullsized":"2024-09-11T00:30:56.172698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997914+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:30:55.997914+0000","last_peered":"2024-09-11T00:30:55.997914+0000","last_clean":"2024-09-11T00:30:55.997914+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:30:55.997914+0000","last_undegraded":"2024-09-11T00:30:55.997914+0000","last_fullsized":"2024-09-11T00:30:55.997914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172531+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:30:56.172531+0000","last_peered":"2024-09-11T00:30:56.172531+0000","last_clean":"2024-09-11T00:30:56.172531+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:30:56.172531+0000","last_undegraded":"2024-09-11T00:30:56.172531+0000","last_fullsized":"2024-09-11T00:30:56.172531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333081+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:30:56.333081+0000","last_peered":"2024-09-11T00:30:56.333081+0000","last_clean":"2024-09-11T00:30:56.333081+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:30:56.333081+0000","last_undegraded":"2024-09-11T00:30:56.333081+0000","last_fullsized":"2024-09-11T00:30:56.333081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509452,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509454,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509453,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509451,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":106706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509453,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":108978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:31:23.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:31:23.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.773+0000 7f6a2d400640 1 -- 172.21.15.31:0/2029367889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a28150020 msgr2=0x7f6a280ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:23.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.773+0000 7f6a2d400640 1 --2- 172.21.15.31:0/2029367889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a28150020 0x7f6a280ad4e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6a28153d20 tx=0x7f6a1802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.773+0000 7f6a2d400640 1 -- 172.21.15.31:0/2029367889 shutdown_connections 2024-09-11T00:31:23.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.773+0000 7f6a2d400640 1 --2- 172.21.15.31:0/2029367889 >> v2:172.21.15.204:3300/0 conn(0x7f6a280ada20 0x7f6a2806e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.773+0000 7f6a2d400640 1 --2- 172.21.15.31:0/2029367889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a28150020 0x7f6a280ad4e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.773+0000 7f6a2d400640 1 -- 172.21.15.31:0/2029367889 >> 172.21.15.31:0/2029367889 conn(0x7f6a28068f10 msgr2=0x7f6a280cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:23.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.773+0000 7f6a2d400640 1 -- 172.21.15.31:0/2029367889 shutdown_connections 2024-09-11T00:31:23.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.773+0000 7f6a2d400640 1 -- 172.21.15.31:0/2029367889 wait complete. 2024-09-11T00:31:23.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.774+0000 7f6a2d400640 1 Processor -- start 2024-09-11T00:31:23.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.774+0000 7f6a2d400640 1 -- start start 2024-09-11T00:31:23.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.775+0000 7f6a2d400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a280ada20 0x7f6a280c1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:23.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.775+0000 7f6a2d400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6a28150020 0x7f6a280bf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:23.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.775+0000 7f6a2d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a28156960 0x7f6a281eba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:23.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.775+0000 7f6a2d400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6a28157550 con 0x7f6a28156960 2024-09-11T00:31:23.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.775+0000 7f6a2d400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6a281573d0 con 0x7f6a280ada20 2024-09-11T00:31:23.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.775+0000 7f6a2d400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6a280cc220 con 0x7f6a28150020 2024-09-11T00:31:23.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a2ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a28156960 0x7f6a281eba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:23.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a2ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a28156960 0x7f6a281eba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47820/0 (socket says 172.21.15.31:47820) 2024-09-11T00:31:23.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a2ca00640 1 -- 172.21.15.31:0/3907671908 learned_addr learned my addr 172.21.15.31:0/3907671908 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a27e00640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a280ada20 0x7f6a280c1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a27400640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6a28150020 0x7f6a280bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a2ca00640 1 -- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6a28150020 msgr2=0x7f6a280bf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a2ca00640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6a28150020 0x7f6a280bf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a2ca00640 1 -- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a280ada20 msgr2=0x7f6a280c1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a2ca00640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a280ada20 0x7f6a280c1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a2ca00640 1 -- 172.21.15.31:0/3907671908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2814e430 con 0x7f6a28156960 2024-09-11T00:31:23.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a27400640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6a28150020 0x7f6a280bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:23.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.776+0000 7f6a2ca00640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a28156960 0x7f6a281eba60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000c7a0 tx=0x7f6a2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:23.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.777+0000 7f6a1fe00640 1 -- 172.21.15.31:0/3907671908 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20003b70 con 0x7f6a28156960 2024-09-11T00:31:23.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.777+0000 7f6a1fe00640 1 -- 172.21.15.31:0/3907671908 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a20003d30 con 0x7f6a28156960 2024-09-11T00:31:23.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.777+0000 7f6a2d400640 1 -- 172.21.15.31:0/3907671908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a280c0e70 con 0x7f6a28156960 2024-09-11T00:31:23.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.777+0000 7f6a1fe00640 1 -- 172.21.15.31:0/3907671908 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2000aab0 con 0x7f6a28156960 2024-09-11T00:31:23.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.777+0000 7f6a2d400640 1 -- 172.21.15.31:0/3907671908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a2814e150 con 0x7f6a28156960 2024-09-11T00:31:23.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.778+0000 7f6a2d400640 1 -- 172.21.15.31:0/3907671908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a280ad4e0 con 0x7f6a28156960 2024-09-11T00:31:23.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.779+0000 7f6a1fe00640 1 -- 172.21.15.31:0/3907671908 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6a2000ac50 con 0x7f6a28156960 2024-09-11T00:31:23.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.780+0000 7f6a1fe00640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6a08077f70 0x7f6a0807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:23.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.780+0000 7f6a1fe00640 1 -- 172.21.15.31:0/3907671908 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f6a20099f30 con 0x7f6a28156960 2024-09-11T00:31:23.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.780+0000 7f6a27e00640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6a08077f70 0x7f6a0807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:23.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.781+0000 7f6a27e00640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6a08077f70 0x7f6a0807a430 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6a18033000 tx=0x7f6a18004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:23.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.784+0000 7f6a1fe00640 1 -- 172.21.15.31:0/3907671908 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6a20062670 con 0x7f6a28156960 2024-09-11T00:31:23.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.975+0000 7f6a2d400640 1 -- 172.21.15.31:0/3907671908 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6a280cc8e0 con 0x7f6a08077f70 2024-09-11T00:31:23.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.984+0000 7f6a1fe00640 1 -- 172.21.15.31:0/3907671908 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490314 (secure 0 0 0) 0x7f6a280cc8e0 con 0x7f6a08077f70 2024-09-11T00:31:23.992 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:23.995 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:31:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.989+0000 7f6a2d400640 1 -- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6a08077f70 msgr2=0x7f6a0807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.989+0000 7f6a2d400640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6a08077f70 0x7f6a0807a430 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6a18033000 tx=0x7f6a18004730 comp rx=0 tx=0).stop 2024-09-11T00:31:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.989+0000 7f6a2d400640 1 -- 172.21.15.31:0/3907671908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a28156960 msgr2=0x7f6a281eba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.989+0000 7f6a2d400640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a28156960 0x7f6a281eba60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000c7a0 tx=0x7f6a2000cc60 comp rx=0 tx=0).stop 2024-09-11T00:31:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.990+0000 7f6a2d400640 1 -- 172.21.15.31:0/3907671908 shutdown_connections 2024-09-11T00:31:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.990+0000 7f6a2d400640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6a08077f70 0x7f6a0807a430 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.990+0000 7f6a2d400640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a28156960 0x7f6a281eba60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.990+0000 7f6a2d400640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6a28150020 0x7f6a280bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.990+0000 7f6a2d400640 1 --2- 172.21.15.31:0/3907671908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a280ada20 0x7f6a280c1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.990+0000 7f6a2d400640 1 -- 172.21.15.31:0/3907671908 >> 172.21.15.31:0/3907671908 conn(0x7f6a28068f10 msgr2=0x7f6a2814f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.990+0000 7f6a2d400640 1 -- 172.21.15.31:0/3907671908 shutdown_connections 2024-09-11T00:31:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:23.990+0000 7f6a2d400640 1 -- 172.21.15.31:0/3907671908 wait complete. 2024-09-11T00:31:24.009 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-11T00:31:22.401600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595629,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":193,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219624,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755096,"statfs":{"total":773094113280,"available":772869218304,"internally_reserved":0,"allocated":3481600,"data_stored":2028210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":441,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003685"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719046+0000","last_change":"2024-09-11T00:30:54.983412+0000","last_active":"2024-09-11T00:30:56.719046+0000","last_peered":"2024-09-11T00:30:56.719046+0000","last_clean":"2024-09-11T00:30:56.719046+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:30:56.719046+0000","last_undegraded":"2024-09-11T00:30:56.719046+0000","last_fullsized":"2024-09-11T00:30:56.719046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:51.807846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970097+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:30:55.970097+0000","last_peered":"2024-09-11T00:30:55.970097+0000","last_clean":"2024-09-11T00:30:55.970097+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:30:55.970097+0000","last_undegraded":"2024-09-11T00:30:55.970097+0000","last_fullsized":"2024-09-11T00:30:55.970097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998047+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:30:55.998047+0000","last_peered":"2024-09-11T00:30:55.998047+0000","last_clean":"2024-09-11T00:30:55.998047+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:30:55.998047+0000","last_undegraded":"2024-09-11T00:30:55.998047+0000","last_fullsized":"2024-09-11T00:30:55.998047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000942+0000","last_change":"2024-09-11T00:30:53.984316+0000","last_active":"2024-09-11T00:30:56.000942+0000","last_peered":"2024-09-11T00:30:56.000942+0000","last_clean":"2024-09-11T00:30:56.000942+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:30:56.000942+0000","last_undegraded":"2024-09-11T00:30:56.000942+0000","last_fullsized":"2024-09-11T00:30:56.000942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:14.425265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968662+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:30:55.968662+0000","last_peered":"2024-09-11T00:30:55.968662+0000","last_clean":"2024-09-11T00:30:55.968662+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:30:55.968662+0000","last_undegraded":"2024-09-11T00:30:55.968662+0000","last_fullsized":"2024-09-11T00:30:55.968662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998255+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:30:55.998255+0000","last_peered":"2024-09-11T00:30:55.998255+0000","last_clean":"2024-09-11T00:30:55.998255+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:30:55.998255+0000","last_undegraded":"2024-09-11T00:30:55.998255+0000","last_fullsized":"2024-09-11T00:30:55.998255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998074+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:30:55.998074+0000","last_peered":"2024-09-11T00:30:55.998074+0000","last_clean":"2024-09-11T00:30:55.998074+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:30:55.998074+0000","last_undegraded":"2024-09-11T00:30:55.998074+0000","last_fullsized":"2024-09-11T00:30:55.998074+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172549+0000","last_change":"2024-09-11T00:30:53.984940+0000","last_active":"2024-09-11T00:30:56.172549+0000","last_peered":"2024-09-11T00:30:56.172549+0000","last_clean":"2024-09-11T00:30:56.172549+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:30:56.172549+0000","last_undegraded":"2024-09-11T00:30:56.172549+0000","last_fullsized":"2024-09-11T00:30:56.172549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:19.647165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719205+0000","last_change":"2024-09-11T00:30:54.993570+0000","last_active":"2024-09-11T00:30:56.719205+0000","last_peered":"2024-09-11T00:30:56.719205+0000","last_clean":"2024-09-11T00:30:56.719205+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:30:56.719205+0000","last_undegraded":"2024-09-11T00:30:56.719205+0000","last_fullsized":"2024-09-11T00:30:56.719205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:45.211259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333273+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:30:56.333273+0000","last_peered":"2024-09-11T00:30:56.333273+0000","last_clean":"2024-09-11T00:30:56.333273+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:30:56.333273+0000","last_undegraded":"2024-09-11T00:30:56.333273+0000","last_fullsized":"2024-09-11T00:30:56.333273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719392+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:30:56.719392+0000","last_peered":"2024-09-11T00:30:56.719392+0000","last_clean":"2024-09-11T00:30:56.719392+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:30:56.719392+0000","last_undegraded":"2024-09-11T00:30:56.719392+0000","last_fullsized":"2024-09-11T00:30:56.719392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172696+0000","last_change":"2024-09-11T00:30:53.974699+0000","last_active":"2024-09-11T00:30:56.172696+0000","last_peered":"2024-09-11T00:30:56.172696+0000","last_clean":"2024-09-11T00:30:56.172696+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:30:56.172696+0000","last_undegraded":"2024-09-11T00:30:56.172696+0000","last_fullsized":"2024-09-11T00:30:56.172696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:28.792487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172498+0000","last_change":"2024-09-11T00:30:54.984890+0000","last_active":"2024-09-11T00:30:56.172498+0000","last_peered":"2024-09-11T00:30:56.172498+0000","last_clean":"2024-09-11T00:30:56.172498+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:30:56.172498+0000","last_undegraded":"2024-09-11T00:30:56.172498+0000","last_fullsized":"2024-09-11T00:30:56.172498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:21.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333166+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:30:56.333166+0000","last_peered":"2024-09-11T00:30:56.333166+0000","last_clean":"2024-09-11T00:30:56.333166+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:30:56.333166+0000","last_undegraded":"2024-09-11T00:30:56.333166+0000","last_fullsized":"2024-09-11T00:30:56.333166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968144+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:30:55.968144+0000","last_peered":"2024-09-11T00:30:55.968144+0000","last_clean":"2024-09-11T00:30:55.968144+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:30:55.968144+0000","last_undegraded":"2024-09-11T00:30:55.968144+0000","last_fullsized":"2024-09-11T00:30:55.968144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172339+0000","last_change":"2024-09-11T00:30:53.977743+0000","last_active":"2024-09-11T00:30:56.172339+0000","last_peered":"2024-09-11T00:30:56.172339+0000","last_clean":"2024-09-11T00:30:56.172339+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:30:56.172339+0000","last_undegraded":"2024-09-11T00:30:56.172339+0000","last_fullsized":"2024-09-11T00:30:56.172339+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:20.618615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997709+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:30:55.997709+0000","last_peered":"2024-09-11T00:30:55.997709+0000","last_clean":"2024-09-11T00:30:55.997709+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:30:55.997709+0000","last_undegraded":"2024-09-11T00:30:55.997709+0000","last_fullsized":"2024-09-11T00:30:55.997709+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967362+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:30:55.967362+0000","last_peered":"2024-09-11T00:30:55.967362+0000","last_clean":"2024-09-11T00:30:55.967362+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:30:55.967362+0000","last_undegraded":"2024-09-11T00:30:55.967362+0000","last_fullsized":"2024-09-11T00:30:55.967362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967452+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:30:55.967452+0000","last_peered":"2024-09-11T00:30:55.967452+0000","last_clean":"2024-09-11T00:30:55.967452+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:30:55.967452+0000","last_undegraded":"2024-09-11T00:30:55.967452+0000","last_fullsized":"2024-09-11T00:30:55.967452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967914+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:30:55.967914+0000","last_peered":"2024-09-11T00:30:55.967914+0000","last_clean":"2024-09-11T00:30:55.967914+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:30:55.967914+0000","last_undegraded":"2024-09-11T00:30:55.967914+0000","last_fullsized":"2024-09-11T00:30:55.967914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001419+0000","last_change":"2024-09-11T00:30:54.997397+0000","last_active":"2024-09-11T00:30:56.001419+0000","last_peered":"2024-09-11T00:30:56.001419+0000","last_clean":"2024-09-11T00:30:56.001419+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:30:56.001419+0000","last_undegraded":"2024-09-11T00:30:56.001419+0000","last_fullsized":"2024-09-11T00:30:56.001419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:53.322576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001116+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:30:56.001116+0000","last_peered":"2024-09-11T00:30:56.001116+0000","last_clean":"2024-09-11T00:30:56.001116+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:30:56.001116+0000","last_undegraded":"2024-09-11T00:30:56.001116+0000","last_fullsized":"2024-09-11T00:30:56.001116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970056+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:30:55.970056+0000","last_peered":"2024-09-11T00:30:55.970056+0000","last_clean":"2024-09-11T00:30:55.970056+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:30:55.970056+0000","last_undegraded":"2024-09-11T00:30:55.970056+0000","last_fullsized":"2024-09-11T00:30:55.970056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332817+0000","last_change":"2024-09-11T00:30:53.986858+0000","last_active":"2024-09-11T00:30:56.332817+0000","last_peered":"2024-09-11T00:30:56.332817+0000","last_clean":"2024-09-11T00:30:56.332817+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:30:56.332817+0000","last_undegraded":"2024-09-11T00:30:56.332817+0000","last_fullsized":"2024-09-11T00:30:56.332817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:33.626193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967911+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:30:55.967911+0000","last_peered":"2024-09-11T00:30:55.967911+0000","last_clean":"2024-09-11T00:30:55.967911+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:30:55.967911+0000","last_undegraded":"2024-09-11T00:30:55.967911+0000","last_fullsized":"2024-09-11T00:30:55.967911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968190+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:30:55.968190+0000","last_peered":"2024-09-11T00:30:55.968190+0000","last_clean":"2024-09-11T00:30:55.968190+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:30:55.968190+0000","last_undegraded":"2024-09-11T00:30:55.968190+0000","last_fullsized":"2024-09-11T00:30:55.968190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968850+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:30:55.968850+0000","last_peered":"2024-09-11T00:30:55.968850+0000","last_clean":"2024-09-11T00:30:55.968850+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:30:55.968850+0000","last_undegraded":"2024-09-11T00:30:55.968850+0000","last_fullsized":"2024-09-11T00:30:55.968850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001447+0000","last_change":"2024-09-11T00:30:53.990765+0000","last_active":"2024-09-11T00:30:56.001447+0000","last_peered":"2024-09-11T00:30:56.001447+0000","last_clean":"2024-09-11T00:30:56.001447+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:30:56.001447+0000","last_undegraded":"2024-09-11T00:30:56.001447+0000","last_fullsized":"2024-09-11T00:30:56.001447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:33.720727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719722+0000","last_change":"2024-09-11T00:30:54.984949+0000","last_active":"2024-09-11T00:30:56.719722+0000","last_peered":"2024-09-11T00:30:56.719722+0000","last_clean":"2024-09-11T00:30:56.719722+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:30:56.719722+0000","last_undegraded":"2024-09-11T00:30:56.719722+0000","last_fullsized":"2024-09-11T00:30:56.719722+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:54.346937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967033+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:30:55.967033+0000","last_peered":"2024-09-11T00:30:55.967033+0000","last_clean":"2024-09-11T00:30:55.967033+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:30:55.967033+0000","last_undegraded":"2024-09-11T00:30:55.967033+0000","last_fullsized":"2024-09-11T00:30:55.967033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332357+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:30:56.332357+0000","last_peered":"2024-09-11T00:30:56.332357+0000","last_clean":"2024-09-11T00:30:56.332357+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:30:56.332357+0000","last_undegraded":"2024-09-11T00:30:56.332357+0000","last_fullsized":"2024-09-11T00:30:56.332357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172821+0000","last_change":"2024-09-11T00:30:53.986940+0000","last_active":"2024-09-11T00:30:56.172821+0000","last_peered":"2024-09-11T00:30:56.172821+0000","last_clean":"2024-09-11T00:30:56.172821+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:30:56.172821+0000","last_undegraded":"2024-09-11T00:30:56.172821+0000","last_fullsized":"2024-09-11T00:30:56.172821+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:11.942187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968462+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:30:55.968462+0000","last_peered":"2024-09-11T00:30:55.968462+0000","last_clean":"2024-09-11T00:30:55.968462+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:30:55.968462+0000","last_undegraded":"2024-09-11T00:30:55.968462+0000","last_fullsized":"2024-09-11T00:30:55.968462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968110+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:30:55.968110+0000","last_peered":"2024-09-11T00:30:55.968110+0000","last_clean":"2024-09-11T00:30:55.968110+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:30:55.968110+0000","last_undegraded":"2024-09-11T00:30:55.968110+0000","last_fullsized":"2024-09-11T00:30:55.968110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172483+0000","last_change":"2024-09-11T00:30:53.988539+0000","last_active":"2024-09-11T00:30:56.172483+0000","last_peered":"2024-09-11T00:30:56.172483+0000","last_clean":"2024-09-11T00:30:56.172483+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:30:56.172483+0000","last_undegraded":"2024-09-11T00:30:56.172483+0000","last_fullsized":"2024-09-11T00:30:56.172483+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:19.083664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966824+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:30:55.966824+0000","last_peered":"2024-09-11T00:30:55.966824+0000","last_clean":"2024-09-11T00:30:55.966824+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:30:55.966824+0000","last_undegraded":"2024-09-11T00:30:55.966824+0000","last_fullsized":"2024-09-11T00:30:55.966824+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970081+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:30:55.970081+0000","last_peered":"2024-09-11T00:30:55.970081+0000","last_clean":"2024-09-11T00:30:55.970081+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:30:55.970081+0000","last_undegraded":"2024-09-11T00:30:55.970081+0000","last_fullsized":"2024-09-11T00:30:55.970081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970376+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:30:55.970376+0000","last_peered":"2024-09-11T00:30:55.970376+0000","last_clean":"2024-09-11T00:30:55.970376+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:30:55.970376+0000","last_undegraded":"2024-09-11T00:30:55.970376+0000","last_fullsized":"2024-09-11T00:30:55.970376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968131+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:30:55.968131+0000","last_peered":"2024-09-11T00:30:55.968131+0000","last_clean":"2024-09-11T00:30:55.968131+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:30:55.968131+0000","last_undegraded":"2024-09-11T00:30:55.968131+0000","last_fullsized":"2024-09-11T00:30:55.968131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332511+0000","last_change":"2024-09-11T00:30:54.993118+0000","last_active":"2024-09-11T00:30:56.332511+0000","last_peered":"2024-09-11T00:30:56.332511+0000","last_clean":"2024-09-11T00:30:56.332511+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:30:56.332511+0000","last_undegraded":"2024-09-11T00:30:56.332511+0000","last_fullsized":"2024-09-11T00:30:56.332511+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:07:41.148370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971378+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:30:55.971378+0000","last_peered":"2024-09-11T00:30:55.971378+0000","last_clean":"2024-09-11T00:30:55.971378+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:30:55.971378+0000","last_undegraded":"2024-09-11T00:30:55.971378+0000","last_fullsized":"2024-09-11T00:30:55.971378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968486+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:30:55.968486+0000","last_peered":"2024-09-11T00:30:55.968486+0000","last_clean":"2024-09-11T00:30:55.968486+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:30:55.968486+0000","last_undegraded":"2024-09-11T00:30:55.968486+0000","last_fullsized":"2024-09-11T00:30:55.968486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970903+0000","last_change":"2024-09-11T00:30:53.986843+0000","last_active":"2024-09-11T00:30:55.970903+0000","last_peered":"2024-09-11T00:30:55.970903+0000","last_clean":"2024-09-11T00:30:55.970903+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:30:55.970903+0000","last_undegraded":"2024-09-11T00:30:55.970903+0000","last_fullsized":"2024-09-11T00:30:55.970903+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:44.684154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969549+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:30:55.969549+0000","last_peered":"2024-09-11T00:30:55.969549+0000","last_clean":"2024-09-11T00:30:55.969549+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:30:55.969549+0000","last_undegraded":"2024-09-11T00:30:55.969549+0000","last_fullsized":"2024-09-11T00:30:55.969549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172919+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:30:56.172919+0000","last_peered":"2024-09-11T00:30:56.172919+0000","last_clean":"2024-09-11T00:30:56.172919+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:30:56.172919+0000","last_undegraded":"2024-09-11T00:30:56.172919+0000","last_fullsized":"2024-09-11T00:30:56.172919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997652+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:30:55.997652+0000","last_peered":"2024-09-11T00:30:55.997652+0000","last_clean":"2024-09-11T00:30:55.997652+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:30:55.997652+0000","last_undegraded":"2024-09-11T00:30:55.997652+0000","last_fullsized":"2024-09-11T00:30:55.997652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969390+0000","last_change":"2024-09-11T00:30:53.986896+0000","last_active":"2024-09-11T00:30:55.969390+0000","last_peered":"2024-09-11T00:30:55.969390+0000","last_clean":"2024-09-11T00:30:55.969390+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:30:55.969390+0000","last_undegraded":"2024-09-11T00:30:55.969390+0000","last_fullsized":"2024-09-11T00:30:55.969390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:06.590278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001631+0000","last_change":"2024-09-11T00:30:55.001370+0000","last_active":"2024-09-11T00:30:56.001631+0000","last_peered":"2024-09-11T00:30:56.001631+0000","last_clean":"2024-09-11T00:30:56.001631+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:30:56.001631+0000","last_undegraded":"2024-09-11T00:30:56.001631+0000","last_fullsized":"2024-09-11T00:30:56.001631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:56.592767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333262+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:30:56.333262+0000","last_peered":"2024-09-11T00:30:56.333262+0000","last_clean":"2024-09-11T00:30:56.333262+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:30:56.333262+0000","last_undegraded":"2024-09-11T00:30:56.333262+0000","last_fullsized":"2024-09-11T00:30:56.333262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967594+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:30:55.967594+0000","last_peered":"2024-09-11T00:30:55.967594+0000","last_clean":"2024-09-11T00:30:55.967594+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:30:55.967594+0000","last_undegraded":"2024-09-11T00:30:55.967594+0000","last_fullsized":"2024-09-11T00:30:55.967594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971192+0000","last_change":"2024-09-11T00:30:53.978480+0000","last_active":"2024-09-11T00:30:55.971192+0000","last_peered":"2024-09-11T00:30:55.971192+0000","last_clean":"2024-09-11T00:30:55.971192+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:30:55.971192+0000","last_undegraded":"2024-09-11T00:30:55.971192+0000","last_fullsized":"2024-09-11T00:30:55.971192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:58.900565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001528+0000","last_change":"2024-09-11T00:30:54.997641+0000","last_active":"2024-09-11T00:30:56.001528+0000","last_peered":"2024-09-11T00:30:56.001528+0000","last_clean":"2024-09-11T00:30:56.001528+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:30:56.001528+0000","last_undegraded":"2024-09-11T00:30:56.001528+0000","last_fullsized":"2024-09-11T00:30:56.001528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:18.437490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997311+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:30:55.997311+0000","last_peered":"2024-09-11T00:30:55.997311+0000","last_clean":"2024-09-11T00:30:55.997311+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:30:55.997311+0000","last_undegraded":"2024-09-11T00:30:55.997311+0000","last_fullsized":"2024-09-11T00:30:55.997311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971163+0000","last_change":"2024-09-11T00:30:53.979031+0000","last_active":"2024-09-11T00:30:55.971163+0000","last_peered":"2024-09-11T00:30:55.971163+0000","last_clean":"2024-09-11T00:30:55.971163+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:30:55.971163+0000","last_undegraded":"2024-09-11T00:30:55.971163+0000","last_fullsized":"2024-09-11T00:30:55.971163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:36.604377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967699+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:30:55.967699+0000","last_peered":"2024-09-11T00:30:55.967699+0000","last_clean":"2024-09-11T00:30:55.967699+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:30:55.967699+0000","last_undegraded":"2024-09-11T00:30:55.967699+0000","last_fullsized":"2024-09-11T00:30:55.967699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:39.629122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968089+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:30:55.968089+0000","last_peered":"2024-09-11T00:30:55.968089+0000","last_clean":"2024-09-11T00:30:55.968089+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:30:55.968089+0000","last_undegraded":"2024-09-11T00:30:55.968089+0000","last_fullsized":"2024-09-11T00:30:55.968089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966595+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:30:55.966595+0000","last_peered":"2024-09-11T00:30:55.966595+0000","last_clean":"2024-09-11T00:30:55.966595+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:30:55.966595+0000","last_undegraded":"2024-09-11T00:30:55.966595+0000","last_fullsized":"2024-09-11T00:30:55.966595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"26'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998301+0000","last_change":"2024-09-11T00:30:53.989299+0000","last_active":"2024-09-11T00:30:55.998301+0000","last_peered":"2024-09-11T00:30:55.998301+0000","last_clean":"2024-09-11T00:30:55.998301+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:30:55.998301+0000","last_undegraded":"2024-09-11T00:30:55.998301+0000","last_fullsized":"2024-09-11T00:30:55.998301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:34:49.298915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332781+0000","last_change":"2024-09-11T00:30:54.990026+0000","last_active":"2024-09-11T00:30:56.332781+0000","last_peered":"2024-09-11T00:30:56.332781+0000","last_clean":"2024-09-11T00:30:56.332781+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:30:56.332781+0000","last_undegraded":"2024-09-11T00:30:56.332781+0000","last_fullsized":"2024-09-11T00:30:56.332781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:02:10.800530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969545+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:30:55.969545+0000","last_peered":"2024-09-11T00:30:55.969545+0000","last_clean":"2024-09-11T00:30:55.969545+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:30:55.969545+0000","last_undegraded":"2024-09-11T00:30:55.969545+0000","last_fullsized":"2024-09-11T00:30:55.969545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967491+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:30:55.967491+0000","last_peered":"2024-09-11T00:30:55.967491+0000","last_clean":"2024-09-11T00:30:55.967491+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:30:55.967491+0000","last_undegraded":"2024-09-11T00:30:55.967491+0000","last_fullsized":"2024-09-11T00:30:55.967491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719269+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:30:56.719269+0000","last_peered":"2024-09-11T00:30:56.719269+0000","last_clean":"2024-09-11T00:30:56.719269+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:30:56.719269+0000","last_undegraded":"2024-09-11T00:30:56.719269+0000","last_fullsized":"2024-09-11T00:30:56.719269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968267+0000","last_change":"2024-09-11T00:30:53.982142+0000","last_active":"2024-09-11T00:30:55.968267+0000","last_peered":"2024-09-11T00:30:55.968267+0000","last_clean":"2024-09-11T00:30:55.968267+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:30:55.968267+0000","last_undegraded":"2024-09-11T00:30:55.968267+0000","last_fullsized":"2024-09-11T00:30:55.968267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:41.558944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998013+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:30:55.998013+0000","last_peered":"2024-09-11T00:30:55.998013+0000","last_clean":"2024-09-11T00:30:55.998013+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:30:55.998013+0000","last_undegraded":"2024-09-11T00:30:55.998013+0000","last_fullsized":"2024-09-11T00:30:55.998013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968362+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:30:55.968362+0000","last_peered":"2024-09-11T00:30:55.968362+0000","last_clean":"2024-09-11T00:30:55.968362+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:30:55.968362+0000","last_undegraded":"2024-09-11T00:30:55.968362+0000","last_fullsized":"2024-09-11T00:30:55.968362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971436+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:30:55.971436+0000","last_peered":"2024-09-11T00:30:55.971436+0000","last_clean":"2024-09-11T00:30:55.971436+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:30:55.971436+0000","last_undegraded":"2024-09-11T00:30:55.971436+0000","last_fullsized":"2024-09-11T00:30:55.971436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172455+0000","last_change":"2024-09-11T00:30:53.989292+0000","last_active":"2024-09-11T00:30:56.172455+0000","last_peered":"2024-09-11T00:30:56.172455+0000","last_clean":"2024-09-11T00:30:56.172455+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:30:56.172455+0000","last_undegraded":"2024-09-11T00:30:56.172455+0000","last_fullsized":"2024-09-11T00:30:56.172455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:44.071338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968473+0000","last_change":"2024-09-11T00:30:54.981630+0000","last_active":"2024-09-11T00:30:55.968473+0000","last_peered":"2024-09-11T00:30:55.968473+0000","last_clean":"2024-09-11T00:30:55.968473+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:30:55.968473+0000","last_undegraded":"2024-09-11T00:30:55.968473+0000","last_fullsized":"2024-09-11T00:30:55.968473+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:12.832327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333061+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:30:56.333061+0000","last_peered":"2024-09-11T00:30:56.333061+0000","last_clean":"2024-09-11T00:30:56.333061+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:30:56.333061+0000","last_undegraded":"2024-09-11T00:30:56.333061+0000","last_fullsized":"2024-09-11T00:30:56.333061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997856+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:30:55.997856+0000","last_peered":"2024-09-11T00:30:55.997856+0000","last_clean":"2024-09-11T00:30:55.997856+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:30:55.997856+0000","last_undegraded":"2024-09-11T00:30:55.997856+0000","last_fullsized":"2024-09-11T00:30:55.997856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997848+0000","last_change":"2024-09-11T00:30:53.984013+0000","last_active":"2024-09-11T00:30:55.997848+0000","last_peered":"2024-09-11T00:30:55.997848+0000","last_clean":"2024-09-11T00:30:55.997848+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:30:55.997848+0000","last_undegraded":"2024-09-11T00:30:55.997848+0000","last_fullsized":"2024-09-11T00:30:55.997848+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:49.963623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719176+0000","last_change":"2024-09-11T00:30:54.996860+0000","last_active":"2024-09-11T00:30:56.719176+0000","last_peered":"2024-09-11T00:30:56.719176+0000","last_clean":"2024-09-11T00:30:56.719176+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:30:56.719176+0000","last_undegraded":"2024-09-11T00:30:56.719176+0000","last_fullsized":"2024-09-11T00:30:56.719176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:43.516748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333114+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:30:56.333114+0000","last_peered":"2024-09-11T00:30:56.333114+0000","last_clean":"2024-09-11T00:30:56.333114+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:30:56.333114+0000","last_undegraded":"2024-09-11T00:30:56.333114+0000","last_fullsized":"2024-09-11T00:30:56.333114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970314+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:30:55.970314+0000","last_peered":"2024-09-11T00:30:55.970314+0000","last_clean":"2024-09-11T00:30:55.970314+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:30:55.970314+0000","last_undegraded":"2024-09-11T00:30:55.970314+0000","last_fullsized":"2024-09-11T00:30:55.970314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966317+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:30:55.966317+0000","last_peered":"2024-09-11T00:30:55.966317+0000","last_clean":"2024-09-11T00:30:55.966317+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:30:55.966317+0000","last_undegraded":"2024-09-11T00:30:55.966317+0000","last_fullsized":"2024-09-11T00:30:55.966317+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:19.891052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967870+0000","last_change":"2024-09-11T00:30:54.993736+0000","last_active":"2024-09-11T00:30:55.967870+0000","last_peered":"2024-09-11T00:30:55.967870+0000","last_clean":"2024-09-11T00:30:55.967870+0000","last_became_active":"2024-09-11T00:30:54.993470+0000","last_became_peered":"2024-09-11T00:30:54.993470+0000","last_unstale":"2024-09-11T00:30:55.967870+0000","last_undegraded":"2024-09-11T00:30:55.967870+0000","last_fullsized":"2024-09-11T00:30:55.967870+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:24.712448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173170+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:30:56.173170+0000","last_peered":"2024-09-11T00:30:56.173170+0000","last_clean":"2024-09-11T00:30:56.173170+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:30:56.173170+0000","last_undegraded":"2024-09-11T00:30:56.173170+0000","last_fullsized":"2024-09-11T00:30:56.173170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333218+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:30:56.333218+0000","last_peered":"2024-09-11T00:30:56.333218+0000","last_clean":"2024-09-11T00:30:56.333218+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:30:56.333218+0000","last_undegraded":"2024-09-11T00:30:56.333218+0000","last_fullsized":"2024-09-11T00:30:56.333218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719247+0000","last_change":"2024-09-11T00:30:53.983352+0000","last_active":"2024-09-11T00:30:56.719247+0000","last_peered":"2024-09-11T00:30:56.719247+0000","last_clean":"2024-09-11T00:30:56.719247+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:30:56.719247+0000","last_undegraded":"2024-09-11T00:30:56.719247+0000","last_fullsized":"2024-09-11T00:30:56.719247+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:43.826114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970030+0000","last_change":"2024-09-11T00:30:54.987533+0000","last_active":"2024-09-11T00:30:55.970030+0000","last_peered":"2024-09-11T00:30:55.970030+0000","last_clean":"2024-09-11T00:30:55.970030+0000","last_became_active":"2024-09-11T00:30:54.987253+0000","last_became_peered":"2024-09-11T00:30:54.987253+0000","last_unstale":"2024-09-11T00:30:55.970030+0000","last_undegraded":"2024-09-11T00:30:55.970030+0000","last_fullsized":"2024-09-11T00:30:55.970030+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.923656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966934+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:30:55.966934+0000","last_peered":"2024-09-11T00:30:55.966934+0000","last_clean":"2024-09-11T00:30:55.966934+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:30:55.966934+0000","last_undegraded":"2024-09-11T00:30:55.966934+0000","last_fullsized":"2024-09-11T00:30:55.966934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968528+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:30:55.968528+0000","last_peered":"2024-09-11T00:30:55.968528+0000","last_clean":"2024-09-11T00:30:55.968528+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:30:55.968528+0000","last_undegraded":"2024-09-11T00:30:55.968528+0000","last_fullsized":"2024-09-11T00:30:55.968528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997651+0000","last_change":"2024-09-11T00:30:53.980703+0000","last_active":"2024-09-11T00:30:55.997651+0000","last_peered":"2024-09-11T00:30:55.997651+0000","last_clean":"2024-09-11T00:30:55.997651+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:30:55.997651+0000","last_undegraded":"2024-09-11T00:30:55.997651+0000","last_fullsized":"2024-09-11T00:30:55.997651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:51.050512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967820+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:30:55.967820+0000","last_peered":"2024-09-11T00:30:55.967820+0000","last_clean":"2024-09-11T00:30:55.967820+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:30:55.967820+0000","last_undegraded":"2024-09-11T00:30:55.967820+0000","last_fullsized":"2024-09-11T00:30:55.967820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719139+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:30:56.719139+0000","last_peered":"2024-09-11T00:30:56.719139+0000","last_clean":"2024-09-11T00:30:56.719139+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:30:56.719139+0000","last_undegraded":"2024-09-11T00:30:56.719139+0000","last_fullsized":"2024-09-11T00:30:56.719139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333070+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:30:56.333070+0000","last_peered":"2024-09-11T00:30:56.333070+0000","last_clean":"2024-09-11T00:30:56.333070+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:30:56.333070+0000","last_undegraded":"2024-09-11T00:30:56.333070+0000","last_fullsized":"2024-09-11T00:30:56.333070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997937+0000","last_change":"2024-09-11T00:30:53.987829+0000","last_active":"2024-09-11T00:30:55.997937+0000","last_peered":"2024-09-11T00:30:55.997937+0000","last_clean":"2024-09-11T00:30:55.997937+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:30:55.997937+0000","last_undegraded":"2024-09-11T00:30:55.997937+0000","last_fullsized":"2024-09-11T00:30:55.997937+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:57:43.792089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967657+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:30:55.967657+0000","last_peered":"2024-09-11T00:30:55.967657+0000","last_clean":"2024-09-11T00:30:55.967657+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:30:55.967657+0000","last_undegraded":"2024-09-11T00:30:55.967657+0000","last_fullsized":"2024-09-11T00:30:55.967657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967382+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:30:55.967382+0000","last_peered":"2024-09-11T00:30:55.967382+0000","last_clean":"2024-09-11T00:30:55.967382+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:30:55.967382+0000","last_undegraded":"2024-09-11T00:30:55.967382+0000","last_fullsized":"2024-09-11T00:30:55.967382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967492+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:30:55.967492+0000","last_peered":"2024-09-11T00:30:55.967492+0000","last_clean":"2024-09-11T00:30:55.967492+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:30:55.967492+0000","last_undegraded":"2024-09-11T00:30:55.967492+0000","last_fullsized":"2024-09-11T00:30:55.967492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970951+0000","last_change":"2024-09-11T00:30:53.984057+0000","last_active":"2024-09-11T00:30:55.970951+0000","last_peered":"2024-09-11T00:30:55.970951+0000","last_clean":"2024-09-11T00:30:55.970951+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:30:55.970951+0000","last_undegraded":"2024-09-11T00:30:55.970951+0000","last_fullsized":"2024-09-11T00:30:55.970951+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:10.477916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969373+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:30:55.969373+0000","last_peered":"2024-09-11T00:30:55.969373+0000","last_clean":"2024-09-11T00:30:55.969373+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:30:55.969373+0000","last_undegraded":"2024-09-11T00:30:55.969373+0000","last_fullsized":"2024-09-11T00:30:55.969373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969960+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:30:55.969960+0000","last_peered":"2024-09-11T00:30:55.969960+0000","last_clean":"2024-09-11T00:30:55.969960+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:30:55.969960+0000","last_undegraded":"2024-09-11T00:30:55.969960+0000","last_fullsized":"2024-09-11T00:30:55.969960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001240+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:30:56.001240+0000","last_peered":"2024-09-11T00:30:56.001240+0000","last_clean":"2024-09-11T00:30:56.001240+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:30:56.001240+0000","last_undegraded":"2024-09-11T00:30:56.001240+0000","last_fullsized":"2024-09-11T00:30:56.001240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968074+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:30:55.968074+0000","last_peered":"2024-09-11T00:30:55.968074+0000","last_clean":"2024-09-11T00:30:55.968074+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:30:55.968074+0000","last_undegraded":"2024-09-11T00:30:55.968074+0000","last_fullsized":"2024-09-11T00:30:55.968074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969893+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:30:55.969893+0000","last_peered":"2024-09-11T00:30:55.969893+0000","last_clean":"2024-09-11T00:30:55.969893+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:30:55.969893+0000","last_undegraded":"2024-09-11T00:30:55.969893+0000","last_fullsized":"2024-09-11T00:30:55.969893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332978+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:30:56.332978+0000","last_peered":"2024-09-11T00:30:56.332978+0000","last_clean":"2024-09-11T00:30:56.332978+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:30:56.332978+0000","last_undegraded":"2024-09-11T00:30:56.332978+0000","last_fullsized":"2024-09-11T00:30:56.332978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:31:14.474646+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:31:14.474646+0000","last_peered":"2024-09-11T00:31:14.474646+0000","last_clean":"2024-09-11T00:31:14.474646+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:31:14.474646+0000","last_undegraded":"2024-09-11T00:31:14.474646+0000","last_fullsized":"2024-09-11T00:31:14.474646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969278+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:30:55.969278+0000","last_peered":"2024-09-11T00:30:55.969278+0000","last_clean":"2024-09-11T00:30:55.969278+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:30:55.969278+0000","last_undegraded":"2024-09-11T00:30:55.969278+0000","last_fullsized":"2024-09-11T00:30:55.969278+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969704+0000","last_change":"2024-09-11T00:30:54.986017+0000","last_active":"2024-09-11T00:30:55.969704+0000","last_peered":"2024-09-11T00:30:55.969704+0000","last_clean":"2024-09-11T00:30:55.969704+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:30:55.969704+0000","last_undegraded":"2024-09-11T00:30:55.969704+0000","last_fullsized":"2024-09-11T00:30:55.969704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:58:12.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967364+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:30:55.967364+0000","last_peered":"2024-09-11T00:30:55.967364+0000","last_clean":"2024-09-11T00:30:55.967364+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:30:55.967364+0000","last_undegraded":"2024-09-11T00:30:55.967364+0000","last_fullsized":"2024-09-11T00:30:55.967364+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001702+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:30:56.001702+0000","last_peered":"2024-09-11T00:30:56.001702+0000","last_clean":"2024-09-11T00:30:56.001702+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:30:56.001702+0000","last_undegraded":"2024-09-11T00:30:56.001702+0000","last_fullsized":"2024-09-11T00:30:56.001702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172588+0000","last_change":"2024-09-11T00:30:53.988462+0000","last_active":"2024-09-11T00:30:56.172588+0000","last_peered":"2024-09-11T00:30:56.172588+0000","last_clean":"2024-09-11T00:30:56.172588+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:30:56.172588+0000","last_undegraded":"2024-09-11T00:30:56.172588+0000","last_fullsized":"2024-09-11T00:30:56.172588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:22.385557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966521+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:30:55.966521+0000","last_peered":"2024-09-11T00:30:55.966521+0000","last_clean":"2024-09-11T00:30:55.966521+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:30:55.966521+0000","last_undegraded":"2024-09-11T00:30:55.966521+0000","last_fullsized":"2024-09-11T00:30:55.966521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998206+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:30:55.998206+0000","last_peered":"2024-09-11T00:30:55.998206+0000","last_clean":"2024-09-11T00:30:55.998206+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:30:55.998206+0000","last_undegraded":"2024-09-11T00:30:55.998206+0000","last_fullsized":"2024-09-11T00:30:55.998206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332956+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:30:56.332956+0000","last_peered":"2024-09-11T00:30:56.332956+0000","last_clean":"2024-09-11T00:30:56.332956+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:30:56.332956+0000","last_undegraded":"2024-09-11T00:30:56.332956+0000","last_fullsized":"2024-09-11T00:30:56.332956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000726+0000","last_change":"2024-09-11T00:30:53.986646+0000","last_active":"2024-09-11T00:30:56.000726+0000","last_peered":"2024-09-11T00:30:56.000726+0000","last_clean":"2024-09-11T00:30:56.000726+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:30:56.000726+0000","last_undegraded":"2024-09-11T00:30:56.000726+0000","last_fullsized":"2024-09-11T00:30:56.000726+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:27.447940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172237+0000","last_change":"2024-09-11T00:30:54.982486+0000","last_active":"2024-09-11T00:30:56.172237+0000","last_peered":"2024-09-11T00:30:56.172237+0000","last_clean":"2024-09-11T00:30:56.172237+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:30:56.172237+0000","last_undegraded":"2024-09-11T00:30:56.172237+0000","last_fullsized":"2024-09-11T00:30:56.172237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:41.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173110+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:30:56.173110+0000","last_peered":"2024-09-11T00:30:56.173110+0000","last_clean":"2024-09-11T00:30:56.173110+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:30:56.173110+0000","last_undegraded":"2024-09-11T00:30:56.173110+0000","last_fullsized":"2024-09-11T00:30:56.173110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719858+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:30:56.719858+0000","last_peered":"2024-09-11T00:30:56.719858+0000","last_clean":"2024-09-11T00:30:56.719858+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:30:56.719858+0000","last_undegraded":"2024-09-11T00:30:56.719858+0000","last_fullsized":"2024-09-11T00:30:56.719858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968520+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:30:55.968520+0000","last_peered":"2024-09-11T00:30:55.968520+0000","last_clean":"2024-09-11T00:30:55.968520+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:30:55.968520+0000","last_undegraded":"2024-09-11T00:30:55.968520+0000","last_fullsized":"2024-09-11T00:30:55.968520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173273+0000","last_change":"2024-09-11T00:30:54.985091+0000","last_active":"2024-09-11T00:30:56.173273+0000","last_peered":"2024-09-11T00:30:56.173273+0000","last_clean":"2024-09-11T00:30:56.173273+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:30:56.173273+0000","last_undegraded":"2024-09-11T00:30:56.173273+0000","last_fullsized":"2024-09-11T00:30:56.173273+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:33.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968811+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:30:55.968811+0000","last_peered":"2024-09-11T00:30:55.968811+0000","last_clean":"2024-09-11T00:30:55.968811+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:30:55.968811+0000","last_undegraded":"2024-09-11T00:30:55.968811+0000","last_fullsized":"2024-09-11T00:30:55.968811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966522+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:30:55.966522+0000","last_peered":"2024-09-11T00:30:55.966522+0000","last_clean":"2024-09-11T00:30:55.966522+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:30:55.966522+0000","last_undegraded":"2024-09-11T00:30:55.966522+0000","last_fullsized":"2024-09-11T00:30:55.966522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971377+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:30:55.971377+0000","last_peered":"2024-09-11T00:30:55.971377+0000","last_clean":"2024-09-11T00:30:55.971377+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:30:55.971377+0000","last_undegraded":"2024-09-11T00:30:55.971377+0000","last_fullsized":"2024-09-11T00:30:55.971377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:11.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967648+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:30:55.967648+0000","last_peered":"2024-09-11T00:30:55.967648+0000","last_clean":"2024-09-11T00:30:55.967648+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:30:55.967648+0000","last_undegraded":"2024-09-11T00:30:55.967648+0000","last_fullsized":"2024-09-11T00:30:55.967648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968196+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:30:55.968196+0000","last_peered":"2024-09-11T00:30:55.968196+0000","last_clean":"2024-09-11T00:30:55.968196+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:30:55.968196+0000","last_undegraded":"2024-09-11T00:30:55.968196+0000","last_fullsized":"2024-09-11T00:30:55.968196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969528+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:30:55.969528+0000","last_peered":"2024-09-11T00:30:55.969528+0000","last_clean":"2024-09-11T00:30:55.969528+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:30:55.969528+0000","last_undegraded":"2024-09-11T00:30:55.969528+0000","last_fullsized":"2024-09-11T00:30:55.969528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967369+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:30:55.967369+0000","last_peered":"2024-09-11T00:30:55.967369+0000","last_clean":"2024-09-11T00:30:55.967369+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:30:55.967369+0000","last_undegraded":"2024-09-11T00:30:55.967369+0000","last_fullsized":"2024-09-11T00:30:55.967369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969812+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:30:55.969812+0000","last_peered":"2024-09-11T00:30:55.969812+0000","last_clean":"2024-09-11T00:30:55.969812+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:30:55.969812+0000","last_undegraded":"2024-09-11T00:30:55.969812+0000","last_fullsized":"2024-09-11T00:30:55.969812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969509+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:30:55.969509+0000","last_peered":"2024-09-11T00:30:55.969509+0000","last_clean":"2024-09-11T00:30:55.969509+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:30:55.969509+0000","last_undegraded":"2024-09-11T00:30:55.969509+0000","last_fullsized":"2024-09-11T00:30:55.969509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173337+0000","last_change":"2024-09-11T00:30:54.998464+0000","last_active":"2024-09-11T00:30:56.173337+0000","last_peered":"2024-09-11T00:30:56.173337+0000","last_clean":"2024-09-11T00:30:56.173337+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:30:56.173337+0000","last_undegraded":"2024-09-11T00:30:56.173337+0000","last_fullsized":"2024-09-11T00:30:56.173337+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:56.469791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998082+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:30:55.998082+0000","last_peered":"2024-09-11T00:30:55.998082+0000","last_clean":"2024-09-11T00:30:55.998082+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:30:55.998082+0000","last_undegraded":"2024-09-11T00:30:55.998082+0000","last_fullsized":"2024-09-11T00:30:55.998082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719278+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:30:56.719278+0000","last_peered":"2024-09-11T00:30:56.719278+0000","last_clean":"2024-09-11T00:30:56.719278+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:30:56.719278+0000","last_undegraded":"2024-09-11T00:30:56.719278+0000","last_fullsized":"2024-09-11T00:30:56.719278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001549+0000","last_change":"2024-09-11T00:30:53.987612+0000","last_active":"2024-09-11T00:30:56.001549+0000","last_peered":"2024-09-11T00:30:56.001549+0000","last_clean":"2024-09-11T00:30:56.001549+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:30:56.001549+0000","last_undegraded":"2024-09-11T00:30:56.001549+0000","last_fullsized":"2024-09-11T00:30:56.001549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:16.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970084+0000","last_change":"2024-09-11T00:30:54.989925+0000","last_active":"2024-09-11T00:30:55.970084+0000","last_peered":"2024-09-11T00:30:55.970084+0000","last_clean":"2024-09-11T00:30:55.970084+0000","last_became_active":"2024-09-11T00:30:54.989401+0000","last_became_peered":"2024-09-11T00:30:54.989401+0000","last_unstale":"2024-09-11T00:30:55.970084+0000","last_undegraded":"2024-09-11T00:30:55.970084+0000","last_fullsized":"2024-09-11T00:30:55.970084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:14.267175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172747+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:30:56.172747+0000","last_peered":"2024-09-11T00:30:56.172747+0000","last_clean":"2024-09-11T00:30:56.172747+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:30:56.172747+0000","last_undegraded":"2024-09-11T00:30:56.172747+0000","last_fullsized":"2024-09-11T00:30:56.172747+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969929+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:30:55.969929+0000","last_peered":"2024-09-11T00:30:55.969929+0000","last_clean":"2024-09-11T00:30:55.969929+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:30:55.969929+0000","last_undegraded":"2024-09-11T00:30:55.969929+0000","last_fullsized":"2024-09-11T00:30:55.969929+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000911+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:30:56.000911+0000","last_peered":"2024-09-11T00:30:56.000911+0000","last_clean":"2024-09-11T00:30:56.000911+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:30:56.000911+0000","last_undegraded":"2024-09-11T00:30:56.000911+0000","last_fullsized":"2024-09-11T00:30:56.000911+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172912+0000","last_change":"2024-09-11T00:30:54.979133+0000","last_active":"2024-09-11T00:30:56.172912+0000","last_peered":"2024-09-11T00:30:56.172912+0000","last_clean":"2024-09-11T00:30:56.172912+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:30:56.172912+0000","last_undegraded":"2024-09-11T00:30:56.172912+0000","last_fullsized":"2024-09-11T00:30:56.172912+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:20.961678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967251+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:30:55.967251+0000","last_peered":"2024-09-11T00:30:55.967251+0000","last_clean":"2024-09-11T00:30:55.967251+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:30:55.967251+0000","last_undegraded":"2024-09-11T00:30:55.967251+0000","last_fullsized":"2024-09-11T00:30:55.967251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997885+0000","last_change":"2024-09-11T00:30:54.988345+0000","last_active":"2024-09-11T00:30:55.997885+0000","last_peered":"2024-09-11T00:30:55.997885+0000","last_clean":"2024-09-11T00:30:55.997885+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:30:55.997885+0000","last_undegraded":"2024-09-11T00:30:55.997885+0000","last_fullsized":"2024-09-11T00:30:55.997885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.112194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998078+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:30:55.998078+0000","last_peered":"2024-09-11T00:30:55.998078+0000","last_clean":"2024-09-11T00:30:55.998078+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:30:55.998078+0000","last_undegraded":"2024-09-11T00:30:55.998078+0000","last_fullsized":"2024-09-11T00:30:55.998078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332489+0000","last_change":"2024-09-11T00:30:54.981185+0000","last_active":"2024-09-11T00:30:56.332489+0000","last_peered":"2024-09-11T00:30:56.332489+0000","last_clean":"2024-09-11T00:30:56.332489+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:30:56.332489+0000","last_undegraded":"2024-09-11T00:30:56.332489+0000","last_fullsized":"2024-09-11T00:30:56.332489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:45.740467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971241+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:30:55.971241+0000","last_peered":"2024-09-11T00:30:55.971241+0000","last_clean":"2024-09-11T00:30:55.971241+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:30:55.971241+0000","last_undegraded":"2024-09-11T00:30:55.971241+0000","last_fullsized":"2024-09-11T00:30:55.971241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332647+0000","last_change":"2024-09-11T00:30:54.992763+0000","last_active":"2024-09-11T00:30:56.332647+0000","last_peered":"2024-09-11T00:30:56.332647+0000","last_clean":"2024-09-11T00:30:56.332647+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:30:56.332647+0000","last_undegraded":"2024-09-11T00:30:56.332647+0000","last_fullsized":"2024-09-11T00:30:56.332647+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:02.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971179+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:30:55.971179+0000","last_peered":"2024-09-11T00:30:55.971179+0000","last_clean":"2024-09-11T00:30:55.971179+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:30:55.971179+0000","last_undegraded":"2024-09-11T00:30:55.971179+0000","last_fullsized":"2024-09-11T00:30:55.971179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966440+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:30:55.966440+0000","last_peered":"2024-09-11T00:30:55.966440+0000","last_clean":"2024-09-11T00:30:55.966440+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:30:55.966440+0000","last_undegraded":"2024-09-11T00:30:55.966440+0000","last_fullsized":"2024-09-11T00:30:55.966440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173066+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:30:56.173066+0000","last_peered":"2024-09-11T00:30:56.173066+0000","last_clean":"2024-09-11T00:30:56.173066+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:30:56.173066+0000","last_undegraded":"2024-09-11T00:30:56.173066+0000","last_fullsized":"2024-09-11T00:30:56.173066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172366+0000","last_change":"2024-09-11T00:30:54.984447+0000","last_active":"2024-09-11T00:30:56.172366+0000","last_peered":"2024-09-11T00:30:56.172366+0000","last_clean":"2024-09-11T00:30:56.172366+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:30:56.172366+0000","last_undegraded":"2024-09-11T00:30:56.172366+0000","last_fullsized":"2024-09-11T00:30:56.172366+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:49.024022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998072+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:30:55.998072+0000","last_peered":"2024-09-11T00:30:55.998072+0000","last_clean":"2024-09-11T00:30:55.998072+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:30:55.998072+0000","last_undegraded":"2024-09-11T00:30:55.998072+0000","last_fullsized":"2024-09-11T00:30:55.998072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001507+0000","last_change":"2024-09-11T00:30:54.996091+0000","last_active":"2024-09-11T00:30:56.001507+0000","last_peered":"2024-09-11T00:30:56.001507+0000","last_clean":"2024-09-11T00:30:56.001507+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:30:56.001507+0000","last_undegraded":"2024-09-11T00:30:56.001507+0000","last_fullsized":"2024-09-11T00:30:56.001507+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:42.987336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966504+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:30:55.966504+0000","last_peered":"2024-09-11T00:30:55.966504+0000","last_clean":"2024-09-11T00:30:55.966504+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:30:55.966504+0000","last_undegraded":"2024-09-11T00:30:55.966504+0000","last_fullsized":"2024-09-11T00:30:55.966504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968347+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:30:55.968347+0000","last_peered":"2024-09-11T00:30:55.968347+0000","last_clean":"2024-09-11T00:30:55.968347+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:30:55.968347+0000","last_undegraded":"2024-09-11T00:30:55.968347+0000","last_fullsized":"2024-09-11T00:30:55.968347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971167+0000","last_change":"2024-09-11T00:30:54.981799+0000","last_active":"2024-09-11T00:30:55.971167+0000","last_peered":"2024-09-11T00:30:55.971167+0000","last_clean":"2024-09-11T00:30:55.971167+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:30:55.971167+0000","last_undegraded":"2024-09-11T00:30:55.971167+0000","last_fullsized":"2024-09-11T00:30:55.971167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:42.670629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719169+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:30:56.719169+0000","last_peered":"2024-09-11T00:30:56.719169+0000","last_clean":"2024-09-11T00:30:56.719169+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:30:56.719169+0000","last_undegraded":"2024-09-11T00:30:56.719169+0000","last_fullsized":"2024-09-11T00:30:56.719169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001453+0000","last_change":"2024-09-11T00:30:54.992697+0000","last_active":"2024-09-11T00:30:56.001453+0000","last_peered":"2024-09-11T00:30:56.001453+0000","last_clean":"2024-09-11T00:30:56.001453+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:30:56.001453+0000","last_undegraded":"2024-09-11T00:30:56.001453+0000","last_fullsized":"2024-09-11T00:30:56.001453+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:54:07.227169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969635+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:30:55.969635+0000","last_peered":"2024-09-11T00:30:55.969635+0000","last_clean":"2024-09-11T00:30:55.969635+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:55.969635+0000","last_undegraded":"2024-09-11T00:30:55.969635+0000","last_fullsized":"2024-09-11T00:30:55.969635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172413+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:30:56.172413+0000","last_peered":"2024-09-11T00:30:56.172413+0000","last_clean":"2024-09-11T00:30:56.172413+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:30:56.172413+0000","last_undegraded":"2024-09-11T00:30:56.172413+0000","last_fullsized":"2024-09-11T00:30:56.172413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332844+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:30:56.332844+0000","last_peered":"2024-09-11T00:30:56.332844+0000","last_clean":"2024-09-11T00:30:56.332844+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:30:56.332844+0000","last_undegraded":"2024-09-11T00:30:56.332844+0000","last_fullsized":"2024-09-11T00:30:56.332844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332895+0000","last_change":"2024-09-11T00:30:54.985827+0000","last_active":"2024-09-11T00:30:56.332895+0000","last_peered":"2024-09-11T00:30:56.332895+0000","last_clean":"2024-09-11T00:30:56.332895+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:30:56.332895+0000","last_undegraded":"2024-09-11T00:30:56.332895+0000","last_fullsized":"2024-09-11T00:30:56.332895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:32.027117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968441+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:30:55.968441+0000","last_peered":"2024-09-11T00:30:55.968441+0000","last_clean":"2024-09-11T00:30:55.968441+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:30:55.968441+0000","last_undegraded":"2024-09-11T00:30:55.968441+0000","last_fullsized":"2024-09-11T00:30:55.968441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172255+0000","last_change":"2024-09-11T00:30:54.989906+0000","last_active":"2024-09-11T00:30:56.172255+0000","last_peered":"2024-09-11T00:30:56.172255+0000","last_clean":"2024-09-11T00:30:56.172255+0000","last_became_active":"2024-09-11T00:30:54.989376+0000","last_became_peered":"2024-09-11T00:30:54.989376+0000","last_unstale":"2024-09-11T00:30:56.172255+0000","last_undegraded":"2024-09-11T00:30:56.172255+0000","last_fullsized":"2024-09-11T00:30:56.172255+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:18:00.138941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001665+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:30:56.001665+0000","last_peered":"2024-09-11T00:30:56.001665+0000","last_clean":"2024-09-11T00:30:56.001665+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:30:56.001665+0000","last_undegraded":"2024-09-11T00:30:56.001665+0000","last_fullsized":"2024-09-11T00:30:56.001665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968974+0000","last_change":"2024-09-11T00:30:54.993156+0000","last_active":"2024-09-11T00:30:55.968974+0000","last_peered":"2024-09-11T00:30:55.968974+0000","last_clean":"2024-09-11T00:30:55.968974+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:30:55.968974+0000","last_undegraded":"2024-09-11T00:30:55.968974+0000","last_fullsized":"2024-09-11T00:30:55.968974+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:25.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968016+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:30:55.968016+0000","last_peered":"2024-09-11T00:30:55.968016+0000","last_clean":"2024-09-11T00:30:55.968016+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:30:55.968016+0000","last_undegraded":"2024-09-11T00:30:55.968016+0000","last_fullsized":"2024-09-11T00:30:55.968016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970335+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:30:55.970335+0000","last_peered":"2024-09-11T00:30:55.970335+0000","last_clean":"2024-09-11T00:30:55.970335+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:30:55.970335+0000","last_undegraded":"2024-09-11T00:30:55.970335+0000","last_fullsized":"2024-09-11T00:30:55.970335+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966434+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:30:55.966434+0000","last_peered":"2024-09-11T00:30:55.966434+0000","last_clean":"2024-09-11T00:30:55.966434+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:30:55.966434+0000","last_undegraded":"2024-09-11T00:30:55.966434+0000","last_fullsized":"2024-09-11T00:30:55.966434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719699+0000","last_change":"2024-09-11T00:30:54.993461+0000","last_active":"2024-09-11T00:30:56.719699+0000","last_peered":"2024-09-11T00:30:56.719699+0000","last_clean":"2024-09-11T00:30:56.719699+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:30:56.719699+0000","last_undegraded":"2024-09-11T00:30:56.719699+0000","last_fullsized":"2024-09-11T00:30:56.719699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:07.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997281+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:30:55.997281+0000","last_peered":"2024-09-11T00:30:55.997281+0000","last_clean":"2024-09-11T00:30:55.997281+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:30:55.997281+0000","last_undegraded":"2024-09-11T00:30:55.997281+0000","last_fullsized":"2024-09-11T00:30:55.997281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719307+0000","last_change":"2024-09-11T00:30:54.986265+0000","last_active":"2024-09-11T00:30:56.719307+0000","last_peered":"2024-09-11T00:30:56.719307+0000","last_clean":"2024-09-11T00:30:56.719307+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:30:56.719307+0000","last_undegraded":"2024-09-11T00:30:56.719307+0000","last_fullsized":"2024-09-11T00:30:56.719307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:49.109328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969766+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:30:55.969766+0000","last_peered":"2024-09-11T00:30:55.969766+0000","last_clean":"2024-09-11T00:30:55.969766+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:30:55.969766+0000","last_undegraded":"2024-09-11T00:30:55.969766+0000","last_fullsized":"2024-09-11T00:30:55.969766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967854+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:30:55.967854+0000","last_peered":"2024-09-11T00:30:55.967854+0000","last_clean":"2024-09-11T00:30:55.967854+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:30:55.967854+0000","last_undegraded":"2024-09-11T00:30:55.967854+0000","last_fullsized":"2024-09-11T00:30:55.967854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971368+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:30:55.971368+0000","last_peered":"2024-09-11T00:30:55.971368+0000","last_clean":"2024-09-11T00:30:55.971368+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:30:55.971368+0000","last_undegraded":"2024-09-11T00:30:55.971368+0000","last_fullsized":"2024-09-11T00:30:55.971368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969452+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:30:55.969452+0000","last_peered":"2024-09-11T00:30:55.969452+0000","last_clean":"2024-09-11T00:30:55.969452+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:30:55.969452+0000","last_undegraded":"2024-09-11T00:30:55.969452+0000","last_fullsized":"2024-09-11T00:30:55.969452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332691+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:30:56.332691+0000","last_peered":"2024-09-11T00:30:56.332691+0000","last_clean":"2024-09-11T00:30:56.332691+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:30:56.332691+0000","last_undegraded":"2024-09-11T00:30:56.332691+0000","last_fullsized":"2024-09-11T00:30:56.332691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970850+0000","last_change":"2024-09-11T00:30:54.994988+0000","last_active":"2024-09-11T00:30:55.970850+0000","last_peered":"2024-09-11T00:30:55.970850+0000","last_clean":"2024-09-11T00:30:55.970850+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:30:55.970850+0000","last_undegraded":"2024-09-11T00:30:55.970850+0000","last_fullsized":"2024-09-11T00:30:55.970850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:31.973641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332924+0000","last_change":"2024-09-11T00:30:45.941635+0000","last_active":"2024-09-11T00:30:56.332924+0000","last_peered":"2024-09-11T00:30:56.332924+0000","last_clean":"2024-09-11T00:30:56.332924+0000","last_became_active":"2024-09-11T00:30:45.941357+0000","last_became_peered":"2024-09-11T00:30:45.941357+0000","last_unstale":"2024-09-11T00:30:56.332924+0000","last_undegraded":"2024-09-11T00:30:56.332924+0000","last_fullsized":"2024-09-11T00:30:56.332924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:12.016406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332421+0000","last_change":"2024-09-11T00:30:54.984767+0000","last_active":"2024-09-11T00:30:56.332421+0000","last_peered":"2024-09-11T00:30:56.332421+0000","last_clean":"2024-09-11T00:30:56.332421+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:30:56.332421+0000","last_undegraded":"2024-09-11T00:30:56.332421+0000","last_fullsized":"2024-09-11T00:30:56.332421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:44.833677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332955+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:30:56.332955+0000","last_peered":"2024-09-11T00:30:56.332955+0000","last_clean":"2024-09-11T00:30:56.332955+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:56.332955+0000","last_undegraded":"2024-09-11T00:30:56.332955+0000","last_fullsized":"2024-09-11T00:30:56.332955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332911+0000","last_change":"2024-09-11T00:30:54.986423+0000","last_active":"2024-09-11T00:30:56.332911+0000","last_peered":"2024-09-11T00:30:56.332911+0000","last_clean":"2024-09-11T00:30:56.332911+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:30:56.332911+0000","last_undegraded":"2024-09-11T00:30:56.332911+0000","last_fullsized":"2024-09-11T00:30:56.332911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:16:50.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719859+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:30:56.719859+0000","last_peered":"2024-09-11T00:30:56.719859+0000","last_clean":"2024-09-11T00:30:56.719859+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:30:56.719859+0000","last_undegraded":"2024-09-11T00:30:56.719859+0000","last_fullsized":"2024-09-11T00:30:56.719859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998062+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:30:55.998062+0000","last_peered":"2024-09-11T00:30:55.998062+0000","last_clean":"2024-09-11T00:30:55.998062+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:30:55.998062+0000","last_undegraded":"2024-09-11T00:30:55.998062+0000","last_fullsized":"2024-09-11T00:30:55.998062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997283+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:30:55.997283+0000","last_peered":"2024-09-11T00:30:55.997283+0000","last_clean":"2024-09-11T00:30:55.997283+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:30:55.997283+0000","last_undegraded":"2024-09-11T00:30:55.997283+0000","last_fullsized":"2024-09-11T00:30:55.997283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173138+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:30:56.173138+0000","last_peered":"2024-09-11T00:30:56.173138+0000","last_clean":"2024-09-11T00:30:56.173138+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:30:56.173138+0000","last_undegraded":"2024-09-11T00:30:56.173138+0000","last_fullsized":"2024-09-11T00:30:56.173138+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333239+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:30:56.333239+0000","last_peered":"2024-09-11T00:30:56.333239+0000","last_clean":"2024-09-11T00:30:56.333239+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:30:56.333239+0000","last_undegraded":"2024-09-11T00:30:56.333239+0000","last_fullsized":"2024-09-11T00:30:56.333239+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970256+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:30:55.970256+0000","last_peered":"2024-09-11T00:30:55.970256+0000","last_clean":"2024-09-11T00:30:55.970256+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:30:55.970256+0000","last_undegraded":"2024-09-11T00:30:55.970256+0000","last_fullsized":"2024-09-11T00:30:55.970256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172539+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:30:56.172539+0000","last_peered":"2024-09-11T00:30:56.172539+0000","last_clean":"2024-09-11T00:30:56.172539+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:30:56.172539+0000","last_undegraded":"2024-09-11T00:30:56.172539+0000","last_fullsized":"2024-09-11T00:30:56.172539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719531+0000","last_change":"2024-09-11T00:30:54.990631+0000","last_active":"2024-09-11T00:30:56.719531+0000","last_peered":"2024-09-11T00:30:56.719531+0000","last_clean":"2024-09-11T00:30:56.719531+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:30:56.719531+0000","last_undegraded":"2024-09-11T00:30:56.719531+0000","last_fullsized":"2024-09-11T00:30:56.719531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:02.422471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333220+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:30:56.333220+0000","last_peered":"2024-09-11T00:30:56.333220+0000","last_clean":"2024-09-11T00:30:56.333220+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:30:56.333220+0000","last_undegraded":"2024-09-11T00:30:56.333220+0000","last_fullsized":"2024-09-11T00:30:56.333220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:54.991047+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:41:48.229261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173027+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:30:56.173027+0000","last_peered":"2024-09-11T00:30:56.173027+0000","last_clean":"2024-09-11T00:30:56.173027+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:30:56.173027+0000","last_undegraded":"2024-09-11T00:30:56.173027+0000","last_fullsized":"2024-09-11T00:30:56.173027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997710+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:30:55.997710+0000","last_peered":"2024-09-11T00:30:55.997710+0000","last_clean":"2024-09-11T00:30:55.997710+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:30:55.997710+0000","last_undegraded":"2024-09-11T00:30:55.997710+0000","last_fullsized":"2024-09-11T00:30:55.997710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997737+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:30:55.997737+0000","last_peered":"2024-09-11T00:30:55.997737+0000","last_clean":"2024-09-11T00:30:55.997737+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:30:55.997737+0000","last_undegraded":"2024-09-11T00:30:55.997737+0000","last_fullsized":"2024-09-11T00:30:55.997737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172698+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:30:56.172698+0000","last_peered":"2024-09-11T00:30:56.172698+0000","last_clean":"2024-09-11T00:30:56.172698+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:30:56.172698+0000","last_undegraded":"2024-09-11T00:30:56.172698+0000","last_fullsized":"2024-09-11T00:30:56.172698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997914+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:30:55.997914+0000","last_peered":"2024-09-11T00:30:55.997914+0000","last_clean":"2024-09-11T00:30:55.997914+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:30:55.997914+0000","last_undegraded":"2024-09-11T00:30:55.997914+0000","last_fullsized":"2024-09-11T00:30:55.997914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172531+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:30:56.172531+0000","last_peered":"2024-09-11T00:30:56.172531+0000","last_clean":"2024-09-11T00:30:56.172531+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:30:56.172531+0000","last_undegraded":"2024-09-11T00:30:56.172531+0000","last_fullsized":"2024-09-11T00:30:56.172531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333081+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:30:56.333081+0000","last_peered":"2024-09-11T00:30:56.333081+0000","last_clean":"2024-09-11T00:30:56.333081+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:30:56.333081+0000","last_undegraded":"2024-09-11T00:30:56.333081+0000","last_fullsized":"2024-09-11T00:30:56.333081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509452,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509454,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509453,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509451,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":106706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509453,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":108978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:31:24.016 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-09-11T00:31:24.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:31:24.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.207+0000 7f1099c00640 1 -- 172.21.15.31:0/1807290885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1094154b60 msgr2=0x7f10940c2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:24.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.207+0000 7f1099c00640 1 --2- 172.21.15.31:0/1807290885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1094154b60 0x7f10940c2fa0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f107c009830 tx=0x7f107c02a4d0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.207+0000 7f1099c00640 1 -- 172.21.15.31:0/1807290885 shutdown_connections 2024-09-11T00:31:24.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.207+0000 7f1099c00640 1 --2- 172.21.15.31:0/1807290885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1094154b60 0x7f10940c2fa0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.207+0000 7f1099c00640 1 --2- 172.21.15.31:0/1807290885 >> v2:172.21.15.204:3300/0 conn(0x7f1094072480 0x7f1094074a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.207+0000 7f1099c00640 1 -- 172.21.15.31:0/1807290885 >> 172.21.15.31:0/1807290885 conn(0x7f10940a9750 msgr2=0x7f10940a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.207+0000 7f1099c00640 1 -- 172.21.15.31:0/1807290885 shutdown_connections 2024-09-11T00:31:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.207+0000 7f1099c00640 1 -- 172.21.15.31:0/1807290885 wait complete. 2024-09-11T00:31:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.209+0000 7f1099c00640 1 Processor -- start 2024-09-11T00:31:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.209+0000 7f1099c00640 1 -- start start 2024-09-11T00:31:24.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.210+0000 7f1099c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1094072480 0x7f10941eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:24.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.210+0000 7f1099c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10940c34e0 0x7f1094148710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:24.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.210+0000 7f1099c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1094154b60 0x7f1094148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:24.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.210+0000 7f1099c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f10940c23a0 con 0x7f1094154b60 2024-09-11T00:31:24.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.210+0000 7f1099c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1094154640 con 0x7f1094072480 2024-09-11T00:31:24.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.210+0000 7f1099c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f10940a7140 con 0x7f10940c34e0 2024-09-11T00:31:24.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.210+0000 7f1093400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1094154b60 0x7f1094148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:24.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.210+0000 7f1093400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1094154b60 0x7f1094148c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47838/0 (socket says 172.21.15.31:47838) 2024-09-11T00:31:24.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.210+0000 7f1093400640 1 -- 172.21.15.31:0/1990588087 learned_addr learned my addr 172.21.15.31:0/1990588087 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:24.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f108a000640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10940c34e0 0x7f1094148710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f1092a00640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1094072480 0x7f10941eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f1093400640 1 -- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10940c34e0 msgr2=0x7f1094148710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f1093400640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10940c34e0 0x7f1094148710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f1093400640 1 -- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1094072480 msgr2=0x7f10941eaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f1093400640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1094072480 0x7f10941eaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f1093400640 1 -- 172.21.15.31:0/1990588087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10941eb1e0 con 0x7f1094154b60 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f108a000640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10940c34e0 0x7f1094148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f1092a00640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1094072480 0x7f10941eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f1093400640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1094154b60 0x7f1094148c50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f108c00cbc0 tx=0x7f108c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f108b400640 1 -- 172.21.15.31:0/1990588087 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c016020 con 0x7f1094154b60 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f108b400640 1 -- 172.21.15.31:0/1990588087 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f108c00ae30 con 0x7f1094154b60 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f108b400640 1 -- 172.21.15.31:0/1990588087 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c012660 con 0x7f1094154b60 2024-09-11T00:31:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f1099c00640 1 -- 172.21.15.31:0/1990588087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10941dd530 con 0x7f1094154b60 2024-09-11T00:31:24.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.211+0000 7f1099c00640 1 -- 172.21.15.31:0/1990588087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10941ddaf0 con 0x7f1094154b60 2024-09-11T00:31:24.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.213+0000 7f1099c00640 1 -- 172.21.15.31:0/1990588087 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1094072860 con 0x7f1094154b60 2024-09-11T00:31:24.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.214+0000 7f108b400640 1 -- 172.21.15.31:0/1990588087 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f108c01b400 con 0x7f1094154b60 2024-09-11T00:31:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.215+0000 7f108b400640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 0x7f107007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.215+0000 7f108b400640 1 -- 172.21.15.31:0/1990588087 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f108c09a9f0 con 0x7f1094154b60 2024-09-11T00:31:24.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.215+0000 7f1092a00640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 0x7f107007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:24.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.216+0000 7f1092a00640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 0x7f107007a430 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1094155b90 tx=0x7f1084005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:24.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.219+0000 7f108b400640 1 -- 172.21.15.31:0/1990588087 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f108c020020 con 0x7f1094154b60 2024-09-11T00:31:24.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.414+0000 7f1099c00640 1 -- 172.21.15.31:0/1990588087 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f10940a6f20 con 0x7f1070077f70 2024-09-11T00:31:24.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.424+0000 7f108b400640 1 -- 172.21.15.31:0/1990588087 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490339 (secure 0 0 0) 0x7f10940a6f20 con 0x7f1070077f70 2024-09-11T00:31:24.432 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:24.435 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:31:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 -- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 msgr2=0x7f107007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 0x7f107007a430 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1094155b90 tx=0x7f1084005f90 comp rx=0 tx=0).stop 2024-09-11T00:31:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 -- 172.21.15.31:0/1990588087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1094154b60 msgr2=0x7f1094148c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1094154b60 0x7f1094148c50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f108c00cbc0 tx=0x7f108c00a430 comp rx=0 tx=0).stop 2024-09-11T00:31:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 -- 172.21.15.31:0/1990588087 shutdown_connections 2024-09-11T00:31:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 0x7f107007a430 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1094154b60 0x7f1094148c50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f10940c34e0 0x7f1094148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 --2- 172.21.15.31:0/1990588087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1094072480 0x7f10941eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 -- 172.21.15.31:0/1990588087 >> 172.21.15.31:0/1990588087 conn(0x7f10940a9750 msgr2=0x7f109414ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:24.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 -- 172.21.15.31:0/1990588087 shutdown_connections 2024-09-11T00:31:24.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.429+0000 7f1099c00640 1 -- 172.21.15.31:0/1990588087 wait complete. 2024-09-11T00:31:24.448 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-11T00:31:24.402108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597700,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":197,"num_write_kb":1662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219620,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755100,"statfs":{"total":773094113280,"available":772869222400,"internally_reserved":0,"allocated":3481600,"data_stored":2032352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003554"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719046+0000","last_change":"2024-09-11T00:30:54.983412+0000","last_active":"2024-09-11T00:30:56.719046+0000","last_peered":"2024-09-11T00:30:56.719046+0000","last_clean":"2024-09-11T00:30:56.719046+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:30:56.719046+0000","last_undegraded":"2024-09-11T00:30:56.719046+0000","last_fullsized":"2024-09-11T00:30:56.719046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:51.807846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970097+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:30:55.970097+0000","last_peered":"2024-09-11T00:30:55.970097+0000","last_clean":"2024-09-11T00:30:55.970097+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:30:55.970097+0000","last_undegraded":"2024-09-11T00:30:55.970097+0000","last_fullsized":"2024-09-11T00:30:55.970097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998047+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:30:55.998047+0000","last_peered":"2024-09-11T00:30:55.998047+0000","last_clean":"2024-09-11T00:30:55.998047+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:30:55.998047+0000","last_undegraded":"2024-09-11T00:30:55.998047+0000","last_fullsized":"2024-09-11T00:30:55.998047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000942+0000","last_change":"2024-09-11T00:30:53.984316+0000","last_active":"2024-09-11T00:30:56.000942+0000","last_peered":"2024-09-11T00:30:56.000942+0000","last_clean":"2024-09-11T00:30:56.000942+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:30:56.000942+0000","last_undegraded":"2024-09-11T00:30:56.000942+0000","last_fullsized":"2024-09-11T00:30:56.000942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:14.425265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968662+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:30:55.968662+0000","last_peered":"2024-09-11T00:30:55.968662+0000","last_clean":"2024-09-11T00:30:55.968662+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:30:55.968662+0000","last_undegraded":"2024-09-11T00:30:55.968662+0000","last_fullsized":"2024-09-11T00:30:55.968662+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998255+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:30:55.998255+0000","last_peered":"2024-09-11T00:30:55.998255+0000","last_clean":"2024-09-11T00:30:55.998255+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:30:55.998255+0000","last_undegraded":"2024-09-11T00:30:55.998255+0000","last_fullsized":"2024-09-11T00:30:55.998255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998074+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:30:55.998074+0000","last_peered":"2024-09-11T00:30:55.998074+0000","last_clean":"2024-09-11T00:30:55.998074+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:30:55.998074+0000","last_undegraded":"2024-09-11T00:30:55.998074+0000","last_fullsized":"2024-09-11T00:30:55.998074+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172549+0000","last_change":"2024-09-11T00:30:53.984940+0000","last_active":"2024-09-11T00:30:56.172549+0000","last_peered":"2024-09-11T00:30:56.172549+0000","last_clean":"2024-09-11T00:30:56.172549+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:30:56.172549+0000","last_undegraded":"2024-09-11T00:30:56.172549+0000","last_fullsized":"2024-09-11T00:30:56.172549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:19.647165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719205+0000","last_change":"2024-09-11T00:30:54.993570+0000","last_active":"2024-09-11T00:30:56.719205+0000","last_peered":"2024-09-11T00:30:56.719205+0000","last_clean":"2024-09-11T00:30:56.719205+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:30:56.719205+0000","last_undegraded":"2024-09-11T00:30:56.719205+0000","last_fullsized":"2024-09-11T00:30:56.719205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:45.211259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333273+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:30:56.333273+0000","last_peered":"2024-09-11T00:30:56.333273+0000","last_clean":"2024-09-11T00:30:56.333273+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:30:56.333273+0000","last_undegraded":"2024-09-11T00:30:56.333273+0000","last_fullsized":"2024-09-11T00:30:56.333273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719392+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:30:56.719392+0000","last_peered":"2024-09-11T00:30:56.719392+0000","last_clean":"2024-09-11T00:30:56.719392+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:30:56.719392+0000","last_undegraded":"2024-09-11T00:30:56.719392+0000","last_fullsized":"2024-09-11T00:30:56.719392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172696+0000","last_change":"2024-09-11T00:30:53.974699+0000","last_active":"2024-09-11T00:30:56.172696+0000","last_peered":"2024-09-11T00:30:56.172696+0000","last_clean":"2024-09-11T00:30:56.172696+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:30:56.172696+0000","last_undegraded":"2024-09-11T00:30:56.172696+0000","last_fullsized":"2024-09-11T00:30:56.172696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:28.792487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172498+0000","last_change":"2024-09-11T00:30:54.984890+0000","last_active":"2024-09-11T00:30:56.172498+0000","last_peered":"2024-09-11T00:30:56.172498+0000","last_clean":"2024-09-11T00:30:56.172498+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:30:56.172498+0000","last_undegraded":"2024-09-11T00:30:56.172498+0000","last_fullsized":"2024-09-11T00:30:56.172498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:21.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333166+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:30:56.333166+0000","last_peered":"2024-09-11T00:30:56.333166+0000","last_clean":"2024-09-11T00:30:56.333166+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:30:56.333166+0000","last_undegraded":"2024-09-11T00:30:56.333166+0000","last_fullsized":"2024-09-11T00:30:56.333166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968144+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:30:55.968144+0000","last_peered":"2024-09-11T00:30:55.968144+0000","last_clean":"2024-09-11T00:30:55.968144+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:30:55.968144+0000","last_undegraded":"2024-09-11T00:30:55.968144+0000","last_fullsized":"2024-09-11T00:30:55.968144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172339+0000","last_change":"2024-09-11T00:30:53.977743+0000","last_active":"2024-09-11T00:30:56.172339+0000","last_peered":"2024-09-11T00:30:56.172339+0000","last_clean":"2024-09-11T00:30:56.172339+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:30:56.172339+0000","last_undegraded":"2024-09-11T00:30:56.172339+0000","last_fullsized":"2024-09-11T00:30:56.172339+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:20.618615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997709+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:30:55.997709+0000","last_peered":"2024-09-11T00:30:55.997709+0000","last_clean":"2024-09-11T00:30:55.997709+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:30:55.997709+0000","last_undegraded":"2024-09-11T00:30:55.997709+0000","last_fullsized":"2024-09-11T00:30:55.997709+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967362+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:30:55.967362+0000","last_peered":"2024-09-11T00:30:55.967362+0000","last_clean":"2024-09-11T00:30:55.967362+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:30:55.967362+0000","last_undegraded":"2024-09-11T00:30:55.967362+0000","last_fullsized":"2024-09-11T00:30:55.967362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967452+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:30:55.967452+0000","last_peered":"2024-09-11T00:30:55.967452+0000","last_clean":"2024-09-11T00:30:55.967452+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:30:55.967452+0000","last_undegraded":"2024-09-11T00:30:55.967452+0000","last_fullsized":"2024-09-11T00:30:55.967452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967914+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:30:55.967914+0000","last_peered":"2024-09-11T00:30:55.967914+0000","last_clean":"2024-09-11T00:30:55.967914+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:30:55.967914+0000","last_undegraded":"2024-09-11T00:30:55.967914+0000","last_fullsized":"2024-09-11T00:30:55.967914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001419+0000","last_change":"2024-09-11T00:30:54.997397+0000","last_active":"2024-09-11T00:30:56.001419+0000","last_peered":"2024-09-11T00:30:56.001419+0000","last_clean":"2024-09-11T00:30:56.001419+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:30:56.001419+0000","last_undegraded":"2024-09-11T00:30:56.001419+0000","last_fullsized":"2024-09-11T00:30:56.001419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:53.322576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001116+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:30:56.001116+0000","last_peered":"2024-09-11T00:30:56.001116+0000","last_clean":"2024-09-11T00:30:56.001116+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:30:56.001116+0000","last_undegraded":"2024-09-11T00:30:56.001116+0000","last_fullsized":"2024-09-11T00:30:56.001116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970056+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:30:55.970056+0000","last_peered":"2024-09-11T00:30:55.970056+0000","last_clean":"2024-09-11T00:30:55.970056+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:30:55.970056+0000","last_undegraded":"2024-09-11T00:30:55.970056+0000","last_fullsized":"2024-09-11T00:30:55.970056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332817+0000","last_change":"2024-09-11T00:30:53.986858+0000","last_active":"2024-09-11T00:30:56.332817+0000","last_peered":"2024-09-11T00:30:56.332817+0000","last_clean":"2024-09-11T00:30:56.332817+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:30:56.332817+0000","last_undegraded":"2024-09-11T00:30:56.332817+0000","last_fullsized":"2024-09-11T00:30:56.332817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:33.626193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967911+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:30:55.967911+0000","last_peered":"2024-09-11T00:30:55.967911+0000","last_clean":"2024-09-11T00:30:55.967911+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:30:55.967911+0000","last_undegraded":"2024-09-11T00:30:55.967911+0000","last_fullsized":"2024-09-11T00:30:55.967911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968190+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:30:55.968190+0000","last_peered":"2024-09-11T00:30:55.968190+0000","last_clean":"2024-09-11T00:30:55.968190+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:30:55.968190+0000","last_undegraded":"2024-09-11T00:30:55.968190+0000","last_fullsized":"2024-09-11T00:30:55.968190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968850+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:30:55.968850+0000","last_peered":"2024-09-11T00:30:55.968850+0000","last_clean":"2024-09-11T00:30:55.968850+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:30:55.968850+0000","last_undegraded":"2024-09-11T00:30:55.968850+0000","last_fullsized":"2024-09-11T00:30:55.968850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001447+0000","last_change":"2024-09-11T00:30:53.990765+0000","last_active":"2024-09-11T00:30:56.001447+0000","last_peered":"2024-09-11T00:30:56.001447+0000","last_clean":"2024-09-11T00:30:56.001447+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:30:56.001447+0000","last_undegraded":"2024-09-11T00:30:56.001447+0000","last_fullsized":"2024-09-11T00:30:56.001447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:33.720727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719722+0000","last_change":"2024-09-11T00:30:54.984949+0000","last_active":"2024-09-11T00:30:56.719722+0000","last_peered":"2024-09-11T00:30:56.719722+0000","last_clean":"2024-09-11T00:30:56.719722+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:30:56.719722+0000","last_undegraded":"2024-09-11T00:30:56.719722+0000","last_fullsized":"2024-09-11T00:30:56.719722+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:54.346937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967033+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:30:55.967033+0000","last_peered":"2024-09-11T00:30:55.967033+0000","last_clean":"2024-09-11T00:30:55.967033+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:30:55.967033+0000","last_undegraded":"2024-09-11T00:30:55.967033+0000","last_fullsized":"2024-09-11T00:30:55.967033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332357+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:30:56.332357+0000","last_peered":"2024-09-11T00:30:56.332357+0000","last_clean":"2024-09-11T00:30:56.332357+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:30:56.332357+0000","last_undegraded":"2024-09-11T00:30:56.332357+0000","last_fullsized":"2024-09-11T00:30:56.332357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172821+0000","last_change":"2024-09-11T00:30:53.986940+0000","last_active":"2024-09-11T00:30:56.172821+0000","last_peered":"2024-09-11T00:30:56.172821+0000","last_clean":"2024-09-11T00:30:56.172821+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:30:56.172821+0000","last_undegraded":"2024-09-11T00:30:56.172821+0000","last_fullsized":"2024-09-11T00:30:56.172821+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:11.942187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968462+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:30:55.968462+0000","last_peered":"2024-09-11T00:30:55.968462+0000","last_clean":"2024-09-11T00:30:55.968462+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:30:55.968462+0000","last_undegraded":"2024-09-11T00:30:55.968462+0000","last_fullsized":"2024-09-11T00:30:55.968462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968110+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:30:55.968110+0000","last_peered":"2024-09-11T00:30:55.968110+0000","last_clean":"2024-09-11T00:30:55.968110+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:30:55.968110+0000","last_undegraded":"2024-09-11T00:30:55.968110+0000","last_fullsized":"2024-09-11T00:30:55.968110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172483+0000","last_change":"2024-09-11T00:30:53.988539+0000","last_active":"2024-09-11T00:30:56.172483+0000","last_peered":"2024-09-11T00:30:56.172483+0000","last_clean":"2024-09-11T00:30:56.172483+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:30:56.172483+0000","last_undegraded":"2024-09-11T00:30:56.172483+0000","last_fullsized":"2024-09-11T00:30:56.172483+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:19.083664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966824+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:30:55.966824+0000","last_peered":"2024-09-11T00:30:55.966824+0000","last_clean":"2024-09-11T00:30:55.966824+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:30:55.966824+0000","last_undegraded":"2024-09-11T00:30:55.966824+0000","last_fullsized":"2024-09-11T00:30:55.966824+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970081+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:30:55.970081+0000","last_peered":"2024-09-11T00:30:55.970081+0000","last_clean":"2024-09-11T00:30:55.970081+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:30:55.970081+0000","last_undegraded":"2024-09-11T00:30:55.970081+0000","last_fullsized":"2024-09-11T00:30:55.970081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970376+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:30:55.970376+0000","last_peered":"2024-09-11T00:30:55.970376+0000","last_clean":"2024-09-11T00:30:55.970376+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:30:55.970376+0000","last_undegraded":"2024-09-11T00:30:55.970376+0000","last_fullsized":"2024-09-11T00:30:55.970376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968131+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:30:55.968131+0000","last_peered":"2024-09-11T00:30:55.968131+0000","last_clean":"2024-09-11T00:30:55.968131+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:30:55.968131+0000","last_undegraded":"2024-09-11T00:30:55.968131+0000","last_fullsized":"2024-09-11T00:30:55.968131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332511+0000","last_change":"2024-09-11T00:30:54.993118+0000","last_active":"2024-09-11T00:30:56.332511+0000","last_peered":"2024-09-11T00:30:56.332511+0000","last_clean":"2024-09-11T00:30:56.332511+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:30:56.332511+0000","last_undegraded":"2024-09-11T00:30:56.332511+0000","last_fullsized":"2024-09-11T00:30:56.332511+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:07:41.148370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971378+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:30:55.971378+0000","last_peered":"2024-09-11T00:30:55.971378+0000","last_clean":"2024-09-11T00:30:55.971378+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:30:55.971378+0000","last_undegraded":"2024-09-11T00:30:55.971378+0000","last_fullsized":"2024-09-11T00:30:55.971378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968486+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:30:55.968486+0000","last_peered":"2024-09-11T00:30:55.968486+0000","last_clean":"2024-09-11T00:30:55.968486+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:30:55.968486+0000","last_undegraded":"2024-09-11T00:30:55.968486+0000","last_fullsized":"2024-09-11T00:30:55.968486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970903+0000","last_change":"2024-09-11T00:30:53.986843+0000","last_active":"2024-09-11T00:30:55.970903+0000","last_peered":"2024-09-11T00:30:55.970903+0000","last_clean":"2024-09-11T00:30:55.970903+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:30:55.970903+0000","last_undegraded":"2024-09-11T00:30:55.970903+0000","last_fullsized":"2024-09-11T00:30:55.970903+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:44.684154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969549+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:30:55.969549+0000","last_peered":"2024-09-11T00:30:55.969549+0000","last_clean":"2024-09-11T00:30:55.969549+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:30:55.969549+0000","last_undegraded":"2024-09-11T00:30:55.969549+0000","last_fullsized":"2024-09-11T00:30:55.969549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172919+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:30:56.172919+0000","last_peered":"2024-09-11T00:30:56.172919+0000","last_clean":"2024-09-11T00:30:56.172919+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:30:56.172919+0000","last_undegraded":"2024-09-11T00:30:56.172919+0000","last_fullsized":"2024-09-11T00:30:56.172919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997652+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:30:55.997652+0000","last_peered":"2024-09-11T00:30:55.997652+0000","last_clean":"2024-09-11T00:30:55.997652+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:30:55.997652+0000","last_undegraded":"2024-09-11T00:30:55.997652+0000","last_fullsized":"2024-09-11T00:30:55.997652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969390+0000","last_change":"2024-09-11T00:30:53.986896+0000","last_active":"2024-09-11T00:30:55.969390+0000","last_peered":"2024-09-11T00:30:55.969390+0000","last_clean":"2024-09-11T00:30:55.969390+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:30:55.969390+0000","last_undegraded":"2024-09-11T00:30:55.969390+0000","last_fullsized":"2024-09-11T00:30:55.969390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:06.590278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001631+0000","last_change":"2024-09-11T00:30:55.001370+0000","last_active":"2024-09-11T00:30:56.001631+0000","last_peered":"2024-09-11T00:30:56.001631+0000","last_clean":"2024-09-11T00:30:56.001631+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:30:56.001631+0000","last_undegraded":"2024-09-11T00:30:56.001631+0000","last_fullsized":"2024-09-11T00:30:56.001631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:56.592767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333262+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:30:56.333262+0000","last_peered":"2024-09-11T00:30:56.333262+0000","last_clean":"2024-09-11T00:30:56.333262+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:30:56.333262+0000","last_undegraded":"2024-09-11T00:30:56.333262+0000","last_fullsized":"2024-09-11T00:30:56.333262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967594+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:30:55.967594+0000","last_peered":"2024-09-11T00:30:55.967594+0000","last_clean":"2024-09-11T00:30:55.967594+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:30:55.967594+0000","last_undegraded":"2024-09-11T00:30:55.967594+0000","last_fullsized":"2024-09-11T00:30:55.967594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971192+0000","last_change":"2024-09-11T00:30:53.978480+0000","last_active":"2024-09-11T00:30:55.971192+0000","last_peered":"2024-09-11T00:30:55.971192+0000","last_clean":"2024-09-11T00:30:55.971192+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:30:55.971192+0000","last_undegraded":"2024-09-11T00:30:55.971192+0000","last_fullsized":"2024-09-11T00:30:55.971192+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:58.900565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001528+0000","last_change":"2024-09-11T00:30:54.997641+0000","last_active":"2024-09-11T00:30:56.001528+0000","last_peered":"2024-09-11T00:30:56.001528+0000","last_clean":"2024-09-11T00:30:56.001528+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:30:56.001528+0000","last_undegraded":"2024-09-11T00:30:56.001528+0000","last_fullsized":"2024-09-11T00:30:56.001528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:18.437490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997311+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:30:55.997311+0000","last_peered":"2024-09-11T00:30:55.997311+0000","last_clean":"2024-09-11T00:30:55.997311+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:30:55.997311+0000","last_undegraded":"2024-09-11T00:30:55.997311+0000","last_fullsized":"2024-09-11T00:30:55.997311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971163+0000","last_change":"2024-09-11T00:30:53.979031+0000","last_active":"2024-09-11T00:30:55.971163+0000","last_peered":"2024-09-11T00:30:55.971163+0000","last_clean":"2024-09-11T00:30:55.971163+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:30:55.971163+0000","last_undegraded":"2024-09-11T00:30:55.971163+0000","last_fullsized":"2024-09-11T00:30:55.971163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:36.604377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967699+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:30:55.967699+0000","last_peered":"2024-09-11T00:30:55.967699+0000","last_clean":"2024-09-11T00:30:55.967699+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:30:55.967699+0000","last_undegraded":"2024-09-11T00:30:55.967699+0000","last_fullsized":"2024-09-11T00:30:55.967699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:39.629122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968089+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:30:55.968089+0000","last_peered":"2024-09-11T00:30:55.968089+0000","last_clean":"2024-09-11T00:30:55.968089+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:30:55.968089+0000","last_undegraded":"2024-09-11T00:30:55.968089+0000","last_fullsized":"2024-09-11T00:30:55.968089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966595+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:30:55.966595+0000","last_peered":"2024-09-11T00:30:55.966595+0000","last_clean":"2024-09-11T00:30:55.966595+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:30:55.966595+0000","last_undegraded":"2024-09-11T00:30:55.966595+0000","last_fullsized":"2024-09-11T00:30:55.966595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"26'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998301+0000","last_change":"2024-09-11T00:30:53.989299+0000","last_active":"2024-09-11T00:30:55.998301+0000","last_peered":"2024-09-11T00:30:55.998301+0000","last_clean":"2024-09-11T00:30:55.998301+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:30:55.998301+0000","last_undegraded":"2024-09-11T00:30:55.998301+0000","last_fullsized":"2024-09-11T00:30:55.998301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:34:49.298915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332781+0000","last_change":"2024-09-11T00:30:54.990026+0000","last_active":"2024-09-11T00:30:56.332781+0000","last_peered":"2024-09-11T00:30:56.332781+0000","last_clean":"2024-09-11T00:30:56.332781+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:30:56.332781+0000","last_undegraded":"2024-09-11T00:30:56.332781+0000","last_fullsized":"2024-09-11T00:30:56.332781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:02:10.800530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969545+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:30:55.969545+0000","last_peered":"2024-09-11T00:30:55.969545+0000","last_clean":"2024-09-11T00:30:55.969545+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:30:55.969545+0000","last_undegraded":"2024-09-11T00:30:55.969545+0000","last_fullsized":"2024-09-11T00:30:55.969545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967491+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:30:55.967491+0000","last_peered":"2024-09-11T00:30:55.967491+0000","last_clean":"2024-09-11T00:30:55.967491+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:30:55.967491+0000","last_undegraded":"2024-09-11T00:30:55.967491+0000","last_fullsized":"2024-09-11T00:30:55.967491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719269+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:30:56.719269+0000","last_peered":"2024-09-11T00:30:56.719269+0000","last_clean":"2024-09-11T00:30:56.719269+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:30:56.719269+0000","last_undegraded":"2024-09-11T00:30:56.719269+0000","last_fullsized":"2024-09-11T00:30:56.719269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968267+0000","last_change":"2024-09-11T00:30:53.982142+0000","last_active":"2024-09-11T00:30:55.968267+0000","last_peered":"2024-09-11T00:30:55.968267+0000","last_clean":"2024-09-11T00:30:55.968267+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:30:55.968267+0000","last_undegraded":"2024-09-11T00:30:55.968267+0000","last_fullsized":"2024-09-11T00:30:55.968267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:41.558944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998013+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:30:55.998013+0000","last_peered":"2024-09-11T00:30:55.998013+0000","last_clean":"2024-09-11T00:30:55.998013+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:30:55.998013+0000","last_undegraded":"2024-09-11T00:30:55.998013+0000","last_fullsized":"2024-09-11T00:30:55.998013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968362+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:30:55.968362+0000","last_peered":"2024-09-11T00:30:55.968362+0000","last_clean":"2024-09-11T00:30:55.968362+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:30:55.968362+0000","last_undegraded":"2024-09-11T00:30:55.968362+0000","last_fullsized":"2024-09-11T00:30:55.968362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971436+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:30:55.971436+0000","last_peered":"2024-09-11T00:30:55.971436+0000","last_clean":"2024-09-11T00:30:55.971436+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:30:55.971436+0000","last_undegraded":"2024-09-11T00:30:55.971436+0000","last_fullsized":"2024-09-11T00:30:55.971436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172455+0000","last_change":"2024-09-11T00:30:53.989292+0000","last_active":"2024-09-11T00:30:56.172455+0000","last_peered":"2024-09-11T00:30:56.172455+0000","last_clean":"2024-09-11T00:30:56.172455+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:30:56.172455+0000","last_undegraded":"2024-09-11T00:30:56.172455+0000","last_fullsized":"2024-09-11T00:30:56.172455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:44.071338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968473+0000","last_change":"2024-09-11T00:30:54.981630+0000","last_active":"2024-09-11T00:30:55.968473+0000","last_peered":"2024-09-11T00:30:55.968473+0000","last_clean":"2024-09-11T00:30:55.968473+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:30:55.968473+0000","last_undegraded":"2024-09-11T00:30:55.968473+0000","last_fullsized":"2024-09-11T00:30:55.968473+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:12.832327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333061+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:30:56.333061+0000","last_peered":"2024-09-11T00:30:56.333061+0000","last_clean":"2024-09-11T00:30:56.333061+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:30:56.333061+0000","last_undegraded":"2024-09-11T00:30:56.333061+0000","last_fullsized":"2024-09-11T00:30:56.333061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997856+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:30:55.997856+0000","last_peered":"2024-09-11T00:30:55.997856+0000","last_clean":"2024-09-11T00:30:55.997856+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:30:55.997856+0000","last_undegraded":"2024-09-11T00:30:55.997856+0000","last_fullsized":"2024-09-11T00:30:55.997856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997848+0000","last_change":"2024-09-11T00:30:53.984013+0000","last_active":"2024-09-11T00:30:55.997848+0000","last_peered":"2024-09-11T00:30:55.997848+0000","last_clean":"2024-09-11T00:30:55.997848+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:30:55.997848+0000","last_undegraded":"2024-09-11T00:30:55.997848+0000","last_fullsized":"2024-09-11T00:30:55.997848+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:49.963623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719176+0000","last_change":"2024-09-11T00:30:54.996860+0000","last_active":"2024-09-11T00:30:56.719176+0000","last_peered":"2024-09-11T00:30:56.719176+0000","last_clean":"2024-09-11T00:30:56.719176+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:30:56.719176+0000","last_undegraded":"2024-09-11T00:30:56.719176+0000","last_fullsized":"2024-09-11T00:30:56.719176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:43.516748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333114+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:30:56.333114+0000","last_peered":"2024-09-11T00:30:56.333114+0000","last_clean":"2024-09-11T00:30:56.333114+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:30:56.333114+0000","last_undegraded":"2024-09-11T00:30:56.333114+0000","last_fullsized":"2024-09-11T00:30:56.333114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970314+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:30:55.970314+0000","last_peered":"2024-09-11T00:30:55.970314+0000","last_clean":"2024-09-11T00:30:55.970314+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:30:55.970314+0000","last_undegraded":"2024-09-11T00:30:55.970314+0000","last_fullsized":"2024-09-11T00:30:55.970314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966317+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:30:55.966317+0000","last_peered":"2024-09-11T00:30:55.966317+0000","last_clean":"2024-09-11T00:30:55.966317+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:30:55.966317+0000","last_undegraded":"2024-09-11T00:30:55.966317+0000","last_fullsized":"2024-09-11T00:30:55.966317+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:19.891052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967870+0000","last_change":"2024-09-11T00:30:54.993736+0000","last_active":"2024-09-11T00:30:55.967870+0000","last_peered":"2024-09-11T00:30:55.967870+0000","last_clean":"2024-09-11T00:30:55.967870+0000","last_became_active":"2024-09-11T00:30:54.993470+0000","last_became_peered":"2024-09-11T00:30:54.993470+0000","last_unstale":"2024-09-11T00:30:55.967870+0000","last_undegraded":"2024-09-11T00:30:55.967870+0000","last_fullsized":"2024-09-11T00:30:55.967870+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:15:24.712448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173170+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:30:56.173170+0000","last_peered":"2024-09-11T00:30:56.173170+0000","last_clean":"2024-09-11T00:30:56.173170+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:30:56.173170+0000","last_undegraded":"2024-09-11T00:30:56.173170+0000","last_fullsized":"2024-09-11T00:30:56.173170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333218+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:30:56.333218+0000","last_peered":"2024-09-11T00:30:56.333218+0000","last_clean":"2024-09-11T00:30:56.333218+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:30:56.333218+0000","last_undegraded":"2024-09-11T00:30:56.333218+0000","last_fullsized":"2024-09-11T00:30:56.333218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719247+0000","last_change":"2024-09-11T00:30:53.983352+0000","last_active":"2024-09-11T00:30:56.719247+0000","last_peered":"2024-09-11T00:30:56.719247+0000","last_clean":"2024-09-11T00:30:56.719247+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:30:56.719247+0000","last_undegraded":"2024-09-11T00:30:56.719247+0000","last_fullsized":"2024-09-11T00:30:56.719247+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:43.826114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970030+0000","last_change":"2024-09-11T00:30:54.987533+0000","last_active":"2024-09-11T00:30:55.970030+0000","last_peered":"2024-09-11T00:30:55.970030+0000","last_clean":"2024-09-11T00:30:55.970030+0000","last_became_active":"2024-09-11T00:30:54.987253+0000","last_became_peered":"2024-09-11T00:30:54.987253+0000","last_unstale":"2024-09-11T00:30:55.970030+0000","last_undegraded":"2024-09-11T00:30:55.970030+0000","last_fullsized":"2024-09-11T00:30:55.970030+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.923656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966934+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:30:55.966934+0000","last_peered":"2024-09-11T00:30:55.966934+0000","last_clean":"2024-09-11T00:30:55.966934+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:30:55.966934+0000","last_undegraded":"2024-09-11T00:30:55.966934+0000","last_fullsized":"2024-09-11T00:30:55.966934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968528+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:30:55.968528+0000","last_peered":"2024-09-11T00:30:55.968528+0000","last_clean":"2024-09-11T00:30:55.968528+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:30:55.968528+0000","last_undegraded":"2024-09-11T00:30:55.968528+0000","last_fullsized":"2024-09-11T00:30:55.968528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997651+0000","last_change":"2024-09-11T00:30:53.980703+0000","last_active":"2024-09-11T00:30:55.997651+0000","last_peered":"2024-09-11T00:30:55.997651+0000","last_clean":"2024-09-11T00:30:55.997651+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:30:55.997651+0000","last_undegraded":"2024-09-11T00:30:55.997651+0000","last_fullsized":"2024-09-11T00:30:55.997651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:51.050512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967820+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:30:55.967820+0000","last_peered":"2024-09-11T00:30:55.967820+0000","last_clean":"2024-09-11T00:30:55.967820+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:30:55.967820+0000","last_undegraded":"2024-09-11T00:30:55.967820+0000","last_fullsized":"2024-09-11T00:30:55.967820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719139+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:30:56.719139+0000","last_peered":"2024-09-11T00:30:56.719139+0000","last_clean":"2024-09-11T00:30:56.719139+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:30:56.719139+0000","last_undegraded":"2024-09-11T00:30:56.719139+0000","last_fullsized":"2024-09-11T00:30:56.719139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333070+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:30:56.333070+0000","last_peered":"2024-09-11T00:30:56.333070+0000","last_clean":"2024-09-11T00:30:56.333070+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:30:56.333070+0000","last_undegraded":"2024-09-11T00:30:56.333070+0000","last_fullsized":"2024-09-11T00:30:56.333070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997937+0000","last_change":"2024-09-11T00:30:53.987829+0000","last_active":"2024-09-11T00:30:55.997937+0000","last_peered":"2024-09-11T00:30:55.997937+0000","last_clean":"2024-09-11T00:30:55.997937+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:30:55.997937+0000","last_undegraded":"2024-09-11T00:30:55.997937+0000","last_fullsized":"2024-09-11T00:30:55.997937+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:57:43.792089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967657+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:30:55.967657+0000","last_peered":"2024-09-11T00:30:55.967657+0000","last_clean":"2024-09-11T00:30:55.967657+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:30:55.967657+0000","last_undegraded":"2024-09-11T00:30:55.967657+0000","last_fullsized":"2024-09-11T00:30:55.967657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967382+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:30:55.967382+0000","last_peered":"2024-09-11T00:30:55.967382+0000","last_clean":"2024-09-11T00:30:55.967382+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:30:55.967382+0000","last_undegraded":"2024-09-11T00:30:55.967382+0000","last_fullsized":"2024-09-11T00:30:55.967382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967492+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:30:55.967492+0000","last_peered":"2024-09-11T00:30:55.967492+0000","last_clean":"2024-09-11T00:30:55.967492+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:30:55.967492+0000","last_undegraded":"2024-09-11T00:30:55.967492+0000","last_fullsized":"2024-09-11T00:30:55.967492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"26'9","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:31:18.986101+0000","last_change":"2024-09-11T00:30:53.984057+0000","last_active":"2024-09-11T00:31:18.986101+0000","last_peered":"2024-09-11T00:31:18.986101+0000","last_clean":"2024-09-11T00:31:18.986101+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:31:18.986101+0000","last_undegraded":"2024-09-11T00:31:18.986101+0000","last_fullsized":"2024-09-11T00:31:18.986101+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:10.477916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969373+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:30:55.969373+0000","last_peered":"2024-09-11T00:30:55.969373+0000","last_clean":"2024-09-11T00:30:55.969373+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:30:55.969373+0000","last_undegraded":"2024-09-11T00:30:55.969373+0000","last_fullsized":"2024-09-11T00:30:55.969373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969960+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:30:55.969960+0000","last_peered":"2024-09-11T00:30:55.969960+0000","last_clean":"2024-09-11T00:30:55.969960+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:30:55.969960+0000","last_undegraded":"2024-09-11T00:30:55.969960+0000","last_fullsized":"2024-09-11T00:30:55.969960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001240+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:30:56.001240+0000","last_peered":"2024-09-11T00:30:56.001240+0000","last_clean":"2024-09-11T00:30:56.001240+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:30:56.001240+0000","last_undegraded":"2024-09-11T00:30:56.001240+0000","last_fullsized":"2024-09-11T00:30:56.001240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968074+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:30:55.968074+0000","last_peered":"2024-09-11T00:30:55.968074+0000","last_clean":"2024-09-11T00:30:55.968074+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:30:55.968074+0000","last_undegraded":"2024-09-11T00:30:55.968074+0000","last_fullsized":"2024-09-11T00:30:55.968074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969893+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:30:55.969893+0000","last_peered":"2024-09-11T00:30:55.969893+0000","last_clean":"2024-09-11T00:30:55.969893+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:30:55.969893+0000","last_undegraded":"2024-09-11T00:30:55.969893+0000","last_fullsized":"2024-09-11T00:30:55.969893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332978+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:30:56.332978+0000","last_peered":"2024-09-11T00:30:56.332978+0000","last_clean":"2024-09-11T00:30:56.332978+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:30:56.332978+0000","last_undegraded":"2024-09-11T00:30:56.332978+0000","last_fullsized":"2024-09-11T00:30:56.332978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'9","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:31:14.475344+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:31:14.475344+0000","last_peered":"2024-09-11T00:31:14.475344+0000","last_clean":"2024-09-11T00:31:14.475344+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:31:14.475344+0000","last_undegraded":"2024-09-11T00:31:14.475344+0000","last_fullsized":"2024-09-11T00:31:14.475344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2868,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969278+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:30:55.969278+0000","last_peered":"2024-09-11T00:30:55.969278+0000","last_clean":"2024-09-11T00:30:55.969278+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:30:55.969278+0000","last_undegraded":"2024-09-11T00:30:55.969278+0000","last_fullsized":"2024-09-11T00:30:55.969278+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969704+0000","last_change":"2024-09-11T00:30:54.986017+0000","last_active":"2024-09-11T00:30:55.969704+0000","last_peered":"2024-09-11T00:30:55.969704+0000","last_clean":"2024-09-11T00:30:55.969704+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:30:55.969704+0000","last_undegraded":"2024-09-11T00:30:55.969704+0000","last_fullsized":"2024-09-11T00:30:55.969704+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:58:12.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967364+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:30:55.967364+0000","last_peered":"2024-09-11T00:30:55.967364+0000","last_clean":"2024-09-11T00:30:55.967364+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:30:55.967364+0000","last_undegraded":"2024-09-11T00:30:55.967364+0000","last_fullsized":"2024-09-11T00:30:55.967364+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001702+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:30:56.001702+0000","last_peered":"2024-09-11T00:30:56.001702+0000","last_clean":"2024-09-11T00:30:56.001702+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:30:56.001702+0000","last_undegraded":"2024-09-11T00:30:56.001702+0000","last_fullsized":"2024-09-11T00:30:56.001702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172588+0000","last_change":"2024-09-11T00:30:53.988462+0000","last_active":"2024-09-11T00:30:56.172588+0000","last_peered":"2024-09-11T00:30:56.172588+0000","last_clean":"2024-09-11T00:30:56.172588+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:30:56.172588+0000","last_undegraded":"2024-09-11T00:30:56.172588+0000","last_fullsized":"2024-09-11T00:30:56.172588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:22.385557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966521+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:30:55.966521+0000","last_peered":"2024-09-11T00:30:55.966521+0000","last_clean":"2024-09-11T00:30:55.966521+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:30:55.966521+0000","last_undegraded":"2024-09-11T00:30:55.966521+0000","last_fullsized":"2024-09-11T00:30:55.966521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998206+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:30:55.998206+0000","last_peered":"2024-09-11T00:30:55.998206+0000","last_clean":"2024-09-11T00:30:55.998206+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:30:55.998206+0000","last_undegraded":"2024-09-11T00:30:55.998206+0000","last_fullsized":"2024-09-11T00:30:55.998206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332956+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:30:56.332956+0000","last_peered":"2024-09-11T00:30:56.332956+0000","last_clean":"2024-09-11T00:30:56.332956+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:30:56.332956+0000","last_undegraded":"2024-09-11T00:30:56.332956+0000","last_fullsized":"2024-09-11T00:30:56.332956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000726+0000","last_change":"2024-09-11T00:30:53.986646+0000","last_active":"2024-09-11T00:30:56.000726+0000","last_peered":"2024-09-11T00:30:56.000726+0000","last_clean":"2024-09-11T00:30:56.000726+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:30:56.000726+0000","last_undegraded":"2024-09-11T00:30:56.000726+0000","last_fullsized":"2024-09-11T00:30:56.000726+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:27.447940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172237+0000","last_change":"2024-09-11T00:30:54.982486+0000","last_active":"2024-09-11T00:30:56.172237+0000","last_peered":"2024-09-11T00:30:56.172237+0000","last_clean":"2024-09-11T00:30:56.172237+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:30:56.172237+0000","last_undegraded":"2024-09-11T00:30:56.172237+0000","last_fullsized":"2024-09-11T00:30:56.172237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:41.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173110+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:30:56.173110+0000","last_peered":"2024-09-11T00:30:56.173110+0000","last_clean":"2024-09-11T00:30:56.173110+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:30:56.173110+0000","last_undegraded":"2024-09-11T00:30:56.173110+0000","last_fullsized":"2024-09-11T00:30:56.173110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719858+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:30:56.719858+0000","last_peered":"2024-09-11T00:30:56.719858+0000","last_clean":"2024-09-11T00:30:56.719858+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:30:56.719858+0000","last_undegraded":"2024-09-11T00:30:56.719858+0000","last_fullsized":"2024-09-11T00:30:56.719858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968520+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:30:55.968520+0000","last_peered":"2024-09-11T00:30:55.968520+0000","last_clean":"2024-09-11T00:30:55.968520+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:30:55.968520+0000","last_undegraded":"2024-09-11T00:30:55.968520+0000","last_fullsized":"2024-09-11T00:30:55.968520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173273+0000","last_change":"2024-09-11T00:30:54.985091+0000","last_active":"2024-09-11T00:30:56.173273+0000","last_peered":"2024-09-11T00:30:56.173273+0000","last_clean":"2024-09-11T00:30:56.173273+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:30:56.173273+0000","last_undegraded":"2024-09-11T00:30:56.173273+0000","last_fullsized":"2024-09-11T00:30:56.173273+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:33.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970181+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:30:55.970181+0000","last_peered":"2024-09-11T00:30:55.970181+0000","last_clean":"2024-09-11T00:30:55.970181+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:30:55.970181+0000","last_undegraded":"2024-09-11T00:30:55.970181+0000","last_fullsized":"2024-09-11T00:30:55.970181+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968811+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:30:55.968811+0000","last_peered":"2024-09-11T00:30:55.968811+0000","last_clean":"2024-09-11T00:30:55.968811+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:30:55.968811+0000","last_undegraded":"2024-09-11T00:30:55.968811+0000","last_fullsized":"2024-09-11T00:30:55.968811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966522+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:30:55.966522+0000","last_peered":"2024-09-11T00:30:55.966522+0000","last_clean":"2024-09-11T00:30:55.966522+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:30:55.966522+0000","last_undegraded":"2024-09-11T00:30:55.966522+0000","last_fullsized":"2024-09-11T00:30:55.966522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971377+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:30:55.971377+0000","last_peered":"2024-09-11T00:30:55.971377+0000","last_clean":"2024-09-11T00:30:55.971377+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:30:55.971377+0000","last_undegraded":"2024-09-11T00:30:55.971377+0000","last_fullsized":"2024-09-11T00:30:55.971377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:11.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967648+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:30:55.967648+0000","last_peered":"2024-09-11T00:30:55.967648+0000","last_clean":"2024-09-11T00:30:55.967648+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:30:55.967648+0000","last_undegraded":"2024-09-11T00:30:55.967648+0000","last_fullsized":"2024-09-11T00:30:55.967648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968196+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:30:55.968196+0000","last_peered":"2024-09-11T00:30:55.968196+0000","last_clean":"2024-09-11T00:30:55.968196+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:30:55.968196+0000","last_undegraded":"2024-09-11T00:30:55.968196+0000","last_fullsized":"2024-09-11T00:30:55.968196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969528+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:30:55.969528+0000","last_peered":"2024-09-11T00:30:55.969528+0000","last_clean":"2024-09-11T00:30:55.969528+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:30:55.969528+0000","last_undegraded":"2024-09-11T00:30:55.969528+0000","last_fullsized":"2024-09-11T00:30:55.969528+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967369+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:30:55.967369+0000","last_peered":"2024-09-11T00:30:55.967369+0000","last_clean":"2024-09-11T00:30:55.967369+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:30:55.967369+0000","last_undegraded":"2024-09-11T00:30:55.967369+0000","last_fullsized":"2024-09-11T00:30:55.967369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969812+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:30:55.969812+0000","last_peered":"2024-09-11T00:30:55.969812+0000","last_clean":"2024-09-11T00:30:55.969812+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:30:55.969812+0000","last_undegraded":"2024-09-11T00:30:55.969812+0000","last_fullsized":"2024-09-11T00:30:55.969812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969509+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:30:55.969509+0000","last_peered":"2024-09-11T00:30:55.969509+0000","last_clean":"2024-09-11T00:30:55.969509+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:30:55.969509+0000","last_undegraded":"2024-09-11T00:30:55.969509+0000","last_fullsized":"2024-09-11T00:30:55.969509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173337+0000","last_change":"2024-09-11T00:30:54.998464+0000","last_active":"2024-09-11T00:30:56.173337+0000","last_peered":"2024-09-11T00:30:56.173337+0000","last_clean":"2024-09-11T00:30:56.173337+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:30:56.173337+0000","last_undegraded":"2024-09-11T00:30:56.173337+0000","last_fullsized":"2024-09-11T00:30:56.173337+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:56.469791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998082+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:30:55.998082+0000","last_peered":"2024-09-11T00:30:55.998082+0000","last_clean":"2024-09-11T00:30:55.998082+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:30:55.998082+0000","last_undegraded":"2024-09-11T00:30:55.998082+0000","last_fullsized":"2024-09-11T00:30:55.998082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719278+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:30:56.719278+0000","last_peered":"2024-09-11T00:30:56.719278+0000","last_clean":"2024-09-11T00:30:56.719278+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:30:56.719278+0000","last_undegraded":"2024-09-11T00:30:56.719278+0000","last_fullsized":"2024-09-11T00:30:56.719278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001549+0000","last_change":"2024-09-11T00:30:53.987612+0000","last_active":"2024-09-11T00:30:56.001549+0000","last_peered":"2024-09-11T00:30:56.001549+0000","last_clean":"2024-09-11T00:30:56.001549+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:30:56.001549+0000","last_undegraded":"2024-09-11T00:30:56.001549+0000","last_fullsized":"2024-09-11T00:30:56.001549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:16.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970084+0000","last_change":"2024-09-11T00:30:54.989925+0000","last_active":"2024-09-11T00:30:55.970084+0000","last_peered":"2024-09-11T00:30:55.970084+0000","last_clean":"2024-09-11T00:30:55.970084+0000","last_became_active":"2024-09-11T00:30:54.989401+0000","last_became_peered":"2024-09-11T00:30:54.989401+0000","last_unstale":"2024-09-11T00:30:55.970084+0000","last_undegraded":"2024-09-11T00:30:55.970084+0000","last_fullsized":"2024-09-11T00:30:55.970084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:14.267175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172747+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:30:56.172747+0000","last_peered":"2024-09-11T00:30:56.172747+0000","last_clean":"2024-09-11T00:30:56.172747+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:30:56.172747+0000","last_undegraded":"2024-09-11T00:30:56.172747+0000","last_fullsized":"2024-09-11T00:30:56.172747+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969929+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:30:55.969929+0000","last_peered":"2024-09-11T00:30:55.969929+0000","last_clean":"2024-09-11T00:30:55.969929+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:30:55.969929+0000","last_undegraded":"2024-09-11T00:30:55.969929+0000","last_fullsized":"2024-09-11T00:30:55.969929+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.000911+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:30:56.000911+0000","last_peered":"2024-09-11T00:30:56.000911+0000","last_clean":"2024-09-11T00:30:56.000911+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:30:56.000911+0000","last_undegraded":"2024-09-11T00:30:56.000911+0000","last_fullsized":"2024-09-11T00:30:56.000911+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172912+0000","last_change":"2024-09-11T00:30:54.979133+0000","last_active":"2024-09-11T00:30:56.172912+0000","last_peered":"2024-09-11T00:30:56.172912+0000","last_clean":"2024-09-11T00:30:56.172912+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:30:56.172912+0000","last_undegraded":"2024-09-11T00:30:56.172912+0000","last_fullsized":"2024-09-11T00:30:56.172912+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:20.961678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967251+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:30:55.967251+0000","last_peered":"2024-09-11T00:30:55.967251+0000","last_clean":"2024-09-11T00:30:55.967251+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:30:55.967251+0000","last_undegraded":"2024-09-11T00:30:55.967251+0000","last_fullsized":"2024-09-11T00:30:55.967251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997885+0000","last_change":"2024-09-11T00:30:54.988345+0000","last_active":"2024-09-11T00:30:55.997885+0000","last_peered":"2024-09-11T00:30:55.997885+0000","last_clean":"2024-09-11T00:30:55.997885+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:30:55.997885+0000","last_undegraded":"2024-09-11T00:30:55.997885+0000","last_fullsized":"2024-09-11T00:30:55.997885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.112194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998078+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:30:55.998078+0000","last_peered":"2024-09-11T00:30:55.998078+0000","last_clean":"2024-09-11T00:30:55.998078+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:30:55.998078+0000","last_undegraded":"2024-09-11T00:30:55.998078+0000","last_fullsized":"2024-09-11T00:30:55.998078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332489+0000","last_change":"2024-09-11T00:30:54.981185+0000","last_active":"2024-09-11T00:30:56.332489+0000","last_peered":"2024-09-11T00:30:56.332489+0000","last_clean":"2024-09-11T00:30:56.332489+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:30:56.332489+0000","last_undegraded":"2024-09-11T00:30:56.332489+0000","last_fullsized":"2024-09-11T00:30:56.332489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:45.740467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971241+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:30:55.971241+0000","last_peered":"2024-09-11T00:30:55.971241+0000","last_clean":"2024-09-11T00:30:55.971241+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:30:55.971241+0000","last_undegraded":"2024-09-11T00:30:55.971241+0000","last_fullsized":"2024-09-11T00:30:55.971241+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332647+0000","last_change":"2024-09-11T00:30:54.992763+0000","last_active":"2024-09-11T00:30:56.332647+0000","last_peered":"2024-09-11T00:30:56.332647+0000","last_clean":"2024-09-11T00:30:56.332647+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:30:56.332647+0000","last_undegraded":"2024-09-11T00:30:56.332647+0000","last_fullsized":"2024-09-11T00:30:56.332647+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:02.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971179+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:30:55.971179+0000","last_peered":"2024-09-11T00:30:55.971179+0000","last_clean":"2024-09-11T00:30:55.971179+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:30:55.971179+0000","last_undegraded":"2024-09-11T00:30:55.971179+0000","last_fullsized":"2024-09-11T00:30:55.971179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966440+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:30:55.966440+0000","last_peered":"2024-09-11T00:30:55.966440+0000","last_clean":"2024-09-11T00:30:55.966440+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:30:55.966440+0000","last_undegraded":"2024-09-11T00:30:55.966440+0000","last_fullsized":"2024-09-11T00:30:55.966440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173066+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:30:56.173066+0000","last_peered":"2024-09-11T00:30:56.173066+0000","last_clean":"2024-09-11T00:30:56.173066+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:30:56.173066+0000","last_undegraded":"2024-09-11T00:30:56.173066+0000","last_fullsized":"2024-09-11T00:30:56.173066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172366+0000","last_change":"2024-09-11T00:30:54.984447+0000","last_active":"2024-09-11T00:30:56.172366+0000","last_peered":"2024-09-11T00:30:56.172366+0000","last_clean":"2024-09-11T00:30:56.172366+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:30:56.172366+0000","last_undegraded":"2024-09-11T00:30:56.172366+0000","last_fullsized":"2024-09-11T00:30:56.172366+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:49.024022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998072+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:30:55.998072+0000","last_peered":"2024-09-11T00:30:55.998072+0000","last_clean":"2024-09-11T00:30:55.998072+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:30:55.998072+0000","last_undegraded":"2024-09-11T00:30:55.998072+0000","last_fullsized":"2024-09-11T00:30:55.998072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001507+0000","last_change":"2024-09-11T00:30:54.996091+0000","last_active":"2024-09-11T00:30:56.001507+0000","last_peered":"2024-09-11T00:30:56.001507+0000","last_clean":"2024-09-11T00:30:56.001507+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:30:56.001507+0000","last_undegraded":"2024-09-11T00:30:56.001507+0000","last_fullsized":"2024-09-11T00:30:56.001507+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:42.987336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966504+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:30:55.966504+0000","last_peered":"2024-09-11T00:30:55.966504+0000","last_clean":"2024-09-11T00:30:55.966504+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:30:55.966504+0000","last_undegraded":"2024-09-11T00:30:55.966504+0000","last_fullsized":"2024-09-11T00:30:55.966504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968347+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:30:55.968347+0000","last_peered":"2024-09-11T00:30:55.968347+0000","last_clean":"2024-09-11T00:30:55.968347+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:30:55.968347+0000","last_undegraded":"2024-09-11T00:30:55.968347+0000","last_fullsized":"2024-09-11T00:30:55.968347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997450+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:30:55.997450+0000","last_peered":"2024-09-11T00:30:55.997450+0000","last_clean":"2024-09-11T00:30:55.997450+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:30:55.997450+0000","last_undegraded":"2024-09-11T00:30:55.997450+0000","last_fullsized":"2024-09-11T00:30:55.997450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971167+0000","last_change":"2024-09-11T00:30:54.981799+0000","last_active":"2024-09-11T00:30:55.971167+0000","last_peered":"2024-09-11T00:30:55.971167+0000","last_clean":"2024-09-11T00:30:55.971167+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:30:55.971167+0000","last_undegraded":"2024-09-11T00:30:55.971167+0000","last_fullsized":"2024-09-11T00:30:55.971167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:42.670629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719169+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:30:56.719169+0000","last_peered":"2024-09-11T00:30:56.719169+0000","last_clean":"2024-09-11T00:30:56.719169+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:30:56.719169+0000","last_undegraded":"2024-09-11T00:30:56.719169+0000","last_fullsized":"2024-09-11T00:30:56.719169+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001453+0000","last_change":"2024-09-11T00:30:54.992697+0000","last_active":"2024-09-11T00:30:56.001453+0000","last_peered":"2024-09-11T00:30:56.001453+0000","last_clean":"2024-09-11T00:30:56.001453+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:30:56.001453+0000","last_undegraded":"2024-09-11T00:30:56.001453+0000","last_fullsized":"2024-09-11T00:30:56.001453+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:54:07.227169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969635+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:30:55.969635+0000","last_peered":"2024-09-11T00:30:55.969635+0000","last_clean":"2024-09-11T00:30:55.969635+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:55.969635+0000","last_undegraded":"2024-09-11T00:30:55.969635+0000","last_fullsized":"2024-09-11T00:30:55.969635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172413+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:30:56.172413+0000","last_peered":"2024-09-11T00:30:56.172413+0000","last_clean":"2024-09-11T00:30:56.172413+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:30:56.172413+0000","last_undegraded":"2024-09-11T00:30:56.172413+0000","last_fullsized":"2024-09-11T00:30:56.172413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332844+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:30:56.332844+0000","last_peered":"2024-09-11T00:30:56.332844+0000","last_clean":"2024-09-11T00:30:56.332844+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:30:56.332844+0000","last_undegraded":"2024-09-11T00:30:56.332844+0000","last_fullsized":"2024-09-11T00:30:56.332844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332895+0000","last_change":"2024-09-11T00:30:54.985827+0000","last_active":"2024-09-11T00:30:56.332895+0000","last_peered":"2024-09-11T00:30:56.332895+0000","last_clean":"2024-09-11T00:30:56.332895+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:30:56.332895+0000","last_undegraded":"2024-09-11T00:30:56.332895+0000","last_fullsized":"2024-09-11T00:30:56.332895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:32.027117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968441+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:30:55.968441+0000","last_peered":"2024-09-11T00:30:55.968441+0000","last_clean":"2024-09-11T00:30:55.968441+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:30:55.968441+0000","last_undegraded":"2024-09-11T00:30:55.968441+0000","last_fullsized":"2024-09-11T00:30:55.968441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172255+0000","last_change":"2024-09-11T00:30:54.989906+0000","last_active":"2024-09-11T00:30:56.172255+0000","last_peered":"2024-09-11T00:30:56.172255+0000","last_clean":"2024-09-11T00:30:56.172255+0000","last_became_active":"2024-09-11T00:30:54.989376+0000","last_became_peered":"2024-09-11T00:30:54.989376+0000","last_unstale":"2024-09-11T00:30:56.172255+0000","last_undegraded":"2024-09-11T00:30:56.172255+0000","last_fullsized":"2024-09-11T00:30:56.172255+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:18:00.138941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001665+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:30:56.001665+0000","last_peered":"2024-09-11T00:30:56.001665+0000","last_clean":"2024-09-11T00:30:56.001665+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:30:56.001665+0000","last_undegraded":"2024-09-11T00:30:56.001665+0000","last_fullsized":"2024-09-11T00:30:56.001665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968974+0000","last_change":"2024-09-11T00:30:54.993156+0000","last_active":"2024-09-11T00:30:55.968974+0000","last_peered":"2024-09-11T00:30:55.968974+0000","last_clean":"2024-09-11T00:30:55.968974+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:30:55.968974+0000","last_undegraded":"2024-09-11T00:30:55.968974+0000","last_fullsized":"2024-09-11T00:30:55.968974+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:25.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.968016+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:30:55.968016+0000","last_peered":"2024-09-11T00:30:55.968016+0000","last_clean":"2024-09-11T00:30:55.968016+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:30:55.968016+0000","last_undegraded":"2024-09-11T00:30:55.968016+0000","last_fullsized":"2024-09-11T00:30:55.968016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970335+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:30:55.970335+0000","last_peered":"2024-09-11T00:30:55.970335+0000","last_clean":"2024-09-11T00:30:55.970335+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:30:55.970335+0000","last_undegraded":"2024-09-11T00:30:55.970335+0000","last_fullsized":"2024-09-11T00:30:55.970335+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.966434+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:30:55.966434+0000","last_peered":"2024-09-11T00:30:55.966434+0000","last_clean":"2024-09-11T00:30:55.966434+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:30:55.966434+0000","last_undegraded":"2024-09-11T00:30:55.966434+0000","last_fullsized":"2024-09-11T00:30:55.966434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719699+0000","last_change":"2024-09-11T00:30:54.993461+0000","last_active":"2024-09-11T00:30:56.719699+0000","last_peered":"2024-09-11T00:30:56.719699+0000","last_clean":"2024-09-11T00:30:56.719699+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:30:56.719699+0000","last_undegraded":"2024-09-11T00:30:56.719699+0000","last_fullsized":"2024-09-11T00:30:56.719699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:07.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997281+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:30:55.997281+0000","last_peered":"2024-09-11T00:30:55.997281+0000","last_clean":"2024-09-11T00:30:55.997281+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:30:55.997281+0000","last_undegraded":"2024-09-11T00:30:55.997281+0000","last_fullsized":"2024-09-11T00:30:55.997281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719307+0000","last_change":"2024-09-11T00:30:54.986265+0000","last_active":"2024-09-11T00:30:56.719307+0000","last_peered":"2024-09-11T00:30:56.719307+0000","last_clean":"2024-09-11T00:30:56.719307+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:30:56.719307+0000","last_undegraded":"2024-09-11T00:30:56.719307+0000","last_fullsized":"2024-09-11T00:30:56.719307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:49.109328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969766+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:30:55.969766+0000","last_peered":"2024-09-11T00:30:55.969766+0000","last_clean":"2024-09-11T00:30:55.969766+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:30:55.969766+0000","last_undegraded":"2024-09-11T00:30:55.969766+0000","last_fullsized":"2024-09-11T00:30:55.969766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.967854+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:30:55.967854+0000","last_peered":"2024-09-11T00:30:55.967854+0000","last_clean":"2024-09-11T00:30:55.967854+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:30:55.967854+0000","last_undegraded":"2024-09-11T00:30:55.967854+0000","last_fullsized":"2024-09-11T00:30:55.967854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.971368+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:30:55.971368+0000","last_peered":"2024-09-11T00:30:55.971368+0000","last_clean":"2024-09-11T00:30:55.971368+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:30:55.971368+0000","last_undegraded":"2024-09-11T00:30:55.971368+0000","last_fullsized":"2024-09-11T00:30:55.971368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.969452+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:30:55.969452+0000","last_peered":"2024-09-11T00:30:55.969452+0000","last_clean":"2024-09-11T00:30:55.969452+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:30:55.969452+0000","last_undegraded":"2024-09-11T00:30:55.969452+0000","last_fullsized":"2024-09-11T00:30:55.969452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332691+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:30:56.332691+0000","last_peered":"2024-09-11T00:30:56.332691+0000","last_clean":"2024-09-11T00:30:56.332691+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:30:56.332691+0000","last_undegraded":"2024-09-11T00:30:56.332691+0000","last_fullsized":"2024-09-11T00:30:56.332691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970850+0000","last_change":"2024-09-11T00:30:54.994988+0000","last_active":"2024-09-11T00:30:55.970850+0000","last_peered":"2024-09-11T00:30:55.970850+0000","last_clean":"2024-09-11T00:30:55.970850+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:30:55.970850+0000","last_undegraded":"2024-09-11T00:30:55.970850+0000","last_fullsized":"2024-09-11T00:30:55.970850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:31.973641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332924+0000","last_change":"2024-09-11T00:30:45.941635+0000","last_active":"2024-09-11T00:30:56.332924+0000","last_peered":"2024-09-11T00:30:56.332924+0000","last_clean":"2024-09-11T00:30:56.332924+0000","last_became_active":"2024-09-11T00:30:45.941357+0000","last_became_peered":"2024-09-11T00:30:45.941357+0000","last_unstale":"2024-09-11T00:30:56.332924+0000","last_undegraded":"2024-09-11T00:30:56.332924+0000","last_fullsized":"2024-09-11T00:30:56.332924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:12.016406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332421+0000","last_change":"2024-09-11T00:30:54.984767+0000","last_active":"2024-09-11T00:30:56.332421+0000","last_peered":"2024-09-11T00:30:56.332421+0000","last_clean":"2024-09-11T00:30:56.332421+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:30:56.332421+0000","last_undegraded":"2024-09-11T00:30:56.332421+0000","last_fullsized":"2024-09-11T00:30:56.332421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:44.833677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332955+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:30:56.332955+0000","last_peered":"2024-09-11T00:30:56.332955+0000","last_clean":"2024-09-11T00:30:56.332955+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:30:56.332955+0000","last_undegraded":"2024-09-11T00:30:56.332955+0000","last_fullsized":"2024-09-11T00:30:56.332955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.332911+0000","last_change":"2024-09-11T00:30:54.986423+0000","last_active":"2024-09-11T00:30:56.332911+0000","last_peered":"2024-09-11T00:30:56.332911+0000","last_clean":"2024-09-11T00:30:56.332911+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:30:56.332911+0000","last_undegraded":"2024-09-11T00:30:56.332911+0000","last_fullsized":"2024-09-11T00:30:56.332911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:16:50.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719859+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:30:56.719859+0000","last_peered":"2024-09-11T00:30:56.719859+0000","last_clean":"2024-09-11T00:30:56.719859+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:30:56.719859+0000","last_undegraded":"2024-09-11T00:30:56.719859+0000","last_fullsized":"2024-09-11T00:30:56.719859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.998062+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:30:55.998062+0000","last_peered":"2024-09-11T00:30:55.998062+0000","last_clean":"2024-09-11T00:30:55.998062+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:30:55.998062+0000","last_undegraded":"2024-09-11T00:30:55.998062+0000","last_fullsized":"2024-09-11T00:30:55.998062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997283+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:30:55.997283+0000","last_peered":"2024-09-11T00:30:55.997283+0000","last_clean":"2024-09-11T00:30:55.997283+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:30:55.997283+0000","last_undegraded":"2024-09-11T00:30:55.997283+0000","last_fullsized":"2024-09-11T00:30:55.997283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173138+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:30:56.173138+0000","last_peered":"2024-09-11T00:30:56.173138+0000","last_clean":"2024-09-11T00:30:56.173138+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:30:56.173138+0000","last_undegraded":"2024-09-11T00:30:56.173138+0000","last_fullsized":"2024-09-11T00:30:56.173138+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333239+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:30:56.333239+0000","last_peered":"2024-09-11T00:30:56.333239+0000","last_clean":"2024-09-11T00:30:56.333239+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:30:56.333239+0000","last_undegraded":"2024-09-11T00:30:56.333239+0000","last_fullsized":"2024-09-11T00:30:56.333239+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.970256+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:30:55.970256+0000","last_peered":"2024-09-11T00:30:55.970256+0000","last_clean":"2024-09-11T00:30:55.970256+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:30:55.970256+0000","last_undegraded":"2024-09-11T00:30:55.970256+0000","last_fullsized":"2024-09-11T00:30:55.970256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172539+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:30:56.172539+0000","last_peered":"2024-09-11T00:30:56.172539+0000","last_clean":"2024-09-11T00:30:56.172539+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:30:56.172539+0000","last_undegraded":"2024-09-11T00:30:56.172539+0000","last_fullsized":"2024-09-11T00:30:56.172539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.719531+0000","last_change":"2024-09-11T00:30:54.990631+0000","last_active":"2024-09-11T00:30:56.719531+0000","last_peered":"2024-09-11T00:30:56.719531+0000","last_clean":"2024-09-11T00:30:56.719531+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:30:56.719531+0000","last_undegraded":"2024-09-11T00:30:56.719531+0000","last_fullsized":"2024-09-11T00:30:56.719531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:02.422471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333220+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:30:56.333220+0000","last_peered":"2024-09-11T00:30:56.333220+0000","last_clean":"2024-09-11T00:30:56.333220+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:30:56.333220+0000","last_undegraded":"2024-09-11T00:30:56.333220+0000","last_fullsized":"2024-09-11T00:30:56.333220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.001045+0000","last_change":"2024-09-11T00:30:54.991047+0000","last_active":"2024-09-11T00:30:56.001045+0000","last_peered":"2024-09-11T00:30:56.001045+0000","last_clean":"2024-09-11T00:30:56.001045+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:30:56.001045+0000","last_undegraded":"2024-09-11T00:30:56.001045+0000","last_fullsized":"2024-09-11T00:30:56.001045+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:41:48.229261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.173027+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:30:56.173027+0000","last_peered":"2024-09-11T00:30:56.173027+0000","last_clean":"2024-09-11T00:30:56.173027+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:30:56.173027+0000","last_undegraded":"2024-09-11T00:30:56.173027+0000","last_fullsized":"2024-09-11T00:30:56.173027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997710+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:30:55.997710+0000","last_peered":"2024-09-11T00:30:55.997710+0000","last_clean":"2024-09-11T00:30:55.997710+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:30:55.997710+0000","last_undegraded":"2024-09-11T00:30:55.997710+0000","last_fullsized":"2024-09-11T00:30:55.997710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997737+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:30:55.997737+0000","last_peered":"2024-09-11T00:30:55.997737+0000","last_clean":"2024-09-11T00:30:55.997737+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:30:55.997737+0000","last_undegraded":"2024-09-11T00:30:55.997737+0000","last_fullsized":"2024-09-11T00:30:55.997737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172698+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:30:56.172698+0000","last_peered":"2024-09-11T00:30:56.172698+0000","last_clean":"2024-09-11T00:30:56.172698+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:30:56.172698+0000","last_undegraded":"2024-09-11T00:30:56.172698+0000","last_fullsized":"2024-09-11T00:30:56.172698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:55.997914+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:30:55.997914+0000","last_peered":"2024-09-11T00:30:55.997914+0000","last_clean":"2024-09-11T00:30:55.997914+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:30:55.997914+0000","last_undegraded":"2024-09-11T00:30:55.997914+0000","last_fullsized":"2024-09-11T00:30:55.997914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.172531+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:30:56.172531+0000","last_peered":"2024-09-11T00:30:56.172531+0000","last_clean":"2024-09-11T00:30:56.172531+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:30:56.172531+0000","last_undegraded":"2024-09-11T00:30:56.172531+0000","last_fullsized":"2024-09-11T00:30:56.172531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T00:30:56.333081+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:30:56.333081+0000","last_peered":"2024-09-11T00:30:56.333081+0000","last_clean":"2024-09-11T00:30:56.333081+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:30:56.333081+0000","last_undegraded":"2024-09-11T00:30:56.333081+0000","last_fullsized":"2024-09-11T00:30:56.333081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3666,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509453,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509455,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":107487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509454,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509452,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509454,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":107521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509454,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542158,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":111049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:31:24.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:24.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.648+0000 7fb59b200640 1 -- 172.21.15.31:0/3424419152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb578003a30 msgr2=0x7fb578003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:24.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.648+0000 7fb59b200640 1 --2- 172.21.15.31:0/3424419152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb578003a30 0x7fb578003e10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb584009b50 tx=0x7fb578005830 comp rx=0 tx=0).stop 2024-09-11T00:31:24.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.648+0000 7fb599200640 1 -- 172.21.15.31:0/3424419152 reap_dead start 2024-09-11T00:31:24.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.648+0000 7fb59b200640 1 -- 172.21.15.31:0/3424419152 shutdown_connections 2024-09-11T00:31:24.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.648+0000 7fb59b200640 1 -- 172.21.15.31:0/3424419152 >> 172.21.15.31:0/3424419152 conn(0x7fb5940a9750 msgr2=0x7fb5940a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:24.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.648+0000 7fb59b200640 1 -- 172.21.15.31:0/3424419152 shutdown_connections 2024-09-11T00:31:24.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.648+0000 7fb59b200640 1 -- 172.21.15.31:0/3424419152 wait complete. 2024-09-11T00:31:24.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.650+0000 7fb59b200640 1 Processor -- start 2024-09-11T00:31:24.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.650+0000 7fb59b200640 1 -- start start 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb59b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb578003a30 0x7fb5941ed110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb59b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5941ed650 0x7fb5940ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb59b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb594155310 0x7fb5940ab750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb59b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb594074200 con 0x7fb594155310 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb59b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb594074080 con 0x7fb578003a30 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb59b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb594072860 con 0x7fb5941ed650 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb599200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb594155310 0x7fb5940ab750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb599200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb594155310 0x7fb5940ab750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47866/0 (socket says 172.21.15.31:47866) 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb599200640 1 -- 172.21.15.31:0/3554904011 learned_addr learned my addr 172.21.15.31:0/3554904011 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb593e00640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb578003a30 0x7fb5941ed110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb593400640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5941ed650 0x7fb5940ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:24.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb599200640 1 -- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5941ed650 msgr2=0x7fb5940ab210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb599200640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5941ed650 0x7fb5940ab210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb599200640 1 -- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb578003a30 msgr2=0x7fb5941ed110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb599200640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb578003a30 0x7fb5941ed110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb599200640 1 -- 172.21.15.31:0/3554904011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5941edb00 con 0x7fb594155310 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.651+0000 7fb593400640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5941ed650 0x7fb5940ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.652+0000 7fb599200640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb594155310 0x7fb5940ab750 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb58c001520 tx=0x7fb58c006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.652+0000 7fb58be00640 1 -- 172.21.15.31:0/3554904011 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c015070 con 0x7fb594155310 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.652+0000 7fb58be00640 1 -- 172.21.15.31:0/3554904011 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb58c011900 con 0x7fb594155310 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.652+0000 7fb58be00640 1 -- 172.21.15.31:0/3554904011 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58c011d30 con 0x7fb594155310 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.652+0000 7fb59b200640 1 -- 172.21.15.31:0/3554904011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5941dfc60 con 0x7fb594155310 2024-09-11T00:31:24.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.652+0000 7fb59b200640 1 -- 172.21.15.31:0/3554904011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5941def10 con 0x7fb594155310 2024-09-11T00:31:24.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.653+0000 7fb59b200640 1 -- 172.21.15.31:0/3554904011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb594151280 con 0x7fb594155310 2024-09-11T00:31:24.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.654+0000 7fb58be00640 1 -- 172.21.15.31:0/3554904011 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb58c0103c0 con 0x7fb594155310 2024-09-11T00:31:24.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.656+0000 7fb58be00640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb574077f70 0x7fb57407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:24.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.656+0000 7fb58be00640 1 -- 172.21.15.31:0/3554904011 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fb58c09ac90 con 0x7fb594155310 2024-09-11T00:31:24.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.656+0000 7fb593e00640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb574077f70 0x7fb57407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:24.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.657+0000 7fb593e00640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb574077f70 0x7fb57407a430 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb584035000 tx=0x7fb58402d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:24.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.660+0000 7fb58be00640 1 -- 172.21.15.31:0/3554904011 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb58c062e00 con 0x7fb594155310 2024-09-11T00:31:24.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.894+0000 7fb59b200640 1 -- 172.21.15.31:0/3554904011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb59414e470 con 0x7fb594155310 2024-09-11T00:31:24.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.894+0000 7fb58be00640 1 -- 172.21.15.31:0/3554904011 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fb58c067830 con 0x7fb594155310 2024-09-11T00:31:24.897 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:24.897 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:24.898 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 -- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb574077f70 msgr2=0x7fb57407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb574077f70 0x7fb57407a430 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb584035000 tx=0x7fb58402d040 comp rx=0 tx=0).stop 2024-09-11T00:31:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 -- 172.21.15.31:0/3554904011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb594155310 msgr2=0x7fb5940ab750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb594155310 0x7fb5940ab750 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb58c001520 tx=0x7fb58c006430 comp rx=0 tx=0).stop 2024-09-11T00:31:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 -- 172.21.15.31:0/3554904011 shutdown_connections 2024-09-11T00:31:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb574077f70 0x7fb57407a430 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb594155310 0x7fb5940ab750 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb5941ed650 0x7fb5940ab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 --2- 172.21.15.31:0/3554904011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb578003a30 0x7fb5941ed110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 -- 172.21.15.31:0/3554904011 >> 172.21.15.31:0/3554904011 conn(0x7fb5940a9750 msgr2=0x7fb59414e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.897+0000 7fb59b200640 1 -- 172.21.15.31:0/3554904011 shutdown_connections 2024-09-11T00:31:24.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:24.898+0000 7fb59b200640 1 -- 172.21.15.31:0/3554904011 wait complete. 2024-09-11T00:31:24.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:31:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.074+0000 7f7f76400640 1 -- 172.21.15.31:0/789838123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f700c5ef0 msgr2=0x7f7f70153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.074+0000 7f7f76400640 1 --2- 172.21.15.31:0/789838123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f700c5ef0 0x7f7f70153480 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7f7006de50 tx=0x7f7f5c02a560 comp rx=0 tx=0).stop 2024-09-11T00:31:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.074+0000 7f7f76400640 1 -- 172.21.15.31:0/789838123 shutdown_connections 2024-09-11T00:31:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.074+0000 7f7f76400640 1 --2- 172.21.15.31:0/789838123 >> v2:172.21.15.204:3300/0 conn(0x7f7f700c6310 0x7f7f701539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.074+0000 7f7f76400640 1 --2- 172.21.15.31:0/789838123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f700c5ef0 0x7f7f70153480 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.074+0000 7f7f76400640 1 --2- 172.21.15.31:0/789838123 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f700c5b10 0x7f7f70156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.074+0000 7f7f76400640 1 -- 172.21.15.31:0/789838123 >> 172.21.15.31:0/789838123 conn(0x7f7f700a9750 msgr2=0x7f7f700a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.074+0000 7f7f76400640 1 -- 172.21.15.31:0/789838123 shutdown_connections 2024-09-11T00:31:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.074+0000 7f7f76400640 1 -- 172.21.15.31:0/789838123 wait complete. 2024-09-11T00:31:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.076+0000 7f7f76400640 1 Processor -- start 2024-09-11T00:31:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.076+0000 7f7f76400640 1 -- start start 2024-09-11T00:31:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.077+0000 7f7f76400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f700c5b10 0x7f7f7014f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.077+0000 7f7f76400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f700c5ef0 0x7f7f7014b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.077+0000 7f7f76400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f700c6310 0x7f7f7014be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.077+0000 7f7f76400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701597f0 con 0x7f7f700c5b10 2024-09-11T00:31:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.077+0000 7f7f76400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7f70159670 con 0x7f7f700c6310 2024-09-11T00:31:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.077+0000 7f7f76400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7f70159970 con 0x7f7f700c5ef0 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.077+0000 7f7f6f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f700c5b10 0x7f7f7014f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.077+0000 7f7f6f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f700c5b10 0x7f7f7014f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47886/0 (socket says 172.21.15.31:47886) 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.077+0000 7f7f6f400640 1 -- 172.21.15.31:0/262203521 learned_addr learned my addr 172.21.15.31:0/262203521 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f6f400640 1 -- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f700c5ef0 msgr2=0x7f7f7014b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f6f400640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f700c5ef0 0x7f7f7014b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f6f400640 1 -- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f700c6310 msgr2=0x7f7f7014be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f6f400640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f700c6310 0x7f7f7014be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f6f400640 1 -- 172.21.15.31:0/262203521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7014c340 con 0x7f7f700c5b10 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f6f400640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f700c5b10 0x7f7f7014f190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000d770 tx=0x7f7f6000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f67400640 1 -- 172.21.15.31:0/262203521 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60017070 con 0x7f7f700c5b10 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f67400640 1 -- 172.21.15.31:0/262203521 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f60009d20 con 0x7f7f700c5b10 2024-09-11T00:31:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f67400640 1 -- 172.21.15.31:0/262203521 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60005280 con 0x7f7f700c5b10 2024-09-11T00:31:25.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f76400640 1 -- 172.21.15.31:0/262203521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7014c5e0 con 0x7f7f700c5b10 2024-09-11T00:31:25.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.078+0000 7f7f76400640 1 -- 172.21.15.31:0/262203521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f701df8c0 con 0x7f7f700c5b10 2024-09-11T00:31:25.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.079+0000 7f7f76400640 1 -- 172.21.15.31:0/262203521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f701df6e0 con 0x7f7f700c5b10 2024-09-11T00:31:25.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.080+0000 7f7f67400640 1 -- 172.21.15.31:0/262203521 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7f60005420 con 0x7f7f700c5b10 2024-09-11T00:31:25.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.082+0000 7f7f67400640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7f50077f70 0x7f7f5007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.082+0000 7f7f67400640 1 -- 172.21.15.31:0/262203521 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f7f6009a240 con 0x7f7f700c5b10 2024-09-11T00:31:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.082+0000 7f7f6ea00640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7f50077f70 0x7f7f5007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.083+0000 7f7f6ea00640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7f50077f70 0x7f7f5007a430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c033000 tx=0x7f7f5c0046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:25.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.086+0000 7f7f67400640 1 -- 172.21.15.31:0/262203521 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7f701df6e0 con 0x7f7f700c5b10 2024-09-11T00:31:25.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.287+0000 7f7f76400640 1 -- 172.21.15.31:0/262203521 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f70156ad0 con 0x7f7f700c5b10 2024-09-11T00:31:25.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.288+0000 7f7f67400640 1 -- 172.21.15.31:0/262203521 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20679 (secure 0 0 0) 0x7f7f60062a10 con 0x7f7f700c5b10 2024-09-11T00:31:25.291 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:25.292 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:31:25.097993+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.35","mappings":[{"from":0,"to":7}]},{"pgid":"5.8","mappings":[{"from":4,"to":5}]},{"pgid":"5.17","mappings":[{"from":4,"to":3}]},{"pgid":"5.26","mappings":[{"from":4,"to":5}]},{"pgid":"5.28","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 -- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7f50077f70 msgr2=0x7f7f5007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7f50077f70 0x7f7f5007a430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c033000 tx=0x7f7f5c0046c0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 -- 172.21.15.31:0/262203521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f700c5b10 msgr2=0x7f7f7014f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f700c5b10 0x7f7f7014f190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000d770 tx=0x7f7f6000dc30 comp rx=0 tx=0).stop 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 -- 172.21.15.31:0/262203521 shutdown_connections 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7f50077f70 0x7f7f5007a430 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f700c6310 0x7f7f7014be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7f700c5ef0 0x7f7f7014b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 --2- 172.21.15.31:0/262203521 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f700c5b10 0x7f7f7014f190 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 -- 172.21.15.31:0/262203521 >> 172.21.15.31:0/262203521 conn(0x7f7f700a9750 msgr2=0x7f7f70150600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:25.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 -- 172.21.15.31:0/262203521 shutdown_connections 2024-09-11T00:31:25.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.291+0000 7f7f76400640 1 -- 172.21.15.31:0/262203521 wait complete. 2024-09-11T00:31:25.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:25.310 INFO:tasks.mds_thrash.fs.[a]:starting mds_do_thrash for fs a 2024-09-11T00:31:25.310 INFO:tasks.mds_thrash.fs.[a]:waiting for 75 secs before thrashing 2024-09-11T00:31:25.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.467+0000 7fde03600640 1 -- 172.21.15.31:0/1706278734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddfc0c5b10 msgr2=0x7fddfc156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.467+0000 7fde03600640 1 --2- 172.21.15.31:0/1706278734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddfc0c5b10 0x7fddfc156ad0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fddfc06e7f0 tx=0x7fddf002a600 comp rx=0 tx=0).stop 2024-09-11T00:31:25.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.467+0000 7fde03600640 1 -- 172.21.15.31:0/1706278734 shutdown_connections 2024-09-11T00:31:25.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.467+0000 7fde03600640 1 --2- 172.21.15.31:0/1706278734 >> v2:172.21.15.204:3300/0 conn(0x7fddfc0c5ef0 0x7fddfc153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.467+0000 7fde03600640 1 --2- 172.21.15.31:0/1706278734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddfc0c5b10 0x7fddfc156ad0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.467+0000 7fde03600640 1 -- 172.21.15.31:0/1706278734 >> 172.21.15.31:0/1706278734 conn(0x7fddfc0a9750 msgr2=0x7fddfc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:25.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.467+0000 7fde03600640 1 -- 172.21.15.31:0/1706278734 shutdown_connections 2024-09-11T00:31:25.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.468+0000 7fde03600640 1 -- 172.21.15.31:0/1706278734 wait complete. 2024-09-11T00:31:25.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.469+0000 7fde03600640 1 Processor -- start 2024-09-11T00:31:25.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.469+0000 7fde03600640 1 -- start start 2024-09-11T00:31:25.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.470+0000 7fde03600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddfc0c5b10 0x7fddfc0acef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.470+0000 7fde03600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddfc0c5ef0 0x7fddfc151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.470+0000 7fde03600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fddfc0c6340 0x7fddfc14cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.470+0000 7fde03600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fddfc1539c0 con 0x7fddfc0c5b10 2024-09-11T00:31:25.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.470+0000 7fde03600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fddfc0c3950 con 0x7fddfc0c5ef0 2024-09-11T00:31:25.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.470+0000 7fde03600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fddfc153b40 con 0x7fddfc0c6340 2024-09-11T00:31:25.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.470+0000 7fde00c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddfc0c5b10 0x7fddfc0acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.470+0000 7fde00c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddfc0c5b10 0x7fddfc0acef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47918/0 (socket says 172.21.15.31:47918) 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.470+0000 7fde00c00640 1 -- 172.21.15.31:0/2797696731 learned_addr learned my addr 172.21.15.31:0/2797696731 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fde00c00640 1 -- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fddfc0c6340 msgr2=0x7fddfc14cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fde00c00640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fddfc0c6340 0x7fddfc14cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fddfbe00640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddfc0c5ef0 0x7fddfc151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fde00c00640 1 -- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddfc0c5ef0 msgr2=0x7fddfc151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fde00c00640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddfc0c5ef0 0x7fddfc151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fde00c00640 1 -- 172.21.15.31:0/2797696731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddfc0748e0 con 0x7fddfc0c5b10 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fddfbe00640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddfc0c5ef0 0x7fddfc151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fde00c00640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddfc0c5b10 0x7fddfc0acef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fddf0009850 tx=0x7fddf0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fddf8c00640 1 -- 172.21.15.31:0/2797696731 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf003f070 con 0x7fddfc0c5b10 2024-09-11T00:31:25.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fddf8c00640 1 -- 172.21.15.31:0/2797696731 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fddf00047a0 con 0x7fddfc0c5b10 2024-09-11T00:31:25.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fddf8c00640 1 -- 172.21.15.31:0/2797696731 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf0035030 con 0x7fddfc0c5b10 2024-09-11T00:31:25.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.471+0000 7fde03600640 1 -- 172.21.15.31:0/2797696731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddfc1ecf30 con 0x7fddfc0c5b10 2024-09-11T00:31:25.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.472+0000 7fde03600640 1 -- 172.21.15.31:0/2797696731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddfc0a70c0 con 0x7fddfc0c5b10 2024-09-11T00:31:25.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.473+0000 7fde03600640 1 -- 172.21.15.31:0/2797696731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddfc0ad960 con 0x7fddfc0c5b10 2024-09-11T00:31:25.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.474+0000 7fddf8c00640 1 -- 172.21.15.31:0/2797696731 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fddf0004bb0 con 0x7fddfc0c5b10 2024-09-11T00:31:25.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.475+0000 7fddf8c00640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdddc077f70 0x7fdddc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.475+0000 7fddf8c00640 1 -- 172.21.15.31:0/2797696731 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6666+0+0 (secure 0 0 0) 0x7fddf00bb290 con 0x7fddfc0c5b10 2024-09-11T00:31:25.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.475+0000 7fddfbe00640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdddc077f70 0x7fdddc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:25.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.476+0000 7fddfbe00640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdddc077f70 0x7fdddc07a430 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0c5140 tx=0x7fdde8005e50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:25.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.479+0000 7fddf8c00640 1 -- 172.21.15.31:0/2797696731 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fddfc0ad960 con 0x7fddfc0c5b10 2024-09-11T00:31:25.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.711+0000 7fde03600640 1 -- 172.21.15.31:0/2797696731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddfc156ad0 con 0x7fddfc0c5b10 2024-09-11T00:31:25.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.711+0000 7fddf8c00640 1 -- 172.21.15.31:0/2797696731 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fddf002ae20 con 0x7fddfc0c5b10 2024-09-11T00:31:25.714 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:25.715 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:25.715 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.713+0000 7fde03600640 1 -- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdddc077f70 msgr2=0x7fdddc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.713+0000 7fde03600640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdddc077f70 0x7fdddc07a430 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0c5140 tx=0x7fdde8005e50 comp rx=0 tx=0).stop 2024-09-11T00:31:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.713+0000 7fde03600640 1 -- 172.21.15.31:0/2797696731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddfc0c5b10 msgr2=0x7fddfc0acef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.713+0000 7fde03600640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddfc0c5b10 0x7fddfc0acef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fddf0009850 tx=0x7fddf0003910 comp rx=0 tx=0).stop 2024-09-11T00:31:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.714+0000 7fde03600640 1 -- 172.21.15.31:0/2797696731 shutdown_connections 2024-09-11T00:31:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.714+0000 7fde03600640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdddc077f70 0x7fdddc07a430 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.714+0000 7fde03600640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fddfc0c6340 0x7fddfc14cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.714+0000 7fde03600640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fddfc0c5ef0 0x7fddfc151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.714+0000 7fde03600640 1 --2- 172.21.15.31:0/2797696731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddfc0c5b10 0x7fddfc0acef0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.714+0000 7fde03600640 1 -- 172.21.15.31:0/2797696731 >> 172.21.15.31:0/2797696731 conn(0x7fddfc0a9750 msgr2=0x7fddfc156240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:25.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.714+0000 7fde03600640 1 -- 172.21.15.31:0/2797696731 shutdown_connections 2024-09-11T00:31:25.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.714+0000 7fde03600640 1 -- 172.21.15.31:0/2797696731 wait complete. 2024-09-11T00:31:25.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:31:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.890+0000 7f83d5a00640 1 -- 172.21.15.31:0/3347595788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83d0072480 msgr2=0x7f83d01511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.890+0000 7f83d5a00640 1 --2- 172.21.15.31:0/3347595788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83d0072480 0x7f83d01511a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f83d00708c0 tx=0x7f83c002a5e0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.890+0000 7f83d5a00640 1 -- 172.21.15.31:0/3347595788 shutdown_connections 2024-09-11T00:31:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.890+0000 7f83d5a00640 1 --2- 172.21.15.31:0/3347595788 >> v2:172.21.15.204:3300/0 conn(0x7f83d0153f20 0x7f83d00c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.890+0000 7f83d5a00640 1 --2- 172.21.15.31:0/3347595788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83d00c23a0 0x7f83d01516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.890+0000 7f83d5a00640 1 --2- 172.21.15.31:0/3347595788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83d0072480 0x7f83d01511a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.890+0000 7f83d5a00640 1 -- 172.21.15.31:0/3347595788 >> 172.21.15.31:0/3347595788 conn(0x7f83d00a9750 msgr2=0x7f83d00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.890+0000 7f83d5a00640 1 -- 172.21.15.31:0/3347595788 shutdown_connections 2024-09-11T00:31:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.890+0000 7f83d5a00640 1 -- 172.21.15.31:0/3347595788 wait complete. 2024-09-11T00:31:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.892+0000 7f83d5a00640 1 Processor -- start 2024-09-11T00:31:25.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.892+0000 7f83d5a00640 1 -- start start 2024-09-11T00:31:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83d5a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83d0072480 0x7f83d0152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83d5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83d00c23a0 0x7f83d01dfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83d5a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83d0153f20 0x7f83d01dd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83d5a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f83d0074200 con 0x7f83d00c23a0 2024-09-11T00:31:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83d5a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f83d0074080 con 0x7f83d0072480 2024-09-11T00:31:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83d5a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f83d0072860 con 0x7f83d0153f20 2024-09-11T00:31:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83c6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83d00c23a0 0x7f83d01dfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83c6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83d00c23a0 0x7f83d01dfa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47930/0 (socket says 172.21.15.31:47930) 2024-09-11T00:31:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83c6000640 1 -- 172.21.15.31:0/3514134333 learned_addr learned my addr 172.21.15.31:0/3514134333 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83cea00640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83d0072480 0x7f83d0152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83cf400640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83d0153f20 0x7f83d01dd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:25.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83c6000640 1 -- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83d0153f20 msgr2=0x7f83d01dd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83c6000640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83d0153f20 0x7f83d01dd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83c6000640 1 -- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83d0072480 msgr2=0x7f83d0152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:25.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83c6000640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83d0072480 0x7f83d0152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:25.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83c6000640 1 -- 172.21.15.31:0/3514134333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d0153210 con 0x7f83d00c23a0 2024-09-11T00:31:25.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83cea00640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83d0072480 0x7f83d0152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:25.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.893+0000 7f83cf400640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83d0153f20 0x7f83d01dd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:31:25.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.894+0000 7f83c6000640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83d00c23a0 0x7f83d01dfa20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f83b800c970 tx=0x7f83b800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:25.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.894+0000 7f83c7400640 1 -- 172.21.15.31:0/3514134333 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8007c70 con 0x7f83d00c23a0 2024-09-11T00:31:25.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.894+0000 7f83c7400640 1 -- 172.21.15.31:0/3514134333 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83b800f450 con 0x7f83d00c23a0 2024-09-11T00:31:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.894+0000 7f83c7400640 1 -- 172.21.15.31:0/3514134333 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b800f8a0 con 0x7f83d00c23a0 2024-09-11T00:31:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.894+0000 7f83d5a00640 1 -- 172.21.15.31:0/3514134333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d014cdd0 con 0x7f83d00c23a0 2024-09-11T00:31:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.894+0000 7f83d5a00640 1 -- 172.21.15.31:0/3514134333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d01dff60 con 0x7f83d00c23a0 2024-09-11T00:31:25.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.895+0000 7f83d5a00640 1 -- 172.21.15.31:0/3514134333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d01511a0 con 0x7f83d00c23a0 2024-09-11T00:31:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.896+0000 7f83c7400640 1 -- 172.21.15.31:0/3514134333 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f83b8020030 con 0x7f83d00c23a0 2024-09-11T00:31:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.897+0000 7f83c7400640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f83ac077fa0 0x7f83ac07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.897+0000 7f83c7400640 1 -- 172.21.15.31:0/3514134333 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6666+0+0 (secure 0 0 0) 0x7f83b809a670 con 0x7f83d00c23a0 2024-09-11T00:31:25.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.898+0000 7f83cea00640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f83ac077fa0 0x7f83ac07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:25.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.899+0000 7f83cea00640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f83ac077fa0 0x7f83ac07a460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f83c0033000 tx=0x7f83c0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:25.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:25.901+0000 7f83c7400640 1 -- 172.21.15.31:0/3514134333 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f83b8014030 con 0x7f83d00c23a0 2024-09-11T00:31:26.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.106+0000 7f83d5a00640 1 -- 172.21.15.31:0/3514134333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83d01dda30 con 0x7f83d00c23a0 2024-09-11T00:31:26.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.107+0000 7f83c7400640 1 -- 172.21.15.31:0/3514134333 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20679 (secure 0 0 0) 0x7f83b8062da0 con 0x7f83d00c23a0 2024-09-11T00:31:26.110 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:26.110 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:31:25.097993+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.35","mappings":[{"from":0,"to":7}]},{"pgid":"5.8","mappings":[{"from":4,"to":5}]},{"pgid":"5.17","mappings":[{"from":4,"to":3}]},{"pgid":"5.26","mappings":[{"from":4,"to":5}]},{"pgid":"5.28","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:31:26.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.110+0000 7f83d5a00640 1 -- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f83ac077fa0 msgr2=0x7f83ac07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:26.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.110+0000 7f83d5a00640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f83ac077fa0 0x7f83ac07a460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f83c0033000 tx=0x7f83c0004730 comp rx=0 tx=0).stop 2024-09-11T00:31:26.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.111+0000 7f83d5a00640 1 -- 172.21.15.31:0/3514134333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83d00c23a0 msgr2=0x7f83d01dfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:26.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.111+0000 7f83d5a00640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83d00c23a0 0x7f83d01dfa20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f83b800c970 tx=0x7f83b800ce30 comp rx=0 tx=0).stop 2024-09-11T00:31:26.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.111+0000 7f83d5a00640 1 -- 172.21.15.31:0/3514134333 shutdown_connections 2024-09-11T00:31:26.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.111+0000 7f83d5a00640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f83ac077fa0 0x7f83ac07a460 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:26.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.111+0000 7f83d5a00640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83d0153f20 0x7f83d01dd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:26.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.111+0000 7f83d5a00640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83d00c23a0 0x7f83d01dfa20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:26.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.111+0000 7f83d5a00640 1 --2- 172.21.15.31:0/3514134333 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83d0072480 0x7f83d0152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:26.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.111+0000 7f83d5a00640 1 -- 172.21.15.31:0/3514134333 >> 172.21.15.31:0/3514134333 conn(0x7f83d00a9750 msgr2=0x7f83d0150360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:26.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.112+0000 7f83d5a00640 1 -- 172.21.15.31:0/3514134333 shutdown_connections 2024-09-11T00:31:26.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:26.112+0000 7f83d5a00640 1 -- 172.21.15.31:0/3514134333 wait complete. 2024-09-11T00:31:26.133 DEBUG:tasks.mds_thrash:Yielding 2024-09-11T00:31:26.133 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T00:31:26.145 INFO:tasks.workunit:Pulling workunits from ref 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T00:31:26.146 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T00:31:26.146 INFO:tasks.workunit:timeout=3h 2024-09-11T00:31:26.146 INFO:tasks.workunit:cleanup=True 2024-09-11T00:31:26.147 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:26.147 INFO:tasks.mds_thrash.fs.[b]:starting mds_do_thrash for fs b 2024-09-11T00:31:26.148 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before thrashing 2024-09-11T00:31:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-09-11T00:31:26.184 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 2ah/42d Inode: 1 Links: 2 2024-09-11T00:31:26.185 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T00:31:26.185 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:fusefs_t:s0 2024-09-11T00:31:26.185 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-11T00:31:26.185 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-09-11 00:30:45.901071655 +0000 2024-09-11T00:31:26.185 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-09-11 00:31:19.516113082 +0000 2024-09-11T00:31:26.185 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2024-09-11T00:31:26.185 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-11T00:31:26.185 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-11T00:31:26.290 DEBUG:teuthology.orchestra.run.smithi204:> stat -- /home/ubuntu/cephtest/mnt.1 2024-09-11T00:31:26.331 INFO:teuthology.orchestra.run.smithi204.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-09-11T00:31:26.331 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-09-11T00:31:26.332 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 2bh/43d Inode: 1 Links: 2 2024-09-11T00:31:26.332 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T00:31:26.332 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:fusefs_t:s0 2024-09-11T00:31:26.332 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-11T00:31:26.332 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-11 00:30:54.982556812 +0000 2024-09-11T00:31:26.332 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-11 00:31:19.650898978 +0000 2024-09-11T00:31:26.332 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2024-09-11T00:31:26.332 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-09-11T00:31:26.332 DEBUG:teuthology.orchestra.run.smithi204:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-09-11T00:31:26.439 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T00:31:26.439 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T00:31:26.477 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-11T00:31:26.478 INFO:tasks.workunit.client.1.smithi204.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-09-11T00:31:28.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:28.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.318+0000 7f1b2f200640 1 -- 172.21.15.31:0/3539277251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b30150020 msgr2=0x7f1b300ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:28.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.318+0000 7f1b2f200640 1 --2- 172.21.15.31:0/3539277251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b30150020 0x7f1b300ad4e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1b20005fd0 tx=0x7f1b2002a5d0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.319+0000 7f1b2f200640 1 -- 172.21.15.31:0/3539277251 shutdown_connections 2024-09-11T00:31:28.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.319+0000 7f1b2f200640 1 --2- 172.21.15.31:0/3539277251 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b300ada20 0x7f1b3006e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.319+0000 7f1b2f200640 1 --2- 172.21.15.31:0/3539277251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b30150020 0x7f1b300ad4e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.319+0000 7f1b2f200640 1 -- 172.21.15.31:0/3539277251 >> 172.21.15.31:0/3539277251 conn(0x7f1b30068f10 msgr2=0x7f1b300cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:28.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.319+0000 7f1b2f200640 1 -- 172.21.15.31:0/3539277251 shutdown_connections 2024-09-11T00:31:28.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.319+0000 7f1b2f200640 1 -- 172.21.15.31:0/3539277251 wait complete. 2024-09-11T00:31:28.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.320+0000 7f1b2f200640 1 Processor -- start 2024-09-11T00:31:28.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.321+0000 7f1b2f200640 1 -- start start 2024-09-11T00:31:28.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.321+0000 7f1b2f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b300ada20 0x7f1b3014ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:28.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.321+0000 7f1b2f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b30150020 0x7f1b30150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:28.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.321+0000 7f1b2f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b30156960 0x7f1b301dfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:28.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.321+0000 7f1b2f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1b30157550 con 0x7f1b30150020 2024-09-11T00:31:28.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.321+0000 7f1b2f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1b301573d0 con 0x7f1b300ada20 2024-09-11T00:31:28.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.321+0000 7f1b2f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1b300cc220 con 0x7f1b30156960 2024-09-11T00:31:28.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.322+0000 7f1b2d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b30150020 0x7f1b30150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:28.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.322+0000 7f1b2d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b30150020 0x7f1b30150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33892/0 (socket says 172.21.15.31:33892) 2024-09-11T00:31:28.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.322+0000 7f1b2d400640 1 -- 172.21.15.31:0/574984340 learned_addr learned my addr 172.21.15.31:0/574984340 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:28.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.322+0000 7f1b2de00640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b300ada20 0x7f1b3014ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:28.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.322+0000 7f1b2e800640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b30156960 0x7f1b301dfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:28.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.322+0000 7f1b2d400640 1 -- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b30156960 msgr2=0x7f1b301dfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:28.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.322+0000 7f1b2d400640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b30156960 0x7f1b301dfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.322+0000 7f1b2d400640 1 -- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b300ada20 msgr2=0x7f1b3014ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:28.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.322+0000 7f1b2d400640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b300ada20 0x7f1b3014ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.322+0000 7f1b2d400640 1 -- 172.21.15.31:0/574984340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b300c07d0 con 0x7f1b30150020 2024-09-11T00:31:28.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.323+0000 7f1b2d400640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b30150020 0x7f1b30150960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00d770 tx=0x7f1b1c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:28.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.323+0000 7f1b26000640 1 -- 172.21.15.31:0/574984340 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c017070 con 0x7f1b30150020 2024-09-11T00:31:28.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.323+0000 7f1b26000640 1 -- 172.21.15.31:0/574984340 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b1c009d20 con 0x7f1b30150020 2024-09-11T00:31:28.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.323+0000 7f1b26000640 1 -- 172.21.15.31:0/574984340 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c005280 con 0x7f1b30150020 2024-09-11T00:31:28.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.323+0000 7f1b2f200640 1 -- 172.21.15.31:0/574984340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b300cb9b0 con 0x7f1b30150020 2024-09-11T00:31:28.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.323+0000 7f1b2f200640 1 -- 172.21.15.31:0/574984340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b300cb3f0 con 0x7f1b30150020 2024-09-11T00:31:28.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.324+0000 7f1b2f200640 1 -- 172.21.15.31:0/574984340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b300ad4e0 con 0x7f1b30150020 2024-09-11T00:31:28.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.325+0000 7f1b26000640 1 -- 172.21.15.31:0/574984340 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1b1c005420 con 0x7f1b30150020 2024-09-11T00:31:28.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.326+0000 7f1b26000640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1b10077f70 0x7f1b1007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:28.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.326+0000 7f1b26000640 1 -- 172.21.15.31:0/574984340 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f1b1c09a2e0 con 0x7f1b30150020 2024-09-11T00:31:28.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.327+0000 7f1b2de00640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1b10077f70 0x7f1b1007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:28.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.327+0000 7f1b2de00640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1b10077f70 0x7f1b1007a430 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1b20033000 tx=0x7f1b20004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:28.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.330+0000 7f1b26000640 1 -- 172.21.15.31:0/574984340 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1b1c0623c0 con 0x7f1b30150020 2024-09-11T00:31:28.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.562+0000 7f1b2f200640 1 -- 172.21.15.31:0/574984340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b301506c0 con 0x7f1b30150020 2024-09-11T00:31:28.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.562+0000 7f1b26000640 1 -- 172.21.15.31:0/574984340 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f1b1c06b130 con 0x7f1b30150020 2024-09-11T00:31:28.565 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:28.565 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:28.566 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:28.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 -- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1b10077f70 msgr2=0x7f1b1007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:28.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1b10077f70 0x7f1b1007a430 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1b20033000 tx=0x7f1b20004730 comp rx=0 tx=0).stop 2024-09-11T00:31:28.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 -- 172.21.15.31:0/574984340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b30150020 msgr2=0x7f1b30150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b30150020 0x7f1b30150960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00d770 tx=0x7f1b1c00dc30 comp rx=0 tx=0).stop 2024-09-11T00:31:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 -- 172.21.15.31:0/574984340 shutdown_connections 2024-09-11T00:31:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1b10077f70 0x7f1b1007a430 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1b30156960 0x7f1b301dfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b30150020 0x7f1b30150960 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 --2- 172.21.15.31:0/574984340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b300ada20 0x7f1b3014ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 -- 172.21.15.31:0/574984340 >> 172.21.15.31:0/574984340 conn(0x7f1b30068f10 msgr2=0x7f1b301576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 -- 172.21.15.31:0/574984340 shutdown_connections 2024-09-11T00:31:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.565+0000 7f1b2f200640 1 -- 172.21.15.31:0/574984340 wait complete. 2024-09-11T00:31:28.583 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-11T00:31:28.583 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-11T00:31:28.584 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:31:28.602 INFO:tasks.ceph.mds.b:Stopped 2024-09-11T00:31:28.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:28.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.762+0000 7f8667400640 1 -- 172.21.15.31:0/840381655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644003aa0 msgr2=0x7f8644003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.762+0000 7f8667400640 1 --2- 172.21.15.31:0/840381655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644003aa0 0x7f8644003ec0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8650001eb0 tx=0x7f8644005830 comp rx=0 tx=0).stop 2024-09-11T00:31:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.762+0000 7f8667400640 1 -- 172.21.15.31:0/840381655 shutdown_connections 2024-09-11T00:31:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.762+0000 7f8667400640 1 --2- 172.21.15.31:0/840381655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644003aa0 0x7f8644003ec0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.762+0000 7f8667400640 1 -- 172.21.15.31:0/840381655 >> 172.21.15.31:0/840381655 conn(0x7f86600a9750 msgr2=0x7f86600a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.762+0000 7f8667400640 1 -- 172.21.15.31:0/840381655 shutdown_connections 2024-09-11T00:31:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.762+0000 7f8667400640 1 -- 172.21.15.31:0/840381655 wait complete. 2024-09-11T00:31:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.764+0000 7f8667400640 1 Processor -- start 2024-09-11T00:31:28.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.764+0000 7f8667400640 1 -- start start 2024-09-11T00:31:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8667400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8644003aa0 0x7f86600a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8667400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8660143fb0 0x7f8660142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8667400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8660143400 0x7f86601ebb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8667400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8660074200 con 0x7f8660143fb0 2024-09-11T00:31:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8667400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8660074080 con 0x7f8660143400 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8667400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8660072860 con 0x7f8644003aa0 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8657e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8660143fb0 0x7f8660142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8657e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8660143fb0 0x7f8660142ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33922/0 (socket says 172.21.15.31:33922) 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8657e00640 1 -- 172.21.15.31:0/3611766020 learned_addr learned my addr 172.21.15.31:0/3611766020 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8664a00640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8644003aa0 0x7f86600a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.765+0000 7f8665400640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8660143400 0x7f86601ebb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f8657e00640 1 -- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8644003aa0 msgr2=0x7f86600a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f8657e00640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8644003aa0 0x7f86600a8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f8657e00640 1 -- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8660143400 msgr2=0x7f86601ebb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f8657e00640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8660143400 0x7f86601ebb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f8657e00640 1 -- 172.21.15.31:0/3611766020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8660143860 con 0x7f8660143fb0 2024-09-11T00:31:28.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f8657e00640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8660143fb0 0x7f8660142ec0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8648004dc0 tx=0x7f8648001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:28.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f865d600640 1 -- 172.21.15.31:0/3611766020 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648015020 con 0x7f8660143fb0 2024-09-11T00:31:28.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f865d600640 1 -- 172.21.15.31:0/3611766020 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8648011960 con 0x7f8660143fb0 2024-09-11T00:31:28.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f865d600640 1 -- 172.21.15.31:0/3611766020 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648001d40 con 0x7f8660143fb0 2024-09-11T00:31:28.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f8667400640 1 -- 172.21.15.31:0/3611766020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8660143cd0 con 0x7f8660143fb0 2024-09-11T00:31:28.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.766+0000 7f8667400640 1 -- 172.21.15.31:0/3611766020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86601ef400 con 0x7f8660143fb0 2024-09-11T00:31:28.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.768+0000 7f8667400640 1 -- 172.21.15.31:0/3611766020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8660154000 con 0x7f8660143fb0 2024-09-11T00:31:28.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.769+0000 7f865d600640 1 -- 172.21.15.31:0/3611766020 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f864801a400 con 0x7f8660143fb0 2024-09-11T00:31:28.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.770+0000 7f865d600640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f863c077f70 0x7f863c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:28.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.770+0000 7f865d600640 1 -- 172.21.15.31:0/3611766020 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f864809acb0 con 0x7f8660143fb0 2024-09-11T00:31:28.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.771+0000 7f8664a00640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f863c077f70 0x7f863c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:28.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.771+0000 7f8664a00640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f863c077f70 0x7f863c07a430 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8650033000 tx=0x7f86600c1d20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:28.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:28.775+0000 7f865d600640 1 -- 172.21.15.31:0/3611766020 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f864801f020 con 0x7f8660143fb0 2024-09-11T00:31:29.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.009+0000 7f8667400640 1 -- 172.21.15.31:0/3611766020 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86601514c0 con 0x7f8660143fb0 2024-09-11T00:31:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.009+0000 7f865d600640 1 -- 172.21.15.31:0/3611766020 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f86480633e0 con 0x7f8660143fb0 2024-09-11T00:31:29.011 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:29.011 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:29.012 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 -- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f863c077f70 msgr2=0x7f863c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f863c077f70 0x7f863c07a430 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8650033000 tx=0x7f86600c1d20 comp rx=0 tx=0).stop 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 -- 172.21.15.31:0/3611766020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8660143fb0 msgr2=0x7f8660142ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8660143fb0 0x7f8660142ec0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8648004dc0 tx=0x7f8648001630 comp rx=0 tx=0).stop 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 -- 172.21.15.31:0/3611766020 shutdown_connections 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f863c077f70 0x7f863c07a430 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8660143400 0x7f86601ebb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8660143fb0 0x7f8660142ec0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 --2- 172.21.15.31:0/3611766020 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8644003aa0 0x7f86600a8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 -- 172.21.15.31:0/3611766020 >> 172.21.15.31:0/3611766020 conn(0x7f86600a9750 msgr2=0x7f86600a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 -- 172.21.15.31:0/3611766020 shutdown_connections 2024-09-11T00:31:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:29.012+0000 7f8667400640 1 -- 172.21.15.31:0/3611766020 wait complete. 2024-09-11T00:31:29.029 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:31:31.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:31.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.200+0000 7f0d7e800640 1 -- 172.21.15.31:0/2009983564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d780ac2c0 msgr2=0x7f0d780ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:31.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.200+0000 7f0d7e800640 1 --2- 172.21.15.31:0/2009983564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d780ac2c0 0x7f0d780ac6a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0d780c5ea0 tx=0x7f0d6c02a420 comp rx=0 tx=0).stop 2024-09-11T00:31:31.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.201+0000 7f0d7e800640 1 -- 172.21.15.31:0/2009983564 shutdown_connections 2024-09-11T00:31:31.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.201+0000 7f0d7e800640 1 --2- 172.21.15.31:0/2009983564 >> v2:172.21.15.204:3300/0 conn(0x7f0d780acbe0 0x7f0d780c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:31.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.201+0000 7f0d7e800640 1 --2- 172.21.15.31:0/2009983564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d780ac2c0 0x7f0d780ac6a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:31.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.201+0000 7f0d7e800640 1 -- 172.21.15.31:0/2009983564 >> 172.21.15.31:0/2009983564 conn(0x7f0d7806caf0 msgr2=0x7f0d7806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:31.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.201+0000 7f0d7e800640 1 -- 172.21.15.31:0/2009983564 shutdown_connections 2024-09-11T00:31:31.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.201+0000 7f0d7e800640 1 -- 172.21.15.31:0/2009983564 wait complete. 2024-09-11T00:31:31.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.202+0000 7f0d7e800640 1 Processor -- start 2024-09-11T00:31:31.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.202+0000 7f0d7e800640 1 -- start start 2024-09-11T00:31:31.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.203+0000 7f0d7e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d780a6e20 0x7f0d7814c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:31.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.203+0000 7f0d7e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d780ac2c0 0x7f0d781eee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.203+0000 7f0d7e800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0d780acbe0 0x7f0d781ef3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.203+0000 7f0d7e800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0d78151950 con 0x7f0d780ac2c0 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.203+0000 7f0d7e800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781517d0 con 0x7f0d780a6e20 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.203+0000 7f0d7e800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0d780c66f0 con 0x7f0d780acbe0 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d780ac2c0 0x7f0d781eee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d780ac2c0 0x7f0d781eee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33934/0 (socket says 172.21.15.31:33934) 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7ca00640 1 -- 172.21.15.31:0/2611609039 learned_addr learned my addr 172.21.15.31:0/2611609039 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7d400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d780a6e20 0x7f0d7814c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7de00640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0d780acbe0 0x7f0d781ef3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7ca00640 1 -- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0d780acbe0 msgr2=0x7f0d781ef3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:31.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7ca00640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0d780acbe0 0x7f0d781ef3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:31.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7ca00640 1 -- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d780a6e20 msgr2=0x7f0d7814c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:31.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7ca00640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d780a6e20 0x7f0d7814c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:31.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7ca00640 1 -- 172.21.15.31:0/2611609039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7814c570 con 0x7f0d780ac2c0 2024-09-11T00:31:31.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.204+0000 7f0d7ca00640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d780ac2c0 0x7f0d781eee70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400c460 tx=0x7f0d6400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:31.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.205+0000 7f0d75600640 1 -- 172.21.15.31:0/2611609039 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d64011790 con 0x7f0d780ac2c0 2024-09-11T00:31:31.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.205+0000 7f0d75600640 1 -- 172.21.15.31:0/2611609039 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d64017070 con 0x7f0d780ac2c0 2024-09-11T00:31:31.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.205+0000 7f0d75600640 1 -- 172.21.15.31:0/2611609039 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d64011d00 con 0x7f0d780ac2c0 2024-09-11T00:31:31.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.205+0000 7f0d7e800640 1 -- 172.21.15.31:0/2611609039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d780c1780 con 0x7f0d780ac2c0 2024-09-11T00:31:31.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.205+0000 7f0d7e800640 1 -- 172.21.15.31:0/2611609039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d781dd990 con 0x7f0d780ac2c0 2024-09-11T00:31:31.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.206+0000 7f0d7e800640 1 -- 172.21.15.31:0/2611609039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d780c6b30 con 0x7f0d780ac2c0 2024-09-11T00:31:31.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.207+0000 7f0d75600640 1 -- 172.21.15.31:0/2611609039 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0d6401c460 con 0x7f0d780ac2c0 2024-09-11T00:31:31.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.208+0000 7f0d75600640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d58077f70 0x7f0d5807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:31.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.208+0000 7f0d75600640 1 -- 172.21.15.31:0/2611609039 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f0d64099ee0 con 0x7f0d780ac2c0 2024-09-11T00:31:31.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.208+0000 7f0d7d400640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d58077f70 0x7f0d5807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:31.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.209+0000 7f0d7d400640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d58077f70 0x7f0d5807a430 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c033000 tx=0x7f0d6c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:31.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.212+0000 7f0d75600640 1 -- 172.21.15.31:0/2611609039 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0d7814c570 con 0x7f0d780ac2c0 2024-09-11T00:31:31.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.442+0000 7f0d7e800640 1 -- 172.21.15.31:0/2611609039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d781ef8f0 con 0x7f0d780ac2c0 2024-09-11T00:31:31.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.443+0000 7f0d75600640 1 -- 172.21.15.31:0/2611609039 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f0d64062610 con 0x7f0d780ac2c0 2024-09-11T00:31:31.445 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:31.445 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:31.445 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:31.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 -- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d58077f70 msgr2=0x7f0d5807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:31.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d58077f70 0x7f0d5807a430 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c033000 tx=0x7f0d6c004730 comp rx=0 tx=0).stop 2024-09-11T00:31:31.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 -- 172.21.15.31:0/2611609039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d780ac2c0 msgr2=0x7f0d781eee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:31.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d780ac2c0 0x7f0d781eee70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400c460 tx=0x7f0d6400c920 comp rx=0 tx=0).stop 2024-09-11T00:31:31.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 -- 172.21.15.31:0/2611609039 shutdown_connections 2024-09-11T00:31:31.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d58077f70 0x7f0d5807a430 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:31.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0d780acbe0 0x7f0d781ef3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:31.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d780ac2c0 0x7f0d781eee70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:31.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 --2- 172.21.15.31:0/2611609039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d780a6e20 0x7f0d7814c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:31.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 -- 172.21.15.31:0/2611609039 >> 172.21.15.31:0/2611609039 conn(0x7f0d7806caf0 msgr2=0x7f0d7806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:31.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 -- 172.21.15.31:0/2611609039 shutdown_connections 2024-09-11T00:31:31.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:31.445+0000 7f0d7e800640 1 -- 172.21.15.31:0/2611609039 wait complete. 2024-09-11T00:31:31.463 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:31:33.463 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:33.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.629+0000 7fa877e00640 1 -- 172.21.15.31:0/1290646604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8780cc220 msgr2=0x7fa87806e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:33.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.629+0000 7fa877e00640 1 --2- 172.21.15.31:0/1290646604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8780cc220 0x7fa87806e680 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa870005fd0 tx=0x7fa87002a820 comp rx=0 tx=0).stop 2024-09-11T00:31:33.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.630+0000 7fa877e00640 1 -- 172.21.15.31:0/1290646604 shutdown_connections 2024-09-11T00:31:33.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.630+0000 7fa877e00640 1 --2- 172.21.15.31:0/1290646604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8780cc220 0x7fa87806e680 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:33.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.630+0000 7fa877e00640 1 --2- 172.21.15.31:0/1290646604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa878156960 0x7fa87806e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:33.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.630+0000 7fa877e00640 1 -- 172.21.15.31:0/1290646604 >> 172.21.15.31:0/1290646604 conn(0x7fa878068f10 msgr2=0x7fa8780cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:33.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.630+0000 7fa877e00640 1 -- 172.21.15.31:0/1290646604 shutdown_connections 2024-09-11T00:31:33.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.630+0000 7fa877e00640 1 -- 172.21.15.31:0/1290646604 wait complete. 2024-09-11T00:31:33.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.631+0000 7fa877e00640 1 Processor -- start 2024-09-11T00:31:33.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.631+0000 7fa877e00640 1 -- start start 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.632+0000 7fa877e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8780cc220 0x7fa8781e3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.632+0000 7fa877e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa878150020 0x7fa87814e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.632+0000 7fa877e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa878156960 0x7fa87814ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.632+0000 7fa877e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa8780c6ca0 con 0x7fa8780cc220 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.632+0000 7fa877e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa8780c6b20 con 0x7fa878156960 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.632+0000 7fa877e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa8781573d0 con 0x7fa878150020 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.632+0000 7fa876a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8780cc220 0x7fa8781e3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.633+0000 7fa876a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8780cc220 0x7fa8781e3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33958/0 (socket says 172.21.15.31:33958) 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.633+0000 7fa876a00640 1 -- 172.21.15.31:0/1597055505 learned_addr learned my addr 172.21.15.31:0/1597055505 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.633+0000 7fa86e000640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa878150020 0x7fa87814e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:33.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.633+0000 7fa877400640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa878156960 0x7fa87814ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:33.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.633+0000 7fa876a00640 1 -- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa878150020 msgr2=0x7fa87814e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:33.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.633+0000 7fa876a00640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa878150020 0x7fa87814e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:33.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.633+0000 7fa876a00640 1 -- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa878156960 msgr2=0x7fa87814ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:33.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.633+0000 7fa876a00640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa878156960 0x7fa87814ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:33.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.633+0000 7fa876a00640 1 -- 172.21.15.31:0/1597055505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8780c2640 con 0x7fa8780cc220 2024-09-11T00:31:33.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.633+0000 7fa876a00640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8780cc220 0x7fa8781e3070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c4c0 tx=0x7fa86800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:33.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.634+0000 7fa86f400640 1 -- 172.21.15.31:0/1597055505 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868016020 con 0x7fa8780cc220 2024-09-11T00:31:33.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.634+0000 7fa86f400640 1 -- 172.21.15.31:0/1597055505 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa868012940 con 0x7fa8780cc220 2024-09-11T00:31:33.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.634+0000 7fa86f400640 1 -- 172.21.15.31:0/1597055505 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868012d90 con 0x7fa8780cc220 2024-09-11T00:31:33.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.634+0000 7fa877e00640 1 -- 172.21.15.31:0/1597055505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa87814f560 con 0x7fa8780cc220 2024-09-11T00:31:33.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.634+0000 7fa877e00640 1 -- 172.21.15.31:0/1597055505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8780bf5a0 con 0x7fa8780cc220 2024-09-11T00:31:33.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.635+0000 7fa877e00640 1 -- 172.21.15.31:0/1597055505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8780ad1d0 con 0x7fa8780cc220 2024-09-11T00:31:33.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.636+0000 7fa86f400640 1 -- 172.21.15.31:0/1597055505 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa86801a3b0 con 0x7fa8780cc220 2024-09-11T00:31:33.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.637+0000 7fa86f400640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa854077f70 0x7fa85407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:33.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.637+0000 7fa86f400640 1 -- 172.21.15.31:0/1597055505 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7fa868099ed0 con 0x7fa8780cc220 2024-09-11T00:31:33.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.637+0000 7fa86e000640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa854077f70 0x7fa85407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:33.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.638+0000 7fa86e000640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa854077f70 0x7fa85407a430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa8780c5e20 tx=0x7fa860009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:33.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.641+0000 7fa86f400640 1 -- 172.21.15.31:0/1597055505 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa8780ad1d0 con 0x7fa8780cc220 2024-09-11T00:31:33.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.890+0000 7fa877e00640 1 -- 172.21.15.31:0/1597055505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8780ad4e0 con 0x7fa8780cc220 2024-09-11T00:31:33.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.891+0000 7fa86f400640 1 -- 172.21.15.31:0/1597055505 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fa868020030 con 0x7fa8780cc220 2024-09-11T00:31:33.893 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:33.893 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:33.893 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 -- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa854077f70 msgr2=0x7fa85407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa854077f70 0x7fa85407a430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa8780c5e20 tx=0x7fa860009060 comp rx=0 tx=0).stop 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 -- 172.21.15.31:0/1597055505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8780cc220 msgr2=0x7fa8781e3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8780cc220 0x7fa8781e3070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c4c0 tx=0x7fa86800c980 comp rx=0 tx=0).stop 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 -- 172.21.15.31:0/1597055505 shutdown_connections 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa854077f70 0x7fa85407a430 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa878156960 0x7fa87814ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa878150020 0x7fa87814e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 --2- 172.21.15.31:0/1597055505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa8780cc220 0x7fa8781e3070 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 -- 172.21.15.31:0/1597055505 >> 172.21.15.31:0/1597055505 conn(0x7fa878068f10 msgr2=0x7fa878159c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:33.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 -- 172.21.15.31:0/1597055505 shutdown_connections 2024-09-11T00:31:33.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:33.894+0000 7fa877e00640 1 -- 172.21.15.31:0/1597055505 wait complete. 2024-09-11T00:31:33.913 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:31:35.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:36.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.082+0000 7f4e12000640 1 -- 172.21.15.31:0/2031240582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e0c150020 msgr2=0x7f4e0c0ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:36.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.082+0000 7f4e12000640 1 --2- 172.21.15.31:0/2031240582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e0c150020 0x7f4e0c0ad4e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c06eda0 tx=0x7f4e0002a4d0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.082+0000 7f4e12000640 1 -- 172.21.15.31:0/2031240582 shutdown_connections 2024-09-11T00:31:36.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.082+0000 7f4e12000640 1 --2- 172.21.15.31:0/2031240582 >> v2:172.21.15.204:3300/0 conn(0x7f4e0c156960 0x7f4e0c06e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.082+0000 7f4e12000640 1 --2- 172.21.15.31:0/2031240582 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e0c0ada20 0x7f4e0c06e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.082+0000 7f4e12000640 1 --2- 172.21.15.31:0/2031240582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e0c150020 0x7f4e0c0ad4e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.082+0000 7f4e12000640 1 -- 172.21.15.31:0/2031240582 >> 172.21.15.31:0/2031240582 conn(0x7f4e0c068f10 msgr2=0x7f4e0c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:36.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.082+0000 7f4e12000640 1 -- 172.21.15.31:0/2031240582 shutdown_connections 2024-09-11T00:31:36.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.082+0000 7f4e12000640 1 -- 172.21.15.31:0/2031240582 wait complete. 2024-09-11T00:31:36.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.084+0000 7f4e12000640 1 Processor -- start 2024-09-11T00:31:36.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.084+0000 7f4e12000640 1 -- start start 2024-09-11T00:31:36.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e12000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e0c0ada20 0x7f4e0c0c19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e12000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e0c150020 0x7f4e0c0c0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e12000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e0c156960 0x7f4e0c0cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e12000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c157550 con 0x7f4e0c150020 2024-09-11T00:31:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e12000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c1573d0 con 0x7f4e0c0ada20 2024-09-11T00:31:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e12000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0c0cc220 con 0x7f4e0c156960 2024-09-11T00:31:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e0c150020 0x7f4e0c0c0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e10c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e0c0ada20 0x7f4e0c0c19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e0c150020 0x7f4e0c0c0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33972/0 (socket says 172.21.15.31:33972) 2024-09-11T00:31:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e0be00640 1 -- 172.21.15.31:0/2471583692 learned_addr learned my addr 172.21.15.31:0/2471583692 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e11600640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e0c156960 0x7f4e0c0cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e0be00640 1 -- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e0c156960 msgr2=0x7f4e0c0cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e0be00640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e0c156960 0x7f4e0c0cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e0be00640 1 -- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e0c0ada20 msgr2=0x7f4e0c0c19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e0be00640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e0c0ada20 0x7f4e0c0c19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.085+0000 7f4e0be00640 1 -- 172.21.15.31:0/2471583692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0c0afb90 con 0x7f4e0c150020 2024-09-11T00:31:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.086+0000 7f4e0be00640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e0c150020 0x7f4e0c0c0de0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4df800e7f0 tx=0x7f4df800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.086+0000 7f4e08c00640 1 -- 172.21.15.31:0/2471583692 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df800eea0 con 0x7f4e0c150020 2024-09-11T00:31:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.086+0000 7f4e08c00640 1 -- 172.21.15.31:0/2471583692 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4df800cc60 con 0x7f4e0c150020 2024-09-11T00:31:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.086+0000 7f4e12000640 1 -- 172.21.15.31:0/2471583692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0c0afe70 con 0x7f4e0c150020 2024-09-11T00:31:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.086+0000 7f4e08c00640 1 -- 172.21.15.31:0/2471583692 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8005020 con 0x7f4e0c150020 2024-09-11T00:31:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.086+0000 7f4e12000640 1 -- 172.21.15.31:0/2471583692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e0c0cb190 con 0x7f4e0c150020 2024-09-11T00:31:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.087+0000 7f4e12000640 1 -- 172.21.15.31:0/2471583692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e0c0ad4e0 con 0x7f4e0c150020 2024-09-11T00:31:36.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.089+0000 7f4e08c00640 1 -- 172.21.15.31:0/2471583692 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4df80051c0 con 0x7f4e0c150020 2024-09-11T00:31:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.090+0000 7f4e08c00640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4dec077fa0 0x7f4dec07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.090+0000 7f4e08c00640 1 -- 172.21.15.31:0/2471583692 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f4df809a8b0 con 0x7f4e0c150020 2024-09-11T00:31:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.090+0000 7f4e10c00640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4dec077fa0 0x7f4dec07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.091+0000 7f4e10c00640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4dec077fa0 0x7f4dec07a460 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4e00033000 tx=0x7f4e00004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:36.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.094+0000 7f4e08c00640 1 -- 172.21.15.31:0/2471583692 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4df8062990 con 0x7f4e0c150020 2024-09-11T00:31:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.344+0000 7f4e12000640 1 -- 172.21.15.31:0/2471583692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e0c0cc8e0 con 0x7f4e0c150020 2024-09-11T00:31:36.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.344+0000 7f4e08c00640 1 -- 172.21.15.31:0/2471583692 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f4df80673c0 con 0x7f4e0c150020 2024-09-11T00:31:36.346 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:36.346 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:36.347 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 -- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4dec077fa0 msgr2=0x7f4dec07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4dec077fa0 0x7f4dec07a460 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4e00033000 tx=0x7f4e00004730 comp rx=0 tx=0).stop 2024-09-11T00:31:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 -- 172.21.15.31:0/2471583692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e0c150020 msgr2=0x7f4e0c0c0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e0c150020 0x7f4e0c0c0de0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4df800e7f0 tx=0x7f4df800ecb0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 -- 172.21.15.31:0/2471583692 shutdown_connections 2024-09-11T00:31:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4dec077fa0 0x7f4dec07a460 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e0c156960 0x7f4e0c0cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e0c150020 0x7f4e0c0c0de0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 --2- 172.21.15.31:0/2471583692 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e0c0ada20 0x7f4e0c0c19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 -- 172.21.15.31:0/2471583692 >> 172.21.15.31:0/2471583692 conn(0x7f4e0c068f10 msgr2=0x7f4e0c159100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:36.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 -- 172.21.15.31:0/2471583692 shutdown_connections 2024-09-11T00:31:36.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:36.347+0000 7f4e12000640 1 -- 172.21.15.31:0/2471583692 wait complete. 2024-09-11T00:31:36.365 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:31:38.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.531+0000 7fb63b200640 1 -- 172.21.15.31:0/1338611093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb63c0c9000 msgr2=0x7fb63c159990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.531+0000 7fb63b200640 1 --2- 172.21.15.31:0/1338611093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb63c0c9000 0x7fb63c159990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb63c06dd40 tx=0x7fb63402a810 comp rx=0 tx=0).stop 2024-09-11T00:31:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.532+0000 7fb63b200640 1 -- 172.21.15.31:0/1338611093 shutdown_connections 2024-09-11T00:31:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.532+0000 7fb63b200640 1 --2- 172.21.15.31:0/1338611093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb63c0c9000 0x7fb63c159990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.532+0000 7fb63b200640 1 --2- 172.21.15.31:0/1338611093 >> v2:172.21.15.204:3300/0 conn(0x7fb63c06cc10 0x7fb63c159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.532+0000 7fb63b200640 1 --2- 172.21.15.31:0/1338611093 >> v2:172.21.15.31:3300/0 conn(0x7fb63c14c880 0x7fb63c0c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.532+0000 7fb63b200640 1 -- 172.21.15.31:0/1338611093 >> 172.21.15.31:0/1338611093 conn(0x7fb63c0adb00 msgr2=0x7fb63c0adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.532+0000 7fb63b200640 1 -- 172.21.15.31:0/1338611093 shutdown_connections 2024-09-11T00:31:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.532+0000 7fb63b200640 1 -- 172.21.15.31:0/1338611093 wait complete. 2024-09-11T00:31:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.533+0000 7fb63b200640 1 Processor -- start 2024-09-11T00:31:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.534+0000 7fb63b200640 1 -- start start 2024-09-11T00:31:38.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.534+0000 7fb63b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb63c06cc10 0x7fb63c0b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:38.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.534+0000 7fb63b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb63c0c9000 0x7fb63c158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.534+0000 7fb63b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63c14c880 0x7fb63c14b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.534+0000 7fb63b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c156810 con 0x7fb63c14c880 2024-09-11T00:31:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.534+0000 7fb63b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c156530 con 0x7fb63c0c9000 2024-09-11T00:31:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.534+0000 7fb63b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb63c14cc60 con 0x7fb63c06cc10 2024-09-11T00:31:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb63a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63c14c880 0x7fb63c14b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb639e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb63c06cc10 0x7fb63c0b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb633e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb63c0c9000 0x7fb63c158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb63a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63c14c880 0x7fb63c14b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44752/0 (socket says 172.21.15.31:44752) 2024-09-11T00:31:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb63a800640 1 -- 172.21.15.31:0/1474201695 learned_addr learned my addr 172.21.15.31:0/1474201695 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb63a800640 1 -- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb63c06cc10 msgr2=0x7fb63c0b0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb63a800640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb63c06cc10 0x7fb63c0b0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb63a800640 1 -- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb63c0c9000 msgr2=0x7fb63c158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb63a800640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb63c0c9000 0x7fb63c158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb63a800640 1 -- 172.21.15.31:0/1474201695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63c147800 con 0x7fb63c14c880 2024-09-11T00:31:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.535+0000 7fb639e00640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb63c06cc10 0x7fb63c0b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.536+0000 7fb63a800640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63c14c880 0x7fb63c14b180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb634009ca0 tx=0x7fb6340315d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.536+0000 7fb632000640 1 -- 172.21.15.31:0/1474201695 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb634041020 con 0x7fb63c14c880 2024-09-11T00:31:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.536+0000 7fb632000640 1 -- 172.21.15.31:0/1474201695 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb634031ac0 con 0x7fb63c14c880 2024-09-11T00:31:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.536+0000 7fb632000640 1 -- 172.21.15.31:0/1474201695 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb634037040 con 0x7fb63c14c880 2024-09-11T00:31:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.536+0000 7fb63b200640 1 -- 172.21.15.31:0/1474201695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb63c0c81b0 con 0x7fb63c14c880 2024-09-11T00:31:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.536+0000 7fb63b200640 1 -- 172.21.15.31:0/1474201695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb63c147520 con 0x7fb63c14c880 2024-09-11T00:31:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.538+0000 7fb63b200640 1 -- 172.21.15.31:0/1474201695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb63c0c59a0 con 0x7fb63c14c880 2024-09-11T00:31:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.540+0000 7fb632000640 1 -- 172.21.15.31:0/1474201695 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb63403c460 con 0x7fb63c14c880 2024-09-11T00:31:38.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.540+0000 7fb632000640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb618077fa0 0x7fb61807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:38.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.541+0000 7fb632000640 1 -- 172.21.15.31:0/1474201695 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7fb6340bb040 con 0x7fb63c14c880 2024-09-11T00:31:38.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.541+0000 7fb639e00640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb618077fa0 0x7fb61807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:38.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.542+0000 7fb639e00640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb618077fa0 0x7fb61807a460 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb62c005fd0 tx=0x7fb62c009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:38.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.545+0000 7fb632000640 1 -- 172.21.15.31:0/1474201695 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb63404b030 con 0x7fb63c14c880 2024-09-11T00:31:38.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.785+0000 7fb63b200640 1 -- 172.21.15.31:0/1474201695 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb63c157380 con 0x7fb63c14c880 2024-09-11T00:31:38.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.785+0000 7fb632000640 1 -- 172.21.15.31:0/1474201695 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fb634083780 con 0x7fb63c14c880 2024-09-11T00:31:38.787 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:38.787 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:38.787 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:38.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 -- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb618077fa0 msgr2=0x7fb61807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:38.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb618077fa0 0x7fb61807a460 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb62c005fd0 tx=0x7fb62c009260 comp rx=0 tx=0).stop 2024-09-11T00:31:38.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 -- 172.21.15.31:0/1474201695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63c14c880 msgr2=0x7fb63c14b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:38.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63c14c880 0x7fb63c14b180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb634009ca0 tx=0x7fb6340315d0 comp rx=0 tx=0).stop 2024-09-11T00:31:38.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 -- 172.21.15.31:0/1474201695 shutdown_connections 2024-09-11T00:31:38.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb618077fa0 0x7fb61807a460 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:38.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb63c14c880 0x7fb63c14b180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:38.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb63c0c9000 0x7fb63c158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:38.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 --2- 172.21.15.31:0/1474201695 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb63c06cc10 0x7fb63c0b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:38.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 -- 172.21.15.31:0/1474201695 >> 172.21.15.31:0/1474201695 conn(0x7fb63c0adb00 msgr2=0x7fb63c0ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:38.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 -- 172.21.15.31:0/1474201695 shutdown_connections 2024-09-11T00:31:38.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:38.788+0000 7fb63b200640 1 -- 172.21.15.31:0/1474201695 wait complete. 2024-09-11T00:31:38.811 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:31:39.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:31:39.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:31:40.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.007+0000 7fb8eb200640 1 -- 172.21.15.31:0/275743724 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8ec156350 msgr2=0x7fb8ec1567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.007+0000 7fb8eb200640 1 --2- 172.21.15.31:0/275743724 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8ec156350 0x7fb8ec1567e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4005fd0 tx=0x7fb8e402a8b0 comp rx=0 tx=0).stop 2024-09-11T00:31:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.008+0000 7fb8eb200640 1 -- 172.21.15.31:0/275743724 shutdown_connections 2024-09-11T00:31:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.008+0000 7fb8eb200640 1 --2- 172.21.15.31:0/275743724 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8ec156350 0x7fb8ec1567e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.008+0000 7fb8eb200640 1 --2- 172.21.15.31:0/275743724 >> v2:172.21.15.204:3300/0 conn(0x7fb8ec0c5aa0 0x7fb8ec0c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.008+0000 7fb8eb200640 1 -- 172.21.15.31:0/275743724 >> 172.21.15.31:0/275743724 conn(0x7fb8ec06caf0 msgr2=0x7fb8ec06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.008+0000 7fb8eb200640 1 -- 172.21.15.31:0/275743724 shutdown_connections 2024-09-11T00:31:41.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.008+0000 7fb8eb200640 1 -- 172.21.15.31:0/275743724 wait complete. 2024-09-11T00:31:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.010+0000 7fb8eb200640 1 Processor -- start 2024-09-11T00:31:41.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.010+0000 7fb8eb200640 1 -- start start 2024-09-11T00:31:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8eb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8ec0c5aa0 0x7fb8ec1df930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8eb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8ec0c6420 0x7fb8ec0a9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8eb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ec156350 0x7fb8ec1e1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8eb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec0c2e80 con 0x7fb8ec156350 2024-09-11T00:31:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8eb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec0c2ba0 con 0x7fb8ec0c6420 2024-09-11T00:31:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8eb200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb8ec156d20 con 0x7fb8ec0c5aa0 2024-09-11T00:31:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8e9e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8ec0c5aa0 0x7fb8ec1df930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:41.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8ea800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ec156350 0x7fb8ec1e1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8e9e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8ec0c5aa0 0x7fb8ec1df930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:38752/0 (socket says 172.21.15.31:38752) 2024-09-11T00:31:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8e9e00640 1 -- 172.21.15.31:0/1370635106 learned_addr learned my addr 172.21.15.31:0/1370635106 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.012+0000 7fb8e9400640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8ec0c6420 0x7fb8ec0a9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.013+0000 7fb8ea800640 1 -- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8ec0c5aa0 msgr2=0x7fb8ec1df930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.013+0000 7fb8ea800640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8ec0c5aa0 0x7fb8ec1df930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:41.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.013+0000 7fb8ea800640 1 -- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8ec0c6420 msgr2=0x7fb8ec0a9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.013+0000 7fb8ea800640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8ec0c6420 0x7fb8ec0a9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.013+0000 7fb8ea800640 1 -- 172.21.15.31:0/1370635106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec1dfe70 con 0x7fb8ec156350 2024-09-11T00:31:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.013+0000 7fb8e9e00640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8ec0c5aa0 0x7fb8ec1df930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.013+0000 7fb8ea800640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ec156350 0x7fb8ec1e1e20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009000 tx=0x7fb8e40316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.014+0000 7fb8e2000640 1 -- 172.21.15.31:0/1370635106 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4031da0 con 0x7fb8ec156350 2024-09-11T00:31:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.014+0000 7fb8e2000640 1 -- 172.21.15.31:0/1370635106 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8e4003a60 con 0x7fb8ec156350 2024-09-11T00:31:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.014+0000 7fb8e2000640 1 -- 172.21.15.31:0/1370635106 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4033070 con 0x7fb8ec156350 2024-09-11T00:31:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.014+0000 7fb8eb200640 1 -- 172.21.15.31:0/1370635106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ec1e0600 con 0x7fb8ec156350 2024-09-11T00:31:41.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.014+0000 7fb8eb200640 1 -- 172.21.15.31:0/1370635106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ec1ec530 con 0x7fb8ec156350 2024-09-11T00:31:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.015+0000 7fb8eb200640 1 -- 172.21.15.31:0/1370635106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8ec1e07d0 con 0x7fb8ec156350 2024-09-11T00:31:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.016+0000 7fb8e2000640 1 -- 172.21.15.31:0/1370635106 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb8e4033210 con 0x7fb8ec156350 2024-09-11T00:31:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.017+0000 7fb8e2000640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb8cc077fa0 0x7fb8cc07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:41.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.017+0000 7fb8e2000640 1 -- 172.21.15.31:0/1370635106 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7fb8e40bb210 con 0x7fb8ec156350 2024-09-11T00:31:41.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.017+0000 7fb8e9e00640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb8cc077fa0 0x7fb8cc07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.018+0000 7fb8e9e00640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb8cc077fa0 0x7fb8cc07a460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0095f0 tx=0x7fb8dc005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:41.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.021+0000 7fb8e2000640 1 -- 172.21.15.31:0/1370635106 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb8e403f030 con 0x7fb8ec156350 2024-09-11T00:31:41.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.274+0000 7fb8eb200640 1 -- 172.21.15.31:0/1370635106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8ec0c5e80 con 0x7fb8ec156350 2024-09-11T00:31:41.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.275+0000 7fb8e2000640 1 -- 172.21.15.31:0/1370635106 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fb8e40832f0 con 0x7fb8ec156350 2024-09-11T00:31:41.276 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:41.277 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:41.277 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 -- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb8cc077fa0 msgr2=0x7fb8cc07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb8cc077fa0 0x7fb8cc07a460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0095f0 tx=0x7fb8dc005f90 comp rx=0 tx=0).stop 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 -- 172.21.15.31:0/1370635106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ec156350 msgr2=0x7fb8ec1e1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ec156350 0x7fb8ec1e1e20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009000 tx=0x7fb8e40316f0 comp rx=0 tx=0).stop 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 -- 172.21.15.31:0/1370635106 shutdown_connections 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb8cc077fa0 0x7fb8cc07a460 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ec156350 0x7fb8ec1e1e20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb8ec0c6420 0x7fb8ec0a9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 --2- 172.21.15.31:0/1370635106 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb8ec0c5aa0 0x7fb8ec1df930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 -- 172.21.15.31:0/1370635106 >> 172.21.15.31:0/1370635106 conn(0x7fb8ec06caf0 msgr2=0x7fb8ec14edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:41.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 -- 172.21.15.31:0/1370635106 shutdown_connections 2024-09-11T00:31:41.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:41.277+0000 7fb8eb200640 1 -- 172.21.15.31:0/1370635106 wait complete. 2024-09-11T00:31:41.295 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:31:43.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:43.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.466+0000 7f6fa2600640 1 -- 172.21.15.31:0/2489058390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c0c64b0 msgr2=0x7f6f9c06dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.466+0000 7f6fa2600640 1 --2- 172.21.15.31:0/2489058390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c0c64b0 0x7f6f9c06dc60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009be0 tx=0x7f6f9402a8d0 comp rx=0 tx=0).stop 2024-09-11T00:31:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.466+0000 7f6fa2600640 1 -- 172.21.15.31:0/2489058390 shutdown_connections 2024-09-11T00:31:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.466+0000 7f6fa2600640 1 --2- 172.21.15.31:0/2489058390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c0c64b0 0x7f6f9c06dc60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.466+0000 7f6fa2600640 1 -- 172.21.15.31:0/2489058390 >> 172.21.15.31:0/2489058390 conn(0x7f6f9c06caf0 msgr2=0x7f6f9c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.467+0000 7f6fa2600640 1 -- 172.21.15.31:0/2489058390 shutdown_connections 2024-09-11T00:31:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.467+0000 7f6fa2600640 1 -- 172.21.15.31:0/2489058390 wait complete. 2024-09-11T00:31:43.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.468+0000 7f6fa2600640 1 Processor -- start 2024-09-11T00:31:43.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.469+0000 7f6fa2600640 1 -- start start 2024-09-11T00:31:43.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.469+0000 7f6fa2600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f9c070410 0x7f6f9c14f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:43.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa2600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f9c0acaf0 0x7f6f9c1dbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:43.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa2600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c0c64b0 0x7f6f9c154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa2600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c0c4ad0 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa2600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c0700f0 con 0x7f6f9c0acaf0 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa2600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6f9c0c4c50 con 0x7f6f9c070410 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c0c64b0 0x7f6f9c154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c0c64b0 0x7f6f9c154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44788/0 (socket says 172.21.15.31:44788) 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa1c00640 1 -- 172.21.15.31:0/2538642316 learned_addr learned my addr 172.21.15.31:0/2538642316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa1200640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f9c070410 0x7f6f9c14f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6f9be00640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f9c0acaf0 0x7f6f9c1dbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa1c00640 1 -- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f9c070410 msgr2=0x7f6f9c14f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa1c00640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f9c070410 0x7f6f9c14f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa1c00640 1 -- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f9c0acaf0 msgr2=0x7f6f9c1dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa1c00640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f9c0acaf0 0x7f6f9c1dbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.470+0000 7f6fa1c00640 1 -- 172.21.15.31:0/2538642316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9c1dfb20 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.471+0000 7f6fa1c00640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c0c64b0 0x7f6f9c154ad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f94006010 tx=0x7f6f94033f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:43.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.471+0000 7f6f98c00640 1 -- 172.21.15.31:0/2538642316 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94031d80 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.471+0000 7f6f98c00640 1 -- 172.21.15.31:0/2538642316 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f94003a60 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.471+0000 7f6fa2600640 1 -- 172.21.15.31:0/2538642316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9c1dff40 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.471+0000 7f6f98c00640 1 -- 172.21.15.31:0/2538642316 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f940330d0 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.471+0000 7f6fa2600640 1 -- 172.21.15.31:0/2538642316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f9c1ede30 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.473+0000 7f6fa2600640 1 -- 172.21.15.31:0/2538642316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f9c06dc60 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.475+0000 7f6f98c00640 1 -- 172.21.15.31:0/2538642316 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6f9404b030 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.476+0000 7f6f98c00640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f7c077fa0 0x7f6f7c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:43.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.476+0000 7f6f98c00640 1 -- 172.21.15.31:0/2538642316 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f6f940bae40 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.476+0000 7f6fa1200640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f7c077fa0 0x7f6f7c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:43.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.477+0000 7f6fa1200640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f7c077fa0 0x7f6f7c07a460 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6f90007fd0 tx=0x7f6f90007b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:43.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.481+0000 7f6f98c00640 1 -- 172.21.15.31:0/2538642316 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6f9403f030 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.767+0000 7f6fa2600640 1 -- 172.21.15.31:0/2538642316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f9c155010 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.767+0000 7f6f98c00640 1 -- 172.21.15.31:0/2538642316 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f6f940869a0 con 0x7f6f9c0c64b0 2024-09-11T00:31:43.769 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:43.769 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":15,"btime":"2024-09-11T00:30:58:999449+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:30:58.435146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6837/3441932736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3441932736},{"type":"v1","addr":"172.21.15.204:6837","nonce":3441932736}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":2}]} 2024-09-11T00:31:43.770 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 15 2024-09-11T00:31:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.770+0000 7f6fa2600640 1 -- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f7c077fa0 msgr2=0x7f6f7c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.770+0000 7f6fa2600640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f7c077fa0 0x7f6f7c07a460 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6f90007fd0 tx=0x7f6f90007b00 comp rx=0 tx=0).stop 2024-09-11T00:31:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.770+0000 7f6fa2600640 1 -- 172.21.15.31:0/2538642316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c0c64b0 msgr2=0x7f6f9c154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.770+0000 7f6fa2600640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c0c64b0 0x7f6f9c154ad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f94006010 tx=0x7f6f94033f30 comp rx=0 tx=0).stop 2024-09-11T00:31:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.770+0000 7f6fa2600640 1 -- 172.21.15.31:0/2538642316 shutdown_connections 2024-09-11T00:31:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.770+0000 7f6fa2600640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f7c077fa0 0x7f6f7c07a460 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.770+0000 7f6fa2600640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c0c64b0 0x7f6f9c154ad0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.770+0000 7f6fa2600640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f9c0acaf0 0x7f6f9c1dbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.770+0000 7f6fa2600640 1 --2- 172.21.15.31:0/2538642316 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f9c070410 0x7f6f9c14f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.770+0000 7f6fa2600640 1 -- 172.21.15.31:0/2538642316 >> 172.21.15.31:0/2538642316 conn(0x7f6f9c06caf0 msgr2=0x7f6f9c0c43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.771+0000 7f6fa2600640 1 -- 172.21.15.31:0/2538642316 shutdown_connections 2024-09-11T00:31:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:43.771+0000 7f6fa2600640 1 -- 172.21.15.31:0/2538642316 wait complete. 2024-09-11T00:31:43.788 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:31:45.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:45.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.955+0000 7fb09da00640 1 -- 172.21.15.31:0/1493995622 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb09806cc10 msgr2=0x7fb098159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:45.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.955+0000 7fb09da00640 1 --2- 172.21.15.31:0/1493995622 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb09806cc10 0x7fb098159450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb080009830 tx=0x7fb08002a5c0 comp rx=0 tx=0).stop 2024-09-11T00:31:45.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.955+0000 7fb09da00640 1 -- 172.21.15.31:0/1493995622 shutdown_connections 2024-09-11T00:31:45.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.955+0000 7fb09da00640 1 --2- 172.21.15.31:0/1493995622 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb09806cc10 0x7fb098159450 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:45.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.955+0000 7fb09da00640 1 -- 172.21.15.31:0/1493995622 >> 172.21.15.31:0/1493995622 conn(0x7fb0980adb00 msgr2=0x7fb0980adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:45.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.955+0000 7fb09da00640 1 -- 172.21.15.31:0/1493995622 shutdown_connections 2024-09-11T00:31:45.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.955+0000 7fb09da00640 1 -- 172.21.15.31:0/1493995622 wait complete. 2024-09-11T00:31:45.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.956+0000 7fb09da00640 1 Processor -- start 2024-09-11T00:31:45.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.956+0000 7fb09da00640 1 -- start start 2024-09-11T00:31:45.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.957+0000 7fb09da00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb09806cc10 0x7fb098157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:45.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.957+0000 7fb09da00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0980caef0 0x7fb098158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:45.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.957+0000 7fb09da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb09814c880 0x7fb0980afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.957+0000 7fb09da00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb09814cc60 con 0x7fb09814c880 2024-09-11T00:31:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.957+0000 7fb09da00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb098156810 con 0x7fb09806cc10 2024-09-11T00:31:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.957+0000 7fb09da00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb09806edc0 con 0x7fb0980caef0 2024-09-11T00:31:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb09d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb09814c880 0x7fb0980afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb09d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb09814c880 0x7fb0980afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44798/0 (socket says 172.21.15.31:44798) 2024-09-11T00:31:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb08fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0980caef0 0x7fb098158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb09d000640 1 -- 172.21.15.31:0/2115699265 learned_addr learned my addr 172.21.15.31:0/2115699265 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:45.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb097e00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb09806cc10 0x7fb098157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb09d000640 1 -- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0980caef0 msgr2=0x7fb098158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb09d000640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0980caef0 0x7fb098158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb09d000640 1 -- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb09806cc10 msgr2=0x7fb098157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb09d000640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb09806cc10 0x7fb098157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb09d000640 1 -- 172.21.15.31:0/2115699265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb098159160 con 0x7fb09814c880 2024-09-11T00:31:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.958+0000 7fb08fe00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0980caef0 0x7fb098158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.959+0000 7fb09d000640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb09814c880 0x7fb0980afb80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb09000c780 tx=0x7fb09000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.959+0000 7fb094c00640 1 -- 172.21.15.31:0/2115699265 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb090003b70 con 0x7fb09814c880 2024-09-11T00:31:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.959+0000 7fb094c00640 1 -- 172.21.15.31:0/2115699265 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb090003d10 con 0x7fb09814c880 2024-09-11T00:31:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.959+0000 7fb09da00640 1 -- 172.21.15.31:0/2115699265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb098159590 con 0x7fb09814c880 2024-09-11T00:31:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.959+0000 7fb094c00640 1 -- 172.21.15.31:0/2115699265 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09000aab0 con 0x7fb09814c880 2024-09-11T00:31:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.959+0000 7fb09da00640 1 -- 172.21.15.31:0/2115699265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb098159b50 con 0x7fb09814c880 2024-09-11T00:31:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.960+0000 7fb09da00640 1 -- 172.21.15.31:0/2115699265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0980c8ac0 con 0x7fb09814c880 2024-09-11T00:31:45.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.961+0000 7fb094c00640 1 -- 172.21.15.31:0/2115699265 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb09000ac50 con 0x7fb09814c880 2024-09-11T00:31:45.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.962+0000 7fb094c00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb074077fa0 0x7fb07407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:45.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.962+0000 7fb094c00640 1 -- 172.21.15.31:0/2115699265 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7fb090099e80 con 0x7fb09814c880 2024-09-11T00:31:45.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.963+0000 7fb097e00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb074077fa0 0x7fb07407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:45.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.963+0000 7fb097e00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb074077fa0 0x7fb07407a460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb088007f20 tx=0x7fb088007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:45.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:45.967+0000 7fb094c00640 1 -- 172.21.15.31:0/2115699265 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb098159160 con 0x7fb09814c880 2024-09-11T00:31:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.201+0000 7fb09da00640 1 -- 172.21.15.31:0/2115699265 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0980c59a0 con 0x7fb09814c880 2024-09-11T00:31:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.202+0000 7fb094c00640 1 -- 172.21.15.31:0/2115699265 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5702 (secure 0 0 0) 0x7fb090061f30 con 0x7fb09814c880 2024-09-11T00:31:46.204 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:46.204 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"btime":"2024-09-11T00:31:45:280524+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:44.500873+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:31:46.204 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 17 2024-09-11T00:31:46.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.204+0000 7fb09da00640 1 -- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb074077fa0 msgr2=0x7fb07407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:46.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.204+0000 7fb09da00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb074077fa0 0x7fb07407a460 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb088007f20 tx=0x7fb088007eb0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.204+0000 7fb09da00640 1 -- 172.21.15.31:0/2115699265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb09814c880 msgr2=0x7fb0980afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:46.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.204+0000 7fb09da00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb09814c880 0x7fb0980afb80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb09000c780 tx=0x7fb09000cc40 comp rx=0 tx=0).stop 2024-09-11T00:31:46.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.204+0000 7fb09da00640 1 -- 172.21.15.31:0/2115699265 shutdown_connections 2024-09-11T00:31:46.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.204+0000 7fb09da00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb074077fa0 0x7fb07407a460 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.204+0000 7fb09da00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb09814c880 0x7fb0980afb80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.204+0000 7fb09da00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0980caef0 0x7fb098158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.205+0000 7fb09da00640 1 --2- 172.21.15.31:0/2115699265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb09806cc10 0x7fb098157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.205+0000 7fb09da00640 1 -- 172.21.15.31:0/2115699265 >> 172.21.15.31:0/2115699265 conn(0x7fb0980adb00 msgr2=0x7fb0980c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:46.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.205+0000 7fb09da00640 1 -- 172.21.15.31:0/2115699265 shutdown_connections 2024-09-11T00:31:46.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.205+0000 7fb09da00640 1 -- 172.21.15.31:0/2115699265 wait complete. 2024-09-11T00:31:46.221 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-11T00:31:46.222 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:31:46.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:46.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.385+0000 7f9efaa00640 1 -- 172.21.15.31:0/433359490 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ef406ff60 msgr2=0x7f9ef4154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:46.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.385+0000 7f9efaa00640 1 --2- 172.21.15.31:0/433359490 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ef406ff60 0x7f9ef4154070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40c4630 tx=0x7f9edc02a540 comp rx=0 tx=0).stop 2024-09-11T00:31:46.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.385+0000 7f9efaa00640 1 -- 172.21.15.31:0/433359490 shutdown_connections 2024-09-11T00:31:46.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.385+0000 7f9efaa00640 1 --2- 172.21.15.31:0/433359490 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ef406ff60 0x7f9ef4154070 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.385+0000 7f9efaa00640 1 --2- 172.21.15.31:0/433359490 >> v2:172.21.15.204:3300/0 conn(0x7f9ef41517a0 0x7f9ef40c39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.385+0000 7f9efaa00640 1 -- 172.21.15.31:0/433359490 >> 172.21.15.31:0/433359490 conn(0x7f9ef40c4aa0 msgr2=0x7f9ef40c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:46.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.385+0000 7f9efaa00640 1 -- 172.21.15.31:0/433359490 shutdown_connections 2024-09-11T00:31:46.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.385+0000 7f9efaa00640 1 -- 172.21.15.31:0/433359490 wait complete. 2024-09-11T00:31:46.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.387+0000 7f9efaa00640 1 Processor -- start 2024-09-11T00:31:46.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.387+0000 7f9efaa00640 1 -- start start 2024-09-11T00:31:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9efaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ef406ff60 0x7f9ef40742b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9efaa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ef40c55c0 0x7f9ef4073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9efaa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef41517a0 0x7f9ef41478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9efaa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef40adfd0 con 0x7f9ef406ff60 2024-09-11T00:31:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9efaa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef40c3ef0 con 0x7f9ef41517a0 2024-09-11T00:31:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9efaa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9ef414d200 con 0x7f9ef40c55c0 2024-09-11T00:31:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9ef3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ef406ff60 0x7f9ef40742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9ef3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ef406ff60 0x7f9ef40742b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44810/0 (socket says 172.21.15.31:44810) 2024-09-11T00:31:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9ef3e00640 1 -- 172.21.15.31:0/2721457461 learned_addr learned my addr 172.21.15.31:0/2721457461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9eeb400640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ef40c55c0 0x7f9ef4073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.388+0000 7f9ef8a00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef41517a0 0x7f9ef41478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9ef3e00640 1 -- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ef40c55c0 msgr2=0x7f9ef4073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9ef3e00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ef40c55c0 0x7f9ef4073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9ef3e00640 1 -- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef41517a0 msgr2=0x7f9ef41478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9ef3e00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef41517a0 0x7f9ef41478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9ef3e00640 1 -- 172.21.15.31:0/2721457461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef414d4e0 con 0x7f9ef406ff60 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9ef3e00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ef406ff60 0x7f9ef40742b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9ee400e790 tx=0x7f9ee400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9ef0c00640 1 -- 172.21.15.31:0/2721457461 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee400eea0 con 0x7f9ef406ff60 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9ef0c00640 1 -- 172.21.15.31:0/2721457461 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ee400cdb0 con 0x7f9ef406ff60 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9efaa00640 1 -- 172.21.15.31:0/2721457461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef414d7c0 con 0x7f9ef406ff60 2024-09-11T00:31:46.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9ef0c00640 1 -- 172.21.15.31:0/2721457461 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4010690 con 0x7f9ef406ff60 2024-09-11T00:31:46.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.389+0000 7f9efaa00640 1 -- 172.21.15.31:0/2721457461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef414afa0 con 0x7f9ef406ff60 2024-09-11T00:31:46.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.391+0000 7f9efaa00640 1 -- 172.21.15.31:0/2721457461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ef414eb90 con 0x7f9ef406ff60 2024-09-11T00:31:46.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.393+0000 7f9ef0c00640 1 -- 172.21.15.31:0/2721457461 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9ee4010830 con 0x7f9ef406ff60 2024-09-11T00:31:46.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.394+0000 7f9ef0c00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ed0077f70 0x7f9ed007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:46.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.394+0000 7f9ef0c00640 1 -- 172.21.15.31:0/2721457461 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f9ee409ab80 con 0x7f9ef406ff60 2024-09-11T00:31:46.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.394+0000 7f9eeb400640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ed0077f70 0x7f9ed007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:46.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.395+0000 7f9eeb400640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ed0077f70 0x7f9ed007a430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9edc033000 tx=0x7f9edc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:46.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.398+0000 7f9ef0c00640 1 -- 172.21.15.31:0/2721457461 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9ef414eb90 con 0x7f9ef406ff60 2024-09-11T00:31:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.727+0000 7f9efaa00640 1 -- 172.21.15.31:0/2721457461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ef40c39b0 con 0x7f9ef406ff60 2024-09-11T00:31:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.727+0000 7f9ef0c00640 1 -- 172.21.15.31:0/2721457461 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5699 (secure 0 0 0) 0x7f9ee4062bb0 con 0x7f9ef406ff60 2024-09-11T00:31:46.729 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:46.729 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":18,"btime":"2024-09-11T00:31:46:288307+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:45.291372+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:31:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 18 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 -- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ed0077f70 msgr2=0x7f9ed007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ed0077f70 0x7f9ed007a430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9edc033000 tx=0x7f9edc004730 comp rx=0 tx=0).stop 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 -- 172.21.15.31:0/2721457461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ef406ff60 msgr2=0x7f9ef40742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ef406ff60 0x7f9ef40742b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9ee400e790 tx=0x7f9ee400ec50 comp rx=0 tx=0).stop 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 -- 172.21.15.31:0/2721457461 shutdown_connections 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ed0077f70 0x7f9ed007a430 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef41517a0 0x7f9ef41478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9ef40c55c0 0x7f9ef4073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 --2- 172.21.15.31:0/2721457461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ef406ff60 0x7f9ef40742b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 -- 172.21.15.31:0/2721457461 >> 172.21.15.31:0/2721457461 conn(0x7f9ef40c4aa0 msgr2=0x7f9ef4152810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:46.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 -- 172.21.15.31:0/2721457461 shutdown_connections 2024-09-11T00:31:46.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:46.730+0000 7f9efaa00640 1 -- 172.21.15.31:0/2721457461 wait complete. 2024-09-11T00:31:46.754 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:31:46:288307+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 18, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4390": { "addr": "172.21.15.204:6835/1801149506", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1801149506, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 1801149506, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4390, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:30:49.364654+0000", "qdb_cluster": [ 4390 ], "qdb_leader": 4390, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4390 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 18, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4310": { "addr": "172.21.15.31:6835/2686855151", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6833", "nonce": 2686855151, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 2686855151, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4310, "incarnation": 16, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:31:45.291372+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4310 } } } ], "standbys": [ { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:31:48.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.923+0000 7f5bd0c00640 1 -- 172.21.15.31:0/515310345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bcc0c23a0 msgr2=0x7f5bcc1516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.923+0000 7f5bd0c00640 1 --2- 172.21.15.31:0/515310345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bcc0c23a0 0x7f5bcc1516e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009830 tx=0x7f5bb802a5a0 comp rx=0 tx=0).stop 2024-09-11T00:31:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.923+0000 7f5bd0c00640 1 -- 172.21.15.31:0/515310345 shutdown_connections 2024-09-11T00:31:48.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.923+0000 7f5bd0c00640 1 --2- 172.21.15.31:0/515310345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bcc0c23a0 0x7f5bcc1516e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:48.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.923+0000 7f5bd0c00640 1 --2- 172.21.15.31:0/515310345 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bcc072480 0x7f5bcc1511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:48.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.923+0000 7f5bd0c00640 1 -- 172.21.15.31:0/515310345 >> 172.21.15.31:0/515310345 conn(0x7f5bcc0a9750 msgr2=0x7f5bcc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:48.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.924+0000 7f5bd0c00640 1 -- 172.21.15.31:0/515310345 shutdown_connections 2024-09-11T00:31:48.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.924+0000 7f5bd0c00640 1 -- 172.21.15.31:0/515310345 wait complete. 2024-09-11T00:31:48.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.926+0000 7f5bd0c00640 1 Processor -- start 2024-09-11T00:31:48.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.926+0000 7f5bd0c00640 1 -- start start 2024-09-11T00:31:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bd0c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bcc072480 0x7f5bcc0c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bd0c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bcc0c23a0 0x7f5bcc1ef680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bd0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bcc153f20 0x7f5bcc14abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bd0c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc074200 con 0x7f5bcc153f20 2024-09-11T00:31:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bd0c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc074080 con 0x7f5bcc0c23a0 2024-09-11T00:31:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bd0c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5bcc072860 con 0x7f5bcc072480 2024-09-11T00:31:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bc9400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bcc0c23a0 0x7f5bcc1ef680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bc9e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bcc072480 0x7f5bcc0c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bc9400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bcc0c23a0 0x7f5bcc1ef680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:42680/0 (socket says 172.21.15.31:42680) 2024-09-11T00:31:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bc9400640 1 -- 172.21.15.31:0/1616943703 learned_addr learned my addr 172.21.15.31:0/1616943703 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bc9e00640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bcc072480 0x7f5bcc0c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:44674/0 (socket says 172.21.15.31:44674) 2024-09-11T00:31:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.927+0000 7f5bca800640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bcc153f20 0x7f5bcc14abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.928+0000 7f5bc9400640 1 -- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bcc072480 msgr2=0x7f5bcc0c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.928+0000 7f5bc9400640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bcc072480 0x7f5bcc0c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.928+0000 7f5bc9400640 1 -- 172.21.15.31:0/1616943703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bcc153f20 msgr2=0x7f5bcc14abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.928+0000 7f5bc9400640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bcc153f20 0x7f5bcc14abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.928+0000 7f5bc9400640 1 -- 172.21.15.31:0/1616943703 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bcc1584e0 con 0x7f5bcc0c23a0 2024-09-11T00:31:48.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.928+0000 7f5bc9400640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bcc0c23a0 0x7f5bcc1ef680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009950 tx=0x7f5bb8032c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:48.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.929+0000 7f5bc2000640 1 -- 172.21.15.31:0/1616943703 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8040070 con 0x7f5bcc0c23a0 2024-09-11T00:31:48.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.929+0000 7f5bc2000640 1 -- 172.21.15.31:0/1616943703 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bb8005310 con 0x7f5bcc0c23a0 2024-09-11T00:31:48.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.929+0000 7f5bc2000640 1 -- 172.21.15.31:0/1616943703 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8005c90 con 0x7f5bcc0c23a0 2024-09-11T00:31:48.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.929+0000 7f5bd0c00640 1 -- 172.21.15.31:0/1616943703 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bcc158200 con 0x7f5bcc0c23a0 2024-09-11T00:31:48.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.929+0000 7f5bd0c00640 1 -- 172.21.15.31:0/1616943703 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bcc0aa6c0 con 0x7f5bcc0c23a0 2024-09-11T00:31:48.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.930+0000 7f5bd0c00640 1 -- 172.21.15.31:0/1616943703 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bcc14e760 con 0x7f5bcc0c23a0 2024-09-11T00:31:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.932+0000 7f5bc2000640 1 -- 172.21.15.31:0/1616943703 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5bb80049b0 con 0x7f5bcc0c23a0 2024-09-11T00:31:48.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.933+0000 7f5bc2000640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bac077f70 0x7f5bac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:48.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.933+0000 7f5bc2000640 1 -- 172.21.15.31:0/1616943703 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f5bb80bb740 con 0x7f5bcc0c23a0 2024-09-11T00:31:48.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.934+0000 7f5bc9e00640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bac077f70 0x7f5bac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:48.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.934+0000 7f5bc9e00640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bac077f70 0x7f5bac07a430 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009530 tx=0x7f5bbc009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:48.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:48.940+0000 7f5bc2000640 1 -- 172.21.15.31:0/1616943703 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5bcc14e760 con 0x7f5bcc0c23a0 2024-09-11T00:31:49.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.172+0000 7f5bd0c00640 1 -- 172.21.15.31:0/1616943703 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bcc1511a0 con 0x7f5bcc0c23a0 2024-09-11T00:31:49.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.173+0000 7f5bc2000640 1 -- 172.21.15.31:0/1616943703 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5706 (secure 0 0 0) 0x7f5bb8083870 con 0x7f5bcc0c23a0 2024-09-11T00:31:49.175 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:49.175 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":19,"btime":"2024-09-11T00:31:47:294344+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:31:49.175 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 19 2024-09-11T00:31:49.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 -- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bac077f70 msgr2=0x7f5bac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:49.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bac077f70 0x7f5bac07a430 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009530 tx=0x7f5bbc009260 comp rx=0 tx=0).stop 2024-09-11T00:31:49.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 -- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bcc0c23a0 msgr2=0x7f5bcc1ef680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:49.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bcc0c23a0 0x7f5bcc1ef680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009950 tx=0x7f5bb8032c10 comp rx=0 tx=0).stop 2024-09-11T00:31:49.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 -- 172.21.15.31:0/1616943703 shutdown_connections 2024-09-11T00:31:49.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bac077f70 0x7f5bac07a430 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:49.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bcc153f20 0x7f5bcc14abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:49.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bcc0c23a0 0x7f5bcc1ef680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:49.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 --2- 172.21.15.31:0/1616943703 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bcc072480 0x7f5bcc0c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:49.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 -- 172.21.15.31:0/1616943703 >> 172.21.15.31:0/1616943703 conn(0x7f5bcc0a9750 msgr2=0x7f5bcc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:49.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 -- 172.21.15.31:0/1616943703 shutdown_connections 2024-09-11T00:31:49.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:49.176+0000 7f5bd0c00640 1 -- 172.21.15.31:0/1616943703 wait complete. 2024-09-11T00:31:49.194 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=4374 2024-09-11T00:31:49.194 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before reviving mds.b 2024-09-11T00:31:50.880 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-11T00:31:50.878+0000 7f554b600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:31:54.194 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-11T00:31:54.195 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-11T00:31:54.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-11T00:31:54.200 INFO:tasks.ceph.mds.b:Started 2024-09-11T00:31:54.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:54.397 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:31:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.476+0000 7fd505400640 1 -- 172.21.15.31:0/517069807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4003a00 msgr2=0x7fd4e4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.476+0000 7fd505400640 1 --2- 172.21.15.31:0/517069807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4e4003a00 0x7fd4e4003e20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000c2e0 tx=0x7fd4e4005830 comp rx=0 tx=0).stop 2024-09-11T00:31:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.476+0000 7fd504a00640 1 -- 172.21.15.31:0/517069807 reap_dead start 2024-09-11T00:31:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.476+0000 7fd505400640 1 -- 172.21.15.31:0/517069807 shutdown_connections 2024-09-11T00:31:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.476+0000 7fd505400640 1 -- 172.21.15.31:0/517069807 >> 172.21.15.31:0/517069807 conn(0x7fd50006caf0 msgr2=0x7fd50006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:54.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.476+0000 7fd505400640 1 -- 172.21.15.31:0/517069807 shutdown_connections 2024-09-11T00:31:54.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.477+0000 7fd505400640 1 -- 172.21.15.31:0/517069807 wait complete. 2024-09-11T00:31:54.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.479+0000 7fd505400640 1 Processor -- start 2024-09-11T00:31:54.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.480+0000 7fd505400640 1 -- start start 2024-09-11T00:31:54.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.481+0000 7fd505400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5001554c0 0x7fd5001dff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:54.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.481+0000 7fd505400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001ef6d0 0x7fd5001ec850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:54.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.481+0000 7fd505400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5001e2110 0x7fd500145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:54.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.481+0000 7fd505400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd5000c5e80 con 0x7fd5001ef6d0 2024-09-11T00:31:54.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.481+0000 7fd505400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd5000c2e80 con 0x7fd5001554c0 2024-09-11T00:31:54.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.481+0000 7fd505400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd5000c6000 con 0x7fd5001e2110 2024-09-11T00:31:54.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.481+0000 7fd4ffe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5001554c0 0x7fd5001dff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:54.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.481+0000 7fd504a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5001e2110 0x7fd500145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:54.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.481+0000 7fd4ffe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5001554c0 0x7fd5001dff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:42712/0 (socket says 172.21.15.31:42712) 2024-09-11T00:31:54.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.481+0000 7fd4ffe00640 1 -- 172.21.15.31:0/628545837 learned_addr learned my addr 172.21.15.31:0/628545837 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:54.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.482+0000 7fd4ff400640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001ef6d0 0x7fd5001ec850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.482+0000 7fd4ffe00640 1 -- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5001e2110 msgr2=0x7fd500145b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.482+0000 7fd4ffe00640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5001e2110 0x7fd500145b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:54.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.482+0000 7fd4ffe00640 1 -- 172.21.15.31:0/628545837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001ef6d0 msgr2=0x7fd5001ec850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.482+0000 7fd4ffe00640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001ef6d0 0x7fd5001ec850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.482+0000 7fd4ffe00640 1 -- 172.21.15.31:0/628545837 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5001e0450 con 0x7fd5001554c0 2024-09-11T00:31:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.482+0000 7fd504a00640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5001e2110 0x7fd500145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:31:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.483+0000 7fd4ffe00640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5001554c0 0x7fd5001dff10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0000c00 tx=0x7fd4f0006ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.483+0000 7fd4f7e00640 1 -- 172.21.15.31:0/628545837 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0031070 con 0x7fd5001554c0 2024-09-11T00:31:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.483+0000 7fd4f7e00640 1 -- 172.21.15.31:0/628545837 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4f00036d0 con 0x7fd5001554c0 2024-09-11T00:31:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.483+0000 7fd4f7e00640 1 -- 172.21.15.31:0/628545837 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0003b20 con 0x7fd5001554c0 2024-09-11T00:31:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.483+0000 7fd505400640 1 -- 172.21.15.31:0/628545837 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5000ac400 con 0x7fd5001554c0 2024-09-11T00:31:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.484+0000 7fd505400640 1 -- 172.21.15.31:0/628545837 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5001e1bc0 con 0x7fd5001554c0 2024-09-11T00:31:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.485+0000 7fd505400640 1 -- 172.21.15.31:0/628545837 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5001ec160 con 0x7fd5001554c0 2024-09-11T00:31:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.487+0000 7fd4f7e00640 1 -- 172.21.15.31:0/628545837 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd4f0003cc0 con 0x7fd5001554c0 2024-09-11T00:31:54.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.489+0000 7fd4f7e00640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4e0077f70 0x7fd4e007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:54.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.489+0000 7fd4f7e00640 1 -- 172.21.15.31:0/628545837 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7fd4f00ba4a0 con 0x7fd5001554c0 2024-09-11T00:31:54.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.490+0000 7fd4ff400640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4e0077f70 0x7fd4e007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:54.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.491+0000 7fd4ff400640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4e0077f70 0x7fd4e007a430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd500147330 tx=0x7fd4ec009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:54.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.497+0000 7fd4f7e00640 1 -- 172.21.15.31:0/628545837 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd4f0082550 con 0x7fd5001554c0 2024-09-11T00:31:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.878+0000 7fd505400640 1 -- 172.21.15.31:0/628545837 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd500153310 con 0x7fd5001554c0 2024-09-11T00:31:54.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.879+0000 7fd4f7e00640 1 -- 172.21.15.31:0/628545837 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5706 (secure 0 0 0) 0x7fd4f0086f80 con 0x7fd5001554c0 2024-09-11T00:31:54.881 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:54.881 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":19,"btime":"2024-09-11T00:31:47:294344+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:31:54.881 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 19 2024-09-11T00:31:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.881+0000 7fd505400640 1 -- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4e0077f70 msgr2=0x7fd4e007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.881+0000 7fd505400640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4e0077f70 0x7fd4e007a430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd500147330 tx=0x7fd4ec009110 comp rx=0 tx=0).stop 2024-09-11T00:31:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.881+0000 7fd505400640 1 -- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5001554c0 msgr2=0x7fd5001dff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:54.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.881+0000 7fd505400640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5001554c0 0x7fd5001dff10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0000c00 tx=0x7fd4f0006ce0 comp rx=0 tx=0).stop 2024-09-11T00:31:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.881+0000 7fd505400640 1 -- 172.21.15.31:0/628545837 shutdown_connections 2024-09-11T00:31:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.881+0000 7fd505400640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4e0077f70 0x7fd4e007a430 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd500147330 tx=0x7fd4ec009110 comp rx=0 tx=0).stop 2024-09-11T00:31:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.881+0000 7fd505400640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5001e2110 0x7fd500145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.881+0000 7fd505400640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5001ef6d0 0x7fd5001ec850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.881+0000 7fd505400640 1 --2- 172.21.15.31:0/628545837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5001554c0 0x7fd5001dff10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.881+0000 7fd505400640 1 -- 172.21.15.31:0/628545837 >> 172.21.15.31:0/628545837 conn(0x7fd50006caf0 msgr2=0x7fd500147c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.882+0000 7fd505400640 1 -- 172.21.15.31:0/628545837 shutdown_connections 2024-09-11T00:31:54.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:54.882+0000 7fd505400640 1 -- 172.21.15.31:0/628545837 wait complete. 2024-09-11T00:31:54.901 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-11T00:31:56.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:31:57.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.081+0000 7f5821800640 1 -- 172.21.15.31:0/120003383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f581c0c67f0 msgr2=0x7f581c159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:57.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.081+0000 7f5821800640 1 --2- 172.21.15.31:0/120003383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f581c0c67f0 0x7f581c159ea0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f581c06e110 tx=0x7f580c02a680 comp rx=0 tx=0).stop 2024-09-11T00:31:57.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.081+0000 7f5821800640 1 -- 172.21.15.31:0/120003383 shutdown_connections 2024-09-11T00:31:57.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.081+0000 7f5821800640 1 --2- 172.21.15.31:0/120003383 >> v2:172.21.15.204:3300/0 conn(0x7f581c06f8d0 0x7f581c15a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:57.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.081+0000 7f5821800640 1 --2- 172.21.15.31:0/120003383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f581c0c67f0 0x7f581c159ea0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:57.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.081+0000 7f5821800640 1 -- 172.21.15.31:0/120003383 >> 172.21.15.31:0/120003383 conn(0x7f581c0694f0 msgr2=0x7f581c0c9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:57.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.081+0000 7f5821800640 1 -- 172.21.15.31:0/120003383 shutdown_connections 2024-09-11T00:31:57.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.081+0000 7f5821800640 1 -- 172.21.15.31:0/120003383 wait complete. 2024-09-11T00:31:57.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.083+0000 7f5821800640 1 Processor -- start 2024-09-11T00:31:57.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.083+0000 7f5821800640 1 -- start start 2024-09-11T00:31:57.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.084+0000 7f5821800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c06f8d0 0x7f581c0c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:57.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.084+0000 7f5821800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0c67f0 0x7f581c1538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:57.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.084+0000 7f5821800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f581c0c8de0 0x7f581c153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:57.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.084+0000 7f5821800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f581c0c5ab0 con 0x7f581c0c8de0 2024-09-11T00:31:57.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.084+0000 7f5821800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f581c0c89d0 con 0x7f581c06f8d0 2024-09-11T00:31:57.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.084+0000 7f5821800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f581c0c5c30 con 0x7f581c0c67f0 2024-09-11T00:31:57.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.084+0000 7f581be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c06f8d0 0x7f581c0c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:57.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.084+0000 7f581be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c06f8d0 0x7f581c0c2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:42722/0 (socket says 172.21.15.31:42722) 2024-09-11T00:31:57.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.084+0000 7f581b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0c67f0 0x7f581c1538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:57.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.084+0000 7f581be00640 1 -- 172.21.15.31:0/2406478025 learned_addr learned my addr 172.21.15.31:0/2406478025 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:31:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.085+0000 7f5820e00640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f581c0c8de0 0x7f581c153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.085+0000 7f581be00640 1 -- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0c67f0 msgr2=0x7f581c1538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.085+0000 7f581be00640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0c67f0 0x7f581c1538f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.085+0000 7f581be00640 1 -- 172.21.15.31:0/2406478025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f581c0c8de0 msgr2=0x7f581c153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.085+0000 7f581be00640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f581c0c8de0 0x7f581c153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.085+0000 7f581be00640 1 -- 172.21.15.31:0/2406478025 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581c0c3d30 con 0x7f581c06f8d0 2024-09-11T00:31:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.085+0000 7f581be00640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c06f8d0 0x7f581c0c2e20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f580c000c00 tx=0x7f580c0038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.086+0000 7f5813e00640 1 -- 172.21.15.31:0/2406478025 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c03f070 con 0x7f581c06f8d0 2024-09-11T00:31:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.086+0000 7f5813e00640 1 -- 172.21.15.31:0/2406478025 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f580c0047a0 con 0x7f581c06f8d0 2024-09-11T00:31:57.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.086+0000 7f5813e00640 1 -- 172.21.15.31:0/2406478025 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c035030 con 0x7f581c06f8d0 2024-09-11T00:31:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.086+0000 7f5821800640 1 -- 172.21.15.31:0/2406478025 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f581c1547f0 con 0x7f581c06f8d0 2024-09-11T00:31:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.087+0000 7f5821800640 1 -- 172.21.15.31:0/2406478025 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f581c1e2240 con 0x7f581c06f8d0 2024-09-11T00:31:57.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.088+0000 7f5811600640 1 -- 172.21.15.31:0/2406478025 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f581c0c6bd0 con 0x7f581c06f8d0 2024-09-11T00:31:57.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.089+0000 7f5813e00640 1 -- 172.21.15.31:0/2406478025 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f580c004bb0 con 0x7f581c06f8d0 2024-09-11T00:31:57.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.090+0000 7f5813e00640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f57fc077f70 0x7f57fc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:31:57.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.090+0000 7f5813e00640 1 -- 172.21.15.31:0/2406478025 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f580c0bb200 con 0x7f581c06f8d0 2024-09-11T00:31:57.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.091+0000 7f581b400640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f57fc077f70 0x7f57fc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:31:57.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.092+0000 7f581b400640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f57fc077f70 0x7f57fc07a430 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f581c0bedd0 tx=0x7f5808005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:31:57.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.099+0000 7f5813e00640 1 -- 172.21.15.31:0/2406478025 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f580c083220 con 0x7f581c06f8d0 2024-09-11T00:31:57.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.350+0000 7f5811600640 1 -- 172.21.15.31:0/2406478025 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f581c0c3360 con 0x7f581c06f8d0 2024-09-11T00:31:57.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.351+0000 7f5813e00640 1 -- 172.21.15.31:0/2406478025 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6534 (secure 0 0 0) 0x7f580c087c50 con 0x7f581c06f8d0 2024-09-11T00:31:57.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:31:57.353 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"btime":"2024-09-11T00:31:55:317441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:31:57.353 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 20 2024-09-11T00:31:57.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.353+0000 7f5811600640 1 -- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f57fc077f70 msgr2=0x7f57fc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:57.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.353+0000 7f5811600640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f57fc077f70 0x7f57fc07a430 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f581c0bedd0 tx=0x7f5808005f90 comp rx=0 tx=0).stop 2024-09-11T00:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.353+0000 7f5811600640 1 -- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c06f8d0 msgr2=0x7f581c0c2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.353+0000 7f5811600640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c06f8d0 0x7f581c0c2e20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f580c000c00 tx=0x7f580c0038a0 comp rx=0 tx=0).stop 2024-09-11T00:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.354+0000 7f5811600640 1 -- 172.21.15.31:0/2406478025 shutdown_connections 2024-09-11T00:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.354+0000 7f5811600640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f57fc077f70 0x7f57fc07a430 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.354+0000 7f5811600640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f581c0c8de0 0x7f581c153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.354+0000 7f5811600640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f581c0c67f0 0x7f581c1538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.354+0000 7f5811600640 1 --2- 172.21.15.31:0/2406478025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f581c06f8d0 0x7f581c0c2e20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.354+0000 7f5811600640 1 -- 172.21.15.31:0/2406478025 >> 172.21.15.31:0/2406478025 conn(0x7f581c0694f0 msgr2=0x7f581c06eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.354+0000 7f5811600640 1 -- 172.21.15.31:0/2406478025 shutdown_connections 2024-09-11T00:31:57.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:31:57.354+0000 7f5811600640 1 -- 172.21.15.31:0/2406478025 wait complete. 2024-09-11T00:31:57.373 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-11T00:31:57.373 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2024-09-11T00:32:08.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:08.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.550+0000 7f4d61600640 1 -- 172.21.15.31:0/3696714057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d5c0ac2c0 msgr2=0x7f4d5c0ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.550+0000 7f4d61600640 1 --2- 172.21.15.31:0/3696714057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d5c0ac2c0 0x7f4d5c0ac6a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c005fd0 tx=0x7f4d4c02a4b0 comp rx=0 tx=0).stop 2024-09-11T00:32:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.550+0000 7f4d61600640 1 -- 172.21.15.31:0/3696714057 shutdown_connections 2024-09-11T00:32:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.550+0000 7f4d61600640 1 --2- 172.21.15.31:0/3696714057 >> v2:172.21.15.204:3300/0 conn(0x7f4d5c0a6e20 0x7f4d5c1510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.550+0000 7f4d61600640 1 --2- 172.21.15.31:0/3696714057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d5c0ac2c0 0x7f4d5c0ac6a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.550+0000 7f4d61600640 1 -- 172.21.15.31:0/3696714057 >> 172.21.15.31:0/3696714057 conn(0x7f4d5c06caf0 msgr2=0x7f4d5c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.550+0000 7f4d61600640 1 -- 172.21.15.31:0/3696714057 shutdown_connections 2024-09-11T00:32:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.551+0000 7f4d61600640 1 -- 172.21.15.31:0/3696714057 wait complete. 2024-09-11T00:32:08.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.552+0000 7f4d61600640 1 Processor -- start 2024-09-11T00:32:08.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.552+0000 7f4d61600640 1 -- start start 2024-09-11T00:32:08.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.553+0000 7f4d61600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d5c0a6e20 0x7f4d5c157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:08.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.553+0000 7f4d61600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d5c0ac2c0 0x7f4d5c072830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:08.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.553+0000 7f4d61600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d5c0acbe0 0x7f4d5c14ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.553+0000 7f4d61600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c151780 con 0x7f4d5c0ac2c0 2024-09-11T00:32:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.553+0000 7f4d61600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c151600 con 0x7f4d5c0a6e20 2024-09-11T00:32:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.553+0000 7f4d61600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4d5c151900 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d5be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d5c0a6e20 0x7f4d5c157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d60c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d5c0acbe0 0x7f4d5c14ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d5b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d5c0ac2c0 0x7f4d5c072830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d60c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d5c0acbe0 0x7f4d5c14ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:59204/0 (socket says 172.21.15.31:59204) 2024-09-11T00:32:08.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d60c00640 1 -- 172.21.15.31:0/271326966 learned_addr learned my addr 172.21.15.31:0/271326966 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:08.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d60c00640 1 -- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d5c0a6e20 msgr2=0x7f4d5c157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:08.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d60c00640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d5c0a6e20 0x7f4d5c157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:08.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d60c00640 1 -- 172.21.15.31:0/271326966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d5c0ac2c0 msgr2=0x7f4d5c072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:08.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d60c00640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d5c0ac2c0 0x7f4d5c072830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d60c00640 1 -- 172.21.15.31:0/271326966 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5c072070 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.554+0000 7f4d5be00640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d5c0a6e20 0x7f4d5c157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:32:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.555+0000 7f4d60c00640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d5c0acbe0 0x7f4d5c14ac10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400c760 tx=0x7f4d5400cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.555+0000 7f4d53e00640 1 -- 172.21.15.31:0/271326966 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d54003b70 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.555+0000 7f4d53e00640 1 -- 172.21.15.31:0/271326966 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d54003d30 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.555+0000 7f4d53e00640 1 -- 172.21.15.31:0/271326966 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5400aab0 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.555+0000 7f4d61600640 1 -- 172.21.15.31:0/271326966 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5c155b80 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.555+0000 7f4d61600640 1 -- 172.21.15.31:0/271326966 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5c156140 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.557+0000 7f4d61600640 1 -- 172.21.15.31:0/271326966 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d5c1510c0 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.558+0000 7f4d53e00640 1 -- 172.21.15.31:0/271326966 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4d5400ac50 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.559+0000 7f4d53e00640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4d3c077fa0 0x7f4d3c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.559+0000 7f4d53e00640 1 -- 172.21.15.31:0/271326966 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f4d5409a440 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.559+0000 7f4d5be00640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4d3c077fa0 0x7f4d3c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:08.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.560+0000 7f4d5be00640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4d3c077fa0 0x7f4d3c07a460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c033000 tx=0x7f4d4c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:08.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.567+0000 7f4d53e00640 1 -- 172.21.15.31:0/271326966 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4d54062bc0 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.919+0000 7f4d61600640 1 -- 172.21.15.31:0/271326966 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d5c0c7590 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.920+0000 7f4d53e00640 1 -- 172.21.15.31:0/271326966 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6534 (secure 0 0 0) 0x7f4d5406b2e0 con 0x7f4d5c0acbe0 2024-09-11T00:32:08.921 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:08.922 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"btime":"2024-09-11T00:31:55:317441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:32:08.922 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 20 2024-09-11T00:32:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 -- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4d3c077fa0 msgr2=0x7f4d3c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4d3c077fa0 0x7f4d3c07a460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c033000 tx=0x7f4d4c004730 comp rx=0 tx=0).stop 2024-09-11T00:32:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 -- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d5c0acbe0 msgr2=0x7f4d5c14ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d5c0acbe0 0x7f4d5c14ac10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400c760 tx=0x7f4d5400cc20 comp rx=0 tx=0).stop 2024-09-11T00:32:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 -- 172.21.15.31:0/271326966 shutdown_connections 2024-09-11T00:32:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4d3c077fa0 0x7f4d3c07a460 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4d5c0acbe0 0x7f4d5c14ac10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d5c0ac2c0 0x7f4d5c072830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 --2- 172.21.15.31:0/271326966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d5c0a6e20 0x7f4d5c157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 -- 172.21.15.31:0/271326966 >> 172.21.15.31:0/271326966 conn(0x7f4d5c06caf0 msgr2=0x7f4d5c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 -- 172.21.15.31:0/271326966 shutdown_connections 2024-09-11T00:32:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:08.923+0000 7f4d61600640 1 -- 172.21.15.31:0/271326966 wait complete. 2024-09-11T00:32:08.941 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-11T00:32:08.941 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-11T00:32:08.943 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:32:08.956 INFO:tasks.ceph.mds.a:Stopped 2024-09-11T00:32:08.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.159+0000 7fd8b3e00640 1 -- 172.21.15.31:0/1391643881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8b40bf600 msgr2=0x7fd8b4157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.159+0000 7fd8b3e00640 1 --2- 172.21.15.31:0/1391643881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8b40bf600 0x7fd8b4157ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac005fd0 tx=0x7fd8ac02aaa0 comp rx=0 tx=0).stop 2024-09-11T00:32:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.159+0000 7fd8b3e00640 1 -- 172.21.15.31:0/1391643881 shutdown_connections 2024-09-11T00:32:09.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.159+0000 7fd8b3e00640 1 --2- 172.21.15.31:0/1391643881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8b40bf600 0x7fd8b4157ff0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:09.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.159+0000 7fd8b3e00640 1 -- 172.21.15.31:0/1391643881 >> 172.21.15.31:0/1391643881 conn(0x7fd8b40c9af0 msgr2=0x7fd8b40c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:09.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.159+0000 7fd8b3e00640 1 -- 172.21.15.31:0/1391643881 shutdown_connections 2024-09-11T00:32:09.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.159+0000 7fd8b3e00640 1 -- 172.21.15.31:0/1391643881 wait complete. 2024-09-11T00:32:09.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.162+0000 7fd8b3e00640 1 Processor -- start 2024-09-11T00:32:09.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.162+0000 7fd8b3e00640 1 -- start start 2024-09-11T00:32:09.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.163+0000 7fd8b3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8b40a8bb0 0x7fd8b41556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:09.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.163+0000 7fd8b3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8b40bf600 0x7fd8b406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:09.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.163+0000 7fd8b3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8b40c67f0 0x7fd8b406f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:09.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.163+0000 7fd8b3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b40a9260 con 0x7fd8b40bf600 2024-09-11T00:32:09.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.163+0000 7fd8b3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b40a90e0 con 0x7fd8b40c67f0 2024-09-11T00:32:09.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.163+0000 7fd8b3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd8b40c5b20 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.164+0000 7fd8b2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8b40a8bb0 0x7fd8b41556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:09.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.164+0000 7fd8b2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8b40a8bb0 0x7fd8b41556d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:59208/0 (socket says 172.21.15.31:59208) 2024-09-11T00:32:09.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.164+0000 7fd8b2a00640 1 -- 172.21.15.31:0/560240459 learned_addr learned my addr 172.21.15.31:0/560240459 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:09.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.164+0000 7fd8b3400640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8b40c67f0 0x7fd8b406f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:09.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.164+0000 7fd8aa000640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8b40bf600 0x7fd8b406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:09.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.164+0000 7fd8b2a00640 1 -- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8b40c67f0 msgr2=0x7fd8b406f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:09.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.164+0000 7fd8b2a00640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8b40c67f0 0x7fd8b406f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:09.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.164+0000 7fd8b2a00640 1 -- 172.21.15.31:0/560240459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8b40bf600 msgr2=0x7fd8b406eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:09.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.164+0000 7fd8b2a00640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8b40bf600 0x7fd8b406eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:09.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.164+0000 7fd8b2a00640 1 -- 172.21.15.31:0/560240459 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b40a84d0 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.165+0000 7fd8aa000640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8b40bf600 0x7fd8b406eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:32:09.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.165+0000 7fd8b2a00640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8b40a8bb0 0x7fd8b41556d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400c430 tx=0x7fd8a400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:09.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.165+0000 7fd8ab400640 1 -- 172.21.15.31:0/560240459 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4017070 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.165+0000 7fd8ab400640 1 -- 172.21.15.31:0/560240459 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8a40053c0 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.165+0000 7fd8ab400640 1 -- 172.21.15.31:0/560240459 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4003850 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.165+0000 7fd8b3e00640 1 -- 172.21.15.31:0/560240459 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b414a740 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.165+0000 7fd8b3e00640 1 -- 172.21.15.31:0/560240459 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8b4155c10 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.167+0000 7fd8b3e00640 1 -- 172.21.15.31:0/560240459 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8b40c6bd0 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.169+0000 7fd8ab400640 1 -- 172.21.15.31:0/560240459 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd8a40039f0 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.169+0000 7fd8ab400640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd890077f70 0x7fd89007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:09.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.170+0000 7fd8ab400640 1 -- 172.21.15.31:0/560240459 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7fd8a4099770 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.170+0000 7fd8aa000640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd890077f70 0x7fd89007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:09.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.171+0000 7fd8aa000640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd890077f70 0x7fd89007a430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40a8b10 tx=0x7fd89c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:09.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.174+0000 7fd8ab400640 1 -- 172.21.15.31:0/560240459 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd8a4061820 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:32:09.250 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:32:09.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.439+0000 7fd8b3e00640 1 -- 172.21.15.31:0/560240459 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8b40c7a60 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.440+0000 7fd8ab400640 1 -- 172.21.15.31:0/560240459 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6534 (secure 0 0 0) 0x7fd8a4066250 con 0x7fd8b40a8bb0 2024-09-11T00:32:09.442 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:09.442 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"btime":"2024-09-11T00:31:55:317441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:32:09.443 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 20 2024-09-11T00:32:09.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 -- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd890077f70 msgr2=0x7fd89007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:09.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd890077f70 0x7fd89007a430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40a8b10 tx=0x7fd89c009040 comp rx=0 tx=0).stop 2024-09-11T00:32:09.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 -- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8b40a8bb0 msgr2=0x7fd8b41556d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:09.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8b40a8bb0 0x7fd8b41556d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400c430 tx=0x7fd8a400c8f0 comp rx=0 tx=0).stop 2024-09-11T00:32:09.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 -- 172.21.15.31:0/560240459 shutdown_connections 2024-09-11T00:32:09.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd890077f70 0x7fd89007a430 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:09.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8b40c67f0 0x7fd8b406f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:09.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8b40bf600 0x7fd8b406eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:09.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 --2- 172.21.15.31:0/560240459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8b40a8bb0 0x7fd8b41556d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:09.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 -- 172.21.15.31:0/560240459 >> 172.21.15.31:0/560240459 conn(0x7fd8b40c9af0 msgr2=0x7fd8b4156630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:09.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 -- 172.21.15.31:0/560240459 shutdown_connections 2024-09-11T00:32:09.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:09.443+0000 7fd8b3e00640 1 -- 172.21.15.31:0/560240459 wait complete. 2024-09-11T00:32:09.465 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:32:11.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:11.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.629+0000 7fe2e2a00640 1 -- 172.21.15.31:0/791611463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2dc156350 msgr2=0x7fe2dc1567c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:11.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.629+0000 7fe2e2a00640 1 --2- 172.21.15.31:0/791611463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2dc156350 0x7fe2dc1567c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4005fd0 tx=0x7fe2d402a940 comp rx=0 tx=0).stop 2024-09-11T00:32:11.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.629+0000 7fe2e2a00640 1 -- 172.21.15.31:0/791611463 shutdown_connections 2024-09-11T00:32:11.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.629+0000 7fe2e2a00640 1 --2- 172.21.15.31:0/791611463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2dc156350 0x7fe2dc1567c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:11.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.629+0000 7fe2e2a00640 1 -- 172.21.15.31:0/791611463 >> 172.21.15.31:0/791611463 conn(0x7fe2dc06caf0 msgr2=0x7fe2dc06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:11.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.629+0000 7fe2e2a00640 1 -- 172.21.15.31:0/791611463 shutdown_connections 2024-09-11T00:32:11.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.629+0000 7fe2e2a00640 1 -- 172.21.15.31:0/791611463 wait complete. 2024-09-11T00:32:11.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.631+0000 7fe2e2a00640 1 Processor -- start 2024-09-11T00:32:11.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.631+0000 7fe2e2a00640 1 -- start start 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2dc0c5aa0 0x7fe2dc14eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0c6420 0x7fe2dc1e0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e2a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2dc156350 0x7fe2dc155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e2a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc153a20 con 0x7fe2dc0c6420 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e2a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe2dc0c3160 con 0x7fe2dc156350 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e2a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe2dc153ba0 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e1600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2dc0c5aa0 0x7fe2dc14eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0c6420 0x7fe2dc1e0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e1600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2dc0c5aa0 0x7fe2dc14eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:59216/0 (socket says 172.21.15.31:59216) 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e1600640 1 -- 172.21.15.31:0/3860727426 learned_addr learned my addr 172.21.15.31:0/3860727426 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.632+0000 7fe2e2000640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2dc156350 0x7fe2dc155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2e1600640 1 -- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2dc156350 msgr2=0x7fe2dc155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2e1600640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2dc156350 0x7fe2dc155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2e1600640 1 -- 172.21.15.31:0/3860727426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0c6420 msgr2=0x7fe2dc1e0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2e1600640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0c6420 0x7fe2dc1e0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2e1600640 1 -- 172.21.15.31:0/3860727426 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2dc14ebf0 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2e1600640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2dc0c5aa0 0x7fe2dc14eed0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000c530 tx=0x7fe2d000c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2d9600640 1 -- 172.21.15.31:0/3860727426 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d00052d0 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2d9600640 1 -- 172.21.15.31:0/3860727426 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2d0005470 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2e2a00640 1 -- 172.21.15.31:0/3860727426 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2dc0ac8c0 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2d9600640 1 -- 172.21.15.31:0/3860727426 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0012870 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.633+0000 7fe2e2a00640 1 -- 172.21.15.31:0/3860727426 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2dc1e2350 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.635+0000 7fe2e2a00640 1 -- 172.21.15.31:0/3860727426 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2dc0ac6b0 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.637+0000 7fe2d9600640 1 -- 172.21.15.31:0/3860727426 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe2d00035f0 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.638+0000 7fe2d9600640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe2bc077f70 0x7fe2bc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:11.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.638+0000 7fe2d9600640 1 -- 172.21.15.31:0/3860727426 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7fe2d0099b20 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.638+0000 7fe2e0c00640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe2bc077f70 0x7fe2bc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:11.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.639+0000 7fe2e0c00640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe2bc077f70 0x7fe2bc07a430 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0c7220 tx=0x7fe2c8007cc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:11.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.642+0000 7fe2d9600640 1 -- 172.21.15.31:0/3860727426 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe2d0061c50 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.963+0000 7fe2e2a00640 1 -- 172.21.15.31:0/3860727426 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2dc156de0 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.964+0000 7fe2d9600640 1 -- 172.21.15.31:0/3860727426 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6534 (secure 0 0 0) 0x7fe2d0066680 con 0x7fe2dc0c5aa0 2024-09-11T00:32:11.967 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:11.967 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"btime":"2024-09-11T00:31:55:317441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:32:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 20 2024-09-11T00:32:11.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 -- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe2bc077f70 msgr2=0x7fe2bc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:11.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe2bc077f70 0x7fe2bc07a430 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0c7220 tx=0x7fe2c8007cc0 comp rx=0 tx=0).stop 2024-09-11T00:32:11.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 -- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2dc0c5aa0 msgr2=0x7fe2dc14eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:11.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2dc0c5aa0 0x7fe2dc14eed0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000c530 tx=0x7fe2d000c9f0 comp rx=0 tx=0).stop 2024-09-11T00:32:11.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 -- 172.21.15.31:0/3860727426 shutdown_connections 2024-09-11T00:32:11.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe2bc077f70 0x7fe2bc07a430 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:11.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2dc156350 0x7fe2dc155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:11.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2dc0c6420 0x7fe2dc1e0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:11.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 --2- 172.21.15.31:0/3860727426 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2dc0c5aa0 0x7fe2dc14eed0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:11.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 -- 172.21.15.31:0/3860727426 >> 172.21.15.31:0/3860727426 conn(0x7fe2dc06caf0 msgr2=0x7fe2dc0a7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:11.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 -- 172.21.15.31:0/3860727426 shutdown_connections 2024-09-11T00:32:11.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:11.967+0000 7fe2e2a00640 1 -- 172.21.15.31:0/3860727426 wait complete. 2024-09-11T00:32:11.988 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:32:13.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.162+0000 7f127f200640 1 -- 172.21.15.31:0/3657718448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12800ac2c0 msgr2=0x7f12800ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.162+0000 7f127f200640 1 --2- 172.21.15.31:0/3657718448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12800ac2c0 0x7f12800ac6a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1270009830 tx=0x7f127002a4a0 comp rx=0 tx=0).stop 2024-09-11T00:32:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.162+0000 7f127f200640 1 -- 172.21.15.31:0/3657718448 shutdown_connections 2024-09-11T00:32:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.162+0000 7f127f200640 1 --2- 172.21.15.31:0/3657718448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12800ac2c0 0x7f12800ac6a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.162+0000 7f127f200640 1 -- 172.21.15.31:0/3657718448 >> 172.21.15.31:0/3657718448 conn(0x7f128006caf0 msgr2=0x7f128006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.163+0000 7f127f200640 1 -- 172.21.15.31:0/3657718448 shutdown_connections 2024-09-11T00:32:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.163+0000 7f127f200640 1 -- 172.21.15.31:0/3657718448 wait complete. 2024-09-11T00:32:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.165+0000 7f127f200640 1 Processor -- start 2024-09-11T00:32:14.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.165+0000 7f127f200640 1 -- start start 2024-09-11T00:32:14.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.167+0000 7f127f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12800a6e20 0x7f12801525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:14.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.167+0000 7f127f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12800ac2c0 0x7f12800a8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:14.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.167+0000 7f127f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12800acbe0 0x7f12800a8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:14.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.167+0000 7f127f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f12800c5d20 con 0x7f12800a6e20 2024-09-11T00:32:14.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.167+0000 7f127f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f12801519e0 con 0x7f12800acbe0 2024-09-11T00:32:14.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.167+0000 7f127f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f12800c68c0 con 0x7f12800ac2c0 2024-09-11T00:32:14.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.167+0000 7f127de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12800a6e20 0x7f12801525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:14.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.167+0000 7f127de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12800a6e20 0x7f12801525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54154/0 (socket says 172.21.15.31:54154) 2024-09-11T00:32:14.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.167+0000 7f127de00640 1 -- 172.21.15.31:0/2957805690 learned_addr learned my addr 172.21.15.31:0/2957805690 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:14.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.167+0000 7f127e800640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12800acbe0 0x7f12800a8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:14.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.168+0000 7f127d400640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12800ac2c0 0x7f12800a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:14.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.168+0000 7f127de00640 1 -- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12800ac2c0 msgr2=0x7f12800a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:14.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.168+0000 7f127de00640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12800ac2c0 0x7f12800a8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:14.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.168+0000 7f127de00640 1 -- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12800acbe0 msgr2=0x7f12800a8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:14.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.168+0000 7f127de00640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12800acbe0 0x7f12800a8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:14.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.168+0000 7f127de00640 1 -- 172.21.15.31:0/2957805690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f128014d2a0 con 0x7f12800a6e20 2024-09-11T00:32:14.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.168+0000 7f127e800640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12800acbe0 0x7f12800a8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:32:14.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.169+0000 7f127de00640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12800a6e20 0x7f12801525d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1270009780 tx=0x7f1270032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:14.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.169+0000 7f1276000640 1 -- 172.21.15.31:0/2957805690 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1270040070 con 0x7f12800a6e20 2024-09-11T00:32:14.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.169+0000 7f1276000640 1 -- 172.21.15.31:0/2957805690 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1270005310 con 0x7f12800a6e20 2024-09-11T00:32:14.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.169+0000 7f1276000640 1 -- 172.21.15.31:0/2957805690 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1270005c90 con 0x7f12800a6e20 2024-09-11T00:32:14.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.169+0000 7f127f200640 1 -- 172.21.15.31:0/2957805690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12801ec3b0 con 0x7f12800a6e20 2024-09-11T00:32:14.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.169+0000 7f127f200640 1 -- 172.21.15.31:0/2957805690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1280152b10 con 0x7f12800a6e20 2024-09-11T00:32:14.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.171+0000 7f127f200640 1 -- 172.21.15.31:0/2957805690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12800a8450 con 0x7f12800a6e20 2024-09-11T00:32:14.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.173+0000 7f1276000640 1 -- 172.21.15.31:0/2957805690 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f12700049b0 con 0x7f12800a6e20 2024-09-11T00:32:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.174+0000 7f1276000640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1260077f70 0x7f126007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.174+0000 7f1276000640 1 -- 172.21.15.31:0/2957805690 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f12700bb5b0 con 0x7f12800a6e20 2024-09-11T00:32:14.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.175+0000 7f127d400640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1260077f70 0x7f126007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:14.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.176+0000 7f127d400640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1260077f70 0x7f126007a430 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f126c007fd0 tx=0x7f126c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:14.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.183+0000 7f1276000640 1 -- 172.21.15.31:0/2957805690 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f12800a8450 con 0x7f12800a6e20 2024-09-11T00:32:14.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.514+0000 7f127f200640 1 -- 172.21.15.31:0/2957805690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f128014ee60 con 0x7f12800a6e20 2024-09-11T00:32:14.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.515+0000 7f1276000640 1 -- 172.21.15.31:0/2957805690 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6534 (secure 0 0 0) 0x7f1270083660 con 0x7f12800a6e20 2024-09-11T00:32:14.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:14.518 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"btime":"2024-09-11T00:31:55:317441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:32:14.518 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 20 2024-09-11T00:32:14.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 -- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1260077f70 msgr2=0x7f126007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:14.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1260077f70 0x7f126007a430 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f126c007fd0 tx=0x7f126c009040 comp rx=0 tx=0).stop 2024-09-11T00:32:14.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 -- 172.21.15.31:0/2957805690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12800a6e20 msgr2=0x7f12801525d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:14.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12800a6e20 0x7f12801525d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1270009780 tx=0x7f1270032c10 comp rx=0 tx=0).stop 2024-09-11T00:32:14.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 -- 172.21.15.31:0/2957805690 shutdown_connections 2024-09-11T00:32:14.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1260077f70 0x7f126007a430 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:14.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12800acbe0 0x7f12800a8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:14.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12800ac2c0 0x7f12800a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:14.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 --2- 172.21.15.31:0/2957805690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12800a6e20 0x7f12801525d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:14.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 -- 172.21.15.31:0/2957805690 >> 172.21.15.31:0/2957805690 conn(0x7f128006caf0 msgr2=0x7f12800c6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:14.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 -- 172.21.15.31:0/2957805690 shutdown_connections 2024-09-11T00:32:14.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:14.517+0000 7f127f200640 1 -- 172.21.15.31:0/2957805690 wait complete. 2024-09-11T00:32:14.537 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:32:16.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.713+0000 7f5f50e00640 1 -- 172.21.15.31:0/885718508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c151470 msgr2=0x7f5f4c154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.713+0000 7f5f50e00640 1 --2- 172.21.15.31:0/885718508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c151470 0x7f5f4c154930 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5f44005fd0 tx=0x7f5f4402ecc0 comp rx=0 tx=0).stop 2024-09-11T00:32:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.713+0000 7f5f50e00640 1 -- 172.21.15.31:0/885718508 shutdown_connections 2024-09-11T00:32:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.713+0000 7f5f50e00640 1 --2- 172.21.15.31:0/885718508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c151470 0x7f5f4c154930 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.713+0000 7f5f50e00640 1 -- 172.21.15.31:0/885718508 >> 172.21.15.31:0/885718508 conn(0x7f5f4c0c4d20 msgr2=0x7f5f4c0c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.713+0000 7f5f50e00640 1 -- 172.21.15.31:0/885718508 shutdown_connections 2024-09-11T00:32:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.713+0000 7f5f50e00640 1 -- 172.21.15.31:0/885718508 wait complete. 2024-09-11T00:32:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.716+0000 7f5f50e00640 1 Processor -- start 2024-09-11T00:32:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.716+0000 7f5f50e00640 1 -- start start 2024-09-11T00:32:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.717+0000 7f5f50e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f4c06dd70 0x7f5f4c14aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.717+0000 7f5f50e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c074a30 0x7f5f4c0acf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.717+0000 7f5f50e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5f4c151470 0x7f5f4c0ab860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.717+0000 7f5f50e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c071260 con 0x7f5f4c074a30 2024-09-11T00:32:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.717+0000 7f5f50e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c070c10 con 0x7f5f4c06dd70 2024-09-11T00:32:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.717+0000 7f5f50e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5f4c06f470 con 0x7f5f4c151470 2024-09-11T00:32:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f4c06dd70 0x7f5f4c14aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f4c06dd70 0x7f5f4c14aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:35958/0 (socket says 172.21.15.31:35958) 2024-09-11T00:32:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4b400640 1 -- 172.21.15.31:0/2314584046 learned_addr learned my addr 172.21.15.31:0/2314584046 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4aa00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c074a30 0x7f5f4c0acf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4be00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5f4c151470 0x7f5f4c0ab860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4b400640 1 -- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5f4c151470 msgr2=0x7f5f4c0ab860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4b400640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5f4c151470 0x7f5f4c0ab860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4b400640 1 -- 172.21.15.31:0/2314584046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c074a30 msgr2=0x7f5f4c0acf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4b400640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c074a30 0x7f5f4c0acf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4b400640 1 -- 172.21.15.31:0/2314584046 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4c0ac250 con 0x7f5f4c06dd70 2024-09-11T00:32:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.718+0000 7f5f4aa00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c074a30 0x7f5f4c0acf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:32:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.719+0000 7f5f4b400640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f4c06dd70 0x7f5f4c14aed0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00c4d0 tx=0x7f5f3c00c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.719+0000 7f5f43400640 1 -- 172.21.15.31:0/2314584046 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c016020 con 0x7f5f4c06dd70 2024-09-11T00:32:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.719+0000 7f5f43400640 1 -- 172.21.15.31:0/2314584046 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f3c012a00 con 0x7f5f4c06dd70 2024-09-11T00:32:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.719+0000 7f5f43400640 1 -- 172.21.15.31:0/2314584046 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c012e50 con 0x7f5f4c06dd70 2024-09-11T00:32:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.719+0000 7f5f50e00640 1 -- 172.21.15.31:0/2314584046 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4c151f30 con 0x7f5f4c06dd70 2024-09-11T00:32:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.719+0000 7f5f50e00640 1 -- 172.21.15.31:0/2314584046 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f4c0ad480 con 0x7f5f4c06dd70 2024-09-11T00:32:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.720+0000 7f5f50e00640 1 -- 172.21.15.31:0/2314584046 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f4c153020 con 0x7f5f4c06dd70 2024-09-11T00:32:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.724+0000 7f5f43400640 1 -- 172.21.15.31:0/2314584046 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5f3c025440 con 0x7f5f4c06dd70 2024-09-11T00:32:16.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.725+0000 7f5f43400640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f2c078110 0x7f5f2c07a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:16.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.725+0000 7f5f43400640 1 -- 172.21.15.31:0/2314584046 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f5f3c09a930 con 0x7f5f4c06dd70 2024-09-11T00:32:16.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.725+0000 7f5f4aa00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f2c078110 0x7f5f2c07a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:16.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.726+0000 7f5f43400640 1 -- 172.21.15.31:0/2314584046 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5f3c020020 con 0x7f5f4c06dd70 2024-09-11T00:32:16.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.726+0000 7f5f4aa00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f2c078110 0x7f5f2c07a5d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c154d30 tx=0x7f5f38009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:16.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.962+0000 7f5f50e00640 1 -- 172.21.15.31:0/2314584046 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f4c06e150 con 0x7f5f4c06dd70 2024-09-11T00:32:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.963+0000 7f5f43400640 1 -- 172.21.15.31:0/2314584046 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6534 (secure 0 0 0) 0x7f5f3c0630b0 con 0x7f5f4c06dd70 2024-09-11T00:32:16.966 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:16.966 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"btime":"2024-09-11T00:31:55:317441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:32:16.966 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 20 2024-09-11T00:32:16.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.965+0000 7f5f50e00640 1 -- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f2c078110 msgr2=0x7f5f2c07a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:16.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.965+0000 7f5f50e00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f2c078110 0x7f5f2c07a5d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c154d30 tx=0x7f5f38009040 comp rx=0 tx=0).stop 2024-09-11T00:32:16.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.965+0000 7f5f50e00640 1 -- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f4c06dd70 msgr2=0x7f5f4c14aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:16.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.965+0000 7f5f50e00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f4c06dd70 0x7f5f4c14aed0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00c4d0 tx=0x7f5f3c00c990 comp rx=0 tx=0).stop 2024-09-11T00:32:16.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.965+0000 7f5f50e00640 1 -- 172.21.15.31:0/2314584046 shutdown_connections 2024-09-11T00:32:16.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.965+0000 7f5f50e00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f2c078110 0x7f5f2c07a5d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:16.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.965+0000 7f5f50e00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5f4c151470 0x7f5f4c0ab860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:16.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.965+0000 7f5f50e00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c074a30 0x7f5f4c0acf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:16.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.965+0000 7f5f50e00640 1 --2- 172.21.15.31:0/2314584046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f4c06dd70 0x7f5f4c14aed0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:16.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.965+0000 7f5f50e00640 1 -- 172.21.15.31:0/2314584046 >> 172.21.15.31:0/2314584046 conn(0x7f5f4c0c4d20 msgr2=0x7f5f4c0c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:16.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.966+0000 7f5f50e00640 1 -- 172.21.15.31:0/2314584046 shutdown_connections 2024-09-11T00:32:16.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:16.966+0000 7f5f50e00640 1 -- 172.21.15.31:0/2314584046 wait complete. 2024-09-11T00:32:16.992 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:32:18.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.154+0000 7f82c4600640 1 -- 172.21.15.31:0/3631978194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc157ac0 msgr2=0x7f82bc151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:19.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.154+0000 7f82c4600640 1 --2- 172.21.15.31:0/3631978194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc157ac0 0x7f82bc151700 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f82a8005fd0 tx=0x7f82a802a540 comp rx=0 tx=0).stop 2024-09-11T00:32:19.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.154+0000 7f82c4600640 1 -- 172.21.15.31:0/3631978194 shutdown_connections 2024-09-11T00:32:19.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.154+0000 7f82c4600640 1 --2- 172.21.15.31:0/3631978194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc157ac0 0x7f82bc151700 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:19.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.154+0000 7f82c4600640 1 --2- 172.21.15.31:0/3631978194 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc1576e0 0x7f82bc1511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:19.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.154+0000 7f82c4600640 1 -- 172.21.15.31:0/3631978194 >> 172.21.15.31:0/3631978194 conn(0x7f82bc0c8320 msgr2=0x7f82bc0c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:19.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.154+0000 7f82c4600640 1 -- 172.21.15.31:0/3631978194 shutdown_connections 2024-09-11T00:32:19.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.155+0000 7f82c4600640 1 -- 172.21.15.31:0/3631978194 wait complete. 2024-09-11T00:32:19.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.156+0000 7f82c4600640 1 Processor -- start 2024-09-11T00:32:19.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.156+0000 7f82c4600640 1 -- start start 2024-09-11T00:32:19.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.157+0000 7f82c4600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc06f820 0x7f82bc1dfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:19.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.157+0000 7f82c4600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc1576e0 0x7f82bc143aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:19.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.157+0000 7f82c4600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82bc157ac0 0x7f82bc143fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:19.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.157+0000 7f82c4600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc074c20 con 0x7f82bc06f820 2024-09-11T00:32:19.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.157+0000 7f82c4600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc0703c0 con 0x7f82bc157ac0 2024-09-11T00:32:19.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.157+0000 7f82c4600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f82bc074da0 con 0x7f82bc1576e0 2024-09-11T00:32:19.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.157+0000 7f82c1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc06f820 0x7f82bc1dfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:19.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.158+0000 7f82c1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc06f820 0x7f82bc1dfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38456/0 (socket says 172.21.15.31:38456) 2024-09-11T00:32:19.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.158+0000 7f82c1c00640 1 -- 172.21.15.31:0/1066186334 learned_addr learned my addr 172.21.15.31:0/1066186334 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:19.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.158+0000 7f82c1200640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc1576e0 0x7f82bc143aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:19.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.158+0000 7f82c2600640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82bc157ac0 0x7f82bc143fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:19.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.158+0000 7f82c1c00640 1 -- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc1576e0 msgr2=0x7f82bc143aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:19.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.158+0000 7f82c1c00640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc1576e0 0x7f82bc143aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:19.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.158+0000 7f82c1c00640 1 -- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82bc157ac0 msgr2=0x7f82bc143fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:19.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.158+0000 7f82c1c00640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82bc157ac0 0x7f82bc143fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:19.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.158+0000 7f82c1c00640 1 -- 172.21.15.31:0/1066186334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82bc150040 con 0x7f82bc06f820 2024-09-11T00:32:19.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.158+0000 7f82c1c00640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc06f820 0x7f82bc1dfd40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f82b000e9f0 tx=0x7f82b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:19.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.159+0000 7f82b9600640 1 -- 172.21.15.31:0/1066186334 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b000cd80 con 0x7f82bc06f820 2024-09-11T00:32:19.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.159+0000 7f82b9600640 1 -- 172.21.15.31:0/1066186334 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82b0010430 con 0x7f82bc06f820 2024-09-11T00:32:19.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.159+0000 7f82b9600640 1 -- 172.21.15.31:0/1066186334 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b0010840 con 0x7f82bc06f820 2024-09-11T00:32:19.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.159+0000 7f82c4600640 1 -- 172.21.15.31:0/1066186334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82bc0c7720 con 0x7f82bc06f820 2024-09-11T00:32:19.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.159+0000 7f82c4600640 1 -- 172.21.15.31:0/1066186334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82bc1505e0 con 0x7f82bc06f820 2024-09-11T00:32:19.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.160+0000 7f82c4600640 1 -- 172.21.15.31:0/1066186334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82bc06e690 con 0x7f82bc06f820 2024-09-11T00:32:19.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.161+0000 7f82b9600640 1 -- 172.21.15.31:0/1066186334 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f82b00109e0 con 0x7f82bc06f820 2024-09-11T00:32:19.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.162+0000 7f82b9600640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c07c380 0x7f829c07e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:19.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.162+0000 7f82b9600640 1 -- 172.21.15.31:0/1066186334 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f82b009aba0 con 0x7f82bc06f820 2024-09-11T00:32:19.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.163+0000 7f82c1200640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c07c380 0x7f829c07e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:19.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.163+0000 7f82c1200640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c07c380 0x7f829c07e840 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f82a8033000 tx=0x7f82a8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:19.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.166+0000 7f82b9600640 1 -- 172.21.15.31:0/1066186334 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f82bc06e690 con 0x7f82bc06f820 2024-09-11T00:32:19.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.399+0000 7f82c4600640 1 -- 172.21.15.31:0/1066186334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82bc06e0a0 con 0x7f82bc06f820 2024-09-11T00:32:19.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.400+0000 7f82b9600640 1 -- 172.21.15.31:0/1066186334 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6534 (secure 0 0 0) 0x7f82b0062d90 con 0x7f82bc06f820 2024-09-11T00:32:19.403 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:19.403 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"btime":"2024-09-11T00:31:55:317441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:32:19.403 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 20 2024-09-11T00:32:19.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 -- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c07c380 msgr2=0x7f829c07e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:19.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c07c380 0x7f829c07e840 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f82a8033000 tx=0x7f82a8004730 comp rx=0 tx=0).stop 2024-09-11T00:32:19.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 -- 172.21.15.31:0/1066186334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc06f820 msgr2=0x7f82bc1dfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:19.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc06f820 0x7f82bc1dfd40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f82b000e9f0 tx=0x7f82b000eeb0 comp rx=0 tx=0).stop 2024-09-11T00:32:19.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 -- 172.21.15.31:0/1066186334 shutdown_connections 2024-09-11T00:32:19.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c07c380 0x7f829c07e840 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:19.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82bc157ac0 0x7f82bc143fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:19.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc1576e0 0x7f82bc143aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:19.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 --2- 172.21.15.31:0/1066186334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc06f820 0x7f82bc1dfd40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:19.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 -- 172.21.15.31:0/1066186334 >> 172.21.15.31:0/1066186334 conn(0x7f82bc0c8320 msgr2=0x7f82bc075c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:19.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 -- 172.21.15.31:0/1066186334 shutdown_connections 2024-09-11T00:32:19.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:19.402+0000 7f82c4600640 1 -- 172.21.15.31:0/1066186334 wait complete. 2024-09-11T00:32:19.421 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:32:21.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:21.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.594+0000 7f1544800640 1 -- 172.21.15.31:0/1234560102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c0c23a0 msgr2=0x7f153c0c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:21.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.594+0000 7f1544800640 1 --2- 172.21.15.31:0/1234560102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c0c23a0 0x7f153c0c30c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f153c0c5500 tx=0x7f152c02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:32:21.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.594+0000 7f1544800640 1 -- 172.21.15.31:0/1234560102 shutdown_connections 2024-09-11T00:32:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.594+0000 7f1544800640 1 --2- 172.21.15.31:0/1234560102 >> v2:172.21.15.204:3300/0 conn(0x7f153c1518b0 0x7f153c0c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.594+0000 7f1544800640 1 --2- 172.21.15.31:0/1234560102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c0c23a0 0x7f153c0c30c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.594+0000 7f1544800640 1 -- 172.21.15.31:0/1234560102 >> 172.21.15.31:0/1234560102 conn(0x7f153c0a9750 msgr2=0x7f153c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.594+0000 7f1544800640 1 -- 172.21.15.31:0/1234560102 shutdown_connections 2024-09-11T00:32:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.594+0000 7f1544800640 1 -- 172.21.15.31:0/1234560102 wait complete. 2024-09-11T00:32:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.595+0000 7f1544800640 1 Processor -- start 2024-09-11T00:32:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.596+0000 7f1544800640 1 -- start start 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1544800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c072480 0x7f153c1e1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1544800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f153c0c23a0 0x7f153c14ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1544800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f153c1518b0 0x7f153c06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1544800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f153c074200 con 0x7f153c072480 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1544800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f153c074080 con 0x7f153c1518b0 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1544800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f153c072860 con 0x7f153c0c23a0 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1541e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c072480 0x7f153c1e1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1541e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c072480 0x7f153c1e1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38474/0 (socket says 172.21.15.31:38474) 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1541e00640 1 -- 172.21.15.31:0/4048517274 learned_addr learned my addr 172.21.15.31:0/4048517274 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1541400640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f153c0c23a0 0x7f153c14ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1542800640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f153c1518b0 0x7f153c06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1541e00640 1 -- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f153c0c23a0 msgr2=0x7f153c14ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1541e00640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f153c0c23a0 0x7f153c14ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1541e00640 1 -- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f153c1518b0 msgr2=0x7f153c06eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1541e00640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f153c1518b0 0x7f153c06eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.597+0000 7f1541e00640 1 -- 172.21.15.31:0/4048517274 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153c06e9d0 con 0x7f153c072480 2024-09-11T00:32:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.598+0000 7f1541e00640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c072480 0x7f153c1e1e70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f153000c4a0 tx=0x7f153000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.598+0000 7f1536000640 1 -- 172.21.15.31:0/4048517274 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530016020 con 0x7f153c072480 2024-09-11T00:32:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.598+0000 7f1536000640 1 -- 172.21.15.31:0/4048517274 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1530012920 con 0x7f153c072480 2024-09-11T00:32:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.598+0000 7f1536000640 1 -- 172.21.15.31:0/4048517274 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530012db0 con 0x7f153c072480 2024-09-11T00:32:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.598+0000 7f1544800640 1 -- 172.21.15.31:0/4048517274 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153c06fcc0 con 0x7f153c072480 2024-09-11T00:32:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.598+0000 7f1544800640 1 -- 172.21.15.31:0/4048517274 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f153c152180 con 0x7f153c072480 2024-09-11T00:32:21.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.599+0000 7f1544800640 1 -- 172.21.15.31:0/4048517274 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f153c06eba0 con 0x7f153c072480 2024-09-11T00:32:21.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.600+0000 7f1536000640 1 -- 172.21.15.31:0/4048517274 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f153001b400 con 0x7f153c072480 2024-09-11T00:32:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.602+0000 7f1536000640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1520077f70 0x7f152007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.602+0000 7f1536000640 1 -- 172.21.15.31:0/4048517274 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f153009a300 con 0x7f153c072480 2024-09-11T00:32:21.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.602+0000 7f1541400640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1520077f70 0x7f152007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:21.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.603+0000 7f1541400640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1520077f70 0x7f152007a430 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f152c033000 tx=0x7f152c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.606+0000 7f1536000640 1 -- 172.21.15.31:0/4048517274 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1530020020 con 0x7f153c072480 2024-09-11T00:32:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.845+0000 7f1544800640 1 -- 172.21.15.31:0/4048517274 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f153c153f20 con 0x7f153c072480 2024-09-11T00:32:21.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.846+0000 7f1536000640 1 -- 172.21.15.31:0/4048517274 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6534 (secure 0 0 0) 0x7f15300623b0 con 0x7f153c072480 2024-09-11T00:32:21.849 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:21.849 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"btime":"2024-09-11T00:31:55:317441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:32:21.849 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 20 2024-09-11T00:32:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 -- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1520077f70 msgr2=0x7f152007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1520077f70 0x7f152007a430 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f152c033000 tx=0x7f152c004730 comp rx=0 tx=0).stop 2024-09-11T00:32:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 -- 172.21.15.31:0/4048517274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c072480 msgr2=0x7f153c1e1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c072480 0x7f153c1e1e70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f153000c4a0 tx=0x7f153000c960 comp rx=0 tx=0).stop 2024-09-11T00:32:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 -- 172.21.15.31:0/4048517274 shutdown_connections 2024-09-11T00:32:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1520077f70 0x7f152007a430 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f152c033000 tx=0x7f152c004730 comp rx=0 tx=0).stop 2024-09-11T00:32:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f153c1518b0 0x7f153c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f153c0c23a0 0x7f153c14ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 --2- 172.21.15.31:0/4048517274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f153c072480 0x7f153c1e1e70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 -- 172.21.15.31:0/4048517274 >> 172.21.15.31:0/4048517274 conn(0x7f153c0a9750 msgr2=0x7f153c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 -- 172.21.15.31:0/4048517274 shutdown_connections 2024-09-11T00:32:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:21.849+0000 7f1544800640 1 -- 172.21.15.31:0/4048517274 wait complete. 2024-09-11T00:32:21.868 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:32:22.036 INFO:tasks.workunit.client.1.smithi204.stderr:Updating files: 66% (7997/12088) Updating files: 67% (8099/12088) Updating files: 68% (8220/12088) Updating files: 69% (8341/12088) Updating files: 70% (8462/12088) Updating files: 71% (8583/12088) Updating files: 72% (8704/12088) Updating files: 73% (8825/12088) Updating files: 74% (8946/12088) Updating files: 75% (9066/12088) Updating files: 76% (9187/12088) Updating files: 77% (9308/12088) Updating files: 78% (9429/12088) Updating files: 79% (9550/12088) Updating files: 80% (9671/12088) Updating files: 81% (9792/12088) Updating files: 82% (9913/12088) Updating files: 83% (10034/12088) Updating files: 84% (10154/12088) Updating files: 85% (10275/12088) Updating files: 86% (10396/12088) Updating files: 87% (10517/12088) Updating files: 88% (10638/12088) Updating files: 89% (10759/12088) Updating files: 90% (10880/12088) Updating files: 91% (11001/12088) Updating files: 92% (11121/12088) Updating files: 93% (11242/12088) Updating files: 94% (11363/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-09-11T00:32:22.970 INFO:tasks.workunit.client.1.smithi204.stderr:Note: switching to '8f219c476cb4f26fc5719c661c0323719a45cb30'. 2024-09-11T00:32:22.970 INFO:tasks.workunit.client.1.smithi204.stderr: 2024-09-11T00:32:22.970 INFO:tasks.workunit.client.1.smithi204.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T00:32:22.970 INFO:tasks.workunit.client.1.smithi204.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T00:32:22.970 INFO:tasks.workunit.client.1.smithi204.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T00:32:22.970 INFO:tasks.workunit.client.1.smithi204.stderr: 2024-09-11T00:32:22.970 INFO:tasks.workunit.client.1.smithi204.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T00:32:22.970 INFO:tasks.workunit.client.1.smithi204.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T00:32:22.971 INFO:tasks.workunit.client.1.smithi204.stderr: 2024-09-11T00:32:22.971 INFO:tasks.workunit.client.1.smithi204.stderr: git switch -c 2024-09-11T00:32:22.971 INFO:tasks.workunit.client.1.smithi204.stderr: 2024-09-11T00:32:22.971 INFO:tasks.workunit.client.1.smithi204.stderr:Or undo this operation with: 2024-09-11T00:32:22.971 INFO:tasks.workunit.client.1.smithi204.stderr: 2024-09-11T00:32:22.971 INFO:tasks.workunit.client.1.smithi204.stderr: git switch - 2024-09-11T00:32:22.971 INFO:tasks.workunit.client.1.smithi204.stderr: 2024-09-11T00:32:22.971 INFO:tasks.workunit.client.1.smithi204.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T00:32:22.971 INFO:tasks.workunit.client.1.smithi204.stderr: 2024-09-11T00:32:22.971 INFO:tasks.workunit.client.1.smithi204.stderr:HEAD is now at 8f219c476cb Merge branch 'mgr-vol-mod-disable' of https://github.com/rishabh-d-dave/ceph into wip-hemanth-testing-2024-09-10-0723 2024-09-11T00:32:22.981 DEBUG:teuthology.orchestra.run.smithi204:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-09-11T00:32:23.026 INFO:tasks.workunit.client.1.smithi204.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T00:32:23.031 INFO:tasks.workunit.client.1.smithi204.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-11T00:32:23.031 INFO:tasks.workunit.client.1.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T00:32:23.683 INFO:tasks.workunit.client.1.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T00:32:23.789 INFO:tasks.workunit.client.1.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T00:32:23.854 INFO:tasks.workunit.client.1.smithi204.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-11T00:32:23.858 INFO:tasks.workunit.client.1.smithi204.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-11T00:32:23.858 INFO:tasks.workunit.client.1.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T00:32:23.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:23.923 INFO:tasks.workunit.client.1.smithi204.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-11T00:32:23.930 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:32:23.930 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-09-11T00:32:24.006 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.1... 2024-09-11T00:32:24.007 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-09-11T00:32:24.007 DEBUG:teuthology.orchestra.run.smithi204:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8f219c476cb4f26fc5719c661c0323719a45cb30 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh 2024-09-11T00:32:24.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.031+0000 7fe84b000640 1 -- 172.21.15.31:0/2386715666 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8441534f0 msgr2=0x7fe8280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:24.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.031+0000 7fe84b000640 1 --2- 172.21.15.31:0/2386715666 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8441534f0 0x7fe8280051c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe83c002230 tx=0x7fe828005830 comp rx=0 tx=0).stop 2024-09-11T00:32:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.031+0000 7fe849000640 1 -- 172.21.15.31:0/2386715666 reap_dead start 2024-09-11T00:32:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.031+0000 7fe84b000640 1 -- 172.21.15.31:0/2386715666 shutdown_connections 2024-09-11T00:32:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.031+0000 7fe84b000640 1 -- 172.21.15.31:0/2386715666 >> 172.21.15.31:0/2386715666 conn(0x7fe844068f10 msgr2=0x7fe8440cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.031+0000 7fe84b000640 1 -- 172.21.15.31:0/2386715666 shutdown_connections 2024-09-11T00:32:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.031+0000 7fe84b000640 1 -- 172.21.15.31:0/2386715666 wait complete. 2024-09-11T00:32:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.033+0000 7fe84b000640 1 Processor -- start 2024-09-11T00:32:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.033+0000 7fe84b000640 1 -- start start 2024-09-11T00:32:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe84b000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8441534f0 0x7fe8440c2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe84b000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe844157450 0x7fe84414d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe84b000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8440c45d0 0x7fe84414d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe84b000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe844153a50 con 0x7fe8440c45d0 2024-09-11T00:32:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe84b000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe8441538d0 con 0x7fe8441534f0 2024-09-11T00:32:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe84b000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe844156ae0 con 0x7fe844157450 2024-09-11T00:32:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe849000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8440c45d0 0x7fe84414d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe849000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8440c45d0 0x7fe84414d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38500/0 (socket says 172.21.15.31:38500) 2024-09-11T00:32:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe849000640 1 -- 172.21.15.31:0/4019352851 learned_addr learned my addr 172.21.15.31:0/4019352851 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe849000640 1 -- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe844157450 msgr2=0x7fe84414d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe849000640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe844157450 0x7fe84414d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe849000640 1 -- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8441534f0 msgr2=0x7fe8440c2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe849000640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8441534f0 0x7fe8440c2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:24.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.034+0000 7fe849000640 1 -- 172.21.15.31:0/4019352851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8440bf080 con 0x7fe8440c45d0 2024-09-11T00:32:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.035+0000 7fe849000640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8440c45d0 0x7fe84414d8f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe83c02b3b0 tx=0x7fe83c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.035+0000 7fe840c00640 1 -- 172.21.15.31:0/4019352851 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c02bb10 con 0x7fe8440c45d0 2024-09-11T00:32:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.035+0000 7fe840c00640 1 -- 172.21.15.31:0/4019352851 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe83c02bcb0 con 0x7fe8440c45d0 2024-09-11T00:32:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.035+0000 7fe840c00640 1 -- 172.21.15.31:0/4019352851 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c02be60 con 0x7fe8440c45d0 2024-09-11T00:32:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.035+0000 7fe84b000640 1 -- 172.21.15.31:0/4019352851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84414dff0 con 0x7fe8440c45d0 2024-09-11T00:32:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.035+0000 7fe84b000640 1 -- 172.21.15.31:0/4019352851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84414e2d0 con 0x7fe8440c45d0 2024-09-11T00:32:24.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.036+0000 7fe84b000640 1 -- 172.21.15.31:0/4019352851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8440cc350 con 0x7fe8440c45d0 2024-09-11T00:32:24.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.037+0000 7fe840c00640 1 -- 172.21.15.31:0/4019352851 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe83c006310 con 0x7fe8440c45d0 2024-09-11T00:32:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.038+0000 7fe840c00640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe820077f70 0x7fe82007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.038+0000 7fe840c00640 1 -- 172.21.15.31:0/4019352851 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7fe83c0b9f70 con 0x7fe8440c45d0 2024-09-11T00:32:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.039+0000 7fe843e00640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe820077f70 0x7fe82007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.040+0000 7fe843e00640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe820077f70 0x7fe82007a430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe834009b40 tx=0x7fe834009660 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.043+0000 7fe840c00640 1 -- 172.21.15.31:0/4019352851 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe83c082660 con 0x7fe8440c45d0 2024-09-11T00:32:24.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.275+0000 7fe84b000640 1 -- 172.21.15.31:0/4019352851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe844155a00 con 0x7fe8440c45d0 2024-09-11T00:32:24.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.276+0000 7fe840c00640 1 -- 172.21.15.31:0/4019352851 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6534 (secure 0 0 0) 0x7fe83c081f60 con 0x7fe8440c45d0 2024-09-11T00:32:24.279 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":20,"btime":"2024-09-11T00:31:55:317441+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4721,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:31:47.294339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4310":{"gid":4310,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":20,"addr":"172.21.15.31:6835/2686855151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":2686855151},{"type":"v1","addr":"172.21.15.31:6835","nonce":2686855151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4310,"qdb_cluster":[4310]},"id":2}]} 2024-09-11T00:32:24.279 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 20 2024-09-11T00:32:24.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 -- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe820077f70 msgr2=0x7fe82007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:24.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe820077f70 0x7fe82007a430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe834009b40 tx=0x7fe834009660 comp rx=0 tx=0).stop 2024-09-11T00:32:24.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 -- 172.21.15.31:0/4019352851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8440c45d0 msgr2=0x7fe84414d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:24.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8440c45d0 0x7fe84414d8f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe83c02b3b0 tx=0x7fe83c003ba0 comp rx=0 tx=0).stop 2024-09-11T00:32:24.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 -- 172.21.15.31:0/4019352851 shutdown_connections 2024-09-11T00:32:24.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe820077f70 0x7fe82007a430 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:24.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8440c45d0 0x7fe84414d8f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:24.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe844157450 0x7fe84414d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:24.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 --2- 172.21.15.31:0/4019352851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8441534f0 0x7fe8440c2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:24.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 -- 172.21.15.31:0/4019352851 >> 172.21.15.31:0/4019352851 conn(0x7fe844068f10 msgr2=0x7fe8441519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:24.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 -- 172.21.15.31:0/4019352851 shutdown_connections 2024-09-11T00:32:24.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:24.278+0000 7fe84b000640 1 -- 172.21.15.31:0/4019352851 wait complete. 2024-09-11T00:32:24.299 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:32:25.817 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 63% (7699/12088) Updating files: 64% (7737/12088) Updating files: 65% (7858/12088) Updating files: 66% (7979/12088) Updating files: 67% (8099/12088) Updating files: 68% (8220/12088) Updating files: 69% (8341/12088) Updating files: 70% (8462/12088) Updating files: 71% (8583/12088) Updating files: 72% (8704/12088) Updating files: 73% (8825/12088) Updating files: 74% (8946/12088) Updating files: 75% (9066/12088) Updating files: 76% (9187/12088) Updating files: 77% (9308/12088) Updating files: 78% (9429/12088) Updating files: 79% (9550/12088) Updating files: 80% (9671/12088) Updating files: 81% (9792/12088) Updating files: 82% (9913/12088) Updating files: 83% (10034/12088) Updating files: 84% (10154/12088) Updating files: 85% (10275/12088) Updating files: 86% (10396/12088) Updating files: 87% (10517/12088) Updating files: 88% (10638/12088) Updating files: 89% (10759/12088) Updating files: 90% (10880/12088) Updating files: 91% (11001/12088) Updating files: 92% (11121/12088) Updating files: 93% (11242/12088) Updating files: 94% (11363/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-09-11T00:32:26.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:26.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.462+0000 7fd201400640 1 -- 172.21.15.31:0/3371810557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1fc0b1160 msgr2=0x7fd1fc15a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:26.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.462+0000 7fd201400640 1 --2- 172.21.15.31:0/3371810557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1fc0b1160 0x7fd1fc15a1a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc08dc70 tx=0x7fd1ec02a4a0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.462+0000 7fd201400640 1 -- 172.21.15.31:0/3371810557 shutdown_connections 2024-09-11T00:32:26.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.462+0000 7fd201400640 1 --2- 172.21.15.31:0/3371810557 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1fc0bf2c0 0x7fd1fc0ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.462+0000 7fd201400640 1 --2- 172.21.15.31:0/3371810557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1fc0b1160 0x7fd1fc15a1a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.462+0000 7fd201400640 1 -- 172.21.15.31:0/3371810557 >> 172.21.15.31:0/3371810557 conn(0x7fd1fc0cc260 msgr2=0x7fd1fc0ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:26.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.462+0000 7fd201400640 1 -- 172.21.15.31:0/3371810557 shutdown_connections 2024-09-11T00:32:26.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.462+0000 7fd201400640 1 -- 172.21.15.31:0/3371810557 wait complete. 2024-09-11T00:32:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.463+0000 7fd201400640 1 Processor -- start 2024-09-11T00:32:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.463+0000 7fd201400640 1 -- start start 2024-09-11T00:32:26.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.464+0000 7fd201400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1fc0b1160 0x7fd1fc1ec830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.464+0000 7fd201400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1fc0bf2c0 0x7fd1fc154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.464+0000 7fd201400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1fc0c9b30 0x7fd1fc14b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.464+0000 7fd201400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc0b1540 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.464+0000 7fd201400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc06f9a0 con 0x7fd1fc0c9b30 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.464+0000 7fd201400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd1fc06f300 con 0x7fd1fc0b1160 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1f9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1fc0bf2c0 0x7fd1fc154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1f9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1fc0bf2c0 0x7fd1fc154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38526/0 (socket says 172.21.15.31:38526) 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1f9e00640 1 -- 172.21.15.31:0/2176178754 learned_addr learned my addr 172.21.15.31:0/2176178754 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1fb200640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1fc0c9b30 0x7fd1fc14b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1fa800640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1fc0b1160 0x7fd1fc1ec830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1f9e00640 1 -- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1fc0b1160 msgr2=0x7fd1fc1ec830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1f9e00640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1fc0b1160 0x7fd1fc1ec830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1f9e00640 1 -- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1fc0c9b30 msgr2=0x7fd1fc14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:26.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1f9e00640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1fc0c9b30 0x7fd1fc14b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1f9e00640 1 -- 172.21.15.31:0/2176178754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1fc1df850 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.465+0000 7fd1f9e00640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1fc0bf2c0 0x7fd1fc154f90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd1e800b3c0 tx=0x7fd1e800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.466+0000 7fd1f2a00640 1 -- 172.21.15.31:0/2176178754 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e8011830 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.466+0000 7fd1f2a00640 1 -- 172.21.15.31:0/2176178754 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1e8016070 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.466+0000 7fd1f2a00640 1 -- 172.21.15.31:0/2176178754 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e8011da0 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.466+0000 7fd201400640 1 -- 172.21.15.31:0/2176178754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1fc1e05f0 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.466+0000 7fd201400640 1 -- 172.21.15.31:0/2176178754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1fc1dfdf0 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.467+0000 7fd201400640 1 -- 172.21.15.31:0/2176178754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1fc0cb5a0 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.468+0000 7fd1f2a00640 1 -- 172.21.15.31:0/2176178754 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd1e801b460 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.469+0000 7fd1f2a00640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd1dc077f70 0x7fd1dc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:26.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.469+0000 7fd1f2a00640 1 -- 172.21.15.31:0/2176178754 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7fd1e8099cf0 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.469+0000 7fd1fa800640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd1dc077f70 0x7fd1dc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:26.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.470+0000 7fd1fa800640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd1dc077f70 0x7fd1dc07a430 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec033000 tx=0x7fd1ec004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:26.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.473+0000 7fd1f2a00640 1 -- 172.21.15.31:0/2176178754 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd1e8020020 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.706+0000 7fd201400640 1 -- 172.21.15.31:0/2176178754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1fc0aebb0 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.707+0000 7fd1f2a00640 1 -- 172.21.15.31:0/2176178754 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5702 (secure 0 0 0) 0x7fd1e8062370 con 0x7fd1fc0bf2c0 2024-09-11T00:32:26.710 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:26.710 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"btime":"2024-09-11T00:32:26:295075+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:25.298184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:32:26.710 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2024-09-11T00:32:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 -- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd1dc077f70 msgr2=0x7fd1dc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd1dc077f70 0x7fd1dc07a430 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec033000 tx=0x7fd1ec004730 comp rx=0 tx=0).stop 2024-09-11T00:32:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 -- 172.21.15.31:0/2176178754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1fc0bf2c0 msgr2=0x7fd1fc154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1fc0bf2c0 0x7fd1fc154f90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd1e800b3c0 tx=0x7fd1e800b880 comp rx=0 tx=0).stop 2024-09-11T00:32:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 -- 172.21.15.31:0/2176178754 shutdown_connections 2024-09-11T00:32:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd1dc077f70 0x7fd1dc07a430 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1fc0c9b30 0x7fd1fc14b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1fc0bf2c0 0x7fd1fc154f90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 --2- 172.21.15.31:0/2176178754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1fc0b1160 0x7fd1fc1ec830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 -- 172.21.15.31:0/2176178754 >> 172.21.15.31:0/2176178754 conn(0x7fd1fc0cc260 msgr2=0x7fd1fc0af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:26.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 -- 172.21.15.31:0/2176178754 shutdown_connections 2024-09-11T00:32:26.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.710+0000 7fd201400640 1 -- 172.21.15.31:0/2176178754 wait complete. 2024-09-11T00:32:26.729 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-11T00:32:26.729 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:32:26.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:26.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.888+0000 7f0928800640 1 -- 172.21.15.31:0/1564179423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09201534f0 msgr2=0x7f0920156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:26.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.888+0000 7f0928800640 1 --2- 172.21.15.31:0/1564179423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09201534f0 0x7f0920156ae0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f092006cd50 tx=0x7f091402a560 comp rx=0 tx=0).stop 2024-09-11T00:32:26.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.888+0000 7f0928800640 1 -- 172.21.15.31:0/1564179423 shutdown_connections 2024-09-11T00:32:26.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.888+0000 7f0928800640 1 --2- 172.21.15.31:0/1564179423 >> v2:172.21.15.204:3300/0 conn(0x7f0920159650 0x7f0920159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.888+0000 7f0928800640 1 --2- 172.21.15.31:0/1564179423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09201534f0 0x7f0920156ae0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.888+0000 7f0928800640 1 -- 172.21.15.31:0/1564179423 >> 172.21.15.31:0/1564179423 conn(0x7f0920068f10 msgr2=0x7f09200cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:26.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.888+0000 7f0928800640 1 -- 172.21.15.31:0/1564179423 shutdown_connections 2024-09-11T00:32:26.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.889+0000 7f0928800640 1 -- 172.21.15.31:0/1564179423 wait complete. 2024-09-11T00:32:26.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.890+0000 7f0928800640 1 Processor -- start 2024-09-11T00:32:26.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.890+0000 7f0928800640 1 -- start start 2024-09-11T00:32:26.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0928800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09200c9b40 0x7f09201e3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:26.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0928800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09201534f0 0x7f09200c3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:26.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0928800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0920159650 0x7f09200acd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:26.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0928800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09200cca60 con 0x7f09201534f0 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0928800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f09201538d0 con 0x7f0920159650 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0928800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f09201ef360 con 0x7f09200c9b40 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0925400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09201534f0 0x7f09200c3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0925400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09201534f0 0x7f09200c3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38538/0 (socket says 172.21.15.31:38538) 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0925400640 1 -- 172.21.15.31:0/758328530 learned_addr learned my addr 172.21.15.31:0/758328530 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0926800640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0920159650 0x7f09200acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.891+0000 7f0925e00640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09200c9b40 0x7f09201e3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f0925400640 1 -- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09200c9b40 msgr2=0x7f09201e3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f0925400640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09200c9b40 0x7f09201e3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f0925400640 1 -- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0920159650 msgr2=0x7f09200acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f0925400640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0920159650 0x7f09200acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:26.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f0925400640 1 -- 172.21.15.31:0/758328530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092014fd70 con 0x7f09201534f0 2024-09-11T00:32:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f0925e00640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09200c9b40 0x7f09201e3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:32:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f0925400640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09201534f0 0x7f09200c3800 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f091000c4d0 tx=0x7f091000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f091a000640 1 -- 172.21.15.31:0/758328530 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910016020 con 0x7f09201534f0 2024-09-11T00:32:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f091a000640 1 -- 172.21.15.31:0/758328530 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0910005310 con 0x7f09201534f0 2024-09-11T00:32:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f091a000640 1 -- 172.21.15.31:0/758328530 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910005500 con 0x7f09201534f0 2024-09-11T00:32:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.892+0000 7f0928800640 1 -- 172.21.15.31:0/758328530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09200c2fc0 con 0x7f09201534f0 2024-09-11T00:32:26.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.893+0000 7f0928800640 1 -- 172.21.15.31:0/758328530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09200c34e0 con 0x7f09201534f0 2024-09-11T00:32:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.894+0000 7f0928800640 1 -- 172.21.15.31:0/758328530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0920151ac0 con 0x7f09201534f0 2024-09-11T00:32:26.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.895+0000 7f091a000640 1 -- 172.21.15.31:0/758328530 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0910006a80 con 0x7f09201534f0 2024-09-11T00:32:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.896+0000 7f091a000640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0904077f70 0x7f090407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.896+0000 7f091a000640 1 -- 172.21.15.31:0/758328530 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f0910099b00 con 0x7f09201534f0 2024-09-11T00:32:26.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.897+0000 7f0925e00640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0904077f70 0x7f090407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:26.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.898+0000 7f0925e00640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0904077f70 0x7f090407a430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0914033000 tx=0x7f0914004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:26.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:26.901+0000 7f091a000640 1 -- 172.21.15.31:0/758328530 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0920151ac0 con 0x7f09201534f0 2024-09-11T00:32:27.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.134+0000 7f0928800640 1 -- 172.21.15.31:0/758328530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0920156ae0 con 0x7f09201534f0 2024-09-11T00:32:27.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.134+0000 7f091a000640 1 -- 172.21.15.31:0/758328530 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5702 (secure 0 0 0) 0x7f0910020020 con 0x7f09201534f0 2024-09-11T00:32:27.137 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:27.137 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"btime":"2024-09-11T00:32:26:295075+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:25.298184+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:32:27.138 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2024-09-11T00:32:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 -- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0904077f70 msgr2=0x7f090407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0904077f70 0x7f090407a430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0914033000 tx=0x7f0914004730 comp rx=0 tx=0).stop 2024-09-11T00:32:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 -- 172.21.15.31:0/758328530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09201534f0 msgr2=0x7f09200c3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09201534f0 0x7f09200c3800 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f091000c4d0 tx=0x7f091000c990 comp rx=0 tx=0).stop 2024-09-11T00:32:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 -- 172.21.15.31:0/758328530 shutdown_connections 2024-09-11T00:32:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0904077f70 0x7f090407a430 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:27.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0920159650 0x7f09200acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09201534f0 0x7f09200c3800 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 --2- 172.21.15.31:0/758328530 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09200c9b40 0x7f09201e3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 -- 172.21.15.31:0/758328530 >> 172.21.15.31:0/758328530 conn(0x7f0920068f10 msgr2=0x7f09200cb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 -- 172.21.15.31:0/758328530 shutdown_connections 2024-09-11T00:32:27.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:27.137+0000 7f0928800640 1 -- 172.21.15.31:0/758328530 wait complete. 2024-09-11T00:32:27.156 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:32:26:295075+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4390": { "addr": "172.21.15.204:6835/1801149506", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1801149506, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 1801149506, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4390, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:30:49.364654+0000", "qdb_cluster": [ 4390 ], "qdb_leader": 4390, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4390 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4721": { "addr": "172.21.15.204:6837/2954547253", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 2954547253, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 2954547253, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4721, "incarnation": 21, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:32:25.298184+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4721 } } } ], "standbys": [ { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:32:27.239 INFO:tasks.workunit.client.0.smithi031.stderr:Note: switching to '8f219c476cb4f26fc5719c661c0323719a45cb30'. 2024-09-11T00:32:27.239 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-11T00:32:27.239 INFO:tasks.workunit.client.0.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T00:32:27.239 INFO:tasks.workunit.client.0.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T00:32:27.239 INFO:tasks.workunit.client.0.smithi031.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T00:32:27.239 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-11T00:32:27.239 INFO:tasks.workunit.client.0.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T00:32:27.239 INFO:tasks.workunit.client.0.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T00:32:27.239 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-11T00:32:27.240 INFO:tasks.workunit.client.0.smithi031.stderr: git switch -c 2024-09-11T00:32:27.240 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-11T00:32:27.240 INFO:tasks.workunit.client.0.smithi031.stderr:Or undo this operation with: 2024-09-11T00:32:27.240 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-11T00:32:27.240 INFO:tasks.workunit.client.0.smithi031.stderr: git switch - 2024-09-11T00:32:27.240 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-11T00:32:27.240 INFO:tasks.workunit.client.0.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T00:32:27.240 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-09-11T00:32:27.240 INFO:tasks.workunit.client.0.smithi031.stderr:HEAD is now at 8f219c476cb Merge branch 'mgr-vol-mod-disable' of https://github.com/rishabh-d-dave/ceph into wip-hemanth-testing-2024-09-10-0723 2024-09-11T00:32:27.251 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-11T00:32:27.313 INFO:tasks.workunit.client.0.smithi031.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T00:32:27.318 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T00:32:27.318 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T00:32:27.350 INFO:tasks.workunit.client.1.smithi204.stderr:++ dirname /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh 2024-09-11T00:32:27.352 INFO:tasks.workunit.client.1.smithi204.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.1/qa/workunits/suites 2024-09-11T00:32:27.353 INFO:tasks.workunit.client.1.smithi204.stderr:+ trap -- retry EXIT 2024-09-11T00:32:27.353 INFO:tasks.workunit.client.1.smithi204.stderr:+ rm -rf ffsb 2024-09-11T00:32:27.356 INFO:tasks.workunit.client.1.smithi204.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-09-11T00:32:27.365 INFO:tasks.workunit.client.1.smithi204.stderr:Cloning into 'ffsb'... 2024-09-11T00:32:27.949 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T00:32:28.043 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T00:32:28.109 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T00:32:28.113 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T00:32:28.113 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T00:32:28.178 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T00:32:28.186 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:32:28.186 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-11T00:32:28.221 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-09-11T00:32:28.223 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-09-11T00:32:28.223 DEBUG:teuthology.orchestra.run.smithi031:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8f219c476cb4f26fc5719c661c0323719a45cb30 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-11T00:32:28.320 INFO:tasks.workunit.client.0.smithi031.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-11T00:32:28.322 INFO:tasks.workunit.client.0.smithi031.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-09-11T00:32:28.322 INFO:tasks.workunit.client.0.smithi031.stderr:+ trap -- retry EXIT 2024-09-11T00:32:28.322 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm -rf ffsb 2024-09-11T00:32:28.325 INFO:tasks.workunit.client.0.smithi031.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-09-11T00:32:28.333 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into 'ffsb'... 2024-09-11T00:32:29.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:29.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.336+0000 7f9563e00640 1 -- 172.21.15.31:0/1734439758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95640c5aa0 msgr2=0x7f95640c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.336+0000 7f9563e00640 1 --2- 172.21.15.31:0/1734439758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95640c5aa0 0x7f95640c7150 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9564072000 tx=0x7f955402a580 comp rx=0 tx=0).stop 2024-09-11T00:32:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.336+0000 7f9563e00640 1 -- 172.21.15.31:0/1734439758 shutdown_connections 2024-09-11T00:32:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.336+0000 7f9563e00640 1 --2- 172.21.15.31:0/1734439758 >> v2:172.21.15.204:3300/0 conn(0x7f9564156350 0x7f95641567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.336+0000 7f9563e00640 1 --2- 172.21.15.31:0/1734439758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95640c5aa0 0x7f95640c7150 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.336+0000 7f9563e00640 1 -- 172.21.15.31:0/1734439758 >> 172.21.15.31:0/1734439758 conn(0x7f956406caf0 msgr2=0x7f956406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.336+0000 7f9563e00640 1 -- 172.21.15.31:0/1734439758 shutdown_connections 2024-09-11T00:32:29.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.337+0000 7f9563e00640 1 -- 172.21.15.31:0/1734439758 wait complete. 2024-09-11T00:32:29.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.338+0000 7f9563e00640 1 Processor -- start 2024-09-11T00:32:29.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.338+0000 7f9563e00640 1 -- start start 2024-09-11T00:32:29.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.339+0000 7f9563e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95640c5aa0 0x7f9564072100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:29.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.339+0000 7f9563e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f95640c6420 0x7f9564155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:29.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.339+0000 7f9563e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9564156350 0x7f95640ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:29.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.339+0000 7f9563e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95640c2e80 con 0x7f9564156350 2024-09-11T00:32:29.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.339+0000 7f9563e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f95640c2ba0 con 0x7f95640c5aa0 2024-09-11T00:32:29.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.339+0000 7f9563e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9564153a20 con 0x7f95640c6420 2024-09-11T00:32:29.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.339+0000 7f9562a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95640c5aa0 0x7f9564072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:29.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.340+0000 7f9562a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95640c5aa0 0x7f9564072100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:37758/0 (socket says 172.21.15.31:37758) 2024-09-11T00:32:29.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.340+0000 7f9562a00640 1 -- 172.21.15.31:0/1865626440 learned_addr learned my addr 172.21.15.31:0/1865626440 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:29.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.340+0000 7f9562000640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f95640c6420 0x7f9564155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:29.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.340+0000 7f9563400640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9564156350 0x7f95640ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.340+0000 7f9562a00640 1 -- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f95640c6420 msgr2=0x7f9564155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.340+0000 7f9562a00640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f95640c6420 0x7f9564155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.340+0000 7f9562a00640 1 -- 172.21.15.31:0/1865626440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9564156350 msgr2=0x7f95640ab190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.340+0000 7f9562a00640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9564156350 0x7f95640ab190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.340+0000 7f9562a00640 1 -- 172.21.15.31:0/1865626440 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956414ea50 con 0x7f95640c5aa0 2024-09-11T00:32:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.340+0000 7f9562a00640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95640c5aa0 0x7f9564072100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9554009850 tx=0x7f95540039e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.341+0000 7f955aa00640 1 -- 172.21.15.31:0/1865626440 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955403f070 con 0x7f95640c5aa0 2024-09-11T00:32:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.341+0000 7f955aa00640 1 -- 172.21.15.31:0/1865626440 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9554004890 con 0x7f95640c5aa0 2024-09-11T00:32:29.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.341+0000 7f955aa00640 1 -- 172.21.15.31:0/1865626440 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9554035030 con 0x7f95640c5aa0 2024-09-11T00:32:29.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.341+0000 7f9563e00640 1 -- 172.21.15.31:0/1865626440 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95640acb40 con 0x7f95640c5aa0 2024-09-11T00:32:29.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.341+0000 7f9563e00640 1 -- 172.21.15.31:0/1865626440 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95640abcb0 con 0x7f95640c5aa0 2024-09-11T00:32:29.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.342+0000 7f9563e00640 1 -- 172.21.15.31:0/1865626440 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95640c5e80 con 0x7f95640c5aa0 2024-09-11T00:32:29.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.343+0000 7f955aa00640 1 -- 172.21.15.31:0/1865626440 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9554004ca0 con 0x7f95640c5aa0 2024-09-11T00:32:29.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.344+0000 7f955aa00640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9544077f70 0x7f954407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:29.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.345+0000 7f955aa00640 1 -- 172.21.15.31:0/1865626440 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f95540bb4b0 con 0x7f95640c5aa0 2024-09-11T00:32:29.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.345+0000 7f9562000640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9544077f70 0x7f954407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:29.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.346+0000 7f9562000640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9544077f70 0x7f954407a430 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f95641534c0 tx=0x7f9550009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:29.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.350+0000 7f955aa00640 1 -- 172.21.15.31:0/1865626440 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f95640c7690 con 0x7f95640c5aa0 2024-09-11T00:32:29.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.611+0000 7f9563e00640 1 -- 172.21.15.31:0/1865626440 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95640a7c40 con 0x7f95640c5aa0 2024-09-11T00:32:29.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.612+0000 7f955aa00640 1 -- 172.21.15.31:0/1865626440 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5709 (secure 0 0 0) 0x7f9554083450 con 0x7f95640c5aa0 2024-09-11T00:32:29.614 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:29.614 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"btime":"2024-09-11T00:32:27:304532+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:29.614 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 24 2024-09-11T00:32:29.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 -- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9544077f70 msgr2=0x7f954407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:29.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9544077f70 0x7f954407a430 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f95641534c0 tx=0x7f9550009040 comp rx=0 tx=0).stop 2024-09-11T00:32:29.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 -- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95640c5aa0 msgr2=0x7f9564072100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:29.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95640c5aa0 0x7f9564072100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9554009850 tx=0x7f95540039e0 comp rx=0 tx=0).stop 2024-09-11T00:32:29.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 -- 172.21.15.31:0/1865626440 shutdown_connections 2024-09-11T00:32:29.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9544077f70 0x7f954407a430 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:29.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9564156350 0x7f95640ab190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f95640c6420 0x7f9564155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 --2- 172.21.15.31:0/1865626440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f95640c5aa0 0x7f9564072100 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 -- 172.21.15.31:0/1865626440 >> 172.21.15.31:0/1865626440 conn(0x7f956406caf0 msgr2=0x7f956414fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 -- 172.21.15.31:0/1865626440 shutdown_connections 2024-09-11T00:32:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:29.615+0000 7f9563e00640 1 -- 172.21.15.31:0/1865626440 wait complete. 2024-09-11T00:32:29.633 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=4310 2024-09-11T00:32:29.633 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.a 2024-09-11T00:32:30.336 INFO:tasks.ceph.mds.b.smithi204.stderr:2024-09-11T00:32:30.334+0000 7f1c34600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:32:32.770 INFO:tasks.workunit.client.1.smithi204.stderr:Updating files: 71% (42/59) Updating files: 72% (43/59) Updating files: 74% (44/59) Updating files: 76% (45/59) Updating files: 77% (46/59) Updating files: 79% (47/59) Updating files: 81% (48/59) Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2024-09-11T00:32:32.856 INFO:tasks.workunit.client.1.smithi204.stderr:+ trap - EXIT 2024-09-11T00:32:32.856 INFO:tasks.workunit.client.1.smithi204.stderr:+ cd ffsb 2024-09-11T00:32:32.858 INFO:tasks.workunit.client.1.smithi204.stderr:+ ./configure 2024-09-11T00:32:33.128 INFO:tasks.workunit.client.1.smithi204.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-11T00:32:33.634 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-11T00:32:33.634 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-11T00:32:33.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-11T00:32:33.638 INFO:tasks.ceph.mds.a:Started 2024-09-11T00:32:33.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:33.769 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-11T00:32:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.799+0000 7ff6d6a00640 1 -- 172.21.15.31:0/1828763778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6d00c71c0 msgr2=0x7ff6d00c5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.799+0000 7ff6d6a00640 1 --2- 172.21.15.31:0/1828763778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6d00c71c0 0x7ff6d00c5920 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009830 tx=0x7ff6bc02a4b0 comp rx=0 tx=0).stop 2024-09-11T00:32:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.799+0000 7ff6d6a00640 1 -- 172.21.15.31:0/1828763778 shutdown_connections 2024-09-11T00:32:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.799+0000 7ff6d6a00640 1 --2- 172.21.15.31:0/1828763778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6d00c71c0 0x7ff6d00c5920 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.799+0000 7ff6d6a00640 1 -- 172.21.15.31:0/1828763778 >> 172.21.15.31:0/1828763778 conn(0x7ff6d00688e0 msgr2=0x7ff6d0068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.799+0000 7ff6d6a00640 1 -- 172.21.15.31:0/1828763778 shutdown_connections 2024-09-11T00:32:33.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.799+0000 7ff6d6a00640 1 -- 172.21.15.31:0/1828763778 wait complete. 2024-09-11T00:32:33.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.801+0000 7ff6d6a00640 1 Processor -- start 2024-09-11T00:32:33.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.801+0000 7ff6d6a00640 1 -- start start 2024-09-11T00:32:33.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.801+0000 7ff6d6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6d00c71c0 0x7ff6d01ec160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:33.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.801+0000 7ff6d6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6d00c8ed0 0x7ff6d00adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:33.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6d6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6d0158260 0x7ff6d01dfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:33.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6d6a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d0158640 con 0x7ff6d00c8ed0 2024-09-11T00:32:33.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6d6a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d00c44c0 con 0x7ff6d00c71c0 2024-09-11T00:32:33.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6d6a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d00c9a70 con 0x7ff6d0158260 2024-09-11T00:32:33.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6cf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6d00c8ed0 0x7ff6d00adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:33.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6cf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6d00c8ed0 0x7ff6d00adc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43946/0 (socket says 172.21.15.31:43946) 2024-09-11T00:32:33.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6cf400640 1 -- 172.21.15.31:0/869857512 learned_addr learned my addr 172.21.15.31:0/869857512 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:33.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6cfe00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6d00c71c0 0x7ff6d01ec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:33.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6d4a00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6d0158260 0x7ff6d01dfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:33.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6cfe00640 1 -- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6d0158260 msgr2=0x7ff6d01dfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6cfe00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6d0158260 0x7ff6d01dfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6cfe00640 1 -- 172.21.15.31:0/869857512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6d00c8ed0 msgr2=0x7ff6d00adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6cfe00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6d00c8ed0 0x7ff6d00adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.802+0000 7ff6cfe00640 1 -- 172.21.15.31:0/869857512 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d00aebb0 con 0x7ff6d00c71c0 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.803+0000 7ff6cfe00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6d00c71c0 0x7ff6d01ec160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000c500 tx=0x7ff6c000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.803+0000 7ff6c7e00640 1 -- 172.21.15.31:0/869857512 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c00117b0 con 0x7ff6d00c71c0 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.803+0000 7ff6c7e00640 1 -- 172.21.15.31:0/869857512 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6c0017070 con 0x7ff6d00c71c0 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.803+0000 7ff6c7e00640 1 -- 172.21.15.31:0/869857512 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0011d00 con 0x7ff6d00c71c0 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.803+0000 7ff6d6a00640 1 -- 172.21.15.31:0/869857512 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d01552a0 con 0x7ff6d00c71c0 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.803+0000 7ff6d6a00640 1 -- 172.21.15.31:0/869857512 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d00ae160 con 0x7ff6d00c71c0 2024-09-11T00:32:33.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.804+0000 7ff6d6a00640 1 -- 172.21.15.31:0/869857512 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d00bee60 con 0x7ff6d00c71c0 2024-09-11T00:32:33.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.805+0000 7ff6c7e00640 1 -- 172.21.15.31:0/869857512 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff6c001c460 con 0x7ff6d00c71c0 2024-09-11T00:32:33.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.806+0000 7ff6c7e00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6b0077f70 0x7ff6b007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:33.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.806+0000 7ff6c7e00640 1 -- 172.21.15.31:0/869857512 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7ff6c0099f20 con 0x7ff6d00c71c0 2024-09-11T00:32:33.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.807+0000 7ff6cf400640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6b0077f70 0x7ff6b007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:33.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.807+0000 7ff6cf400640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6b0077f70 0x7ff6b007a430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0071fc0 tx=0x7ff6bc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:33.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:33.811+0000 7ff6c7e00640 1 -- 172.21.15.31:0/869857512 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff6d00bee60 con 0x7ff6d00c71c0 2024-09-11T00:32:34.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.042+0000 7ff6d6a00640 1 -- 172.21.15.31:0/869857512 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6d0154bd0 con 0x7ff6d00c71c0 2024-09-11T00:32:34.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.043+0000 7ff6c7e00640 1 -- 172.21.15.31:0/869857512 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5709 (secure 0 0 0) 0x7ff6c0061ed0 con 0x7ff6d00c71c0 2024-09-11T00:32:34.045 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:34.045 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":24,"btime":"2024-09-11T00:32:27:304532+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:34.045 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 24 2024-09-11T00:32:34.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 -- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6b0077f70 msgr2=0x7ff6b007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:34.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6b0077f70 0x7ff6b007a430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0071fc0 tx=0x7ff6bc004730 comp rx=0 tx=0).stop 2024-09-11T00:32:34.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 -- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6d00c71c0 msgr2=0x7ff6d01ec160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:34.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6d00c71c0 0x7ff6d01ec160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff6c000c500 tx=0x7ff6c000c9c0 comp rx=0 tx=0).stop 2024-09-11T00:32:34.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 -- 172.21.15.31:0/869857512 shutdown_connections 2024-09-11T00:32:34.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6b0077f70 0x7ff6b007a430 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:34.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6d0158260 0x7ff6d01dfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:34.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6d00c8ed0 0x7ff6d00adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:34.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 --2- 172.21.15.31:0/869857512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6d00c71c0 0x7ff6d01ec160 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:34.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 -- 172.21.15.31:0/869857512 >> 172.21.15.31:0/869857512 conn(0x7ff6d00688e0 msgr2=0x7ff6d0156620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:34.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 -- 172.21.15.31:0/869857512 shutdown_connections 2024-09-11T00:32:34.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:34.046+0000 7ff6d6a00640 1 -- 172.21.15.31:0/869857512 wait complete. 2024-09-11T00:32:34.064 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-11T00:32:34.145 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether build environment is sane... yes 2024-09-11T00:32:34.177 INFO:tasks.workunit.client.1.smithi204.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-11T00:32:34.178 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gawk... gawk 2024-09-11T00:32:34.213 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether make sets $(MAKE)... yes 2024-09-11T00:32:34.304 INFO:tasks.workunit.client.1.smithi204.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-11T00:32:34.304 INFO:tasks.workunit.client.1.smithi204.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-11T00:32:34.332 INFO:tasks.workunit.client.1.smithi204.stdout:checking for style of include used by make... GNU 2024-09-11T00:32:34.339 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gcc... gcc 2024-09-11T00:32:34.488 INFO:tasks.workunit.client.1.smithi204.stdout:checking for C compiler default output file name... a.out 2024-09-11T00:32:34.497 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether the C compiler works... yes 2024-09-11T00:32:34.504 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether we are cross compiling... no 2024-09-11T00:32:34.600 INFO:tasks.workunit.client.1.smithi204.stdout:checking for suffix of executables... 2024-09-11T00:32:34.681 INFO:tasks.workunit.client.1.smithi204.stdout:checking for suffix of object files... o 2024-09-11T00:32:34.688 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 40% (24/59) Updating files: 42% (25/59) Updating files: 44% (26/59) Updating files: 45% (27/59) Updating files: 47% (28/59) Updating files: 49% (29/59) Updating files: 50% (30/59) Updating files: 52% (31/59) Updating files: 54% (32/59) Updating files: 55% (33/59) Updating files: 57% (34/59) Updating files: 59% (35/59) Updating files: 61% (36/59) Updating files: 62% (37/59) Updating files: 64% (38/59) Updating files: 66% (39/59) Updating files: 67% (40/59) Updating files: 69% (41/59) Updating files: 71% (42/59) Updating files: 72% (43/59) Updating files: 74% (44/59) Updating files: 76% (45/59) Updating files: 77% (46/59) Updating files: 79% (47/59) Updating files: 81% (48/59) Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2024-09-11T00:32:34.755 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether we are using the GNU C compiler... yes 2024-09-11T00:32:34.777 INFO:tasks.workunit.client.0.smithi031.stderr:+ trap - EXIT 2024-09-11T00:32:34.778 INFO:tasks.workunit.client.0.smithi031.stderr:+ cd ffsb 2024-09-11T00:32:34.780 INFO:tasks.workunit.client.0.smithi031.stderr:+ ./configure 2024-09-11T00:32:34.833 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether gcc accepts -g... yes 2024-09-11T00:32:34.971 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-11T00:32:35.085 INFO:tasks.workunit.client.0.smithi031.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-11T00:32:35.204 INFO:tasks.workunit.client.1.smithi204.stdout:checking dependency style of gcc... gcc3 2024-09-11T00:32:35.526 INFO:tasks.workunit.client.1.smithi204.stdout:checking how to run the C preprocessor... gcc -E 2024-09-11T00:32:35.633 INFO:tasks.workunit.client.1.smithi204.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-11T00:32:35.637 INFO:tasks.workunit.client.1.smithi204.stdout:checking for egrep... /usr/bin/grep -E 2024-09-11T00:32:35.663 INFO:tasks.workunit.client.1.smithi204.stdout:checking for AIX... no 2024-09-11T00:32:35.690 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gcc... (cached) gcc 2024-09-11T00:32:35.718 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-11T00:32:35.720 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-11T00:32:35.720 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-11T00:32:35.721 INFO:tasks.workunit.client.1.smithi204.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-11T00:32:36.065 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:36.102 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether build environment is sane... yes 2024-09-11T00:32:36.135 INFO:tasks.workunit.client.0.smithi031.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-11T00:32:36.136 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gawk... gawk 2024-09-11T00:32:36.171 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether make sets $(MAKE)... yes 2024-09-11T00:32:36.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.247+0000 7fc571800640 1 -- 172.21.15.31:0/2170184230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc56c072480 msgr2=0x7fc56c1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:36.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.247+0000 7fc571800640 1 --2- 172.21.15.31:0/2170184230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc56c072480 0x7fc56c1511a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009830 tx=0x7fc55c02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:32:36.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.247+0000 7fc571800640 1 -- 172.21.15.31:0/2170184230 shutdown_connections 2024-09-11T00:32:36.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.247+0000 7fc571800640 1 --2- 172.21.15.31:0/2170184230 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc56c153f20 0x7fc56c1516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:36.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.247+0000 7fc571800640 1 --2- 172.21.15.31:0/2170184230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc56c072480 0x7fc56c1511a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:36.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.247+0000 7fc571800640 1 -- 172.21.15.31:0/2170184230 >> 172.21.15.31:0/2170184230 conn(0x7fc56c0a9750 msgr2=0x7fc56c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:36.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.247+0000 7fc571800640 1 -- 172.21.15.31:0/2170184230 shutdown_connections 2024-09-11T00:32:36.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.247+0000 7fc571800640 1 -- 172.21.15.31:0/2170184230 wait complete. 2024-09-11T00:32:36.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.249+0000 7fc571800640 1 Processor -- start 2024-09-11T00:32:36.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.249+0000 7fc571800640 1 -- start start 2024-09-11T00:32:36.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.250+0000 7fc571800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc56c072480 0x7fc56c06f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:36.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.250+0000 7fc571800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc56c0c23a0 0x7fc56c14c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:36.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.250+0000 7fc571800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc56c153f20 0x7fc56c14cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:36.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.250+0000 7fc571800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c074200 con 0x7fc56c153f20 2024-09-11T00:32:36.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.250+0000 7fc571800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c074080 con 0x7fc56c0c23a0 2024-09-11T00:32:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.250+0000 7fc571800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc56c072860 con 0x7fc56c072480 2024-09-11T00:32:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.250+0000 7fc56b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc56c153f20 0x7fc56c14cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc56c153f20 0x7fc56c14cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43948/0 (socket says 172.21.15.31:43948) 2024-09-11T00:32:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56b400640 1 -- 172.21.15.31:0/3813237056 learned_addr learned my addr 172.21.15.31:0/3813237056 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56aa00640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc56c072480 0x7fc56c06f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56a000640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc56c0c23a0 0x7fc56c14c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56b400640 1 -- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc56c072480 msgr2=0x7fc56c06f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56b400640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc56c072480 0x7fc56c06f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56b400640 1 -- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc56c0c23a0 msgr2=0x7fc56c14c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56b400640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc56c0c23a0 0x7fc56c14c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56b400640 1 -- 172.21.15.31:0/3813237056 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c0c7510 con 0x7fc56c153f20 2024-09-11T00:32:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56a000640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc56c0c23a0 0x7fc56c14c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:32:36.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.251+0000 7fc56b400640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc56c153f20 0x7fc56c14cc00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc56400ea80 tx=0x7fc56400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:36.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.252+0000 7fc562a00640 1 -- 172.21.15.31:0/3813237056 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564004b20 con 0x7fc56c153f20 2024-09-11T00:32:36.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.252+0000 7fc562a00640 1 -- 172.21.15.31:0/3813237056 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc564004ce0 con 0x7fc56c153f20 2024-09-11T00:32:36.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.252+0000 7fc562a00640 1 -- 172.21.15.31:0/3813237056 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564005280 con 0x7fc56c153f20 2024-09-11T00:32:36.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.252+0000 7fc571800640 1 -- 172.21.15.31:0/3813237056 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56c0c62e0 con 0x7fc56c153f20 2024-09-11T00:32:36.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.252+0000 7fc571800640 1 -- 172.21.15.31:0/3813237056 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc56c14d140 con 0x7fc56c153f20 2024-09-11T00:32:36.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.253+0000 7fc571800640 1 -- 172.21.15.31:0/3813237056 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc56c1511a0 con 0x7fc56c153f20 2024-09-11T00:32:36.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.254+0000 7fc562a00640 1 -- 172.21.15.31:0/3813237056 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc564005420 con 0x7fc56c153f20 2024-09-11T00:32:36.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.255+0000 7fc562a00640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc54c077f70 0x7fc54c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:36.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.255+0000 7fc562a00640 1 -- 172.21.15.31:0/3813237056 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7fc56409ae10 con 0x7fc56c153f20 2024-09-11T00:32:36.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.256+0000 7fc56aa00640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc54c077f70 0x7fc54c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:36.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.257+0000 7fc56aa00640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc54c077f70 0x7fc54c07a430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc55c034000 tx=0x7fc55c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:36.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.260+0000 7fc562a00640 1 -- 172.21.15.31:0/3813237056 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc56409e030 con 0x7fc56c153f20 2024-09-11T00:32:36.271 INFO:tasks.workunit.client.0.smithi031.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-11T00:32:36.272 INFO:tasks.workunit.client.0.smithi031.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-11T00:32:36.300 INFO:tasks.workunit.client.0.smithi031.stdout:checking for style of include used by make... GNU 2024-09-11T00:32:36.310 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gcc... gcc 2024-09-11T00:32:36.470 INFO:tasks.workunit.client.0.smithi031.stdout:checking for C compiler default output file name... a.out 2024-09-11T00:32:36.478 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether the C compiler works... yes 2024-09-11T00:32:36.484 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether we are cross compiling... no 2024-09-11T00:32:36.486 INFO:tasks.workunit.client.1.smithi204.stdout:checking for main in -lm... yes 2024-09-11T00:32:36.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.524+0000 7fc571800640 1 -- 172.21.15.31:0/3813237056 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc56c0c69e0 con 0x7fc56c153f20 2024-09-11T00:32:36.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.525+0000 7fc562a00640 1 -- 172.21.15.31:0/3813237056 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6534 (secure 0 0 0) 0x7fc564063490 con 0x7fc56c153f20 2024-09-11T00:32:36.527 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:36.527 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"btime":"2024-09-11T00:32:34:318805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5076,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:36.527 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 -- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc54c077f70 msgr2=0x7fc54c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc54c077f70 0x7fc54c07a430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc55c034000 tx=0x7fc55c004730 comp rx=0 tx=0).stop 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 -- 172.21.15.31:0/3813237056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc56c153f20 msgr2=0x7fc56c14cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc56c153f20 0x7fc56c14cc00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc56400ea80 tx=0x7fc56400ef40 comp rx=0 tx=0).stop 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 -- 172.21.15.31:0/3813237056 shutdown_connections 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc54c077f70 0x7fc54c07a430 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc56c153f20 0x7fc56c14cc00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc56c0c23a0 0x7fc56c14c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 --2- 172.21.15.31:0/3813237056 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc56c072480 0x7fc56c06f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 -- 172.21.15.31:0/3813237056 >> 172.21.15.31:0/3813237056 conn(0x7fc56c0a9750 msgr2=0x7fc56c14fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 -- 172.21.15.31:0/3813237056 shutdown_connections 2024-09-11T00:32:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:36.528+0000 7fc571800640 1 -- 172.21.15.31:0/3813237056 wait complete. 2024-09-11T00:32:36.546 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-11T00:32:36.548 INFO:tasks.mds_thrash.fs.[b]:waiting for 38 secs before thrashing 2024-09-11T00:32:36.576 INFO:tasks.workunit.client.0.smithi031.stdout:checking for suffix of executables... 2024-09-11T00:32:36.617 INFO:tasks.workunit.client.1.smithi204.stdout:checking for main in -lpthread... yes 2024-09-11T00:32:36.653 INFO:tasks.workunit.client.0.smithi031.stdout:checking for suffix of object files... o 2024-09-11T00:32:36.728 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether we are using the GNU C compiler... yes 2024-09-11T00:32:36.809 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether gcc accepts -g... yes 2024-09-11T00:32:36.949 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-11T00:32:36.958 INFO:tasks.workunit.client.1.smithi204.stdout:checking for ANSI C header files... yes 2024-09-11T00:32:37.150 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-11T00:32:37.201 INFO:tasks.workunit.client.0.smithi031.stdout:checking dependency style of gcc... gcc3 2024-09-11T00:32:37.263 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/types.h... yes 2024-09-11T00:32:37.343 INFO:tasks.workunit.client.0.smithi031.stdout:checking how to run the C preprocessor... gcc -E 2024-09-11T00:32:37.455 INFO:tasks.workunit.client.0.smithi031.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-11T00:32:37.458 INFO:tasks.workunit.client.0.smithi031.stdout:checking for egrep... /usr/bin/grep -E 2024-09-11T00:32:37.485 INFO:tasks.workunit.client.0.smithi031.stdout:checking for AIX... no 2024-09-11T00:32:37.546 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gcc... (cached) gcc 2024-09-11T00:32:37.574 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-11T00:32:37.576 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-11T00:32:37.577 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-11T00:32:37.578 INFO:tasks.workunit.client.0.smithi031.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-11T00:32:37.720 INFO:tasks.workunit.client.0.smithi031.stdout:checking for main in -lm... yes 2024-09-11T00:32:37.850 INFO:tasks.workunit.client.0.smithi031.stdout:checking for main in -lpthread... yes 2024-09-11T00:32:38.034 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/stat.h... yes 2024-09-11T00:32:38.151 INFO:tasks.workunit.client.1.smithi204.stdout:checking for stdlib.h... yes 2024-09-11T00:32:38.222 INFO:tasks.workunit.client.0.smithi031.stdout:checking for ANSI C header files... yes 2024-09-11T00:32:38.267 INFO:tasks.workunit.client.1.smithi204.stdout:checking for string.h... yes 2024-09-11T00:32:38.351 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-11T00:32:38.398 INFO:tasks.workunit.client.1.smithi204.stdout:checking for memory.h... yes 2024-09-11T00:32:38.452 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/types.h... yes 2024-09-11T00:32:38.517 INFO:tasks.workunit.client.1.smithi204.stdout:checking for strings.h... yes 2024-09-11T00:32:38.567 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/stat.h... yes 2024-09-11T00:32:38.633 INFO:tasks.workunit.client.1.smithi204.stdout:checking for inttypes.h... yes 2024-09-11T00:32:38.685 INFO:tasks.workunit.client.0.smithi031.stdout:checking for stdlib.h... yes 2024-09-11T00:32:38.803 INFO:tasks.workunit.client.0.smithi031.stdout:checking for string.h... yes 2024-09-11T00:32:38.918 INFO:tasks.workunit.client.0.smithi031.stdout:checking for memory.h... yes 2024-09-11T00:32:38.978 INFO:tasks.workunit.client.1.smithi204.stdout:checking for stdint.h... yes 2024-09-11T00:32:39.040 INFO:tasks.workunit.client.0.smithi031.stdout:checking for strings.h... yes 2024-09-11T00:32:39.104 INFO:tasks.workunit.client.1.smithi204.stdout:checking for unistd.h... yes 2024-09-11T00:32:39.235 INFO:tasks.workunit.client.1.smithi204.stdout:checking pthread.h usability... yes 2024-09-11T00:32:39.289 INFO:tasks.workunit.client.1.smithi204.stdout:checking pthread.h presence... yes 2024-09-11T00:32:39.290 INFO:tasks.workunit.client.1.smithi204.stdout:checking for pthread.h... yes 2024-09-11T00:32:39.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:32:39.360 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:32:39.388 INFO:tasks.workunit.client.0.smithi031.stdout:checking for inttypes.h... yes 2024-09-11T00:32:39.427 INFO:tasks.workunit.client.1.smithi204.stdout:checking fcntl.h usability... yes 2024-09-11T00:32:39.478 INFO:tasks.workunit.client.1.smithi204.stdout:checking fcntl.h presence... yes 2024-09-11T00:32:39.480 INFO:tasks.workunit.client.1.smithi204.stdout:checking for fcntl.h... yes 2024-09-11T00:32:39.509 INFO:tasks.workunit.client.0.smithi031.stdout:checking for stdint.h... yes 2024-09-11T00:32:39.601 INFO:tasks.workunit.client.1.smithi204.stdout:checking limits.h usability... yes 2024-09-11T00:32:39.650 INFO:tasks.workunit.client.1.smithi204.stdout:checking limits.h presence... yes 2024-09-11T00:32:39.652 INFO:tasks.workunit.client.1.smithi204.stdout:checking for limits.h... yes 2024-09-11T00:32:39.683 INFO:tasks.workunit.client.1.smithi204.stdout:checking for stdint.h... (cached) yes 2024-09-11T00:32:39.801 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/time.h usability... yes 2024-09-11T00:32:39.866 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/time.h presence... yes 2024-09-11T00:32:39.867 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/time.h... yes 2024-09-11T00:32:39.884 INFO:tasks.workunit.client.1.smithi204.stdout:checking for unistd.h... (cached) yes 2024-09-11T00:32:40.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:40.381 INFO:tasks.workunit.client.0.smithi031.stdout:checking for unistd.h... yes 2024-09-11T00:32:40.421 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/vfs.h usability... yes 2024-09-11T00:32:40.472 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/vfs.h presence... yes 2024-09-11T00:32:40.473 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/vfs.h... yes 2024-09-11T00:32:40.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.480+0000 7f5321600640 1 -- 172.21.15.31:0/2743313633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f531c156960 msgr2=0x7f531c06e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.480+0000 7f5321600640 1 --2- 172.21.15.31:0/2743313633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f531c156960 0x7f531c06e140 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f531c153d20 tx=0x7f530802a560 comp rx=0 tx=0).stop 2024-09-11T00:32:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.480+0000 7f5321600640 1 -- 172.21.15.31:0/2743313633 shutdown_connections 2024-09-11T00:32:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.480+0000 7f5321600640 1 --2- 172.21.15.31:0/2743313633 >> v2:172.21.15.204:3300/0 conn(0x7f531c0cc220 0x7f531c06e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.480+0000 7f5321600640 1 --2- 172.21.15.31:0/2743313633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f531c156960 0x7f531c06e140 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.480+0000 7f5321600640 1 -- 172.21.15.31:0/2743313633 >> 172.21.15.31:0/2743313633 conn(0x7f531c068f10 msgr2=0x7f531c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.480+0000 7f5321600640 1 -- 172.21.15.31:0/2743313633 shutdown_connections 2024-09-11T00:32:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.480+0000 7f5321600640 1 -- 172.21.15.31:0/2743313633 wait complete. 2024-09-11T00:32:40.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.481+0000 7f5321600640 1 Processor -- start 2024-09-11T00:32:40.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.482+0000 7f5321600640 1 -- start start 2024-09-11T00:32:40.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f5321600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f531c0cc220 0x7f531c1eef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:40.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f5321600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f531c150020 0x7f531c1eb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:40.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f5321600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f531c156960 0x7f531c1ebe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:40.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f5321600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f531c0c6ca0 con 0x7f531c156960 2024-09-11T00:32:40.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f5321600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f531c0c6b20 con 0x7f531c150020 2024-09-11T00:32:40.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f5321600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f531c1573d0 con 0x7f531c0cc220 2024-09-11T00:32:40.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f531be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f531c0cc220 0x7f531c1eef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f531b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f531c150020 0x7f531c1eb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f5320c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f531c156960 0x7f531c1ebe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f531b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f531c150020 0x7f531c1eb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:53762/0 (socket says 172.21.15.31:53762) 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f531be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f531c0cc220 0x7f531c1eef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50324/0 (socket says 172.21.15.31:50324) 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f531b400640 1 -- 172.21.15.31:0/24395862 learned_addr learned my addr 172.21.15.31:0/24395862 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f5320c00640 1 -- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f531c0cc220 msgr2=0x7f531c1eef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.483+0000 7f5320c00640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f531c0cc220 0x7f531c1eef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.484+0000 7f5320c00640 1 -- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f531c150020 msgr2=0x7f531c1eb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.484+0000 7f5320c00640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f531c150020 0x7f531c1eb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.484+0000 7f5320c00640 1 -- 172.21.15.31:0/24395862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531c0c5d90 con 0x7f531c156960 2024-09-11T00:32:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.484+0000 7f531b400640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f531c150020 0x7f531c1eb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:32:40.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.484+0000 7f5320c00640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f531c156960 0x7f531c1ebe50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f531400be20 tx=0x7f531400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:40.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.484+0000 7f5313e00640 1 -- 172.21.15.31:0/24395862 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5314019070 con 0x7f531c156960 2024-09-11T00:32:40.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.484+0000 7f5313e00640 1 -- 172.21.15.31:0/24395862 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5314014440 con 0x7f531c156960 2024-09-11T00:32:40.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.484+0000 7f5313e00640 1 -- 172.21.15.31:0/24395862 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5314014890 con 0x7f531c156960 2024-09-11T00:32:40.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.484+0000 7f5321600640 1 -- 172.21.15.31:0/24395862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f531c151bc0 con 0x7f531c156960 2024-09-11T00:32:40.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.484+0000 7f5321600640 1 -- 172.21.15.31:0/24395862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f531c14de90 con 0x7f531c156960 2024-09-11T00:32:40.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.485+0000 7f5321600640 1 -- 172.21.15.31:0/24395862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f531c0ad4e0 con 0x7f531c156960 2024-09-11T00:32:40.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.486+0000 7f5313e00640 1 -- 172.21.15.31:0/24395862 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f531401d3b0 con 0x7f531c156960 2024-09-11T00:32:40.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.487+0000 7f5313e00640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f52fc077f70 0x7f52fc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:40.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.488+0000 7f5313e00640 1 -- 172.21.15.31:0/24395862 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f531409a3b0 con 0x7f531c156960 2024-09-11T00:32:40.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.488+0000 7f531be00640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f52fc077f70 0x7f52fc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:40.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.489+0000 7f531be00640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f52fc077f70 0x7f52fc07a430 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f530c0017a0 tx=0x7f530c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:40.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.493+0000 7f5313e00640 1 -- 172.21.15.31:0/24395862 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f53140623e0 con 0x7f531c156960 2024-09-11T00:32:40.529 INFO:tasks.workunit.client.0.smithi031.stdout:checking pthread.h usability... yes 2024-09-11T00:32:40.584 INFO:tasks.workunit.client.0.smithi031.stdout:checking pthread.h presence... yes 2024-09-11T00:32:40.586 INFO:tasks.workunit.client.0.smithi031.stdout:checking for pthread.h... yes 2024-09-11T00:32:40.588 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/limits.h usability... no 2024-09-11T00:32:40.646 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/limits.h presence... no 2024-09-11T00:32:40.647 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/limits.h... no 2024-09-11T00:32:40.716 INFO:tasks.workunit.client.0.smithi031.stdout:checking fcntl.h usability... yes 2024-09-11T00:32:40.730 INFO:tasks.workunit.client.1.smithi204.stdout:checking for an ANSI C-conforming const... yes 2024-09-11T00:32:40.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.755+0000 7f5321600640 1 -- 172.21.15.31:0/24395862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f531c14dc40 con 0x7f531c156960 2024-09-11T00:32:40.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.756+0000 7f5313e00640 1 -- 172.21.15.31:0/24395862 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6534 (secure 0 0 0) 0x7f5314066e10 con 0x7f531c156960 2024-09-11T00:32:40.758 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:40.758 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"btime":"2024-09-11T00:32:34:318805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5076,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:40.758 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-09-11T00:32:40.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.758+0000 7f5321600640 1 -- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f52fc077f70 msgr2=0x7f52fc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:40.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.758+0000 7f5321600640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f52fc077f70 0x7f52fc07a430 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f530c0017a0 tx=0x7f530c009040 comp rx=0 tx=0).stop 2024-09-11T00:32:40.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.758+0000 7f5321600640 1 -- 172.21.15.31:0/24395862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f531c156960 msgr2=0x7f531c1ebe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:40.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.758+0000 7f5321600640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f531c156960 0x7f531c1ebe50 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f531400be20 tx=0x7f531400bf10 comp rx=0 tx=0).stop 2024-09-11T00:32:40.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.758+0000 7f5321600640 1 -- 172.21.15.31:0/24395862 shutdown_connections 2024-09-11T00:32:40.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.758+0000 7f5321600640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f52fc077f70 0x7f52fc07a430 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.758+0000 7f5321600640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f531c156960 0x7f531c1ebe50 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.758+0000 7f5321600640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f531c150020 0x7f531c1eb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.758+0000 7f5321600640 1 --2- 172.21.15.31:0/24395862 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f531c0cc220 0x7f531c1eef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.758+0000 7f5321600640 1 -- 172.21.15.31:0/24395862 >> 172.21.15.31:0/24395862 conn(0x7f531c068f10 msgr2=0x7f531c158150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:40.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.759+0000 7f5321600640 1 -- 172.21.15.31:0/24395862 shutdown_connections 2024-09-11T00:32:40.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.759+0000 7f5321600640 1 -- 172.21.15.31:0/24395862 wait complete. 2024-09-11T00:32:40.771 INFO:tasks.workunit.client.0.smithi031.stdout:checking fcntl.h presence... yes 2024-09-11T00:32:40.773 INFO:tasks.workunit.client.0.smithi031.stdout:checking for fcntl.h... yes 2024-09-11T00:32:40.777 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-11T00:32:40.778 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:32:40.778 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:32:40.796 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:32:40.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:40.813 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-11T00:32:40.817 INFO:tasks.workunit.client.1.smithi204.stdout:checking for function prototypes... yes 2024-09-11T00:32:40.903 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether setvbuf arguments are reversed... no 2024-09-11T00:32:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.958+0000 7f0791200640 1 -- 172.21.15.31:0/150597197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c0c6420 msgr2=0x7f078c153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.958+0000 7f0791200640 1 --2- 172.21.15.31:0/150597197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c0c6420 0x7f078c153310 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0774009830 tx=0x7f077402a540 comp rx=0 tx=0).stop 2024-09-11T00:32:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.958+0000 7f0791200640 1 -- 172.21.15.31:0/150597197 shutdown_connections 2024-09-11T00:32:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.958+0000 7f0791200640 1 --2- 172.21.15.31:0/150597197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c0c6420 0x7f078c153310 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.958+0000 7f0791200640 1 -- 172.21.15.31:0/150597197 >> 172.21.15.31:0/150597197 conn(0x7f078c06caf0 msgr2=0x7f078c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:40.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.958+0000 7f0791200640 1 -- 172.21.15.31:0/150597197 shutdown_connections 2024-09-11T00:32:40.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.958+0000 7f0791200640 1 -- 172.21.15.31:0/150597197 wait complete. 2024-09-11T00:32:40.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.960+0000 7f0791200640 1 Processor -- start 2024-09-11T00:32:40.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.960+0000 7f0791200640 1 -- start start 2024-09-11T00:32:40.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f0791200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f078c0c5aa0 0x7f078c154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:40.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f0791200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c0c6420 0x7f078c14dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:40.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f0791200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078c153850 0x7f078c14e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:40.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f0791200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0c5e80 con 0x7f078c153850 2024-09-11T00:32:40.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f0791200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0c3160 con 0x7f078c0c6420 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f0791200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f078c0c6000 con 0x7f078c0c5aa0 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f078be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078c153850 0x7f078c14e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f078be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078c153850 0x7f078c14e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32862/0 (socket says 172.21.15.31:32862) 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f078be00640 1 -- 172.21.15.31:0/160930246 learned_addr learned my addr 172.21.15.31:0/160930246 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f078b400640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f078c0c5aa0 0x7f078c154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f0782a00640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c0c6420 0x7f078c14dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f078be00640 1 -- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f078c0c5aa0 msgr2=0x7f078c154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.961+0000 7f078be00640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f078c0c5aa0 0x7f078c154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.962+0000 7f078be00640 1 -- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c0c6420 msgr2=0x7f078c14dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.962+0000 7f078be00640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c0c6420 0x7f078c14dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.962+0000 7f078be00640 1 -- 172.21.15.31:0/160930246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c14e760 con 0x7f078c153850 2024-09-11T00:32:40.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.962+0000 7f078be00640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078c153850 0x7f078c14e220 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f078400c700 tx=0x7f078400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:40.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.962+0000 7f0783e00640 1 -- 172.21.15.31:0/160930246 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0784003b70 con 0x7f078c153850 2024-09-11T00:32:40.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.962+0000 7f0783e00640 1 -- 172.21.15.31:0/160930246 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0784003d10 con 0x7f078c153850 2024-09-11T00:32:40.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.962+0000 7f0791200640 1 -- 172.21.15.31:0/160930246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078c14ebe0 con 0x7f078c153850 2024-09-11T00:32:40.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.962+0000 7f0783e00640 1 -- 172.21.15.31:0/160930246 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078400aab0 con 0x7f078c153850 2024-09-11T00:32:40.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.962+0000 7f0791200640 1 -- 172.21.15.31:0/160930246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f078c0aa190 con 0x7f078c153850 2024-09-11T00:32:40.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.964+0000 7f0791200640 1 -- 172.21.15.31:0/160930246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f078c153310 con 0x7f078c153850 2024-09-11T00:32:40.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.965+0000 7f0783e00640 1 -- 172.21.15.31:0/160930246 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f078400ac50 con 0x7f078c153850 2024-09-11T00:32:40.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.965+0000 7f0783e00640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0768077f70 0x7f076807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:40.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.965+0000 7f0783e00640 1 -- 172.21.15.31:0/160930246 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f0784099e00 con 0x7f078c153850 2024-09-11T00:32:40.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.966+0000 7f078b400640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0768077f70 0x7f076807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:40.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.967+0000 7f078b400640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0768077f70 0x7f076807a430 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f077c001eb0 tx=0x7f077c001660 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:40.970+0000 7f0783e00640 1 -- 172.21.15.31:0/160930246 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0784061e50 con 0x7f078c153850 2024-09-11T00:32:41.062 INFO:tasks.workunit.client.1.smithi204.stdout:checking for system... yes 2024-09-11T00:32:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.202+0000 7f0791200640 1 -- 172.21.15.31:0/160930246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f078c156ab0 con 0x7f078c153850 2024-09-11T00:32:41.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.203+0000 7f0783e00640 1 -- 172.21.15.31:0/160930246 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6534 (secure 0 0 0) 0x7f0784066880 con 0x7f078c153850 2024-09-11T00:32:41.205 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:41.205 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"btime":"2024-09-11T00:32:34:318805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5076,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:41.205 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-09-11T00:32:41.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 -- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0768077f70 msgr2=0x7f076807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:41.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0768077f70 0x7f076807a430 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f077c001eb0 tx=0x7f077c001660 comp rx=0 tx=0).stop 2024-09-11T00:32:41.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 -- 172.21.15.31:0/160930246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078c153850 msgr2=0x7f078c14e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:41.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078c153850 0x7f078c14e220 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f078400c700 tx=0x7f078400cbc0 comp rx=0 tx=0).stop 2024-09-11T00:32:41.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 -- 172.21.15.31:0/160930246 shutdown_connections 2024-09-11T00:32:41.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0768077f70 0x7f076807a430 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:41.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f078c153850 0x7f078c14e220 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:41.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078c0c6420 0x7f078c14dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:41.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 --2- 172.21.15.31:0/160930246 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f078c0c5aa0 0x7f078c154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:41.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 -- 172.21.15.31:0/160930246 >> 172.21.15.31:0/160930246 conn(0x7f078c06caf0 msgr2=0x7f078c14fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:41.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 -- 172.21.15.31:0/160930246 shutdown_connections 2024-09-11T00:32:41.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:41.205+0000 7f0791200640 1 -- 172.21.15.31:0/160930246 wait complete. 2024-09-11T00:32:41.209 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gettimeofday... yes 2024-09-11T00:32:41.223 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:32:41.364 INFO:tasks.workunit.client.1.smithi204.stdout:checking for mkdir... yes 2024-09-11T00:32:41.967 INFO:tasks.workunit.client.1.smithi204.stdout:checking for strerror... yes 2024-09-11T00:32:42.122 INFO:tasks.workunit.client.1.smithi204.stdout:checking for open64... yes 2024-09-11T00:32:42.548 INFO:tasks.workunit.client.1.smithi204.stdout:checking for stat64... yes 2024-09-11T00:32:42.698 INFO:tasks.workunit.client.1.smithi204.stdout:checking for fseeko64... yes 2024-09-11T00:32:42.849 INFO:tasks.workunit.client.1.smithi204.stdout:checking for lrand48_r... yes 2024-09-11T00:32:42.999 INFO:tasks.workunit.client.1.smithi204.stdout:checking for srand48_r... yes 2024-09-11T00:32:43.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:43.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.398+0000 7f6dce000640 1 -- 172.21.15.31:0/2764620134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dc8072480 msgr2=0x7f6dc81511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:43.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.398+0000 7f6dce000640 1 --2- 172.21.15.31:0/2764620134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dc8072480 0x7f6dc81511a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80c5500 tx=0x7f6db802a680 comp rx=0 tx=0).stop 2024-09-11T00:32:43.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.398+0000 7f6dce000640 1 -- 172.21.15.31:0/2764620134 shutdown_connections 2024-09-11T00:32:43.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.398+0000 7f6dce000640 1 --2- 172.21.15.31:0/2764620134 >> v2:172.21.15.204:3300/0 conn(0x7f6dc8153f20 0x7f6dc81516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:43.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.398+0000 7f6dce000640 1 --2- 172.21.15.31:0/2764620134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dc8072480 0x7f6dc81511a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:43.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.398+0000 7f6dce000640 1 -- 172.21.15.31:0/2764620134 >> 172.21.15.31:0/2764620134 conn(0x7f6dc80a9750 msgr2=0x7f6dc80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:43.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.398+0000 7f6dce000640 1 -- 172.21.15.31:0/2764620134 shutdown_connections 2024-09-11T00:32:43.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.398+0000 7f6dce000640 1 -- 172.21.15.31:0/2764620134 wait complete. 2024-09-11T00:32:43.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.400+0000 7f6dce000640 1 Processor -- start 2024-09-11T00:32:43.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.400+0000 7f6dce000640 1 -- start start 2024-09-11T00:32:43.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dce000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dc8072480 0x7f6dc80c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:43.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dce000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dc80c23a0 0x7f6dc80c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dce000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dc8153f20 0x7f6dc81ea3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dce000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc8074200 con 0x7f6dc80c23a0 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dce000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc8074080 con 0x7f6dc8072480 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dce000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc8072860 con 0x7f6dc8153f20 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dbea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dc80c23a0 0x7f6dc80c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dbea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dc80c23a0 0x7f6dc80c67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32884/0 (socket says 172.21.15.31:32884) 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dbea00640 1 -- 172.21.15.31:0/3359338744 learned_addr learned my addr 172.21.15.31:0/3359338744 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dc7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dc8072480 0x7f6dc80c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dc7e00640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dc8153f20 0x7f6dc81ea3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dbea00640 1 -- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dc8153f20 msgr2=0x7f6dc81ea3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dbea00640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dc8153f20 0x7f6dc81ea3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:43.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dbea00640 1 -- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dc8072480 msgr2=0x7f6dc80c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:43.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dbea00640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dc8072480 0x7f6dc80c3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:43.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.401+0000 7f6dbea00640 1 -- 172.21.15.31:0/3359338744 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc8143e40 con 0x7f6dc80c23a0 2024-09-11T00:32:43.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.402+0000 7f6dbea00640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dc80c23a0 0x7f6dc80c67f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6db000c420 tx=0x7f6db000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:43.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.402+0000 7f6dbfe00640 1 -- 172.21.15.31:0/3359338744 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db0017070 con 0x7f6dc80c23a0 2024-09-11T00:32:43.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.402+0000 7f6dbfe00640 1 -- 172.21.15.31:0/3359338744 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6db00053c0 con 0x7f6dc80c23a0 2024-09-11T00:32:43.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.402+0000 7f6dbfe00640 1 -- 172.21.15.31:0/3359338744 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db0003850 con 0x7f6dc80c23a0 2024-09-11T00:32:43.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.402+0000 7f6dce000640 1 -- 172.21.15.31:0/3359338744 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc80c4df0 con 0x7f6dc80c23a0 2024-09-11T00:32:43.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.402+0000 7f6dce000640 1 -- 172.21.15.31:0/3359338744 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc80c7560 con 0x7f6dc80c23a0 2024-09-11T00:32:43.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.403+0000 7f6dce000640 1 -- 172.21.15.31:0/3359338744 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc81511a0 con 0x7f6dc80c23a0 2024-09-11T00:32:43.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.404+0000 7f6dbfe00640 1 -- 172.21.15.31:0/3359338744 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6db00039f0 con 0x7f6dc80c23a0 2024-09-11T00:32:43.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.405+0000 7f6dbfe00640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6da4077f70 0x7f6da407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:43.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.406+0000 7f6dbfe00640 1 -- 172.21.15.31:0/3359338744 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f6db00997f0 con 0x7f6dc80c23a0 2024-09-11T00:32:43.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.406+0000 7f6dc7400640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6da4077f70 0x7f6da407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:43.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.407+0000 7f6dc7400640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6da4077f70 0x7f6da407a430 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6db8033000 tx=0x7f6db8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:43.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.410+0000 7f6dbfe00640 1 -- 172.21.15.31:0/3359338744 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6dc80c4df0 con 0x7f6dc80c23a0 2024-09-11T00:32:43.436 INFO:tasks.workunit.client.1.smithi204.stdout:configure: creating ./config.status 2024-09-11T00:32:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.643+0000 7f6dce000640 1 -- 172.21.15.31:0/3359338744 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dc81e00a0 con 0x7f6dc80c23a0 2024-09-11T00:32:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.644+0000 7f6dbfe00640 1 -- 172.21.15.31:0/3359338744 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6534 (secure 0 0 0) 0x7f6db0061820 con 0x7f6dc80c23a0 2024-09-11T00:32:43.646 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:43.646 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"btime":"2024-09-11T00:32:34:318805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5076,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:43.646 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-09-11T00:32:43.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.646+0000 7f6dce000640 1 -- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6da4077f70 msgr2=0x7f6da407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.646+0000 7f6dce000640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6da4077f70 0x7f6da407a430 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6db8033000 tx=0x7f6db8004730 comp rx=0 tx=0).stop 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.646+0000 7f6dce000640 1 -- 172.21.15.31:0/3359338744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dc80c23a0 msgr2=0x7f6dc80c67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.646+0000 7f6dce000640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dc80c23a0 0x7f6dc80c67f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6db000c420 tx=0x7f6db000c8e0 comp rx=0 tx=0).stop 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.646+0000 7f6dce000640 1 -- 172.21.15.31:0/3359338744 shutdown_connections 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.646+0000 7f6dce000640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6da4077f70 0x7f6da407a430 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.646+0000 7f6dce000640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6dc8153f20 0x7f6dc81ea3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.646+0000 7f6dce000640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dc80c23a0 0x7f6dc80c67f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.646+0000 7f6dce000640 1 --2- 172.21.15.31:0/3359338744 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dc8072480 0x7f6dc80c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.646+0000 7f6dce000640 1 -- 172.21.15.31:0/3359338744 >> 172.21.15.31:0/3359338744 conn(0x7f6dc80a9750 msgr2=0x7f6dc814fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.647+0000 7f6dce000640 1 -- 172.21.15.31:0/3359338744 shutdown_connections 2024-09-11T00:32:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:43.647+0000 7f6dce000640 1 -- 172.21.15.31:0/3359338744 wait complete. 2024-09-11T00:32:43.665 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:32:45.316 INFO:tasks.workunit.client.1.smithi204.stdout:config.status: creating Makefile 2024-09-11T00:32:45.357 INFO:tasks.workunit.client.1.smithi204.stdout:config.status: creating config.h 2024-09-11T00:32:45.407 INFO:tasks.workunit.client.1.smithi204.stdout:config.status: executing depfiles commands 2024-09-11T00:32:45.543 INFO:tasks.workunit.client.1.smithi204.stdout:config.status: executing default commands 2024-09-11T00:32:45.632 INFO:tasks.workunit.client.1.smithi204.stderr:+ make 2024-09-11T00:32:45.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:45.679 INFO:tasks.workunit.client.1.smithi204.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run aclocal-1.10 2024-09-11T00:32:45.694 INFO:tasks.workunit.client.1.smithi204.stderr:/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-09-11T00:32:45.697 INFO:tasks.workunit.client.1.smithi204.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-09-11T00:32:45.697 INFO:tasks.workunit.client.1.smithi204.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-09-11T00:32:45.697 INFO:tasks.workunit.client.1.smithi204.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-09-11T00:32:45.697 INFO:tasks.workunit.client.1.smithi204.stderr: any GNU archive site. 2024-09-11T00:32:45.708 INFO:tasks.workunit.client.1.smithi204.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run automake-1.10 --gnu 2024-09-11T00:32:45.718 INFO:tasks.workunit.client.1.smithi204.stderr:/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-09-11T00:32:45.720 INFO:tasks.workunit.client.1.smithi204.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-09-11T00:32:45.720 INFO:tasks.workunit.client.1.smithi204.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-09-11T00:32:45.720 INFO:tasks.workunit.client.1.smithi204.stderr: You might want to install the `Automake' and `Perl' packages. 2024-09-11T00:32:45.720 INFO:tasks.workunit.client.1.smithi204.stderr: Grab them from any GNU archive site. 2024-09-11T00:32:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.836+0000 7f8c6fe00640 1 -- 172.21.15.31:0/1130251112 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c700a8bb0 msgr2=0x7f8c700bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.836+0000 7f8c6fe00640 1 --2- 172.21.15.31:0/1130251112 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c700a8bb0 0x7f8c700bf600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009860 tx=0x7f8c5c02a510 comp rx=0 tx=0).stop 2024-09-11T00:32:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.836+0000 7f8c6fe00640 1 -- 172.21.15.31:0/1130251112 shutdown_connections 2024-09-11T00:32:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.836+0000 7f8c6fe00640 1 --2- 172.21.15.31:0/1130251112 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c700a8bb0 0x7f8c700bf600 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.836+0000 7f8c6fe00640 1 -- 172.21.15.31:0/1130251112 >> 172.21.15.31:0/1130251112 conn(0x7f8c700c9af0 msgr2=0x7f8c700c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.836+0000 7f8c6fe00640 1 -- 172.21.15.31:0/1130251112 shutdown_connections 2024-09-11T00:32:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.836+0000 7f8c6fe00640 1 -- 172.21.15.31:0/1130251112 wait complete. 2024-09-11T00:32:45.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.837+0000 7f8c6fe00640 1 Processor -- start 2024-09-11T00:32:45.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.838+0000 7f8c6fe00640 1 -- start start 2024-09-11T00:32:45.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.838+0000 7f8c6fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c700a8bb0 0x7f8c7014caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:45.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.838+0000 7f8c6fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c700c67f0 0x7f8c70155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:45.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.838+0000 7f8c6fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c701578e0 0x7f8c700a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:45.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.838+0000 7f8c6fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8c700a9260 con 0x7f8c700c67f0 2024-09-11T00:32:45.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.838+0000 7f8c6fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8c700a90e0 con 0x7f8c700a8bb0 2024-09-11T00:32:45.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.838+0000 7f8c6fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8c700c5b20 con 0x7f8c701578e0 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.839+0000 7f8c6e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c700c67f0 0x7f8c70155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.839+0000 7f8c6e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c700c67f0 0x7f8c70155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32890/0 (socket says 172.21.15.31:32890) 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.839+0000 7f8c6e000640 1 -- 172.21.15.31:0/285280193 learned_addr learned my addr 172.21.15.31:0/285280193 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.839+0000 7f8c6ea00640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c700a8bb0 0x7f8c7014caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.839+0000 7f8c6f400640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c701578e0 0x7f8c700a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.839+0000 7f8c6e000640 1 -- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c701578e0 msgr2=0x7f8c700a9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.839+0000 7f8c6e000640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c701578e0 0x7f8c700a9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.839+0000 7f8c6e000640 1 -- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c700a8bb0 msgr2=0x7f8c7014caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.839+0000 7f8c6e000640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c700a8bb0 0x7f8c7014caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.839+0000 7f8c6e000640 1 -- 172.21.15.31:0/285280193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7014d030 con 0x7f8c700c67f0 2024-09-11T00:32:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.840+0000 7f8c6e000640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c700c67f0 0x7f8c70155460 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c02ab00 tx=0x7f8c5c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.840+0000 7f8c66a00640 1 -- 172.21.15.31:0/285280193 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c040070 con 0x7f8c700c67f0 2024-09-11T00:32:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.840+0000 7f8c66a00640 1 -- 172.21.15.31:0/285280193 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c5c005310 con 0x7f8c700c67f0 2024-09-11T00:32:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.840+0000 7f8c66a00640 1 -- 172.21.15.31:0/285280193 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c005c90 con 0x7f8c700c67f0 2024-09-11T00:32:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.840+0000 7f8c6fe00640 1 -- 172.21.15.31:0/285280193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c701eccb0 con 0x7f8c700c67f0 2024-09-11T00:32:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.840+0000 7f8c6fe00640 1 -- 172.21.15.31:0/285280193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c700aae30 con 0x7f8c700c67f0 2024-09-11T00:32:45.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.841+0000 7f8c6fe00640 1 -- 172.21.15.31:0/285280193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c700ab030 con 0x7f8c700c67f0 2024-09-11T00:32:45.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.842+0000 7f8c66a00640 1 -- 172.21.15.31:0/285280193 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8c5c0049b0 con 0x7f8c700c67f0 2024-09-11T00:32:45.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.844+0000 7f8c66a00640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8c50077f70 0x7f8c5007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:45.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.844+0000 7f8c66a00640 1 -- 172.21.15.31:0/285280193 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f8c5c0bb530 con 0x7f8c700c67f0 2024-09-11T00:32:45.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.844+0000 7f8c6ea00640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8c50077f70 0x7f8c5007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:45.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.845+0000 7f8c6ea00640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8c50077f70 0x7f8c5007a430 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8c60001f00 tx=0x7f8c60009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:45.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:45.849+0000 7f8c66a00640 1 -- 172.21.15.31:0/285280193 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8c700ab030 con 0x7f8c700c67f0 2024-09-11T00:32:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.092+0000 7f8c6fe00640 1 -- 172.21.15.31:0/285280193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c700c6c20 con 0x7f8c700c67f0 2024-09-11T00:32:46.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.092+0000 7f8c66a00640 1 -- 172.21.15.31:0/285280193 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6534 (secure 0 0 0) 0x7f8c5c083560 con 0x7f8c700c67f0 2024-09-11T00:32:46.094 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:46.095 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"btime":"2024-09-11T00:32:34:318805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5076,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:46.095 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-09-11T00:32:46.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 -- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8c50077f70 msgr2=0x7f8c5007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:46.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8c50077f70 0x7f8c5007a430 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8c60001f00 tx=0x7f8c60009040 comp rx=0 tx=0).stop 2024-09-11T00:32:46.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 -- 172.21.15.31:0/285280193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c700c67f0 msgr2=0x7f8c70155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:46.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c700c67f0 0x7f8c70155460 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c02ab00 tx=0x7f8c5c032c10 comp rx=0 tx=0).stop 2024-09-11T00:32:46.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 -- 172.21.15.31:0/285280193 shutdown_connections 2024-09-11T00:32:46.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8c50077f70 0x7f8c5007a430 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8c701578e0 0x7f8c700a9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c700c67f0 0x7f8c70155460 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 --2- 172.21.15.31:0/285280193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c700a8bb0 0x7f8c7014caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 -- 172.21.15.31:0/285280193 >> 172.21.15.31:0/285280193 conn(0x7f8c700c9af0 msgr2=0x7f8c700720f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 -- 172.21.15.31:0/285280193 shutdown_connections 2024-09-11T00:32:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:46.095+0000 7f8c6fe00640 1 -- 172.21.15.31:0/285280193 wait complete. 2024-09-11T00:32:46.113 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:32:47.261 INFO:tasks.workunit.client.1.smithi204.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run autoconf 2024-09-11T00:32:47.800 INFO:tasks.workunit.client.1.smithi204.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-11T00:32:47.800 INFO:tasks.workunit.client.1.smithi204.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-11T00:32:47.800 INFO:tasks.workunit.client.1.smithi204.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-11T00:32:47.800 INFO:tasks.workunit.client.1.smithi204.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-11T00:32:47.950 INFO:tasks.workunit.client.1.smithi204.stdout:/bin/sh ./config.status --recheck 2024-09-11T00:32:48.084 INFO:tasks.workunit.client.1.smithi204.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-09-11T00:32:48.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:48.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.285+0000 7f03ebe00640 1 -- 172.21.15.31:0/3154119660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03cc004020 msgr2=0x7f03cc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:48.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.285+0000 7f03ebe00640 1 --2- 172.21.15.31:0/3154119660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03cc004020 0x7f03cc004800 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f03e4154fd0 tx=0x7f03cc005830 comp rx=0 tx=0).stop 2024-09-11T00:32:48.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.286+0000 7f03e9e00640 1 -- 172.21.15.31:0/3154119660 reap_dead start 2024-09-11T00:32:48.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.286+0000 7f03ebe00640 1 -- 172.21.15.31:0/3154119660 shutdown_connections 2024-09-11T00:32:48.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.286+0000 7f03ebe00640 1 -- 172.21.15.31:0/3154119660 >> 172.21.15.31:0/3154119660 conn(0x7f03e40a9750 msgr2=0x7f03e40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:48.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.286+0000 7f03ebe00640 1 -- 172.21.15.31:0/3154119660 shutdown_connections 2024-09-11T00:32:48.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.286+0000 7f03ebe00640 1 -- 172.21.15.31:0/3154119660 wait complete. 2024-09-11T00:32:48.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.287+0000 7f03ebe00640 1 Processor -- start 2024-09-11T00:32:48.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.287+0000 7f03ebe00640 1 -- start start 2024-09-11T00:32:48.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.288+0000 7f03ebe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03cc004020 0x7f03e40c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:48.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.288+0000 7f03ebe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f03e41dfb90 0x7f03e4149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:48.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.288+0000 7f03ebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03e414a1b0 0x7f03e41ebaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:48.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.288+0000 7f03ebe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f03e4074200 con 0x7f03e414a1b0 2024-09-11T00:32:48.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.288+0000 7f03ebe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f03e4074080 con 0x7f03cc004020 2024-09-11T00:32:48.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.288+0000 7f03ebe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f03e4072860 con 0x7f03e41dfb90 2024-09-11T00:32:48.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03e414a1b0 0x7f03e41ebaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03e414a1b0 0x7f03e41ebaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50782/0 (socket says 172.21.15.31:50782) 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e9e00640 1 -- 172.21.15.31:0/912203844 learned_addr learned my addr 172.21.15.31:0/912203844 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e9400640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03cc004020 0x7f03e40c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e8a00640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f03e41dfb90 0x7f03e4149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e9e00640 1 -- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f03e41dfb90 msgr2=0x7f03e4149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e9e00640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f03e41dfb90 0x7f03e4149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e9e00640 1 -- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03cc004020 msgr2=0x7f03e40c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e9e00640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03cc004020 0x7f03e40c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e9e00640 1 -- 172.21.15.31:0/912203844 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e414a660 con 0x7f03e414a1b0 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e9e00640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03e414a1b0 0x7f03e41ebaf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f03e4073200 tx=0x7f03dc0099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.289+0000 7f03e1600640 1 -- 172.21.15.31:0/912203844 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc004d20 con 0x7f03e414a1b0 2024-09-11T00:32:48.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.290+0000 7f03e1600640 1 -- 172.21.15.31:0/912203844 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03dc011710 con 0x7f03e414a1b0 2024-09-11T00:32:48.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.290+0000 7f03ebe00640 1 -- 172.21.15.31:0/912203844 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e41ddb40 con 0x7f03e414a1b0 2024-09-11T00:32:48.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.290+0000 7f03e1600640 1 -- 172.21.15.31:0/912203844 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc011b60 con 0x7f03e414a1b0 2024-09-11T00:32:48.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.290+0000 7f03ebe00640 1 -- 172.21.15.31:0/912203844 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e4153dc0 con 0x7f03e414a1b0 2024-09-11T00:32:48.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.291+0000 7f03ebe00640 1 -- 172.21.15.31:0/912203844 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e4154000 con 0x7f03e414a1b0 2024-09-11T00:32:48.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.292+0000 7f03e1600640 1 -- 172.21.15.31:0/912203844 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f03dc004fb0 con 0x7f03e414a1b0 2024-09-11T00:32:48.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.293+0000 7f03e1600640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f03c4077fa0 0x7f03c407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:48.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.293+0000 7f03e1600640 1 -- 172.21.15.31:0/912203844 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f03dc09abc0 con 0x7f03e414a1b0 2024-09-11T00:32:48.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.293+0000 7f03e9400640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f03c4077fa0 0x7f03c407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:48.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.294+0000 7f03e9400640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f03c4077fa0 0x7f03c407a460 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f03d8001400 tx=0x7f03d8006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:48.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.297+0000 7f03e1600640 1 -- 172.21.15.31:0/912203844 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f03e4154000 con 0x7f03e414a1b0 2024-09-11T00:32:48.353 INFO:tasks.workunit.client.1.smithi204.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-11T00:32:48.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.535+0000 7f03ebe00640 1 -- 172.21.15.31:0/912203844 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03e40c3d00 con 0x7f03e414a1b0 2024-09-11T00:32:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.535+0000 7f03e1600640 1 -- 172.21.15.31:0/912203844 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6534 (secure 0 0 0) 0x7f03dc015030 con 0x7f03e414a1b0 2024-09-11T00:32:48.537 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:48.538 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"btime":"2024-09-11T00:32:34:318805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5076,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-09-11T00:32:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 -- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f03c4077fa0 msgr2=0x7f03c407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f03c4077fa0 0x7f03c407a460 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f03d8001400 tx=0x7f03d8006040 comp rx=0 tx=0).stop 2024-09-11T00:32:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 -- 172.21.15.31:0/912203844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03e414a1b0 msgr2=0x7f03e41ebaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03e414a1b0 0x7f03e41ebaf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f03e4073200 tx=0x7f03dc0099c0 comp rx=0 tx=0).stop 2024-09-11T00:32:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 -- 172.21.15.31:0/912203844 shutdown_connections 2024-09-11T00:32:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f03c4077fa0 0x7f03c407a460 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f03e414a1b0 0x7f03e41ebaf0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f03e41dfb90 0x7f03e4149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 --2- 172.21.15.31:0/912203844 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f03cc004020 0x7f03e40c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:48.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 -- 172.21.15.31:0/912203844 >> 172.21.15.31:0/912203844 conn(0x7f03e40a9750 msgr2=0x7f03e40a8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:48.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 -- 172.21.15.31:0/912203844 shutdown_connections 2024-09-11T00:32:48.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:48.538+0000 7f03ebe00640 1 -- 172.21.15.31:0/912203844 wait complete. 2024-09-11T00:32:48.556 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:32:49.369 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether build environment is sane... yes 2024-09-11T00:32:49.390 INFO:tasks.workunit.client.1.smithi204.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-11T00:32:49.391 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gawk... gawk 2024-09-11T00:32:49.423 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether make sets $(MAKE)... yes 2024-09-11T00:32:49.517 INFO:tasks.workunit.client.1.smithi204.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-11T00:32:49.518 INFO:tasks.workunit.client.1.smithi204.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-11T00:32:49.543 INFO:tasks.workunit.client.1.smithi204.stdout:checking for style of include used by make... GNU 2024-09-11T00:32:49.551 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gcc... gcc 2024-09-11T00:32:49.739 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether the C compiler works... yes 2024-09-11T00:32:49.740 INFO:tasks.workunit.client.1.smithi204.stdout:checking for C compiler default output file name... a.out 2024-09-11T00:32:49.828 INFO:tasks.workunit.client.1.smithi204.stdout:checking for suffix of executables... 2024-09-11T00:32:49.920 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether we are cross compiling... no 2024-09-11T00:32:49.993 INFO:tasks.workunit.client.1.smithi204.stdout:checking for suffix of object files... o 2024-09-11T00:32:50.048 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether we are using the GNU C compiler... yes 2024-09-11T00:32:50.104 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether gcc accepts -g... yes 2024-09-11T00:32:50.174 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-11T00:32:50.403 INFO:tasks.workunit.client.1.smithi204.stdout:checking dependency style of gcc... gcc3 2024-09-11T00:32:50.500 INFO:tasks.workunit.client.1.smithi204.stdout:checking how to run the C preprocessor... gcc -E 2024-09-11T00:32:50.557 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:50.601 INFO:tasks.workunit.client.1.smithi204.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-11T00:32:50.605 INFO:tasks.workunit.client.1.smithi204.stdout:checking for egrep... /usr/bin/grep -E 2024-09-11T00:32:50.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.728+0000 7f40f5400640 1 -- 172.21.15.31:0/2060887422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40f00c67f0 msgr2=0x7f40f00bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:50.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.728+0000 7f40f5400640 1 --2- 172.21.15.31:0/2060887422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40f00c67f0 0x7f40f00bf600 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f40e0005fd0 tx=0x7f40e002a5f0 comp rx=0 tx=0).stop 2024-09-11T00:32:50.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.728+0000 7f40f5400640 1 -- 172.21.15.31:0/2060887422 shutdown_connections 2024-09-11T00:32:50.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.728+0000 7f40f5400640 1 --2- 172.21.15.31:0/2060887422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40f00c67f0 0x7f40f00bf600 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:50.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.728+0000 7f40f5400640 1 -- 172.21.15.31:0/2060887422 >> 172.21.15.31:0/2060887422 conn(0x7f40f00c9af0 msgr2=0x7f40f00c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:50.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.729+0000 7f40f5400640 1 -- 172.21.15.31:0/2060887422 shutdown_connections 2024-09-11T00:32:50.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.729+0000 7f40f5400640 1 -- 172.21.15.31:0/2060887422 wait complete. 2024-09-11T00:32:50.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.730+0000 7f40f5400640 1 Processor -- start 2024-09-11T00:32:50.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.730+0000 7f40f5400640 1 -- start start 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.731+0000 7f40f5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40f00a8bb0 0x7f40f01ebef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.731+0000 7f40f5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40f00c67f0 0x7f40f01def20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.731+0000 7f40f5400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40f01578e0 0x7f40f00bff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.731+0000 7f40f5400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f40f00a9260 con 0x7f40f00a8bb0 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.731+0000 7f40f5400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f40f00a90e0 con 0x7f40f00c67f0 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.731+0000 7f40f5400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f40f00c5b20 con 0x7f40f01578e0 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.731+0000 7f40efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40f00a8bb0 0x7f40f01ebef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.731+0000 7f40efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40f00a8bb0 0x7f40f01ebef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50790/0 (socket says 172.21.15.31:50790) 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.731+0000 7f40efe00640 1 -- 172.21.15.31:0/4288075517 learned_addr learned my addr 172.21.15.31:0/4288075517 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40e7400640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40f00c67f0 0x7f40f01def20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:50.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40f4a00640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40f01578e0 0x7f40f00bff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:50.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40efe00640 1 -- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40f01578e0 msgr2=0x7f40f00bff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:50.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40efe00640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40f01578e0 0x7f40f00bff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:50.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40efe00640 1 -- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40f00c67f0 msgr2=0x7f40f01def20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:50.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40efe00640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40f00c67f0 0x7f40f01def20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:50.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40efe00640 1 -- 172.21.15.31:0/4288075517 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f00a8630 con 0x7f40f00a8bb0 2024-09-11T00:32:50.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40efe00640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40f00a8bb0 0x7f40f01ebef0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f40e0009850 tx=0x7f40e00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:50.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40ecc00640 1 -- 172.21.15.31:0/4288075517 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e003f070 con 0x7f40f00a8bb0 2024-09-11T00:32:50.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40ecc00640 1 -- 172.21.15.31:0/4288075517 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40e0004890 con 0x7f40f00a8bb0 2024-09-11T00:32:50.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.732+0000 7f40ecc00640 1 -- 172.21.15.31:0/4288075517 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e0035030 con 0x7f40f00a8bb0 2024-09-11T00:32:50.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.733+0000 7f40f5400640 1 -- 172.21.15.31:0/4288075517 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f00a88f0 con 0x7f40f00a8bb0 2024-09-11T00:32:50.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.733+0000 7f40f5400640 1 -- 172.21.15.31:0/4288075517 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f01e0640 con 0x7f40f00a8bb0 2024-09-11T00:32:50.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.734+0000 7f40f5400640 1 -- 172.21.15.31:0/4288075517 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f00c6bd0 con 0x7f40f00a8bb0 2024-09-11T00:32:50.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.735+0000 7f40ecc00640 1 -- 172.21.15.31:0/4288075517 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f40e0004c60 con 0x7f40f00a8bb0 2024-09-11T00:32:50.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.736+0000 7f40ecc00640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40cc077f70 0x7f40cc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:50.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.736+0000 7f40ecc00640 1 -- 172.21.15.31:0/4288075517 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f40e00bb2c0 con 0x7f40f00a8bb0 2024-09-11T00:32:50.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.736+0000 7f40e7400640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40cc077f70 0x7f40cc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:50.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.737+0000 7f40e7400640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40cc077f70 0x7f40cc07a430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f40d80017c0 tx=0x7f40d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:50.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.740+0000 7f40ecc00640 1 -- 172.21.15.31:0/4288075517 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f40e0083390 con 0x7f40f00a8bb0 2024-09-11T00:32:50.871 INFO:tasks.workunit.client.1.smithi204.stdout:checking for ANSI C header files... yes 2024-09-11T00:32:50.945 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/types.h... yes 2024-09-11T00:32:50.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.975+0000 7f40f5400640 1 -- 172.21.15.31:0/4288075517 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40f00c7a60 con 0x7f40f00a8bb0 2024-09-11T00:32:50.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.975+0000 7f40ecc00640 1 -- 172.21.15.31:0/4288075517 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6534 (secure 0 0 0) 0x7f40e0087dc0 con 0x7f40f00a8bb0 2024-09-11T00:32:50.977 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:50.978 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"btime":"2024-09-11T00:32:34:318805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5076,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:50.978 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-09-11T00:32:50.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 -- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40cc077f70 msgr2=0x7f40cc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:50.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40cc077f70 0x7f40cc07a430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f40d80017c0 tx=0x7f40d8009040 comp rx=0 tx=0).stop 2024-09-11T00:32:50.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 -- 172.21.15.31:0/4288075517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40f00a8bb0 msgr2=0x7f40f01ebef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:50.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40f00a8bb0 0x7f40f01ebef0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f40e0009850 tx=0x7f40e00039e0 comp rx=0 tx=0).stop 2024-09-11T00:32:50.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 -- 172.21.15.31:0/4288075517 shutdown_connections 2024-09-11T00:32:50.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40cc077f70 0x7f40cc07a430 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:50.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40f01578e0 0x7f40f00bff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:50.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40f00c67f0 0x7f40f01def20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:50.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 --2- 172.21.15.31:0/4288075517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40f00a8bb0 0x7f40f01ebef0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:50.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 -- 172.21.15.31:0/4288075517 >> 172.21.15.31:0/4288075517 conn(0x7f40f00c9af0 msgr2=0x7f40f0070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:50.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 -- 172.21.15.31:0/4288075517 shutdown_connections 2024-09-11T00:32:50.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:50.978+0000 7f40f5400640 1 -- 172.21.15.31:0/4288075517 wait complete. 2024-09-11T00:32:50.996 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:32:51.034 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/stat.h... yes 2024-09-11T00:32:51.125 INFO:tasks.workunit.client.1.smithi204.stdout:checking for stdlib.h... yes 2024-09-11T00:32:51.218 INFO:tasks.workunit.client.1.smithi204.stdout:checking for string.h... yes 2024-09-11T00:32:51.694 INFO:tasks.workunit.client.1.smithi204.stdout:checking for memory.h... yes 2024-09-11T00:32:51.784 INFO:tasks.workunit.client.1.smithi204.stdout:checking for strings.h... yes 2024-09-11T00:32:51.875 INFO:tasks.workunit.client.1.smithi204.stdout:checking for inttypes.h... yes 2024-09-11T00:32:51.968 INFO:tasks.workunit.client.1.smithi204.stdout:checking for stdint.h... yes 2024-09-11T00:32:52.061 INFO:tasks.workunit.client.1.smithi204.stdout:checking for unistd.h... yes 2024-09-11T00:32:52.158 INFO:tasks.workunit.client.1.smithi204.stdout:checking minix/config.h usability... no 2024-09-11T00:32:52.221 INFO:tasks.workunit.client.1.smithi204.stdout:checking minix/config.h presence... no 2024-09-11T00:32:52.222 INFO:tasks.workunit.client.1.smithi204.stdout:checking for minix/config.h... no 2024-09-11T00:32:52.294 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-09-11T00:32:52.302 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gcc... (cached) gcc 2024-09-11T00:32:52.392 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-11T00:32:52.393 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-11T00:32:52.393 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-11T00:32:52.394 INFO:tasks.workunit.client.1.smithi204.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-11T00:32:52.499 INFO:tasks.workunit.client.1.smithi204.stdout:checking for main in -lm... yes 2024-09-11T00:32:52.613 INFO:tasks.workunit.client.1.smithi204.stdout:checking for main in -lpthread... yes 2024-09-11T00:32:52.618 INFO:tasks.workunit.client.1.smithi204.stdout:checking for ANSI C header files... (cached) yes 2024-09-11T00:32:52.691 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-11T00:32:52.784 INFO:tasks.workunit.client.1.smithi204.stdout:checking pthread.h usability... yes 2024-09-11T00:32:52.824 INFO:tasks.workunit.client.1.smithi204.stdout:checking pthread.h presence... yes 2024-09-11T00:32:52.825 INFO:tasks.workunit.client.1.smithi204.stdout:checking for pthread.h... yes 2024-09-11T00:32:52.997 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.168+0000 7f6fde600640 1 -- 172.21.15.31:0/3989565303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fd8072480 msgr2=0x7f6fd81511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.168+0000 7f6fde600640 1 --2- 172.21.15.31:0/3989565303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fd8072480 0x7f6fd81511a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009830 tx=0x7f6fc802a5a0 comp rx=0 tx=0).stop 2024-09-11T00:32:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.168+0000 7f6fde600640 1 -- 172.21.15.31:0/3989565303 shutdown_connections 2024-09-11T00:32:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.168+0000 7f6fde600640 1 --2- 172.21.15.31:0/3989565303 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6fd8153f20 0x7f6fd81516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.168+0000 7f6fde600640 1 --2- 172.21.15.31:0/3989565303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fd8072480 0x7f6fd81511a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.168+0000 7f6fde600640 1 -- 172.21.15.31:0/3989565303 >> 172.21.15.31:0/3989565303 conn(0x7f6fd80a9750 msgr2=0x7f6fd80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.168+0000 7f6fde600640 1 -- 172.21.15.31:0/3989565303 shutdown_connections 2024-09-11T00:32:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.168+0000 7f6fde600640 1 -- 172.21.15.31:0/3989565303 wait complete. 2024-09-11T00:32:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.170+0000 7f6fde600640 1 Processor -- start 2024-09-11T00:32:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.170+0000 7f6fde600640 1 -- start start 2024-09-11T00:32:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fde600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fd8072480 0x7f6fd80c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fde600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fd80c23a0 0x7f6fd81e2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fde600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6fd8153f20 0x7f6fd80c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fde600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8074200 con 0x7f6fd80c23a0 2024-09-11T00:32:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fde600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8074080 con 0x7f6fd8072480 2024-09-11T00:32:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fde600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6fd8072860 con 0x7f6fd8153f20 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fcea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fd80c23a0 0x7f6fd81e2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fcea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fd80c23a0 0x7f6fd81e2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50798/0 (socket says 172.21.15.31:50798) 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fcea00640 1 -- 172.21.15.31:0/1841167726 learned_addr learned my addr 172.21.15.31:0/1841167726 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fd7400640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fd8072480 0x7f6fd80c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fd7e00640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6fd8153f20 0x7f6fd80c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fcea00640 1 -- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6fd8153f20 msgr2=0x7f6fd80c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fcea00640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6fd8153f20 0x7f6fd80c4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fcea00640 1 -- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fd8072480 msgr2=0x7f6fd80c40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fcea00640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fd8072480 0x7f6fd80c40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fcea00640 1 -- 172.21.15.31:0/1841167726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd806fe10 con 0x7f6fd80c23a0 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.171+0000 7f6fd7400640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fd8072480 0x7f6fd80c40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:32:53.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.172+0000 7f6fcea00640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fd80c23a0 0x7f6fd81e2070 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000c430 tx=0x7f6fc000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:53.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.172+0000 7f6fcfe00640 1 -- 172.21.15.31:0/1841167726 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc00052d0 con 0x7f6fd80c23a0 2024-09-11T00:32:53.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.172+0000 7f6fcfe00640 1 -- 172.21.15.31:0/1841167726 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6fc0007020 con 0x7f6fd80c23a0 2024-09-11T00:32:53.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.172+0000 7f6fcfe00640 1 -- 172.21.15.31:0/1841167726 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0007470 con 0x7f6fd80c23a0 2024-09-11T00:32:53.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.172+0000 7f6fde600640 1 -- 172.21.15.31:0/1841167726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd81e25b0 con 0x7f6fd80c23a0 2024-09-11T00:32:53.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.172+0000 7f6fde600640 1 -- 172.21.15.31:0/1841167726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd8070080 con 0x7f6fd80c23a0 2024-09-11T00:32:53.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.173+0000 7f6fde600640 1 -- 172.21.15.31:0/1841167726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fd80c1a10 con 0x7f6fd80c23a0 2024-09-11T00:32:53.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.174+0000 7f6fcfe00640 1 -- 172.21.15.31:0/1841167726 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6fc00067d0 con 0x7f6fd80c23a0 2024-09-11T00:32:53.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.175+0000 7f6fcfe00640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6fb4077fa0 0x7f6fb407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:53.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.176+0000 7f6fcfe00640 1 -- 172.21.15.31:0/1841167726 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f6fc0099740 con 0x7f6fd80c23a0 2024-09-11T00:32:53.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.176+0000 7f6fd7400640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6fb4077fa0 0x7f6fb407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:53.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.177+0000 7f6fd7400640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6fb4077fa0 0x7f6fb407a460 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8034000 tx=0x7f6fc8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:53.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.180+0000 7f6fcfe00640 1 -- 172.21.15.31:0/1841167726 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6fd80c1a10 con 0x7f6fd80c23a0 2024-09-11T00:32:53.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.417+0000 7f6fde600640 1 -- 172.21.15.31:0/1841167726 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fd81511a0 con 0x7f6fd80c23a0 2024-09-11T00:32:53.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.417+0000 7f6fcfe00640 1 -- 172.21.15.31:0/1841167726 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6534 (secure 0 0 0) 0x7f6fc00616e0 con 0x7f6fd80c23a0 2024-09-11T00:32:53.419 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:53.419 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":25,"btime":"2024-09-11T00:32:34:318805+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5076,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:30:49.364654+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4390},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4390":{"gid":4390,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.204:6835/1801149506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1801149506},{"type":"v1","addr":"172.21.15.204:6835","nonce":1801149506}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4390,"qdb_cluster":[4390]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:53.420 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 25 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 -- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6fb4077fa0 msgr2=0x7f6fb407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6fb4077fa0 0x7f6fb407a460 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8034000 tx=0x7f6fc8004730 comp rx=0 tx=0).stop 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 -- 172.21.15.31:0/1841167726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fd80c23a0 msgr2=0x7f6fd81e2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fd80c23a0 0x7f6fd81e2070 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000c430 tx=0x7f6fc000c8f0 comp rx=0 tx=0).stop 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 -- 172.21.15.31:0/1841167726 shutdown_connections 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6fb4077fa0 0x7f6fb407a460 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6fd8153f20 0x7f6fd80c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fd80c23a0 0x7f6fd81e2070 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 --2- 172.21.15.31:0/1841167726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fd8072480 0x7f6fd80c40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 -- 172.21.15.31:0/1841167726 >> 172.21.15.31:0/1841167726 conn(0x7f6fd80a9750 msgr2=0x7f6fd80a8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:53.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 -- 172.21.15.31:0/1841167726 shutdown_connections 2024-09-11T00:32:53.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:53.420+0000 7f6fde600640 1 -- 172.21.15.31:0/1841167726 wait complete. 2024-09-11T00:32:53.438 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:32:54.009 INFO:tasks.workunit.client.1.smithi204.stdout:checking fcntl.h usability... yes 2024-09-11T00:32:54.046 INFO:tasks.workunit.client.1.smithi204.stdout:checking fcntl.h presence... yes 2024-09-11T00:32:54.047 INFO:tasks.workunit.client.1.smithi204.stdout:checking for fcntl.h... yes 2024-09-11T00:32:54.220 INFO:tasks.workunit.client.1.smithi204.stdout:checking limits.h usability... yes 2024-09-11T00:32:54.255 INFO:tasks.workunit.client.1.smithi204.stdout:checking limits.h presence... yes 2024-09-11T00:32:54.255 INFO:tasks.workunit.client.1.smithi204.stdout:checking for limits.h... yes 2024-09-11T00:32:54.288 INFO:tasks.workunit.client.1.smithi204.stdout:checking for stdint.h... (cached) yes 2024-09-11T00:32:54.387 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/time.h usability... yes 2024-09-11T00:32:54.427 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/time.h presence... yes 2024-09-11T00:32:54.428 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/time.h... yes 2024-09-11T00:32:54.454 INFO:tasks.workunit.client.1.smithi204.stdout:checking for unistd.h... (cached) yes 2024-09-11T00:32:54.624 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/vfs.h usability... yes 2024-09-11T00:32:54.659 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/vfs.h presence... yes 2024-09-11T00:32:54.659 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/vfs.h... yes 2024-09-11T00:32:54.767 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/limits.h usability... no 2024-09-11T00:32:54.828 INFO:tasks.workunit.client.1.smithi204.stdout:checking sys/limits.h presence... no 2024-09-11T00:32:54.828 INFO:tasks.workunit.client.1.smithi204.stdout:checking for sys/limits.h... no 2024-09-11T00:32:54.888 INFO:tasks.workunit.client.1.smithi204.stdout:checking for an ANSI C-conforming const... yes 2024-09-11T00:32:55.299 INFO:tasks.workunit.client.1.smithi204.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-11T00:32:55.437 INFO:tasks.workunit.client.1.smithi204.stdout:checking for system... yes 2024-09-11T00:32:55.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:55.568 INFO:tasks.workunit.client.1.smithi204.stdout:checking for gettimeofday... yes 2024-09-11T00:32:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.597+0000 7f8b0aa00640 1 -- 172.21.15.31:0/3919219598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b040c23a0 msgr2=0x7f8b041516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.597+0000 7f8b0aa00640 1 --2- 172.21.15.31:0/3919219598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b040c23a0 0x7f8b041516e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8b040c4400 tx=0x7f8aec02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:32:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.597+0000 7f8b0aa00640 1 -- 172.21.15.31:0/3919219598 shutdown_connections 2024-09-11T00:32:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.597+0000 7f8b0aa00640 1 --2- 172.21.15.31:0/3919219598 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b04153f20 0x7f8b040c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.597+0000 7f8b0aa00640 1 --2- 172.21.15.31:0/3919219598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b040c23a0 0x7f8b041516e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.597+0000 7f8b0aa00640 1 --2- 172.21.15.31:0/3919219598 >> v2:172.21.15.204:3300/0 conn(0x7f8b04072480 0x7f8b041511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.597+0000 7f8b0aa00640 1 -- 172.21.15.31:0/3919219598 >> 172.21.15.31:0/3919219598 conn(0x7f8b040a9750 msgr2=0x7f8b040a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.597+0000 7f8b0aa00640 1 -- 172.21.15.31:0/3919219598 shutdown_connections 2024-09-11T00:32:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.597+0000 7f8b0aa00640 1 -- 172.21.15.31:0/3919219598 wait complete. 2024-09-11T00:32:55.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.599+0000 7f8b0aa00640 1 Processor -- start 2024-09-11T00:32:55.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.599+0000 7f8b0aa00640 1 -- start start 2024-09-11T00:32:55.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b0aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b04072480 0x7f8b040c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:55.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b0aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040c23a0 0x7f8b040c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:55.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b0aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b04153f20 0x7f8b041e0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:55.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b0aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8b04074200 con 0x7f8b04153f20 2024-09-11T00:32:55.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b0aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8b04074080 con 0x7f8b04072480 2024-09-11T00:32:55.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b0aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8b04072860 con 0x7f8b040c23a0 2024-09-11T00:32:55.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b08a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b04153f20 0x7f8b041e0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:55.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b08a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b04153f20 0x7f8b041e0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50816/0 (socket says 172.21.15.31:50816) 2024-09-11T00:32:55.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b03e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b04072480 0x7f8b040c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:55.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b08a00640 1 -- 172.21.15.31:0/2169373922 learned_addr learned my addr 172.21.15.31:0/2169373922 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:55.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8afb400640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040c23a0 0x7f8b040c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b08a00640 1 -- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040c23a0 msgr2=0x7f8b040c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b08a00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040c23a0 0x7f8b040c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b08a00640 1 -- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b04072480 msgr2=0x7f8b040c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b08a00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b04072480 0x7f8b040c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.600+0000 7f8b08a00640 1 -- 172.21.15.31:0/2169373922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b040c3a50 con 0x7f8b04153f20 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.601+0000 7f8b08a00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b04153f20 0x7f8b041e0a80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00eab0 tx=0x7f8afc00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.601+0000 7f8b00c00640 1 -- 172.21.15.31:0/2169373922 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc00cea0 con 0x7f8b04153f20 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.601+0000 7f8b00c00640 1 -- 172.21.15.31:0/2169373922 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8afc004b20 con 0x7f8b04153f20 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.601+0000 7f8b00c00640 1 -- 172.21.15.31:0/2169373922 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc005020 con 0x7f8b04153f20 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.601+0000 7f8b0aa00640 1 -- 172.21.15.31:0/2169373922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b040c3d30 con 0x7f8b04153f20 2024-09-11T00:32:55.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.601+0000 7f8b0aa00640 1 -- 172.21.15.31:0/2169373922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b041e28c0 con 0x7f8b04153f20 2024-09-11T00:32:55.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.602+0000 7f8b0aa00640 1 -- 172.21.15.31:0/2169373922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b040c3f70 con 0x7f8b04153f20 2024-09-11T00:32:55.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.603+0000 7f8b00c00640 1 -- 172.21.15.31:0/2169373922 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8afc0051c0 con 0x7f8b04153f20 2024-09-11T00:32:55.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.604+0000 7f8b00c00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ae0077f70 0x7f8ae007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:55.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.604+0000 7f8b00c00640 1 -- 172.21.15.31:0/2169373922 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f8afc09acd0 con 0x7f8b04153f20 2024-09-11T00:32:55.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.605+0000 7f8b03e00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ae0077f70 0x7f8ae007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:55.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.605+0000 7f8b03e00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ae0077f70 0x7f8ae007a430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8af4009bd0 tx=0x7f8af4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:55.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.609+0000 7f8b00c00640 1 -- 172.21.15.31:0/2169373922 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8b040c3f70 con 0x7f8b04153f20 2024-09-11T00:32:55.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.840+0000 7f8b0aa00640 1 -- 172.21.15.31:0/2169373922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b041511a0 con 0x7f8b04153f20 2024-09-11T00:32:55.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.840+0000 7f8b00c00640 1 -- 172.21.15.31:0/2169373922 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5705 (secure 0 0 0) 0x7f8afc062cb0 con 0x7f8b04153f20 2024-09-11T00:32:55.842 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:55.843 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"btime":"2024-09-11T00:32:55:293099+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:54.461539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:55.843 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-09-11T00:32:55.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 -- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ae0077f70 msgr2=0x7f8ae007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:55.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ae0077f70 0x7f8ae007a430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8af4009bd0 tx=0x7f8af4005f90 comp rx=0 tx=0).stop 2024-09-11T00:32:55.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 -- 172.21.15.31:0/2169373922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b04153f20 msgr2=0x7f8b041e0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:55.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b04153f20 0x7f8b041e0a80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00eab0 tx=0x7f8afc00ef70 comp rx=0 tx=0).stop 2024-09-11T00:32:55.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 -- 172.21.15.31:0/2169373922 shutdown_connections 2024-09-11T00:32:55.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ae0077f70 0x7f8ae007a430 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:55.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b04153f20 0x7f8b041e0a80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:55.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b040c23a0 0x7f8b040c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:55.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 --2- 172.21.15.31:0/2169373922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b04072480 0x7f8b040c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:55.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 -- 172.21.15.31:0/2169373922 >> 172.21.15.31:0/2169373922 conn(0x7f8b040a9750 msgr2=0x7f8b04152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:55.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 -- 172.21.15.31:0/2169373922 shutdown_connections 2024-09-11T00:32:55.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:55.843+0000 7f8b0aa00640 1 -- 172.21.15.31:0/2169373922 wait complete. 2024-09-11T00:32:55.860 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-11T00:32:55.861 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:32:55.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:56.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.018+0000 7f12e8a00640 1 -- 172.21.15.31:0/3506211824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e40c5aa0 msgr2=0x7f12e40c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:56.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.018+0000 7f12e8a00640 1 --2- 172.21.15.31:0/3506211824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e40c5aa0 0x7f12e40c7150 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f12e406df10 tx=0x7f12d402a5f0 comp rx=0 tx=0).stop 2024-09-11T00:32:56.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.018+0000 7f12e8a00640 1 -- 172.21.15.31:0/3506211824 shutdown_connections 2024-09-11T00:32:56.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.018+0000 7f12e8a00640 1 --2- 172.21.15.31:0/3506211824 >> v2:172.21.15.204:3300/0 conn(0x7f12e40c6420 0x7f12e4153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:56.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.018+0000 7f12e8a00640 1 --2- 172.21.15.31:0/3506211824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e40c5aa0 0x7f12e40c7150 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:56.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.018+0000 7f12e8a00640 1 -- 172.21.15.31:0/3506211824 >> 172.21.15.31:0/3506211824 conn(0x7f12e406caf0 msgr2=0x7f12e406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:56.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.019+0000 7f12e8a00640 1 -- 172.21.15.31:0/3506211824 shutdown_connections 2024-09-11T00:32:56.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.019+0000 7f12e8a00640 1 -- 172.21.15.31:0/3506211824 wait complete. 2024-09-11T00:32:56.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.020+0000 7f12e8a00640 1 Processor -- start 2024-09-11T00:32:56.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.020+0000 7f12e8a00640 1 -- start start 2024-09-11T00:32:56.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.021+0000 7f12e8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e40c5aa0 0x7f12e41e1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:56.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.021+0000 7f12e8a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12e40c6420 0x7f12e414ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.021+0000 7f12e8a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12e4153850 0x7f12e40aa7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.021+0000 7f12e8a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f12e40c2e80 con 0x7f12e40c5aa0 2024-09-11T00:32:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.021+0000 7f12e8a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f12e40c2ba0 con 0x7f12e40c6420 2024-09-11T00:32:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.021+0000 7f12e8a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f12e40c5e80 con 0x7f12e4153850 2024-09-11T00:32:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.021+0000 7f12e2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e40c5aa0 0x7f12e41e1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.021+0000 7f12e2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e40c5aa0 0x7f12e41e1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50826/0 (socket says 172.21.15.31:50826) 2024-09-11T00:32:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.021+0000 7f12e2800640 1 -- 172.21.15.31:0/4059369592 learned_addr learned my addr 172.21.15.31:0/4059369592 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.021+0000 7f12d9e00640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12e40c6420 0x7f12e414ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12e3200640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12e4153850 0x7f12e40aa7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:56.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12e2800640 1 -- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12e4153850 msgr2=0x7f12e40aa7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:56.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12e2800640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12e4153850 0x7f12e40aa7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:56.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12e2800640 1 -- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12e40c6420 msgr2=0x7f12e414ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:56.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12e2800640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12e40c6420 0x7f12e414ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:56.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12e2800640 1 -- 172.21.15.31:0/4059369592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e414d390 con 0x7f12e40c5aa0 2024-09-11T00:32:56.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12d9e00640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12e40c6420 0x7f12e414ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:32:56.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12e2800640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e40c5aa0 0x7f12e41e1fb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009850 tx=0x7f12d40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:56.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12db400640 1 -- 172.21.15.31:0/4059369592 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d403f070 con 0x7f12e40c5aa0 2024-09-11T00:32:56.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12db400640 1 -- 172.21.15.31:0/4059369592 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12d4004890 con 0x7f12e40c5aa0 2024-09-11T00:32:56.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12db400640 1 -- 172.21.15.31:0/4059369592 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4035030 con 0x7f12e40c5aa0 2024-09-11T00:32:56.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.022+0000 7f12e8a00640 1 -- 172.21.15.31:0/4059369592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e414cbb0 con 0x7f12e40c5aa0 2024-09-11T00:32:56.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.023+0000 7f12e8a00640 1 -- 172.21.15.31:0/4059369592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e40ac360 con 0x7f12e40c5aa0 2024-09-11T00:32:56.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.024+0000 7f12e8a00640 1 -- 172.21.15.31:0/4059369592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12e406dc60 con 0x7f12e40c5aa0 2024-09-11T00:32:56.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.025+0000 7f12db400640 1 -- 172.21.15.31:0/4059369592 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f12d4004ca0 con 0x7f12e40c5aa0 2024-09-11T00:32:56.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.025+0000 7f12db400640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12c0077f70 0x7f12c007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:56.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.026+0000 7f12db400640 1 -- 172.21.15.31:0/4059369592 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f12d40bb5c0 con 0x7f12e40c5aa0 2024-09-11T00:32:56.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.026+0000 7f12d9e00640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12c0077f70 0x7f12c007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:56.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.026+0000 7f12d9e00640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12c0077f70 0x7f12c007a430 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f12e4153270 tx=0x7f12cc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:56.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.030+0000 7f12db400640 1 -- 172.21.15.31:0/4059369592 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f12e414d390 con 0x7f12e40c5aa0 2024-09-11T00:32:56.125 INFO:tasks.workunit.client.1.smithi204.stdout:checking for mkdir... yes 2024-09-11T00:32:56.258 INFO:tasks.workunit.client.1.smithi204.stdout:checking for strerror... yes 2024-09-11T00:32:56.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.266+0000 7f12e8a00640 1 -- 172.21.15.31:0/4059369592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12e4156d90 con 0x7f12e40c5aa0 2024-09-11T00:32:56.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.267+0000 7f12db400640 1 -- 172.21.15.31:0/4059369592 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5705 (secure 0 0 0) 0x7f12d4083b60 con 0x7f12e40c5aa0 2024-09-11T00:32:56.269 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:56.269 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"btime":"2024-09-11T00:32:55:293099+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:54.461539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:56.269 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-09-11T00:32:56.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.269+0000 7f12e8a00640 1 -- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12c0077f70 msgr2=0x7f12c007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.269+0000 7f12e8a00640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12c0077f70 0x7f12c007a430 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f12e4153270 tx=0x7f12cc008040 comp rx=0 tx=0).stop 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.269+0000 7f12e8a00640 1 -- 172.21.15.31:0/4059369592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e40c5aa0 msgr2=0x7f12e41e1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.269+0000 7f12e8a00640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e40c5aa0 0x7f12e41e1fb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009850 tx=0x7f12d40039e0 comp rx=0 tx=0).stop 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.270+0000 7f12e8a00640 1 -- 172.21.15.31:0/4059369592 shutdown_connections 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.270+0000 7f12e8a00640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12c0077f70 0x7f12c007a430 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.270+0000 7f12e8a00640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f12e4153850 0x7f12e40aa7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.270+0000 7f12e8a00640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12e40c6420 0x7f12e414ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.270+0000 7f12e8a00640 1 --2- 172.21.15.31:0/4059369592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12e40c5aa0 0x7f12e41e1fb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.270+0000 7f12e8a00640 1 -- 172.21.15.31:0/4059369592 >> 172.21.15.31:0/4059369592 conn(0x7f12e406caf0 msgr2=0x7f12e4150b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.270+0000 7f12e8a00640 1 -- 172.21.15.31:0/4059369592 shutdown_connections 2024-09-11T00:32:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:56.270+0000 7f12e8a00640 1 -- 172.21.15.31:0/4059369592 wait complete. 2024-09-11T00:32:56.288 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:32:55:293099+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5076": { "addr": "172.21.15.31:6835/4202305151", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6833", "nonce": 4202305151, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 4202305151, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5076, "incarnation": 26, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:32:54.461539+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5076 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4721": { "addr": "172.21.15.204:6837/2954547253", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 2954547253, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 2954547253, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4721, "incarnation": 21, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:32:27.304525+0000", "qdb_cluster": [ 4721 ], "qdb_leader": 4721, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4721 } } } ], "standbys": [ { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:32:56.399 INFO:tasks.workunit.client.1.smithi204.stdout:checking for open64... yes 2024-09-11T00:32:56.532 INFO:tasks.workunit.client.1.smithi204.stdout:checking for stat64... yes 2024-09-11T00:32:56.923 INFO:tasks.workunit.client.1.smithi204.stdout:checking for fseeko64... yes 2024-09-11T00:32:57.059 INFO:tasks.workunit.client.1.smithi204.stdout:checking for lrand48_r... yes 2024-09-11T00:32:57.190 INFO:tasks.workunit.client.1.smithi204.stdout:checking for srand48_r... yes 2024-09-11T00:32:57.237 INFO:tasks.workunit.client.1.smithi204.stdout:configure: creating ./config.status 2024-09-11T00:32:57.436 INFO:tasks.workunit.client.1.smithi204.stdout: /bin/sh ./config.status 2024-09-11T00:32:58.036 INFO:tasks.workunit.client.1.smithi204.stdout:config.status: creating Makefile 2024-09-11T00:32:58.079 INFO:tasks.workunit.client.1.smithi204.stdout:config.status: creating config.h 2024-09-11T00:32:58.127 INFO:tasks.workunit.client.1.smithi204.stdout:config.status: executing depfiles commands 2024-09-11T00:32:58.172 INFO:tasks.workunit.client.1.smithi204.stdout:config.status: executing default commands 2024-09-11T00:32:58.248 INFO:tasks.workunit.client.1.smithi204.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run autoheader 2024-09-11T00:32:58.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:32:58.437 INFO:tasks.workunit.client.0.smithi031.stdout:checking limits.h usability... yes 2024-09-11T00:32:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.475+0000 7fee59200640 1 -- 172.21.15.31:0/1469047442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee5414c8e0 msgr2=0x7fee54156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.475+0000 7fee59200640 1 --2- 172.21.15.31:0/1469047442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee5414c8e0 0x7fee54156b00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fee540ae660 tx=0x7fee4402a5f0 comp rx=0 tx=0).stop 2024-09-11T00:32:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.475+0000 7fee59200640 1 -- 172.21.15.31:0/1469047442 shutdown_connections 2024-09-11T00:32:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.475+0000 7fee59200640 1 --2- 172.21.15.31:0/1469047442 >> v2:172.21.15.204:3300/0 conn(0x7fee540c9a10 0x7fee54159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.475+0000 7fee59200640 1 --2- 172.21.15.31:0/1469047442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee5414c8e0 0x7fee54156b00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.475+0000 7fee59200640 1 -- 172.21.15.31:0/1469047442 >> 172.21.15.31:0/1469047442 conn(0x7fee54068ee0 msgr2=0x7fee540cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:58.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.475+0000 7fee59200640 1 -- 172.21.15.31:0/1469047442 shutdown_connections 2024-09-11T00:32:58.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.475+0000 7fee59200640 1 -- 172.21.15.31:0/1469047442 wait complete. 2024-09-11T00:32:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.478+0000 7fee59200640 1 Processor -- start 2024-09-11T00:32:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.478+0000 7fee59200640 1 -- start start 2024-09-11T00:32:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee59200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee540c9a10 0x7fee54071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee59200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee5414c8e0 0x7fee54158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee59200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee5414ccc0 0x7fee540c0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee59200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fee540c6490 con 0x7fee5414ccc0 2024-09-11T00:32:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee59200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fee540c6800 con 0x7fee540c9a10 2024-09-11T00:32:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee59200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fee540c6c90 con 0x7fee5414c8e0 2024-09-11T00:32:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee52a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee5414c8e0 0x7fee54158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee52a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee5414c8e0 0x7fee54158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:51056/0 (socket says 172.21.15.31:51056) 2024-09-11T00:32:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee53400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee540c9a10 0x7fee54071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee52a00640 1 -- 172.21.15.31:0/2826474535 learned_addr learned my addr 172.21.15.31:0/2826474535 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:32:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.480+0000 7fee53e00640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee5414ccc0 0x7fee540c0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.481+0000 7fee53400640 1 -- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee5414c8e0 msgr2=0x7fee54158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.481+0000 7fee53400640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee5414c8e0 0x7fee54158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.481+0000 7fee53400640 1 -- 172.21.15.31:0/2826474535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee5414ccc0 msgr2=0x7fee540c0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.481+0000 7fee53400640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee5414ccc0 0x7fee540c0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.481+0000 7fee53400640 1 -- 172.21.15.31:0/2826474535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee54073920 con 0x7fee540c9a10 2024-09-11T00:32:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.481+0000 7fee52a00640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee5414c8e0 0x7fee54158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:32:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.481+0000 7fee53400640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee540c9a10 0x7fee54071d20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fee44031000 tx=0x7fee44005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.482+0000 7fee4b400640 1 -- 172.21.15.31:0/2826474535 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4403f070 con 0x7fee540c9a10 2024-09-11T00:32:58.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.482+0000 7fee4b400640 1 -- 172.21.15.31:0/2826474535 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee440047a0 con 0x7fee540c9a10 2024-09-11T00:32:58.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.482+0000 7fee4b400640 1 -- 172.21.15.31:0/2826474535 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee44035030 con 0x7fee540c9a10 2024-09-11T00:32:58.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.482+0000 7fee59200640 1 -- 172.21.15.31:0/2826474535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee54072260 con 0x7fee540c9a10 2024-09-11T00:32:58.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.483+0000 7fee59200640 1 -- 172.21.15.31:0/2826474535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee54159050 con 0x7fee540c9a10 2024-09-11T00:32:58.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.484+0000 7fee59200640 1 -- 172.21.15.31:0/2826474535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee540bffb0 con 0x7fee540c9a10 2024-09-11T00:32:58.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.485+0000 7fee4b400640 1 -- 172.21.15.31:0/2826474535 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fee44004bb0 con 0x7fee540c9a10 2024-09-11T00:32:58.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.487+0000 7fee4b400640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee34077f70 0x7fee3407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:32:58.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.487+0000 7fee4b400640 1 -- 172.21.15.31:0/2826474535 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fee440bb320 con 0x7fee540c9a10 2024-09-11T00:32:58.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.487+0000 7fee52a00640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee34077f70 0x7fee3407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:32:58.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.488+0000 7fee52a00640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee34077f70 0x7fee3407a430 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee54156890 tx=0x7fee40009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:32:58.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.491+0000 7fee4b400640 1 -- 172.21.15.31:0/2826474535 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fee54072260 con 0x7fee540c9a10 2024-09-11T00:32:58.496 INFO:tasks.workunit.client.0.smithi031.stdout:checking limits.h presence... yes 2024-09-11T00:32:58.498 INFO:tasks.workunit.client.0.smithi031.stdout:checking for limits.h... yes 2024-09-11T00:32:58.534 INFO:tasks.workunit.client.0.smithi031.stdout:checking for stdint.h... (cached) yes 2024-09-11T00:32:58.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.744+0000 7fee59200640 1 -- 172.21.15.31:0/2826474535 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee54154cb0 con 0x7fee540c9a10 2024-09-11T00:32:58.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.744+0000 7fee4b400640 1 -- 172.21.15.31:0/2826474535 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5710 (secure 0 0 0) 0x7fee440bb0e0 con 0x7fee540c9a10 2024-09-11T00:32:58.746 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:32:58.746 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"btime":"2024-09-11T00:32:58:315123+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:32:58.747 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 -- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee34077f70 msgr2=0x7fee3407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee34077f70 0x7fee3407a430 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee54156890 tx=0x7fee40009040 comp rx=0 tx=0).stop 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 -- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee540c9a10 msgr2=0x7fee54071d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee540c9a10 0x7fee54071d20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fee44031000 tx=0x7fee44005e80 comp rx=0 tx=0).stop 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 -- 172.21.15.31:0/2826474535 shutdown_connections 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee34077f70 0x7fee3407a430 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee5414ccc0 0x7fee540c0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee5414c8e0 0x7fee54158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 --2- 172.21.15.31:0/2826474535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee540c9a10 0x7fee54071d20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 -- 172.21.15.31:0/2826474535 >> 172.21.15.31:0/2826474535 conn(0x7fee54068ee0 msgr2=0x7fee5406dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:32:58.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 -- 172.21.15.31:0/2826474535 shutdown_connections 2024-09-11T00:32:58.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:32:58.747+0000 7fee59200640 1 -- 172.21.15.31:0/2826474535 wait complete. 2024-09-11T00:32:58.763 INFO:tasks.workunit.client.1.smithi204.stdout:rm -f stamp-h1 2024-09-11T00:32:58.765 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=4390 2024-09-11T00:32:58.765 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.d 2024-09-11T00:32:58.767 INFO:tasks.workunit.client.1.smithi204.stdout:touch config.h.in 2024-09-11T00:32:58.768 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/time.h usability... yes 2024-09-11T00:32:58.772 INFO:tasks.workunit.client.1.smithi204.stdout:cd . && /bin/sh ./config.status config.h 2024-09-11T00:32:58.820 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/time.h presence... yes 2024-09-11T00:32:58.821 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/time.h... yes 2024-09-11T00:32:58.850 INFO:tasks.workunit.client.0.smithi031.stdout:checking for unistd.h... (cached) yes 2024-09-11T00:32:58.873 INFO:tasks.workunit.client.1.smithi204.stdout:config.status: creating config.h 2024-09-11T00:32:58.930 INFO:tasks.workunit.client.1.smithi204.stdout:make all-am 2024-09-11T00:32:58.973 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/vfs.h usability... yes 2024-09-11T00:32:59.023 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/vfs.h presence... yes 2024-09-11T00:32:59.025 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/vfs.h... yes 2024-09-11T00:32:59.035 INFO:tasks.workunit.client.1.smithi204.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb' 2024-09-11T00:32:59.035 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-09-11T00:32:59.132 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/limits.h usability... no 2024-09-11T00:32:59.186 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/limits.h presence... no 2024-09-11T00:32:59.188 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/limits.h... no 2024-09-11T00:32:59.268 INFO:tasks.workunit.client.0.smithi031.stdout:checking for an ANSI C-conforming const... yes 2024-09-11T00:32:59.333 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-11T00:32:59.331+0000 7fd387c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:32:59.356 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-11T00:32:59.361 INFO:tasks.workunit.client.0.smithi031.stdout:checking for function prototypes... yes 2024-09-11T00:32:59.449 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether setvbuf arguments are reversed... no 2024-09-11T00:32:59.604 INFO:tasks.workunit.client.0.smithi031.stdout:checking for system... yes 2024-09-11T00:33:00.263 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gettimeofday... yes 2024-09-11T00:33:00.418 INFO:tasks.workunit.client.0.smithi031.stdout:checking for mkdir... yes 2024-09-11T00:33:00.569 INFO:tasks.workunit.client.0.smithi031.stdout:checking for strerror... yes 2024-09-11T00:33:00.720 INFO:tasks.workunit.client.0.smithi031.stdout:checking for open64... yes 2024-09-11T00:33:00.766 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-11T00:33:00.766 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:33:00.766 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:33:00.770 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:33:00.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:00.912 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:33:00.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.972+0000 7fb6e1200640 1 -- 172.21.15.31:0/3300634675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6dc0c2560 msgr2=0x7fb6dc164150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:00.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.972+0000 7fb6e1200640 1 --2- 172.21.15.31:0/3300634675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6dc0c2560 0x7fb6dc164150 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4005fd0 tx=0x7fb6d402b8d0 comp rx=0 tx=0).stop 2024-09-11T00:33:00.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.972+0000 7fb6e1200640 1 -- 172.21.15.31:0/3300634675 shutdown_connections 2024-09-11T00:33:00.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.972+0000 7fb6e1200640 1 --2- 172.21.15.31:0/3300634675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6dc0c2560 0x7fb6dc164150 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:00.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.972+0000 7fb6e1200640 1 -- 172.21.15.31:0/3300634675 >> 172.21.15.31:0/3300634675 conn(0x7fb6dc0a9750 msgr2=0x7fb6dc0a9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:00.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.972+0000 7fb6e1200640 1 -- 172.21.15.31:0/3300634675 shutdown_connections 2024-09-11T00:33:00.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.972+0000 7fb6e1200640 1 -- 172.21.15.31:0/3300634675 wait complete. 2024-09-11T00:33:00.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.974+0000 7fb6e1200640 1 Processor -- start 2024-09-11T00:33:00.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.974+0000 7fb6e1200640 1 -- start start 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.975+0000 7fb6e1200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6dc08c340 0x7fb6dc1fb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.975+0000 7fb6e1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6dc0c2560 0x7fb6dc0893b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.975+0000 7fb6e1200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6dc0c2c90 0x7fb6dc1fe160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.975+0000 7fb6e1200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb6dc08cb50 con 0x7fb6dc0c2560 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.975+0000 7fb6e1200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb6dc0a7510 con 0x7fb6dc0c2c90 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.975+0000 7fb6e1200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb6dc08c820 con 0x7fb6dc08c340 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.975+0000 7fb6d1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6dc0c2560 0x7fb6dc0893b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.975+0000 7fb6d1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6dc0c2560 0x7fb6dc0893b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33566/0 (socket says 172.21.15.31:33566) 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.975+0000 7fb6d1400640 1 -- 172.21.15.31:0/2065425362 learned_addr learned my addr 172.21.15.31:0/2065425362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.975+0000 7fb6d9e00640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6dc08c340 0x7fb6dc1fb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:00.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6da800640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6dc0c2c90 0x7fb6dc1fe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6d1400640 1 -- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6dc08c340 msgr2=0x7fb6dc1fb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6d1400640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6dc08c340 0x7fb6dc1fb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6d1400640 1 -- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6dc0c2c90 msgr2=0x7fb6dc1fe160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6d1400640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6dc0c2c90 0x7fb6dc1fe160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6d1400640 1 -- 172.21.15.31:0/2065425362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6dc083620 con 0x7fb6dc0c2560 2024-09-11T00:33:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6d1400640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6dc0c2560 0x7fb6dc0893b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb6c400c500 tx=0x7fb6c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6d2a00640 1 -- 172.21.15.31:0/2065425362 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4018070 con 0x7fb6dc0c2560 2024-09-11T00:33:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6d2a00640 1 -- 172.21.15.31:0/2065425362 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6c40035f0 con 0x7fb6dc0c2560 2024-09-11T00:33:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6d2a00640 1 -- 172.21.15.31:0/2065425362 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4003a40 con 0x7fb6dc0c2560 2024-09-11T00:33:00.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.976+0000 7fb6e1200640 1 -- 172.21.15.31:0/2065425362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6dc1627f0 con 0x7fb6dc0c2560 2024-09-11T00:33:00.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.977+0000 7fb6e1200640 1 -- 172.21.15.31:0/2065425362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6dc0c5bb0 con 0x7fb6dc0c2560 2024-09-11T00:33:00.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.978+0000 7fb6e1200640 1 -- 172.21.15.31:0/2065425362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6dc0a94d0 con 0x7fb6dc0c2560 2024-09-11T00:33:00.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.979+0000 7fb6d2a00640 1 -- 172.21.15.31:0/2065425362 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb6c4003be0 con 0x7fb6dc0c2560 2024-09-11T00:33:00.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.980+0000 7fb6d2a00640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb6b8077fa0 0x7fb6b807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:00.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.980+0000 7fb6d2a00640 1 -- 172.21.15.31:0/2065425362 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fb6c4099840 con 0x7fb6dc0c2560 2024-09-11T00:33:00.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.981+0000 7fb6d9e00640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb6b8077fa0 0x7fb6b807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:00.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.982+0000 7fb6d9e00640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb6b8077fa0 0x7fb6b807a460 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc000f60 tx=0x7fb6cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:00.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:00.985+0000 7fb6d2a00640 1 -- 172.21.15.31:0/2065425362 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb6dc0a94d0 con 0x7fb6dc0c2560 2024-09-11T00:33:01.028 INFO:tasks.workunit.client.0.smithi031.stdout:checking for stat64... yes 2024-09-11T00:33:01.058 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-09-11T00:33:01.068 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-09-11T00:33:01.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.234+0000 7fb6e1200640 1 -- 172.21.15.31:0/2065425362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6dc164150 con 0x7fb6dc0c2560 2024-09-11T00:33:01.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.235+0000 7fb6d2a00640 1 -- 172.21.15.31:0/2065425362 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5710 (secure 0 0 0) 0x7fb6dc083620 con 0x7fb6dc0c2560 2024-09-11T00:33:01.237 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:01.237 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"btime":"2024-09-11T00:32:58:315123+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:01.237 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2024-09-11T00:33:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 -- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb6b8077fa0 msgr2=0x7fb6b807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb6b8077fa0 0x7fb6b807a460 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc000f60 tx=0x7fb6cc009040 comp rx=0 tx=0).stop 2024-09-11T00:33:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 -- 172.21.15.31:0/2065425362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6dc0c2560 msgr2=0x7fb6dc0893b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6dc0c2560 0x7fb6dc0893b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb6c400c500 tx=0x7fb6c400c9c0 comp rx=0 tx=0).stop 2024-09-11T00:33:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 -- 172.21.15.31:0/2065425362 shutdown_connections 2024-09-11T00:33:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb6b8077fa0 0x7fb6b807a460 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6dc0c2c90 0x7fb6dc1fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6dc0c2560 0x7fb6dc0893b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 --2- 172.21.15.31:0/2065425362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6dc08c340 0x7fb6dc1fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 -- 172.21.15.31:0/2065425362 >> 172.21.15.31:0/2065425362 conn(0x7fb6dc0a9750 msgr2=0x7fb6dc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 -- 172.21.15.31:0/2065425362 shutdown_connections 2024-09-11T00:33:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:01.237+0000 7fb6e1200640 1 -- 172.21.15.31:0/2065425362 wait complete. 2024-09-11T00:33:01.255 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:33:01.600 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-09-11T00:33:01.609 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-09-11T00:33:01.625 INFO:tasks.workunit.client.0.smithi031.stdout:checking for fseeko64... yes 2024-09-11T00:33:01.779 INFO:tasks.workunit.client.0.smithi031.stdout:checking for lrand48_r... yes 2024-09-11T00:33:01.825 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-09-11T00:33:01.833 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-09-11T00:33:01.932 INFO:tasks.workunit.client.0.smithi031.stdout:checking for srand48_r... yes 2024-09-11T00:33:01.981 INFO:tasks.workunit.client.0.smithi031.stdout:configure: creating ./config.status 2024-09-11T00:33:01.999 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-09-11T00:33:02.008 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-09-11T00:33:02.205 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-09-11T00:33:02.214 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-09-11T00:33:02.330 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-09-11T00:33:02.338 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-09-11T00:33:02.424 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-09-11T00:33:02.432 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-09-11T00:33:02.531 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-09-11T00:33:02.539 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-09-11T00:33:03.021 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-09-11T00:33:03.029 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-09-11T00:33:03.224 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-09-11T00:33:03.231 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-09-11T00:33:03.256 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:03.283 INFO:tasks.workunit.client.0.smithi031.stdout:config.status: creating Makefile 2024-09-11T00:33:03.331 INFO:tasks.workunit.client.0.smithi031.stdout:config.status: creating config.h 2024-09-11T00:33:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.433+0000 7efc6da00640 1 -- 172.21.15.31:0/342093205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6806e850 msgr2=0x7efc68160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.433+0000 7efc6da00640 1 --2- 172.21.15.31:0/342093205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6806e850 0x7efc68160860 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7efc68153d20 tx=0x7efc6002aac0 comp rx=0 tx=0).stop 2024-09-11T00:33:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.433+0000 7efc6da00640 1 -- 172.21.15.31:0/342093205 shutdown_connections 2024-09-11T00:33:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.433+0000 7efc6da00640 1 --2- 172.21.15.31:0/342093205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6806e850 0x7efc68160860 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.433+0000 7efc6da00640 1 --2- 172.21.15.31:0/342093205 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc68150020 0x7efc680ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.433+0000 7efc6da00640 1 -- 172.21.15.31:0/342093205 >> 172.21.15.31:0/342093205 conn(0x7efc68068f10 msgr2=0x7efc680cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.433+0000 7efc6da00640 1 -- 172.21.15.31:0/342093205 shutdown_connections 2024-09-11T00:33:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.433+0000 7efc6da00640 1 -- 172.21.15.31:0/342093205 wait complete. 2024-09-11T00:33:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.434+0000 7efc6da00640 1 Processor -- start 2024-09-11T00:33:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.435+0000 7efc6da00640 1 -- start start 2024-09-11T00:33:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.435+0000 7efc6da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6806e850 0x7efc680c09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.435+0000 7efc6da00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc680ada20 0x7efc680c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.435+0000 7efc6da00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc68150020 0x7efc680c3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.435+0000 7efc6da00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efc68157550 con 0x7efc6806e850 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.435+0000 7efc6da00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efc681573d0 con 0x7efc680ada20 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.435+0000 7efc6da00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efc68156960 con 0x7efc68150020 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc67e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6806e850 0x7efc680c09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc67e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6806e850 0x7efc680c09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33588/0 (socket says 172.21.15.31:33588) 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc67e00640 1 -- 172.21.15.31:0/3402276933 learned_addr learned my addr 172.21.15.31:0/3402276933 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc6d000640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc68150020 0x7efc680c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc5f400640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc680ada20 0x7efc680c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc67e00640 1 -- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc68150020 msgr2=0x7efc680c3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc67e00640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc68150020 0x7efc680c3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc67e00640 1 -- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc680ada20 msgr2=0x7efc680c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc67e00640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc680ada20 0x7efc680c1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc67e00640 1 -- 172.21.15.31:0/3402276933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6814e650 con 0x7efc6806e850 2024-09-11T00:33:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.436+0000 7efc67e00640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6806e850 0x7efc680c09e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7efc5800e9f0 tx=0x7efc5800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.437+0000 7efc64c00640 1 -- 172.21.15.31:0/3402276933 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5800cd80 con 0x7efc6806e850 2024-09-11T00:33:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.437+0000 7efc64c00640 1 -- 172.21.15.31:0/3402276933 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc58010430 con 0x7efc6806e850 2024-09-11T00:33:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.437+0000 7efc6da00640 1 -- 172.21.15.31:0/3402276933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc680c1f60 con 0x7efc6806e850 2024-09-11T00:33:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.437+0000 7efc64c00640 1 -- 172.21.15.31:0/3402276933 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc58010840 con 0x7efc6806e850 2024-09-11T00:33:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.437+0000 7efc6da00640 1 -- 172.21.15.31:0/3402276933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc680cb9e0 con 0x7efc6806e850 2024-09-11T00:33:03.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.438+0000 7efc6da00640 1 -- 172.21.15.31:0/3402276933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc680c21a0 con 0x7efc6806e850 2024-09-11T00:33:03.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.439+0000 7efc64c00640 1 -- 172.21.15.31:0/3402276933 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7efc580109e0 con 0x7efc6806e850 2024-09-11T00:33:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.440+0000 7efc64c00640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efc44077f70 0x7efc4407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.441+0000 7efc64c00640 1 -- 172.21.15.31:0/3402276933 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7efc5809ad30 con 0x7efc6806e850 2024-09-11T00:33:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.441+0000 7efc5f400640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efc44077f70 0x7efc4407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:03.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.442+0000 7efc5f400640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efc44077f70 0x7efc4407a430 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efc6814fd60 tx=0x7efc50009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:03.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.445+0000 7efc64c00640 1 -- 172.21.15.31:0/3402276933 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efc680c21a0 con 0x7efc6806e850 2024-09-11T00:33:03.459 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-09-11T00:33:03.467 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-09-11T00:33:03.512 INFO:tasks.workunit.client.0.smithi031.stdout:config.status: executing depfiles commands 2024-09-11T00:33:03.650 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-09-11T00:33:03.659 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-09-11T00:33:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.681+0000 7efc6da00640 1 -- 172.21.15.31:0/3402276933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc680cc220 con 0x7efc6806e850 2024-09-11T00:33:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.682+0000 7efc64c00640 1 -- 172.21.15.31:0/3402276933 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6535 (secure 0 0 0) 0x7efc58062dd0 con 0x7efc6806e850 2024-09-11T00:33:03.684 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:03.684 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"btime":"2024-09-11T00:33:01:330912+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:03.684 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 -- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efc44077f70 msgr2=0x7efc4407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efc44077f70 0x7efc4407a430 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efc6814fd60 tx=0x7efc50009040 comp rx=0 tx=0).stop 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 -- 172.21.15.31:0/3402276933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6806e850 msgr2=0x7efc680c09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6806e850 0x7efc680c09e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7efc5800e9f0 tx=0x7efc5800eeb0 comp rx=0 tx=0).stop 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 -- 172.21.15.31:0/3402276933 shutdown_connections 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efc44077f70 0x7efc4407a430 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efc68150020 0x7efc680c3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc680ada20 0x7efc680c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 --2- 172.21.15.31:0/3402276933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc6806e850 0x7efc680c09e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 -- 172.21.15.31:0/3402276933 >> 172.21.15.31:0/3402276933 conn(0x7efc68068f10 msgr2=0x7efc68154fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 -- 172.21.15.31:0/3402276933 shutdown_connections 2024-09-11T00:33:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:03.685+0000 7efc6da00640 1 -- 172.21.15.31:0/3402276933 wait complete. 2024-09-11T00:33:03.702 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-11T00:33:03.702 INFO:tasks.mds_thrash.fs.[a]:waiting for 111 secs before thrashing 2024-09-11T00:33:03.800 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-09-11T00:33:03.807 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-09-11T00:33:03.988 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-09-11T00:33:03.996 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-09-11T00:33:04.071 INFO:tasks.workunit.client.1.smithi204.stderr:In file included from parser.h:22, 2024-09-11T00:33:04.071 INFO:tasks.workunit.client.1.smithi204.stderr: from parser.c:25: 2024-09-11T00:33:04.071 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c: In function ‘init_filesys’: 2024-09-11T00:33:04.074 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:04.074 INFO:tasks.workunit.client.1.smithi204.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:04.074 INFO:tasks.workunit.client.1.smithi204.stderr: | ^ 2024-09-11T00:33:04.074 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:04.074 INFO:tasks.workunit.client.1.smithi204.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:04.074 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~ 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:04.075 INFO:tasks.workunit.client.1.smithi204.stderr: | ^ 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~ 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-11T00:33:04.076 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr: | ^ 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~ 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:04.077 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr: | ^ 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~ 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:04.078 INFO:tasks.workunit.client.1.smithi204.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c: In function ‘init_tg_stats’: 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr: | ^ 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~ 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:04.079 INFO:tasks.workunit.client.1.smithi204.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr: | ^ 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:04.080 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~ 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:04.081 INFO:tasks.workunit.client.1.smithi204.stderr: | ^ 2024-09-11T00:33:04.082 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:04.082 INFO:tasks.workunit.client.1.smithi204.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:04.082 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~ 2024-09-11T00:33:04.082 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:04.082 INFO:tasks.workunit.client.1.smithi204.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:04.082 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr: | ^ 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~ 2024-09-11T00:33:04.083 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:04.084 INFO:tasks.workunit.client.1.smithi204.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:04.084 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:04.084 INFO:tasks.workunit.client.1.smithi204.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:04.084 INFO:tasks.workunit.client.1.smithi204.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-11T00:33:04.084 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:04.084 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:04.084 INFO:tasks.workunit.client.1.smithi204.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-11T00:33:04.084 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:04.125 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c: In function ‘get_optstr’: 2024-09-11T00:33:04.125 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-09-11T00:33:04.125 INFO:tasks.workunit.client.1.smithi204.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-09-11T00:33:04.125 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:04.125 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:200:14: note: source object declared here 2024-09-11T00:33:04.125 INFO:tasks.workunit.client.1.smithi204.stderr: 200 | char temp[BUFSIZE]; 2024-09-11T00:33:04.126 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~ 2024-09-11T00:33:04.131 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c: In function ‘size64_convert’: 2024-09-11T00:33:04.131 INFO:tasks.workunit.client.1.smithi204.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-09-11T00:33:04.131 INFO:tasks.workunit.client.1.smithi204.stderr: 119 | memcpy(unit, "\0", 3); 2024-09-11T00:33:04.131 INFO:tasks.workunit.client.1.smithi204.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:04.414 INFO:tasks.workunit.client.0.smithi031.stdout:config.status: executing default commands 2024-09-11T00:33:04.516 INFO:tasks.workunit.client.0.smithi031.stderr:+ make 2024-09-11T00:33:04.567 INFO:tasks.workunit.client.0.smithi031.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-09-11T00:33:04.569 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-09-11T00:33:04.577 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-09-11T00:33:04.584 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-09-11T00:33:04.586 INFO:tasks.workunit.client.0.smithi031.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-09-11T00:33:04.586 INFO:tasks.workunit.client.0.smithi031.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-09-11T00:33:04.586 INFO:tasks.workunit.client.0.smithi031.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-09-11T00:33:04.586 INFO:tasks.workunit.client.0.smithi031.stderr: any GNU archive site. 2024-09-11T00:33:04.598 INFO:tasks.workunit.client.0.smithi031.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-09-11T00:33:04.608 INFO:tasks.workunit.client.0.smithi031.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-09-11T00:33:04.610 INFO:tasks.workunit.client.0.smithi031.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-09-11T00:33:04.610 INFO:tasks.workunit.client.0.smithi031.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-09-11T00:33:04.610 INFO:tasks.workunit.client.0.smithi031.stderr: You might want to install the `Automake' and `Perl' packages. 2024-09-11T00:33:04.610 INFO:tasks.workunit.client.0.smithi031.stderr: Grab them from any GNU archive site. 2024-09-11T00:33:04.696 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-09-11T00:33:04.705 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-09-11T00:33:04.850 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-09-11T00:33:04.859 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-09-11T00:33:04.925 INFO:tasks.workunit.client.1.smithi204.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-09-11T00:33:04.935 INFO:tasks.workunit.client.1.smithi204.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-09-11T00:33:05.121 INFO:tasks.workunit.client.1.smithi204.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb' 2024-09-11T00:33:05.123 INFO:tasks.workunit.client.1.smithi204.stderr:+ cd .. 2024-09-11T00:33:05.125 INFO:tasks.workunit.client.1.smithi204.stderr:+ mkdir tmp 2024-09-11T00:33:05.129 INFO:tasks.workunit.client.1.smithi204.stderr:+ cd tmp 2024-09-11T00:33:05.131 INFO:tasks.workunit.client.1.smithi204.stderr:+ for f in $mydir/*.ffsb 2024-09-11T00:33:05.131 INFO:tasks.workunit.client.1.smithi204.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/random_write.32.ffsb 2024-09-11T00:33:05.133 INFO:tasks.workunit.client.1.smithi204.stdout:FFSB version 6.0-RC2 started 2024-09-11T00:33:05.133 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:33:05.139 INFO:tasks.workunit.client.1.smithi204.stdout:benchmark time = 300 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout:ThreadGroup 0 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout:================ 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout: num_threads = 32 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout: read_random = off 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout: read_size = 0 (0B) 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout: read_blocksize = 0 (0B) 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout: read_skip = off 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout: read_skipsize = 0 (0B) 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout: write_random = on 2024-09-11T00:33:05.140 INFO:tasks.workunit.client.1.smithi204.stdout: write_size = 5242880 (5MB) 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: fsync_file = 0 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: write_blocksize = 4096 (4KB) 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: wait time = 0 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: op weights 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: read = 0 (0.00%) 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: readall = 0 (0.00%) 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: write = 1 (100.00%) 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: create = 0 (0.00%) 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: append = 0 (0.00%) 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: delete = 0 (0.00%) 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: metaop = 0 (0.00%) 2024-09-11T00:33:05.141 INFO:tasks.workunit.client.1.smithi204.stdout: createdir = 0 (0.00%) 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: stat = 0 (0.00%) 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: writeall = 0 (0.00%) 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: writeall_fsync = 0 (0.00%) 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: open_close = 0 (0.00%) 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: write_fsync = 0 (0.00%) 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: create_fsync = 0 (0.00%) 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: append_fsync = 0 (0.00%) 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout:FileSystem . 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout:========== 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: num_dirs = 0 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: starting files = 128 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:33:05.142 INFO:tasks.workunit.client.1.smithi204.stdout: min file size = 104857600 (100MB) 2024-09-11T00:33:05.143 INFO:tasks.workunit.client.1.smithi204.stdout: max file size = 104857600 (100MB) 2024-09-11T00:33:05.143 INFO:tasks.workunit.client.1.smithi204.stdout: directio = off 2024-09-11T00:33:05.143 INFO:tasks.workunit.client.1.smithi204.stdout: alignedio = on 2024-09-11T00:33:05.143 INFO:tasks.workunit.client.1.smithi204.stdout: bufferedio = off 2024-09-11T00:33:05.143 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:33:05.143 INFO:tasks.workunit.client.1.smithi204.stdout: aging is off 2024-09-11T00:33:05.143 INFO:tasks.workunit.client.1.smithi204.stdout: current utilization = 0.00% 2024-09-11T00:33:05.143 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:33:05.143 INFO:tasks.workunit.client.1.smithi204.stdout:checking existing fs: . 2024-09-11T00:33:05.144 INFO:tasks.workunit.client.1.smithi204.stdout:recreating new fileset 2024-09-11T00:33:05.144 INFO:tasks.workunit.client.1.smithi204.stderr:opendir: No such file or directory 2024-09-11T00:33:06.203 INFO:tasks.workunit.client.0.smithi031.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-09-11T00:33:06.621 INFO:tasks.workunit.client.0.smithi031.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-11T00:33:06.621 INFO:tasks.workunit.client.0.smithi031.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-11T00:33:06.621 INFO:tasks.workunit.client.0.smithi031.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-11T00:33:06.621 INFO:tasks.workunit.client.0.smithi031.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-11T00:33:06.777 INFO:tasks.workunit.client.0.smithi031.stdout:/bin/sh ./config.status --recheck 2024-09-11T00:33:06.906 INFO:tasks.workunit.client.0.smithi031.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-09-11T00:33:07.185 INFO:tasks.workunit.client.0.smithi031.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-11T00:33:08.202 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether build environment is sane... yes 2024-09-11T00:33:08.224 INFO:tasks.workunit.client.0.smithi031.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-11T00:33:08.226 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gawk... gawk 2024-09-11T00:33:08.257 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether make sets $(MAKE)... yes 2024-09-11T00:33:08.354 INFO:tasks.workunit.client.0.smithi031.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-11T00:33:08.354 INFO:tasks.workunit.client.0.smithi031.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-11T00:33:08.383 INFO:tasks.workunit.client.0.smithi031.stdout:checking for style of include used by make... GNU 2024-09-11T00:33:08.390 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gcc... gcc 2024-09-11T00:33:08.577 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether the C compiler works... yes 2024-09-11T00:33:08.578 INFO:tasks.workunit.client.0.smithi031.stdout:checking for C compiler default output file name... a.out 2024-09-11T00:33:08.670 INFO:tasks.workunit.client.0.smithi031.stdout:checking for suffix of executables... 2024-09-11T00:33:08.764 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether we are cross compiling... no 2024-09-11T00:33:08.834 INFO:tasks.workunit.client.0.smithi031.stdout:checking for suffix of object files... o 2024-09-11T00:33:08.894 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether we are using the GNU C compiler... yes 2024-09-11T00:33:08.951 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether gcc accepts -g... yes 2024-09-11T00:33:09.031 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-11T00:33:09.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:33:09.458 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:33:10.095 INFO:tasks.workunit.client.0.smithi031.stdout:checking dependency style of gcc... gcc3 2024-09-11T00:33:10.199 INFO:tasks.workunit.client.0.smithi031.stdout:checking how to run the C preprocessor... gcc -E 2024-09-11T00:33:10.300 INFO:tasks.workunit.client.0.smithi031.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-11T00:33:10.303 INFO:tasks.workunit.client.0.smithi031.stdout:checking for egrep... /usr/bin/grep -E 2024-09-11T00:33:10.580 INFO:tasks.workunit.client.0.smithi031.stdout:checking for ANSI C header files... yes 2024-09-11T00:33:10.654 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/types.h... yes 2024-09-11T00:33:10.744 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/stat.h... yes 2024-09-11T00:33:10.833 INFO:tasks.workunit.client.0.smithi031.stdout:checking for stdlib.h... yes 2024-09-11T00:33:11.555 INFO:tasks.workunit.client.0.smithi031.stdout:checking for string.h... yes 2024-09-11T00:33:11.647 INFO:tasks.workunit.client.0.smithi031.stdout:checking for memory.h... yes 2024-09-11T00:33:11.739 INFO:tasks.workunit.client.0.smithi031.stdout:checking for strings.h... yes 2024-09-11T00:33:12.392 INFO:tasks.workunit.client.0.smithi031.stdout:checking for inttypes.h... yes 2024-09-11T00:33:12.483 INFO:tasks.workunit.client.0.smithi031.stdout:checking for stdint.h... yes 2024-09-11T00:33:12.579 INFO:tasks.workunit.client.0.smithi031.stdout:checking for unistd.h... yes 2024-09-11T00:33:12.676 INFO:tasks.workunit.client.0.smithi031.stdout:checking minix/config.h usability... no 2024-09-11T00:33:12.736 INFO:tasks.workunit.client.0.smithi031.stdout:checking minix/config.h presence... no 2024-09-11T00:33:12.737 INFO:tasks.workunit.client.0.smithi031.stdout:checking for minix/config.h... no 2024-09-11T00:33:12.810 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-09-11T00:33:12.823 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gcc... (cached) gcc 2024-09-11T00:33:13.751 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-11T00:33:13.752 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-11T00:33:13.753 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-11T00:33:13.754 INFO:tasks.workunit.client.0.smithi031.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-11T00:33:14.392 INFO:tasks.workunit.client.0.smithi031.stdout:checking for main in -lm... yes 2024-09-11T00:33:14.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:14.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.721+0000 7fbc0e600640 1 -- 172.21.15.31:0/4133372822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc08153850 msgr2=0x7fbc08156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:14.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.721+0000 7fbc0e600640 1 --2- 172.21.15.31:0/4133372822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc08153850 0x7fbc08156350 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbc0806df10 tx=0x7fbc0002a8b0 comp rx=0 tx=0).stop 2024-09-11T00:33:14.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.721+0000 7fbc0e600640 1 -- 172.21.15.31:0/4133372822 shutdown_connections 2024-09-11T00:33:14.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.721+0000 7fbc0e600640 1 --2- 172.21.15.31:0/4133372822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc08153850 0x7fbc08156350 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:14.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.721+0000 7fbc0e600640 1 --2- 172.21.15.31:0/4133372822 >> v2:172.21.15.204:3300/0 conn(0x7fbc080c6420 0x7fbc08153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:14.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.721+0000 7fbc0e600640 1 -- 172.21.15.31:0/4133372822 >> 172.21.15.31:0/4133372822 conn(0x7fbc0806caf0 msgr2=0x7fbc0806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:14.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.721+0000 7fbc0e600640 1 -- 172.21.15.31:0/4133372822 shutdown_connections 2024-09-11T00:33:14.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.721+0000 7fbc0e600640 1 -- 172.21.15.31:0/4133372822 wait complete. 2024-09-11T00:33:14.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.723+0000 7fbc0e600640 1 Processor -- start 2024-09-11T00:33:14.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.723+0000 7fbc0e600640 1 -- start start 2024-09-11T00:33:14.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.724+0000 7fbc0e600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc080c5aa0 0x7fbc0814ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:14.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.724+0000 7fbc0e600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc080c6420 0x7fbc080abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.724+0000 7fbc0e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc08153850 0x7fbc081e1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.724+0000 7fbc0e600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbc080c5e80 con 0x7fbc08153850 2024-09-11T00:33:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.724+0000 7fbc0e600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc080c2e80 con 0x7fbc080c6420 2024-09-11T00:33:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.724+0000 7fbc0e600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbc080c6000 con 0x7fbc080c5aa0 2024-09-11T00:33:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc0dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc08153850 0x7fbc081e1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc0dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc08153850 0x7fbc081e1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33344/0 (socket says 172.21.15.31:33344) 2024-09-11T00:33:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc07e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc080c6420 0x7fbc080abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc0dc00640 1 -- 172.21.15.31:0/4073953516 learned_addr learned my addr 172.21.15.31:0/4073953516 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc0d200640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc080c5aa0 0x7fbc0814ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:14.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc0dc00640 1 -- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc080c5aa0 msgr2=0x7fbc0814ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:14.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc0dc00640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc080c5aa0 0x7fbc0814ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc0dc00640 1 -- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc080c6420 msgr2=0x7fbc080abb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc0dc00640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc080c6420 0x7fbc080abb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc0dc00640 1 -- 172.21.15.31:0/4073953516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc080acee0 con 0x7fbc08153850 2024-09-11T00:33:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.725+0000 7fbc0dc00640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc08153850 0x7fbc081e1a30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbc00005fd0 tx=0x7fbc000316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.726+0000 7fbc04c00640 1 -- 172.21.15.31:0/4073953516 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00031990 con 0x7fbc08153850 2024-09-11T00:33:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.726+0000 7fbc04c00640 1 -- 172.21.15.31:0/4073953516 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc00031c20 con 0x7fbc08153850 2024-09-11T00:33:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.726+0000 7fbc0e600640 1 -- 172.21.15.31:0/4073953516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc080acc00 con 0x7fbc08153850 2024-09-11T00:33:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.726+0000 7fbc04c00640 1 -- 172.21.15.31:0/4073953516 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc000330d0 con 0x7fbc08153850 2024-09-11T00:33:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.726+0000 7fbc0e600640 1 -- 172.21.15.31:0/4073953516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0814e800 con 0x7fbc08153850 2024-09-11T00:33:14.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.727+0000 7fbc0e600640 1 -- 172.21.15.31:0/4073953516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc08153310 con 0x7fbc08153850 2024-09-11T00:33:14.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.729+0000 7fbc04c00640 1 -- 172.21.15.31:0/4073953516 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbc00033270 con 0x7fbc08153850 2024-09-11T00:33:14.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.730+0000 7fbc04c00640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbe8077f70 0x7fbbe807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:14.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.730+0000 7fbc04c00640 1 -- 172.21.15.31:0/4073953516 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fbc000bb530 con 0x7fbc08153850 2024-09-11T00:33:14.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.730+0000 7fbc0d200640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbe8077f70 0x7fbbe807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:14.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.731+0000 7fbc0d200640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbe8077f70 0x7fbbe807a430 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc007fd0 tx=0x7fbbfc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:14.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:14.735+0000 7fbc04c00640 1 -- 172.21.15.31:0/4073953516 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbc0003f030 con 0x7fbc08153850 2024-09-11T00:33:14.747 INFO:tasks.workunit.client.0.smithi031.stdout:checking for main in -lpthread... yes 2024-09-11T00:33:14.753 INFO:tasks.workunit.client.0.smithi031.stdout:checking for ANSI C header files... (cached) yes 2024-09-11T00:33:14.838 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-11T00:33:14.937 INFO:tasks.workunit.client.0.smithi031.stdout:checking pthread.h usability... yes 2024-09-11T00:33:14.980 INFO:tasks.workunit.client.0.smithi031.stdout:checking pthread.h presence... yes 2024-09-11T00:33:14.980 INFO:tasks.workunit.client.0.smithi031.stdout:checking for pthread.h... yes 2024-09-11T00:33:15.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.003+0000 7fbc0e600640 1 -- 172.21.15.31:0/4073953516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc08156ab0 con 0x7fbc08153850 2024-09-11T00:33:15.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.004+0000 7fbc04c00640 1 -- 172.21.15.31:0/4073953516 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6535 (secure 0 0 0) 0x7fbc0008c030 con 0x7fbc08153850 2024-09-11T00:33:15.007 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:15.007 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"btime":"2024-09-11T00:33:01:330912+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:15.007 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-09-11T00:33:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 -- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbe8077f70 msgr2=0x7fbbe807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbe8077f70 0x7fbbe807a430 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc007fd0 tx=0x7fbbfc009040 comp rx=0 tx=0).stop 2024-09-11T00:33:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 -- 172.21.15.31:0/4073953516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc08153850 msgr2=0x7fbc081e1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc08153850 0x7fbc081e1a30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbc00005fd0 tx=0x7fbc000316d0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 -- 172.21.15.31:0/4073953516 shutdown_connections 2024-09-11T00:33:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbe8077f70 0x7fbbe807a430 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc08153850 0x7fbc081e1a30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc080c6420 0x7fbc080abb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 --2- 172.21.15.31:0/4073953516 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbc080c5aa0 0x7fbc0814ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 -- 172.21.15.31:0/4073953516 >> 172.21.15.31:0/4073953516 conn(0x7fbc0806caf0 msgr2=0x7fbc0814fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:15.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 -- 172.21.15.31:0/4073953516 shutdown_connections 2024-09-11T00:33:15.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.007+0000 7fbc0e600640 1 -- 172.21.15.31:0/4073953516 wait complete. 2024-09-11T00:33:15.027 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-11T00:33:15.028 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-11T00:33:15.028 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:33:15.048 INFO:tasks.ceph.mds.b:Stopped 2024-09-11T00:33:15.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:15.092 INFO:tasks.workunit.client.0.smithi031.stdout:checking fcntl.h usability... yes 2024-09-11T00:33:15.131 INFO:tasks.workunit.client.0.smithi031.stdout:checking fcntl.h presence... yes 2024-09-11T00:33:15.131 INFO:tasks.workunit.client.0.smithi031.stdout:checking for fcntl.h... yes 2024-09-11T00:33:15.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.223+0000 7f9f5d600640 1 -- 172.21.15.31:0/363786226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f58070570 msgr2=0x7f9f5806e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:15.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.223+0000 7f9f5d600640 1 --2- 172.21.15.31:0/363786226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f58070570 0x7f9f5806e480 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9f48005fd0 tx=0x7f9f4802a660 comp rx=0 tx=0).stop 2024-09-11T00:33:15.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.223+0000 7f9f5d600640 1 -- 172.21.15.31:0/363786226 shutdown_connections 2024-09-11T00:33:15.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.223+0000 7f9f5d600640 1 --2- 172.21.15.31:0/363786226 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f5806e9c0 0x7f9f58155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.223+0000 7f9f5d600640 1 --2- 172.21.15.31:0/363786226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f58070570 0x7f9f5806e480 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.223+0000 7f9f5d600640 1 -- 172.21.15.31:0/363786226 >> 172.21.15.31:0/363786226 conn(0x7f9f580a9750 msgr2=0x7f9f580a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:15.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.223+0000 7f9f5d600640 1 -- 172.21.15.31:0/363786226 shutdown_connections 2024-09-11T00:33:15.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.223+0000 7f9f5d600640 1 -- 172.21.15.31:0/363786226 wait complete. 2024-09-11T00:33:15.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.224+0000 7f9f5d600640 1 Processor -- start 2024-09-11T00:33:15.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.225+0000 7f9f5d600640 1 -- start start 2024-09-11T00:33:15.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.225+0000 7f9f5d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f5806e9c0 0x7f9f580c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:15.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.225+0000 7f9f5d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f58070570 0x7f9f581483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:15.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f5d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f580c2ff0 0x7f9f58148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:15.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f5d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5814fb20 con 0x7f9f5806e9c0 2024-09-11T00:33:15.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f5d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5806ef90 con 0x7f9f580c2ff0 2024-09-11T00:33:15.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f5d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9f580c7690 con 0x7f9f58070570 2024-09-11T00:33:15.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f56a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f5806e9c0 0x7f9f580c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f56a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f5806e9c0 0x7f9f580c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33368/0 (socket says 172.21.15.31:33368) 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f56a00640 1 -- 172.21.15.31:0/3712609604 learned_addr learned my addr 172.21.15.31:0/3712609604 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f56000640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f58070570 0x7f9f581483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f57400640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f580c2ff0 0x7f9f58148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f56a00640 1 -- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f58070570 msgr2=0x7f9f581483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f56a00640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f58070570 0x7f9f581483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f56a00640 1 -- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f580c2ff0 msgr2=0x7f9f58148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f56a00640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f580c2ff0 0x7f9f58148910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f56a00640 1 -- 172.21.15.31:0/3712609604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f58149150 con 0x7f9f5806e9c0 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.226+0000 7f9f56000640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f58070570 0x7f9f581483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:33:15.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.227+0000 7f9f56a00640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f5806e9c0 0x7f9f580c1040 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009850 tx=0x7f9f480038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:15.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.227+0000 7f9f4ea00640 1 -- 172.21.15.31:0/3712609604 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f48035030 con 0x7f9f5806e9c0 2024-09-11T00:33:15.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.227+0000 7f9f4ea00640 1 -- 172.21.15.31:0/3712609604 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f48042070 con 0x7f9f5806e9c0 2024-09-11T00:33:15.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.227+0000 7f9f4ea00640 1 -- 172.21.15.31:0/3712609604 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f480049e0 con 0x7f9f5806e9c0 2024-09-11T00:33:15.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.227+0000 7f9f5d600640 1 -- 172.21.15.31:0/3712609604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f581df800 con 0x7f9f5806e9c0 2024-09-11T00:33:15.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.227+0000 7f9f5d600640 1 -- 172.21.15.31:0/3712609604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f58148e50 con 0x7f9f5806e9c0 2024-09-11T00:33:15.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.228+0000 7f9f5d600640 1 -- 172.21.15.31:0/3712609604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f5814f740 con 0x7f9f5806e9c0 2024-09-11T00:33:15.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.229+0000 7f9f4ea00640 1 -- 172.21.15.31:0/3712609604 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9f48004b80 con 0x7f9f5806e9c0 2024-09-11T00:33:15.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.230+0000 7f9f4ea00640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9f38077fa0 0x7f9f3807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:15.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.230+0000 7f9f4ea00640 1 -- 172.21.15.31:0/3712609604 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f9f480baf70 con 0x7f9f5806e9c0 2024-09-11T00:33:15.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.231+0000 7f9f56000640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9f38077fa0 0x7f9f3807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:15.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.231+0000 7f9f56000640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9f38077fa0 0x7f9f3807a460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400a700 tx=0x7f9f44010040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:15.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.235+0000 7f9f4ea00640 1 -- 172.21.15.31:0/3712609604 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9f48082e30 con 0x7f9f5806e9c0 2024-09-11T00:33:15.260 INFO:tasks.workunit.client.0.smithi031.stdout:checking limits.h usability... yes 2024-09-11T00:33:15.298 INFO:tasks.workunit.client.0.smithi031.stdout:checking limits.h presence... yes 2024-09-11T00:33:15.299 INFO:tasks.workunit.client.0.smithi031.stdout:checking for limits.h... yes 2024-09-11T00:33:15.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.484+0000 7f9f5d600640 1 -- 172.21.15.31:0/3712609604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f581e0470 con 0x7f9f5806e9c0 2024-09-11T00:33:15.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.485+0000 7f9f4ea00640 1 -- 172.21.15.31:0/3712609604 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6535 (secure 0 0 0) 0x7f9f48087860 con 0x7f9f5806e9c0 2024-09-11T00:33:15.488 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:15.488 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"btime":"2024-09-11T00:33:01:330912+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:15.488 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-09-11T00:33:15.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.487+0000 7f9f5d600640 1 -- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9f38077fa0 msgr2=0x7f9f3807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:15.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.487+0000 7f9f5d600640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9f38077fa0 0x7f9f3807a460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400a700 tx=0x7f9f44010040 comp rx=0 tx=0).stop 2024-09-11T00:33:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.487+0000 7f9f5d600640 1 -- 172.21.15.31:0/3712609604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f5806e9c0 msgr2=0x7f9f580c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.487+0000 7f9f5d600640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f5806e9c0 0x7f9f580c1040 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009850 tx=0x7f9f480038a0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.487+0000 7f9f5d600640 1 -- 172.21.15.31:0/3712609604 shutdown_connections 2024-09-11T00:33:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.488+0000 7f9f5d600640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9f38077fa0 0x7f9f3807a460 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.488+0000 7f9f5d600640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f580c2ff0 0x7f9f58148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.488+0000 7f9f5d600640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9f58070570 0x7f9f581483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.488+0000 7f9f5d600640 1 --2- 172.21.15.31:0/3712609604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f5806e9c0 0x7f9f580c1040 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.488+0000 7f9f5d600640 1 -- 172.21.15.31:0/3712609604 >> 172.21.15.31:0/3712609604 conn(0x7f9f580a9750 msgr2=0x7f9f580c3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.488+0000 7f9f5d600640 1 -- 172.21.15.31:0/3712609604 shutdown_connections 2024-09-11T00:33:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:15.488+0000 7f9f5d600640 1 -- 172.21.15.31:0/3712609604 wait complete. 2024-09-11T00:33:15.506 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:33:15.611 INFO:tasks.workunit.client.0.smithi031.stdout:checking for stdint.h... (cached) yes 2024-09-11T00:33:15.700 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/time.h usability... yes 2024-09-11T00:33:15.735 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/time.h presence... yes 2024-09-11T00:33:15.736 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/time.h... yes 2024-09-11T00:33:15.766 INFO:tasks.workunit.client.0.smithi031.stdout:checking for unistd.h... (cached) yes 2024-09-11T00:33:15.861 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/vfs.h usability... yes 2024-09-11T00:33:15.897 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/vfs.h presence... yes 2024-09-11T00:33:15.897 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/vfs.h... yes 2024-09-11T00:33:16.011 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/limits.h usability... no 2024-09-11T00:33:16.074 INFO:tasks.workunit.client.0.smithi031.stdout:checking sys/limits.h presence... no 2024-09-11T00:33:16.075 INFO:tasks.workunit.client.0.smithi031.stdout:checking for sys/limits.h... no 2024-09-11T00:33:16.138 INFO:tasks.workunit.client.0.smithi031.stdout:checking for an ANSI C-conforming const... yes 2024-09-11T00:33:16.201 INFO:tasks.workunit.client.0.smithi031.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-11T00:33:16.333 INFO:tasks.workunit.client.0.smithi031.stdout:checking for system... yes 2024-09-11T00:33:16.765 INFO:tasks.workunit.client.0.smithi031.stdout:checking for gettimeofday... yes 2024-09-11T00:33:16.901 INFO:tasks.workunit.client.0.smithi031.stdout:checking for mkdir... yes 2024-09-11T00:33:17.035 INFO:tasks.workunit.client.0.smithi031.stdout:checking for strerror... yes 2024-09-11T00:33:17.171 INFO:tasks.workunit.client.0.smithi031.stdout:checking for open64... yes 2024-09-11T00:33:17.304 INFO:tasks.workunit.client.0.smithi031.stdout:checking for stat64... yes 2024-09-11T00:33:17.444 INFO:tasks.workunit.client.0.smithi031.stdout:checking for fseeko64... yes 2024-09-11T00:33:17.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:17.587 INFO:tasks.workunit.client.0.smithi031.stdout:checking for lrand48_r... yes 2024-09-11T00:33:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.684+0000 7fc893600640 1 -- 172.21.15.31:0/3188337383 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc88c0c4fe0 msgr2=0x7fc88c157530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.684+0000 7fc893600640 1 --2- 172.21.15.31:0/3188337383 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc88c0c4fe0 0x7fc88c157530 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc878009830 tx=0x7fc87802a530 comp rx=0 tx=0).stop 2024-09-11T00:33:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.684+0000 7fc893600640 1 -- 172.21.15.31:0/3188337383 shutdown_connections 2024-09-11T00:33:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.684+0000 7fc893600640 1 --2- 172.21.15.31:0/3188337383 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc88c0c4fe0 0x7fc88c157530 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.684+0000 7fc893600640 1 -- 172.21.15.31:0/3188337383 >> 172.21.15.31:0/3188337383 conn(0x7fc88c0ab8a0 msgr2=0x7fc88c0abcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:17.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.684+0000 7fc893600640 1 -- 172.21.15.31:0/3188337383 shutdown_connections 2024-09-11T00:33:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.684+0000 7fc893600640 1 -- 172.21.15.31:0/3188337383 wait complete. 2024-09-11T00:33:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.686+0000 7fc893600640 1 Processor -- start 2024-09-11T00:33:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.687+0000 7fc893600640 1 -- start start 2024-09-11T00:33:17.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.688+0000 7fc893600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc88c0c44f0 0x7fc88c150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:17.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.688+0000 7fc893600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c157a70 0x7fc88c0c3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:17.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.688+0000 7fc893600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc88c14fd70 0x7fc88c150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:17.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.688+0000 7fc893600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c0701b0 con 0x7fc88c157a70 2024-09-11T00:33:17.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.688+0000 7fc893600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c151af0 con 0x7fc88c14fd70 2024-09-11T00:33:17.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.688+0000 7fc893600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc88c0a9380 con 0x7fc88c0c44f0 2024-09-11T00:33:17.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc88be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c157a70 0x7fc88c0c3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:17.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc890c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc88c0c44f0 0x7fc88c150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:17.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc88be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c157a70 0x7fc88c0c3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33374/0 (socket says 172.21.15.31:33374) 2024-09-11T00:33:17.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc88be00640 1 -- 172.21.15.31:0/507934670 learned_addr learned my addr 172.21.15.31:0/507934670 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:17.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc891600640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc88c14fd70 0x7fc88c150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:17.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc88be00640 1 -- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc88c0c44f0 msgr2=0x7fc88c150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:17.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc88be00640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc88c0c44f0 0x7fc88c150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:17.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc88be00640 1 -- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc88c14fd70 msgr2=0x7fc88c150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:17.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc88be00640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc88c14fd70 0x7fc88c150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:17.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc88be00640 1 -- 172.21.15.31:0/507934670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c150be0 con 0x7fc88c157a70 2024-09-11T00:33:17.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.689+0000 7fc88be00640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c157a70 0x7fc88c0c3190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc878009780 tx=0x7fc878032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:17.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.690+0000 7fc888c00640 1 -- 172.21.15.31:0/507934670 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc878040070 con 0x7fc88c157a70 2024-09-11T00:33:17.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.690+0000 7fc888c00640 1 -- 172.21.15.31:0/507934670 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc878005310 con 0x7fc88c157a70 2024-09-11T00:33:17.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.690+0000 7fc888c00640 1 -- 172.21.15.31:0/507934670 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8780047a0 con 0x7fc88c157a70 2024-09-11T00:33:17.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.690+0000 7fc893600640 1 -- 172.21.15.31:0/507934670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c1e0380 con 0x7fc88c157a70 2024-09-11T00:33:17.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.690+0000 7fc893600640 1 -- 172.21.15.31:0/507934670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc88c157740 con 0x7fc88c157a70 2024-09-11T00:33:17.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.691+0000 7fc893600640 1 -- 172.21.15.31:0/507934670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc88c1e0170 con 0x7fc88c157a70 2024-09-11T00:33:17.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.692+0000 7fc888c00640 1 -- 172.21.15.31:0/507934670 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc87804c030 con 0x7fc88c157a70 2024-09-11T00:33:17.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.693+0000 7fc888c00640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc86c077f70 0x7fc86c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:17.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.694+0000 7fc888c00640 1 -- 172.21.15.31:0/507934670 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fc8780bb150 con 0x7fc88c157a70 2024-09-11T00:33:17.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.694+0000 7fc890c00640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc86c077f70 0x7fc86c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:17.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.695+0000 7fc890c00640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc86c077f70 0x7fc86c07a430 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc880007fd0 tx=0x7fc880009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:17.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.699+0000 7fc888c00640 1 -- 172.21.15.31:0/507934670 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc878083130 con 0x7fc88c157a70 2024-09-11T00:33:17.738 INFO:tasks.workunit.client.0.smithi031.stdout:checking for srand48_r... yes 2024-09-11T00:33:17.785 INFO:tasks.workunit.client.0.smithi031.stdout:configure: creating ./config.status 2024-09-11T00:33:17.990 INFO:tasks.workunit.client.0.smithi031.stdout: /bin/sh ./config.status 2024-09-11T00:33:17.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.996+0000 7fc893600640 1 -- 172.21.15.31:0/507934670 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc88c157f20 con 0x7fc88c157a70 2024-09-11T00:33:17.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.997+0000 7fc888c00640 1 -- 172.21.15.31:0/507934670 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6535 (secure 0 0 0) 0x7fc878087b60 con 0x7fc88c157a70 2024-09-11T00:33:18.000 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:18.000 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"btime":"2024-09-11T00:33:01:330912+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-09-11T00:33:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.999+0000 7fc893600640 1 -- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc86c077f70 msgr2=0x7fc86c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:17.999+0000 7fc893600640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc86c077f70 0x7fc86c07a430 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc880007fd0 tx=0x7fc880009040 comp rx=0 tx=0).stop 2024-09-11T00:33:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:18.000+0000 7fc893600640 1 -- 172.21.15.31:0/507934670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c157a70 msgr2=0x7fc88c0c3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:18.000+0000 7fc893600640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c157a70 0x7fc88c0c3190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc878009780 tx=0x7fc878032b90 comp rx=0 tx=0).stop 2024-09-11T00:33:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:18.000+0000 7fc893600640 1 -- 172.21.15.31:0/507934670 shutdown_connections 2024-09-11T00:33:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:18.000+0000 7fc893600640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc86c077f70 0x7fc86c07a430 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:18.000+0000 7fc893600640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc88c14fd70 0x7fc88c150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:18.000+0000 7fc893600640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc88c157a70 0x7fc88c0c3190 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:18.000+0000 7fc893600640 1 --2- 172.21.15.31:0/507934670 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc88c0c44f0 0x7fc88c150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:18.000+0000 7fc893600640 1 -- 172.21.15.31:0/507934670 >> 172.21.15.31:0/507934670 conn(0x7fc88c0ab8a0 msgr2=0x7fc88c0c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:18.000+0000 7fc893600640 1 -- 172.21.15.31:0/507934670 shutdown_connections 2024-09-11T00:33:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:18.000+0000 7fc893600640 1 -- 172.21.15.31:0/507934670 wait complete. 2024-09-11T00:33:18.019 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:33:18.112 INFO:tasks.workunit.client.0.smithi031.stdout:config.status: creating Makefile 2024-09-11T00:33:18.147 INFO:tasks.workunit.client.0.smithi031.stdout:config.status: creating config.h 2024-09-11T00:33:18.197 INFO:tasks.workunit.client.0.smithi031.stdout:config.status: executing depfiles commands 2024-09-11T00:33:18.242 INFO:tasks.workunit.client.0.smithi031.stdout:config.status: executing default commands 2024-09-11T00:33:18.320 INFO:tasks.workunit.client.0.smithi031.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-09-11T00:33:18.577 INFO:tasks.workunit.client.0.smithi031.stdout:rm -f stamp-h1 2024-09-11T00:33:18.581 INFO:tasks.workunit.client.0.smithi031.stdout:touch config.h.in 2024-09-11T00:33:18.585 INFO:tasks.workunit.client.0.smithi031.stdout:cd . && /bin/sh ./config.status config.h 2024-09-11T00:33:18.689 INFO:tasks.workunit.client.0.smithi031.stdout:config.status: creating config.h 2024-09-11T00:33:18.748 INFO:tasks.workunit.client.0.smithi031.stdout:make all-am 2024-09-11T00:33:18.792 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-11T00:33:18.792 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-09-11T00:33:19.073 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-09-11T00:33:19.082 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-09-11T00:33:19.245 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-09-11T00:33:19.253 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-09-11T00:33:19.418 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-09-11T00:33:19.427 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-09-11T00:33:19.588 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-09-11T00:33:19.596 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-09-11T00:33:19.795 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-09-11T00:33:19.803 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-09-11T00:33:19.918 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-09-11T00:33:19.926 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-09-11T00:33:20.011 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-09-11T00:33:20.019 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-09-11T00:33:20.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:20.124 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-09-11T00:33:20.132 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-09-11T00:33:20.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.188+0000 7fdec1c00640 1 -- 172.21.15.31:0/4011162992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdea0004d40 msgr2=0x7fdea00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:20.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.188+0000 7fdec1c00640 1 --2- 172.21.15.31:0/4011162992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdea0004d40 0x7fdea00051c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4009f00 tx=0x7fdea0005830 comp rx=0 tx=0).stop 2024-09-11T00:33:20.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.188+0000 7fdec1200640 1 -- 172.21.15.31:0/4011162992 reap_dead start 2024-09-11T00:33:20.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.188+0000 7fdec1c00640 1 -- 172.21.15.31:0/4011162992 shutdown_connections 2024-09-11T00:33:20.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.188+0000 7fdec1c00640 1 -- 172.21.15.31:0/4011162992 >> 172.21.15.31:0/4011162992 conn(0x7fdebc0ab800 msgr2=0x7fdebc0abbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:20.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.188+0000 7fdec1c00640 1 -- 172.21.15.31:0/4011162992 shutdown_connections 2024-09-11T00:33:20.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.188+0000 7fdec1c00640 1 -- 172.21.15.31:0/4011162992 wait complete. 2024-09-11T00:33:20.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.190+0000 7fdec1c00640 1 Processor -- start 2024-09-11T00:33:20.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.190+0000 7fdec1c00640 1 -- start start 2024-09-11T00:33:20.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdea0004d40 0x7fdebc1e0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdebc157680 0x7fdebc14a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdebc14aae0 0x7fdebc1ed370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc075150 con 0x7fdebc14aae0 2024-09-11T00:33:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc074fd0 con 0x7fdea0004d40 2024-09-11T00:33:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdebc074800 con 0x7fdebc157680 2024-09-11T00:33:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdebc14aae0 0x7fdebc1ed370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdebc14aae0 0x7fdebc1ed370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46954/0 (socket says 172.21.15.31:46954) 2024-09-11T00:33:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1200640 1 -- 172.21.15.31:0/3463619808 learned_addr learned my addr 172.21.15.31:0/3463619808 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdebbe00640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdea0004d40 0x7fdebc1e0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:20.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdebb400640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdebc157680 0x7fdebc14a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1200640 1 -- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdebc157680 msgr2=0x7fdebc14a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1200640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdebc157680 0x7fdebc14a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1200640 1 -- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdea0004d40 msgr2=0x7fdebc1e0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1200640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdea0004d40 0x7fdebc1e0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.191+0000 7fdec1200640 1 -- 172.21.15.31:0/3463619808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc158050 con 0x7fdebc14aae0 2024-09-11T00:33:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.192+0000 7fdec1200640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdebc14aae0 0x7fdebc1ed370 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400e380 tx=0x7fdeb4033d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.192+0000 7fdeb3e00640 1 -- 172.21.15.31:0/3463619808 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4042070 con 0x7fdebc14aae0 2024-09-11T00:33:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.192+0000 7fdeb3e00640 1 -- 172.21.15.31:0/3463619808 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdeb4003c60 con 0x7fdebc14aae0 2024-09-11T00:33:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.192+0000 7fdeb3e00640 1 -- 172.21.15.31:0/3463619808 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4038040 con 0x7fdebc14aae0 2024-09-11T00:33:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.192+0000 7fdec1c00640 1 -- 172.21.15.31:0/3463619808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdebc0af930 con 0x7fdebc14aae0 2024-09-11T00:33:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.192+0000 7fdec1c00640 1 -- 172.21.15.31:0/3463619808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdebc1e0d50 con 0x7fdebc14aae0 2024-09-11T00:33:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.193+0000 7fdec1c00640 1 -- 172.21.15.31:0/3463619808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdebc153e70 con 0x7fdebc14aae0 2024-09-11T00:33:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.194+0000 7fdeb3e00640 1 -- 172.21.15.31:0/3463619808 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdeb403b070 con 0x7fdebc14aae0 2024-09-11T00:33:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.195+0000 7fdeb3e00640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde9c077fa0 0x7fde9c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.195+0000 7fdeb3e00640 1 -- 172.21.15.31:0/3463619808 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fdeb40ba170 con 0x7fdebc14aae0 2024-09-11T00:33:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.196+0000 7fdebbe00640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde9c077fa0 0x7fde9c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.197+0000 7fdebbe00640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde9c077fa0 0x7fde9c07a460 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdeac001010 tx=0x7fdeac006720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:20.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.200+0000 7fdeb3e00640 1 -- 172.21.15.31:0/3463619808 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdeb4003e50 con 0x7fdebc14aae0 2024-09-11T00:33:20.412 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-09-11T00:33:20.421 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-09-11T00:33:20.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.441+0000 7fdec1c00640 1 -- 172.21.15.31:0/3463619808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdebc0c7aa0 con 0x7fdebc14aae0 2024-09-11T00:33:20.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.442+0000 7fdeb3e00640 1 -- 172.21.15.31:0/3463619808 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6535 (secure 0 0 0) 0x7fdeb40820c0 con 0x7fdebc14aae0 2024-09-11T00:33:20.445 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:20.445 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"btime":"2024-09-11T00:33:01:330912+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:20.445 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-09-11T00:33:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 -- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde9c077fa0 msgr2=0x7fde9c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde9c077fa0 0x7fde9c07a460 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdeac001010 tx=0x7fdeac006720 comp rx=0 tx=0).stop 2024-09-11T00:33:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 -- 172.21.15.31:0/3463619808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdebc14aae0 msgr2=0x7fdebc1ed370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdebc14aae0 0x7fdebc1ed370 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400e380 tx=0x7fdeb4033d50 comp rx=0 tx=0).stop 2024-09-11T00:33:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 -- 172.21.15.31:0/3463619808 shutdown_connections 2024-09-11T00:33:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde9c077fa0 0x7fde9c07a460 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdebc14aae0 0x7fdebc1ed370 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdebc157680 0x7fdebc14a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 --2- 172.21.15.31:0/3463619808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdea0004d40 0x7fdebc1e0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:20.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 -- 172.21.15.31:0/3463619808 >> 172.21.15.31:0/3463619808 conn(0x7fdebc0ab800 msgr2=0x7fdebc0c6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 -- 172.21.15.31:0/3463619808 shutdown_connections 2024-09-11T00:33:20.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:20.445+0000 7fdec1c00640 1 -- 172.21.15.31:0/3463619808 wait complete. 2024-09-11T00:33:20.465 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:33:20.639 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-09-11T00:33:20.647 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-09-11T00:33:20.877 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-09-11T00:33:20.885 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-09-11T00:33:21.015 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-09-11T00:33:21.023 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-09-11T00:33:21.159 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-09-11T00:33:21.168 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-09-11T00:33:21.356 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-09-11T00:33:21.364 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-09-11T00:33:21.441 INFO:tasks.workunit.client.0.smithi031.stderr:In file included from parser.h:22, 2024-09-11T00:33:21.441 INFO:tasks.workunit.client.0.smithi031.stderr: from parser.c:25: 2024-09-11T00:33:21.441 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c: In function ‘init_filesys’: 2024-09-11T00:33:21.444 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:21.444 INFO:tasks.workunit.client.0.smithi031.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:21.444 INFO:tasks.workunit.client.0.smithi031.stderr: | ^ 2024-09-11T00:33:21.444 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:21.444 INFO:tasks.workunit.client.0.smithi031.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:21.444 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~ 2024-09-11T00:33:21.444 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr: | ^ 2024-09-11T00:33:21.445 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~ 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-11T00:33:21.446 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr: | ^ 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~ 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:21.447 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr: | ^ 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~ 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:21.448 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c: In function ‘init_tg_stats’: 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr: | ^ 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:21.449 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~ 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr: | ^ 2024-09-11T00:33:21.450 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~ 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:21.451 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr: | ^ 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~ 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-11T00:33:21.452 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:21.453 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-11T00:33:21.453 INFO:tasks.workunit.client.0.smithi031.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-11T00:33:21.453 INFO:tasks.workunit.client.0.smithi031.stderr: | ^ 2024-09-11T00:33:21.453 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-09-11T00:33:21.453 INFO:tasks.workunit.client.0.smithi031.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-11T00:33:21.453 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~ 2024-09-11T00:33:21.453 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-09-11T00:33:21.453 INFO:tasks.workunit.client.0.smithi031.stderr: 36 | container_of(ptr, type, member) 2024-09-11T00:33:21.453 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~ 2024-09-11T00:33:21.454 INFO:tasks.workunit.client.0.smithi031.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-09-11T00:33:21.454 INFO:tasks.workunit.client.0.smithi031.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-11T00:33:21.454 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~ 2024-09-11T00:33:21.454 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-09-11T00:33:21.454 INFO:tasks.workunit.client.0.smithi031.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-11T00:33:21.454 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:21.496 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c: In function ‘get_optstr’: 2024-09-11T00:33:21.497 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-09-11T00:33:21.497 INFO:tasks.workunit.client.0.smithi031.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-09-11T00:33:21.497 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:21.497 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:200:14: note: source object declared here 2024-09-11T00:33:21.497 INFO:tasks.workunit.client.0.smithi031.stderr: 200 | char temp[BUFSIZE]; 2024-09-11T00:33:21.497 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~ 2024-09-11T00:33:21.503 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c: In function ‘size64_convert’: 2024-09-11T00:33:21.504 INFO:tasks.workunit.client.0.smithi031.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-09-11T00:33:21.504 INFO:tasks.workunit.client.0.smithi031.stderr: 119 | memcpy(unit, "\0", 3); 2024-09-11T00:33:21.504 INFO:tasks.workunit.client.0.smithi031.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-09-11T00:33:21.842 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-09-11T00:33:21.850 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-09-11T00:33:21.970 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-09-11T00:33:21.978 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-09-11T00:33:22.127 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-09-11T00:33:22.135 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-09-11T00:33:22.204 INFO:tasks.workunit.client.0.smithi031.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-09-11T00:33:22.211 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-09-11T00:33:22.403 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-11T00:33:22.404 INFO:tasks.workunit.client.0.smithi031.stderr:+ cd .. 2024-09-11T00:33:22.408 INFO:tasks.workunit.client.0.smithi031.stderr:+ mkdir tmp 2024-09-11T00:33:22.412 INFO:tasks.workunit.client.0.smithi031.stderr:+ cd tmp 2024-09-11T00:33:22.414 INFO:tasks.workunit.client.0.smithi031.stderr:+ for f in $mydir/*.ffsb 2024-09-11T00:33:22.414 INFO:tasks.workunit.client.0.smithi031.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-09-11T00:33:22.416 INFO:tasks.workunit.client.0.smithi031.stdout:FFSB version 6.0-RC2 started 2024-09-11T00:33:22.416 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:33:22.422 INFO:tasks.workunit.client.0.smithi031.stdout:benchmark time = 300 2024-09-11T00:33:22.422 INFO:tasks.workunit.client.0.smithi031.stdout:ThreadGroup 0 2024-09-11T00:33:22.422 INFO:tasks.workunit.client.0.smithi031.stdout:================ 2024-09-11T00:33:22.422 INFO:tasks.workunit.client.0.smithi031.stdout: num_threads = 32 2024-09-11T00:33:22.422 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:33:22.422 INFO:tasks.workunit.client.0.smithi031.stdout: read_random = off 2024-09-11T00:33:22.422 INFO:tasks.workunit.client.0.smithi031.stdout: read_size = 0 (0B) 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: read_blocksize = 0 (0B) 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: read_skip = off 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: read_skipsize = 0 (0B) 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: write_random = on 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: write_size = 5242880 (5MB) 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: fsync_file = 0 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: write_blocksize = 4096 (4KB) 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: wait time = 0 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: op weights 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: read = 0 (0.00%) 2024-09-11T00:33:22.423 INFO:tasks.workunit.client.0.smithi031.stdout: readall = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: write = 1 (100.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: create = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: append = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: delete = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: metaop = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: createdir = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: stat = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: writeall = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: writeall_fsync = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: open_close = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: write_fsync = 0 (0.00%) 2024-09-11T00:33:22.424 INFO:tasks.workunit.client.0.smithi031.stdout: create_fsync = 0 (0.00%) 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout: append_fsync = 0 (0.00%) 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout:FileSystem . 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout:========== 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout: num_dirs = 0 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout: starting files = 128 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout: min file size = 104857600 (100MB) 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout: max file size = 104857600 (100MB) 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout: directio = off 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout: alignedio = on 2024-09-11T00:33:22.425 INFO:tasks.workunit.client.0.smithi031.stdout: bufferedio = off 2024-09-11T00:33:22.426 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:33:22.426 INFO:tasks.workunit.client.0.smithi031.stdout: aging is off 2024-09-11T00:33:22.426 INFO:tasks.workunit.client.0.smithi031.stdout: current utilization = 0.00% 2024-09-11T00:33:22.426 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:33:22.426 INFO:tasks.workunit.client.0.smithi031.stdout:checking existing fs: . 2024-09-11T00:33:22.426 INFO:tasks.workunit.client.0.smithi031.stdout:recreating new fileset 2024-09-11T00:33:22.426 INFO:tasks.workunit.client.0.smithi031.stderr:opendir: No such file or directory 2024-09-11T00:33:22.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:22.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.667+0000 7fd1a3e00640 1 -- 172.21.15.31:0/3094521483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1a406fda0 msgr2=0x7fd1a406db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.667+0000 7fd1a3e00640 1 --2- 172.21.15.31:0/3094521483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1a406fda0 0x7fd1a406db00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd19c005fd0 tx=0x7fd19c02a710 comp rx=0 tx=0).stop 2024-09-11T00:33:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.668+0000 7fd1a3e00640 1 -- 172.21.15.31:0/3094521483 shutdown_connections 2024-09-11T00:33:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.668+0000 7fd1a3e00640 1 --2- 172.21.15.31:0/3094521483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1a406fda0 0x7fd1a406db00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.668+0000 7fd1a3e00640 1 --2- 172.21.15.31:0/3094521483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1a4071100 0x7fd1a40bd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.668+0000 7fd1a3e00640 1 -- 172.21.15.31:0/3094521483 >> 172.21.15.31:0/3094521483 conn(0x7fd1a40c4d20 msgr2=0x7fd1a40c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.668+0000 7fd1a3e00640 1 -- 172.21.15.31:0/3094521483 shutdown_connections 2024-09-11T00:33:22.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.668+0000 7fd1a3e00640 1 -- 172.21.15.31:0/3094521483 wait complete. 2024-09-11T00:33:22.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.669+0000 7fd1a3e00640 1 Processor -- start 2024-09-11T00:33:22.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.669+0000 7fd1a3e00640 1 -- start start 2024-09-11T00:33:22.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.670+0000 7fd1a3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1a406fda0 0x7fd1a4149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:22.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.670+0000 7fd1a3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a4071100 0x7fd1a414bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:22.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.670+0000 7fd1a3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1a414cb80 0x7fd1a41ebc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:22.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.670+0000 7fd1a3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a414d0e0 con 0x7fd1a4071100 2024-09-11T00:33:22.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.670+0000 7fd1a3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a414cf60 con 0x7fd1a406fda0 2024-09-11T00:33:22.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.670+0000 7fd1a3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd1a40701f0 con 0x7fd1a414cb80 2024-09-11T00:33:22.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.671+0000 7fd1a2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a4071100 0x7fd1a414bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.671+0000 7fd1a2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a4071100 0x7fd1a414bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46960/0 (socket says 172.21.15.31:46960) 2024-09-11T00:33:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.671+0000 7fd1a2000640 1 -- 172.21.15.31:0/1959428038 learned_addr learned my addr 172.21.15.31:0/1959428038 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.671+0000 7fd1a3400640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1a414cb80 0x7fd1a41ebc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.671+0000 7fd1a2a00640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1a406fda0 0x7fd1a4149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.671+0000 7fd1a2000640 1 -- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1a414cb80 msgr2=0x7fd1a41ebc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.671+0000 7fd1a2000640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1a414cb80 0x7fd1a41ebc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.671+0000 7fd1a2000640 1 -- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1a406fda0 msgr2=0x7fd1a4149ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.671+0000 7fd1a2000640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1a406fda0 0x7fd1a4149ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:22.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.671+0000 7fd1a2000640 1 -- 172.21.15.31:0/1959428038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a40adab0 con 0x7fd1a4071100 2024-09-11T00:33:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.672+0000 7fd1a2000640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a4071100 0x7fd1a414bb70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd19000e9f0 tx=0x7fd19000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.672+0000 7fd19aa00640 1 -- 172.21.15.31:0/1959428038 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19000cd80 con 0x7fd1a4071100 2024-09-11T00:33:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.672+0000 7fd19aa00640 1 -- 172.21.15.31:0/1959428038 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd190010430 con 0x7fd1a4071100 2024-09-11T00:33:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.672+0000 7fd1a3e00640 1 -- 172.21.15.31:0/1959428038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a4156fd0 con 0x7fd1a4071100 2024-09-11T00:33:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.672+0000 7fd1a3e00640 1 -- 172.21.15.31:0/1959428038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a40add50 con 0x7fd1a4071100 2024-09-11T00:33:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.672+0000 7fd19aa00640 1 -- 172.21.15.31:0/1959428038 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd190010840 con 0x7fd1a4071100 2024-09-11T00:33:22.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.673+0000 7fd1a3e00640 1 -- 172.21.15.31:0/1959428038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd17c005290 con 0x7fd1a4071100 2024-09-11T00:33:22.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.675+0000 7fd19aa00640 1 -- 172.21.15.31:0/1959428038 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd1900109e0 con 0x7fd1a4071100 2024-09-11T00:33:22.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.676+0000 7fd19aa00640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd184077fa0 0x7fd18407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:22.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.677+0000 7fd19aa00640 1 -- 172.21.15.31:0/1959428038 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fd19009ac50 con 0x7fd1a4071100 2024-09-11T00:33:22.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.677+0000 7fd1a2a00640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd184077fa0 0x7fd18407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:22.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.678+0000 7fd1a2a00640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd184077fa0 0x7fd18407a460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd194001220 tx=0x7fd194001c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:22.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:22.682+0000 7fd19aa00640 1 -- 172.21.15.31:0/1959428038 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd190062cf0 con 0x7fd1a4071100 2024-09-11T00:33:23.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.022+0000 7fd1a3e00640 1 -- 172.21.15.31:0/1959428038 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd17c005580 con 0x7fd1a4071100 2024-09-11T00:33:23.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.022+0000 7fd19aa00640 1 -- 172.21.15.31:0/1959428038 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6535 (secure 0 0 0) 0x7fd190067720 con 0x7fd1a4071100 2024-09-11T00:33:23.025 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:23.025 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"btime":"2024-09-11T00:33:01:330912+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:23.026 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-09-11T00:33:23.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.025+0000 7fd1a3e00640 1 -- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd184077fa0 msgr2=0x7fd18407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.025+0000 7fd1a3e00640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd184077fa0 0x7fd18407a460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd194001220 tx=0x7fd194001c90 comp rx=0 tx=0).stop 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.025+0000 7fd1a3e00640 1 -- 172.21.15.31:0/1959428038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a4071100 msgr2=0x7fd1a414bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.025+0000 7fd1a3e00640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a4071100 0x7fd1a414bb70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd19000e9f0 tx=0x7fd19000eeb0 comp rx=0 tx=0).stop 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.025+0000 7fd1a3e00640 1 -- 172.21.15.31:0/1959428038 shutdown_connections 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.025+0000 7fd1a3e00640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd184077fa0 0x7fd18407a460 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.025+0000 7fd1a3e00640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1a414cb80 0x7fd1a41ebc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.025+0000 7fd1a3e00640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a4071100 0x7fd1a414bb70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.025+0000 7fd1a3e00640 1 --2- 172.21.15.31:0/1959428038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1a406fda0 0x7fd1a4149ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.025+0000 7fd1a3e00640 1 -- 172.21.15.31:0/1959428038 >> 172.21.15.31:0/1959428038 conn(0x7fd1a40c4d20 msgr2=0x7fd1a40c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.026+0000 7fd1a3e00640 1 -- 172.21.15.31:0/1959428038 shutdown_connections 2024-09-11T00:33:23.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:23.026+0000 7fd1a3e00640 1 -- 172.21.15.31:0/1959428038 wait complete. 2024-09-11T00:33:23.045 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:33:25.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:25.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.207+0000 7fe238e00640 1 -- 172.21.15.31:0/1294206404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2340c7240 msgr2=0x7fe23406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:25.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.207+0000 7fe238e00640 1 --2- 172.21.15.31:0/1294206404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2340c7240 0x7fe23406dc60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe22400a940 tx=0x7fe22402b7b0 comp rx=0 tx=0).stop 2024-09-11T00:33:25.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.207+0000 7fe238e00640 1 -- 172.21.15.31:0/1294206404 shutdown_connections 2024-09-11T00:33:25.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.207+0000 7fe238e00640 1 --2- 172.21.15.31:0/1294206404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2340c7240 0x7fe23406dc60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:25.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.207+0000 7fe238e00640 1 -- 172.21.15.31:0/1294206404 >> 172.21.15.31:0/1294206404 conn(0x7fe23406caf0 msgr2=0x7fe23406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:25.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.207+0000 7fe238e00640 1 -- 172.21.15.31:0/1294206404 shutdown_connections 2024-09-11T00:33:25.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.207+0000 7fe238e00640 1 -- 172.21.15.31:0/1294206404 wait complete. 2024-09-11T00:33:25.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.209+0000 7fe238e00640 1 Processor -- start 2024-09-11T00:33:25.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.209+0000 7fe238e00640 1 -- start start 2024-09-11T00:33:25.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe238e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2340acaf0 0x7fe2340a8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:25.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe238e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2340c64b0 0x7fe234149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:25.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe238e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2340c7240 0x7fe2341f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:25.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe238e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe234155b90 con 0x7fe2340c64b0 2024-09-11T00:33:25.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe238e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe2340700f0 con 0x7fe2340acaf0 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe238e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe2340c4ad0 con 0x7fe2340c7240 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe232a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2340c64b0 0x7fe234149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe232a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2340c64b0 0x7fe234149010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46980/0 (socket says 172.21.15.31:46980) 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe232a00640 1 -- 172.21.15.31:0/2417400915 learned_addr learned my addr 172.21.15.31:0/2417400915 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe233400640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2340acaf0 0x7fe2340a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.210+0000 7fe233e00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2340c7240 0x7fe2341f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe232a00640 1 -- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2340c7240 msgr2=0x7fe2341f01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe232a00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2340c7240 0x7fe2341f01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe232a00640 1 -- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2340acaf0 msgr2=0x7fe2340a8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe232a00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2340acaf0 0x7fe2340a8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe232a00640 1 -- 172.21.15.31:0/2417400915 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe234148d30 con 0x7fe2340c64b0 2024-09-11T00:33:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe233e00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2340c7240 0x7fe2341f01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:33:25.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe232a00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2340c64b0 0x7fe234149010 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe22000c450 tx=0x7fe22000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:25.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe22b400640 1 -- 172.21.15.31:0/2417400915 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220016020 con 0x7fe2340c64b0 2024-09-11T00:33:25.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe22b400640 1 -- 172.21.15.31:0/2417400915 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe220012920 con 0x7fe2340c64b0 2024-09-11T00:33:25.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe22b400640 1 -- 172.21.15.31:0/2417400915 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220012d70 con 0x7fe2340c64b0 2024-09-11T00:33:25.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe238e00640 1 -- 172.21.15.31:0/2417400915 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2341ef9e0 con 0x7fe2340c64b0 2024-09-11T00:33:25.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.211+0000 7fe238e00640 1 -- 172.21.15.31:0/2417400915 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe234147370 con 0x7fe2340c64b0 2024-09-11T00:33:25.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.213+0000 7fe238e00640 1 -- 172.21.15.31:0/2417400915 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2340c7620 con 0x7fe2340c64b0 2024-09-11T00:33:25.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.214+0000 7fe22b400640 1 -- 172.21.15.31:0/2417400915 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe22001b400 con 0x7fe2340c64b0 2024-09-11T00:33:25.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.216+0000 7fe22b400640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe214077fa0 0x7fe21407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:25.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.216+0000 7fe22b400640 1 -- 172.21.15.31:0/2417400915 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fe22009a530 con 0x7fe2340c64b0 2024-09-11T00:33:25.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.216+0000 7fe233400640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe214077fa0 0x7fe21407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:25.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.217+0000 7fe233400640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe214077fa0 0x7fe21407a460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe23406f6f0 tx=0x7fe224004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:25.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.220+0000 7fe22b400640 1 -- 172.21.15.31:0/2417400915 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe220020020 con 0x7fe2340c64b0 2024-09-11T00:33:25.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.531+0000 7fe238e00640 1 -- 172.21.15.31:0/2417400915 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2341e2600 con 0x7fe2340c64b0 2024-09-11T00:33:25.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.532+0000 7fe22b400640 1 -- 172.21.15.31:0/2417400915 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6535 (secure 0 0 0) 0x7fe23406e520 con 0x7fe2340c64b0 2024-09-11T00:33:25.535 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:25.535 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"btime":"2024-09-11T00:33:01:330912+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:25.535 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-09-11T00:33:25.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 -- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe214077fa0 msgr2=0x7fe21407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:25.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe214077fa0 0x7fe21407a460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe23406f6f0 tx=0x7fe224004730 comp rx=0 tx=0).stop 2024-09-11T00:33:25.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 -- 172.21.15.31:0/2417400915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2340c64b0 msgr2=0x7fe234149010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:25.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2340c64b0 0x7fe234149010 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe22000c450 tx=0x7fe22000c910 comp rx=0 tx=0).stop 2024-09-11T00:33:25.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 -- 172.21.15.31:0/2417400915 shutdown_connections 2024-09-11T00:33:25.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe214077fa0 0x7fe21407a460 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:25.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2340c7240 0x7fe2341f01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:25.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2340c64b0 0x7fe234149010 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:25.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 --2- 172.21.15.31:0/2417400915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2340acaf0 0x7fe2340a8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:25.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 -- 172.21.15.31:0/2417400915 >> 172.21.15.31:0/2417400915 conn(0x7fe23406caf0 msgr2=0x7fe23414c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:25.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 -- 172.21.15.31:0/2417400915 shutdown_connections 2024-09-11T00:33:25.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:25.534+0000 7fe238e00640 1 -- 172.21.15.31:0/2417400915 wait complete. 2024-09-11T00:33:25.554 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:33:27.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:27.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.729+0000 7f2bbf200640 1 -- 172.21.15.31:0/1029900398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc0150020 msgr2=0x7f2bc00ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:27.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.729+0000 7f2bbf200640 1 --2- 172.21.15.31:0/1029900398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc0150020 0x7f2bc00ad4e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0005fd0 tx=0x7f2bb002a670 comp rx=0 tx=0).stop 2024-09-11T00:33:27.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.729+0000 7f2bbf200640 1 -- 172.21.15.31:0/1029900398 shutdown_connections 2024-09-11T00:33:27.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.729+0000 7f2bbf200640 1 --2- 172.21.15.31:0/1029900398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc0150020 0x7f2bc00ad4e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:27.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.729+0000 7f2bbf200640 1 -- 172.21.15.31:0/1029900398 >> 172.21.15.31:0/1029900398 conn(0x7f2bc0068f10 msgr2=0x7f2bc00cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:27.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.729+0000 7f2bbf200640 1 -- 172.21.15.31:0/1029900398 shutdown_connections 2024-09-11T00:33:27.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.729+0000 7f2bbf200640 1 -- 172.21.15.31:0/1029900398 wait complete. 2024-09-11T00:33:27.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.731+0000 7f2bbf200640 1 Processor -- start 2024-09-11T00:33:27.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.731+0000 7f2bbf200640 1 -- start start 2024-09-11T00:33:27.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.732+0000 7f2bbf200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bc006e850 0x7f2bc00bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.732+0000 7f2bbf200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bc00ada20 0x7f2bc01580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.732+0000 7f2bbf200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc0150020 0x7f2bc01ef570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.732+0000 7f2bbf200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc0157550 con 0x7f2bc0150020 2024-09-11T00:33:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.732+0000 7f2bbf200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc01573d0 con 0x7f2bc006e850 2024-09-11T00:33:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.732+0000 7f2bbf200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc0156960 con 0x7f2bc00ada20 2024-09-11T00:33:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.732+0000 7f2bbe800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc0150020 0x7f2bc01ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.732+0000 7f2bbe800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc0150020 0x7f2bc01ef570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47000/0 (socket says 172.21.15.31:47000) 2024-09-11T00:33:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.732+0000 7f2bbe800640 1 -- 172.21.15.31:0/3325104227 learned_addr learned my addr 172.21.15.31:0/3325104227 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bbde00640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bc006e850 0x7f2bc00bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:27.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bbd400640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bc00ada20 0x7f2bc01580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:27.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bbe800640 1 -- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bc00ada20 msgr2=0x7f2bc01580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:27.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bbe800640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bc00ada20 0x7f2bc01580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:27.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bbe800640 1 -- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bc006e850 msgr2=0x7f2bc00bf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:27.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bbe800640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bc006e850 0x7f2bc00bf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:27.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bbe800640 1 -- 172.21.15.31:0/3325104227 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc014ae90 con 0x7f2bc0150020 2024-09-11T00:33:27.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bbe800640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc0150020 0x7f2bc01ef570 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800c6c0 tx=0x7f2bb800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:27.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bb6000640 1 -- 172.21.15.31:0/3325104227 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8003b70 con 0x7f2bc0150020 2024-09-11T00:33:27.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bb6000640 1 -- 172.21.15.31:0/3325104227 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bb8003d30 con 0x7f2bc0150020 2024-09-11T00:33:27.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bbf200640 1 -- 172.21.15.31:0/3325104227 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc00c25f0 con 0x7f2bc0150020 2024-09-11T00:33:27.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.733+0000 7f2bb6000640 1 -- 172.21.15.31:0/3325104227 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8005330 con 0x7f2bc0150020 2024-09-11T00:33:27.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.734+0000 7f2bbf200640 1 -- 172.21.15.31:0/3325104227 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc00ad610 con 0x7f2bc0150020 2024-09-11T00:33:27.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.735+0000 7f2bbf200640 1 -- 172.21.15.31:0/3325104227 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bc00c23e0 con 0x7f2bc0150020 2024-09-11T00:33:27.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.737+0000 7f2bb6000640 1 -- 172.21.15.31:0/3325104227 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2bb80054d0 con 0x7f2bc0150020 2024-09-11T00:33:27.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.738+0000 7f2bb6000640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ba0077f70 0x7f2ba007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:27.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.738+0000 7f2bb6000640 1 -- 172.21.15.31:0/3325104227 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f2bb8099ff0 con 0x7f2bc0150020 2024-09-11T00:33:27.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.739+0000 7f2bbde00640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ba0077f70 0x7f2ba007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:27.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.741+0000 7f2bbde00640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ba0077f70 0x7f2ba007a430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0033000 tx=0x7f2bb0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:27.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:27.742+0000 7f2bb6000640 1 -- 172.21.15.31:0/3325104227 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2bb80057f0 con 0x7f2bc0150020 2024-09-11T00:33:28.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.048+0000 7f2bbf200640 1 -- 172.21.15.31:0/3325104227 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2bc00cc220 con 0x7f2bc0150020 2024-09-11T00:33:28.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.049+0000 7f2bb6000640 1 -- 172.21.15.31:0/3325104227 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6535 (secure 0 0 0) 0x7f2bb80625d0 con 0x7f2bc0150020 2024-09-11T00:33:28.051 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:28.052 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"btime":"2024-09-11T00:33:01:330912+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:28.052 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-09-11T00:33:28.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 -- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ba0077f70 msgr2=0x7f2ba007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:28.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ba0077f70 0x7f2ba007a430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0033000 tx=0x7f2bb0004730 comp rx=0 tx=0).stop 2024-09-11T00:33:28.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 -- 172.21.15.31:0/3325104227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc0150020 msgr2=0x7f2bc01ef570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:28.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc0150020 0x7f2bc01ef570 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800c6c0 tx=0x7f2bb800cb80 comp rx=0 tx=0).stop 2024-09-11T00:33:28.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 -- 172.21.15.31:0/3325104227 shutdown_connections 2024-09-11T00:33:28.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ba0077f70 0x7f2ba007a430 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:28.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bc0150020 0x7f2bc01ef570 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:28.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bc00ada20 0x7f2bc01580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:28.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 --2- 172.21.15.31:0/3325104227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bc006e850 0x7f2bc00bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:28.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 -- 172.21.15.31:0/3325104227 >> 172.21.15.31:0/3325104227 conn(0x7f2bc0068f10 msgr2=0x7f2bc0159100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:28.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 -- 172.21.15.31:0/3325104227 shutdown_connections 2024-09-11T00:33:28.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:28.052+0000 7f2bbf200640 1 -- 172.21.15.31:0/3325104227 wait complete. 2024-09-11T00:33:28.074 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:33:30.075 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:30.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.239+0000 7f08e2400640 1 -- 172.21.15.31:0/3849577913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08dc157ac0 msgr2=0x7f08dc151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.239+0000 7f08e2400640 1 --2- 172.21.15.31:0/3849577913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08dc157ac0 0x7f08dc151700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009830 tx=0x7f08c802a480 comp rx=0 tx=0).stop 2024-09-11T00:33:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.239+0000 7f08e2400640 1 -- 172.21.15.31:0/3849577913 shutdown_connections 2024-09-11T00:33:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.239+0000 7f08e2400640 1 --2- 172.21.15.31:0/3849577913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08dc157ac0 0x7f08dc151700 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.239+0000 7f08e2400640 1 --2- 172.21.15.31:0/3849577913 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08dc1576e0 0x7f08dc1511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.239+0000 7f08e2400640 1 -- 172.21.15.31:0/3849577913 >> 172.21.15.31:0/3849577913 conn(0x7f08dc0c8320 msgr2=0x7f08dc0c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.239+0000 7f08e2400640 1 -- 172.21.15.31:0/3849577913 shutdown_connections 2024-09-11T00:33:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.239+0000 7f08e2400640 1 -- 172.21.15.31:0/3849577913 wait complete. 2024-09-11T00:33:30.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.241+0000 7f08e2400640 1 Processor -- start 2024-09-11T00:33:30.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.242+0000 7f08e2400640 1 -- start start 2024-09-11T00:33:30.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08e2400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08dc06f820 0x7f08dc1dd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:30.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08e2400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08dc1576e0 0x7f08dc149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08e2400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08dc157ac0 0x7f08dc1ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08e2400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc074c20 con 0x7f08dc06f820 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08e2400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc0703c0 con 0x7f08dc157ac0 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08e2400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f08dc074da0 con 0x7f08dc1576e0 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08db400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08dc06f820 0x7f08dc1dd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08db400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08dc06f820 0x7f08dc1dd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35202/0 (socket says 172.21.15.31:35202) 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08db400640 1 -- 172.21.15.31:0/381314573 learned_addr learned my addr 172.21.15.31:0/381314573 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08daa00640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08dc1576e0 0x7f08dc149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.243+0000 7f08dbe00640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08dc157ac0 0x7f08dc1ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08db400640 1 -- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08dc1576e0 msgr2=0x7f08dc149500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08db400640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08dc1576e0 0x7f08dc149500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:30.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08db400640 1 -- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08dc157ac0 msgr2=0x7f08dc1ebb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:30.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08db400640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08dc157ac0 0x7f08dc1ebb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:30.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08db400640 1 -- 172.21.15.31:0/381314573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08dc1eff10 con 0x7f08dc06f820 2024-09-11T00:33:30.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08dbe00640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08dc157ac0 0x7f08dc1ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:33:30.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08db400640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08dc06f820 0x7f08dc1dd480 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00d980 tx=0x7f08cc00de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:30.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08d3400640 1 -- 172.21.15.31:0/381314573 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc017070 con 0x7f08dc06f820 2024-09-11T00:33:30.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08d3400640 1 -- 172.21.15.31:0/381314573 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08cc009d20 con 0x7f08dc06f820 2024-09-11T00:33:30.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08e2400640 1 -- 172.21.15.31:0/381314573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08dc1ef150 con 0x7f08dc06f820 2024-09-11T00:33:30.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08d3400640 1 -- 172.21.15.31:0/381314573 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc004280 con 0x7f08dc06f820 2024-09-11T00:33:30.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.244+0000 7f08e2400640 1 -- 172.21.15.31:0/381314573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08dc1ef710 con 0x7f08dc06f820 2024-09-11T00:33:30.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.245+0000 7f08e2400640 1 -- 172.21.15.31:0/381314573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08dc149b00 con 0x7f08dc06f820 2024-09-11T00:33:30.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.247+0000 7f08d3400640 1 -- 172.21.15.31:0/381314573 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f08cc0035f0 con 0x7f08dc06f820 2024-09-11T00:33:30.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.248+0000 7f08d3400640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f08bc077f70 0x7f08bc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:30.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.248+0000 7f08d3400640 1 -- 172.21.15.31:0/381314573 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f08cc09a290 con 0x7f08dc06f820 2024-09-11T00:33:30.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.249+0000 7f08daa00640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f08bc077f70 0x7f08bc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:30.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.250+0000 7f08daa00640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f08bc077f70 0x7f08bc07a430 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f08c8034000 tx=0x7f08c8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:30.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.253+0000 7f08d3400640 1 -- 172.21.15.31:0/381314573 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f08cc0622f0 con 0x7f08dc06f820 2024-09-11T00:33:30.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.534+0000 7f08e2400640 1 -- 172.21.15.31:0/381314573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08dc06e0a0 con 0x7f08dc06f820 2024-09-11T00:33:30.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.535+0000 7f08d3400640 1 -- 172.21.15.31:0/381314573 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6535 (secure 0 0 0) 0x7f08cc066d20 con 0x7f08dc06f820 2024-09-11T00:33:30.537 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:30.537 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"btime":"2024-09-11T00:33:01:330912+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:30.537 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-09-11T00:33:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 -- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f08bc077f70 msgr2=0x7f08bc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f08bc077f70 0x7f08bc07a430 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f08c8034000 tx=0x7f08c8004730 comp rx=0 tx=0).stop 2024-09-11T00:33:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 -- 172.21.15.31:0/381314573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08dc06f820 msgr2=0x7f08dc1dd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08dc06f820 0x7f08dc1dd480 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f08cc00d980 tx=0x7f08cc00de40 comp rx=0 tx=0).stop 2024-09-11T00:33:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 -- 172.21.15.31:0/381314573 shutdown_connections 2024-09-11T00:33:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f08bc077f70 0x7f08bc07a430 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f08dc157ac0 0x7f08dc1ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f08dc1576e0 0x7f08dc149500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 --2- 172.21.15.31:0/381314573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08dc06f820 0x7f08dc1dd480 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 -- 172.21.15.31:0/381314573 >> 172.21.15.31:0/381314573 conn(0x7f08dc0c8320 msgr2=0x7f08dc0738a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 -- 172.21.15.31:0/381314573 shutdown_connections 2024-09-11T00:33:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:30.538+0000 7f08e2400640 1 -- 172.21.15.31:0/381314573 wait complete. 2024-09-11T00:33:30.559 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:33:32.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.731+0000 7f05ca600640 1 -- 172.21.15.31:0/1674584012 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05c4157c40 msgr2=0x7f05c415e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.731+0000 7f05ca600640 1 --2- 172.21.15.31:0/1674584012 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05c4157c40 0x7f05c415e8c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f05bc005fd0 tx=0x7f05bc02a810 comp rx=0 tx=0).stop 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.731+0000 7f05ca600640 1 -- 172.21.15.31:0/1674584012 shutdown_connections 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.731+0000 7f05ca600640 1 --2- 172.21.15.31:0/1674584012 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05c4157c40 0x7f05c415e8c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.731+0000 7f05ca600640 1 --2- 172.21.15.31:0/1674584012 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05c40c44f0 0x7f05c40c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.731+0000 7f05ca600640 1 -- 172.21.15.31:0/1674584012 >> 172.21.15.31:0/1674584012 conn(0x7f05c40ab8a0 msgr2=0x7f05c40abcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.731+0000 7f05ca600640 1 -- 172.21.15.31:0/1674584012 shutdown_connections 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.731+0000 7f05ca600640 1 -- 172.21.15.31:0/1674584012 wait complete. 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.733+0000 7f05ca600640 1 Processor -- start 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.734+0000 7f05ca600640 1 -- start start 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05ca600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05c40c44f0 0x7f05c41e01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:32.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05ca600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c40c4e10 0x7f05c4150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05ca600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05c4157c40 0x7f05c4156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05ca600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f05c4151af0 con 0x7f05c40c4e10 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05ca600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f05c40c54e0 con 0x7f05c40c44f0 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05ca600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f05c40701b0 con 0x7f05c4157c40 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c40c4e10 0x7f05c4150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05c2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c40c4e10 0x7f05c4150cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35210/0 (socket says 172.21.15.31:35210) 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05c2a00640 1 -- 172.21.15.31:0/1765859395 learned_addr learned my addr 172.21.15.31:0/1765859395 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05c3e00640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05c4157c40 0x7f05c4156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.735+0000 7f05c3400640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05c40c44f0 0x7f05c41e01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05c2a00640 1 -- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05c4157c40 msgr2=0x7f05c4156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05c2a00640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05c4157c40 0x7f05c4156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05c2a00640 1 -- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05c40c44f0 msgr2=0x7f05c41e01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:32.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05c2a00640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05c40c44f0 0x7f05c41e01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:32.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05c2a00640 1 -- 172.21.15.31:0/1765859395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c41509d0 con 0x7f05c40c4e10 2024-09-11T00:33:32.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05c2a00640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c40c4e10 0x7f05c4150cb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f05b000c460 tx=0x7f05b000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:32.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05bb400640 1 -- 172.21.15.31:0/1765859395 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0016020 con 0x7f05c40c4e10 2024-09-11T00:33:32.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05bb400640 1 -- 172.21.15.31:0/1765859395 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05b0012920 con 0x7f05c40c4e10 2024-09-11T00:33:32.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05bb400640 1 -- 172.21.15.31:0/1765859395 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0012d50 con 0x7f05c40c4e10 2024-09-11T00:33:32.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05ca600640 1 -- 172.21.15.31:0/1765859395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c406f0d0 con 0x7f05c40c4e10 2024-09-11T00:33:32.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.736+0000 7f05ca600640 1 -- 172.21.15.31:0/1765859395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c41570d0 con 0x7f05c40c4e10 2024-09-11T00:33:32.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.738+0000 7f05ca600640 1 -- 172.21.15.31:0/1765859395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c41580f0 con 0x7f05c40c4e10 2024-09-11T00:33:32.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.739+0000 7f05bb400640 1 -- 172.21.15.31:0/1765859395 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f05b001a3b0 con 0x7f05c40c4e10 2024-09-11T00:33:32.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.740+0000 7f05bb400640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f05a4077f70 0x7f05a407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:32.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.740+0000 7f05bb400640 1 -- 172.21.15.31:0/1765859395 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f05b009a080 con 0x7f05c40c4e10 2024-09-11T00:33:32.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.740+0000 7f05c3400640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f05a4077f70 0x7f05a407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:32.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.741+0000 7f05c3400640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f05a4077f70 0x7f05a407a430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009bd0 tx=0x7f05b4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:32.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:32.745+0000 7f05bb400640 1 -- 172.21.15.31:0/1765859395 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f05b0020030 con 0x7f05c40c4e10 2024-09-11T00:33:33.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.030+0000 7f05ca600640 1 -- 172.21.15.31:0/1765859395 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05c41e2310 con 0x7f05c40c4e10 2024-09-11T00:33:33.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.031+0000 7f05bb400640 1 -- 172.21.15.31:0/1765859395 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6535 (secure 0 0 0) 0x7f05b0062120 con 0x7f05c40c4e10 2024-09-11T00:33:33.033 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:33.033 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"btime":"2024-09-11T00:33:01:330912+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:32:27.304525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4721":{"gid":4721,"name":"b","rank":0,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/2954547253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2954547253},{"type":"v1","addr":"172.21.15.204:6837","nonce":2954547253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4721,"qdb_cluster":[4721]},"id":2}]} 2024-09-11T00:33:33.033 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2024-09-11T00:33:33.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.033+0000 7f05ca600640 1 -- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f05a4077f70 msgr2=0x7f05a407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:33.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.033+0000 7f05ca600640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f05a4077f70 0x7f05a407a430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009bd0 tx=0x7f05b4005f90 comp rx=0 tx=0).stop 2024-09-11T00:33:33.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.033+0000 7f05ca600640 1 -- 172.21.15.31:0/1765859395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c40c4e10 msgr2=0x7f05c4150cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:33.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.033+0000 7f05ca600640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c40c4e10 0x7f05c4150cb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f05b000c460 tx=0x7f05b000c920 comp rx=0 tx=0).stop 2024-09-11T00:33:33.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.033+0000 7f05ca600640 1 -- 172.21.15.31:0/1765859395 shutdown_connections 2024-09-11T00:33:33.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.033+0000 7f05ca600640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f05a4077f70 0x7f05a407a430 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:33.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.033+0000 7f05ca600640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05c4157c40 0x7f05c4156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:33.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.033+0000 7f05ca600640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05c40c4e10 0x7f05c4150cb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:33.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.033+0000 7f05ca600640 1 --2- 172.21.15.31:0/1765859395 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05c40c44f0 0x7f05c41e01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:33.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.033+0000 7f05ca600640 1 -- 172.21.15.31:0/1765859395 >> 172.21.15.31:0/1765859395 conn(0x7f05c40ab8a0 msgr2=0x7f05c40c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:33.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.034+0000 7f05ca600640 1 -- 172.21.15.31:0/1765859395 shutdown_connections 2024-09-11T00:33:33.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:33.034+0000 7f05ca600640 1 -- 172.21.15.31:0/1765859395 wait complete. 2024-09-11T00:33:33.052 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:33:35.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.234+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2462994771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb080bf240 msgr2=0x7fbb080b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.234+0000 7fbb0ec00640 1 --2- 172.21.15.31:0/2462994771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb080bf240 0x7fbb080b0be0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40098e0 tx=0x7fbaf402a540 comp rx=0 tx=0).stop 2024-09-11T00:33:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.234+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2462994771 shutdown_connections 2024-09-11T00:33:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.234+0000 7fbb0ec00640 1 --2- 172.21.15.31:0/2462994771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb080bf240 0x7fbb080b0be0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.234+0000 7fbb0ec00640 1 --2- 172.21.15.31:0/2462994771 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb080c6730 0x7fbb081596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.234+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2462994771 >> 172.21.15.31:0/2462994771 conn(0x7fbb08068ac0 msgr2=0x7fbb08068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.234+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2462994771 shutdown_connections 2024-09-11T00:33:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.235+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2462994771 wait complete. 2024-09-11T00:33:35.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.236+0000 7fbb0ec00640 1 Processor -- start 2024-09-11T00:33:35.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.236+0000 7fbb0ec00640 1 -- start start 2024-09-11T00:33:35.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.237+0000 7fbb0ec00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb080bf240 0x7fbb080c5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:35.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.237+0000 7fbb0ec00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb080c6730 0x7fbb081e1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:35.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.237+0000 7fbb0ec00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb08156e40 0x7fbb081e1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:35.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ec00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbb080c6c90 con 0x7fbb080c6730 2024-09-11T00:33:35.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ec00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbb080c6b10 con 0x7fbb08156e40 2024-09-11T00:33:35.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ec00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbb0806e850 con 0x7fbb080bf240 2024-09-11T00:33:35.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb080c6730 0x7fbb081e1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:35.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb080c6730 0x7fbb081e1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35222/0 (socket says 172.21.15.31:35222) 2024-09-11T00:33:35.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ce00640 1 -- 172.21.15.31:0/2613272478 learned_addr learned my addr 172.21.15.31:0/2613272478 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:35.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0d800640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb080bf240 0x7fbb080c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:35.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0e200640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb08156e40 0x7fbb081e1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:35.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ce00640 1 -- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb080bf240 msgr2=0x7fbb080c5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:35.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ce00640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb080bf240 0x7fbb080c5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ce00640 1 -- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb08156e40 msgr2=0x7fbb081e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:35.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ce00640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb08156e40 0x7fbb081e1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0ce00640 1 -- 172.21.15.31:0/2613272478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0814cae0 con 0x7fbb080c6730 2024-09-11T00:33:35.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.238+0000 7fbb0e200640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb08156e40 0x7fbb081e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:33:35.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.239+0000 7fbb0ce00640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb080c6730 0x7fbb081e1990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4009a00 tx=0x7fbaf40052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:35.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.239+0000 7fbb05600640 1 -- 172.21.15.31:0/2613272478 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4005cf0 con 0x7fbb080c6730 2024-09-11T00:33:35.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.239+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2613272478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb081df7e0 con 0x7fbb080c6730 2024-09-11T00:33:35.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.239+0000 7fbb05600640 1 -- 172.21.15.31:0/2613272478 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbaf4004340 con 0x7fbb080c6730 2024-09-11T00:33:35.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.239+0000 7fbb05600640 1 -- 172.21.15.31:0/2613272478 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4004a00 con 0x7fbb080c6730 2024-09-11T00:33:35.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.239+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2613272478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb081dfda0 con 0x7fbb080c6730 2024-09-11T00:33:35.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.240+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2613272478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb080c9a40 con 0x7fbb080c6730 2024-09-11T00:33:35.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.241+0000 7fbb05600640 1 -- 172.21.15.31:0/2613272478 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbaf404c030 con 0x7fbb080c6730 2024-09-11T00:33:35.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.242+0000 7fbb05600640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbae8077f70 0x7fbae807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:35.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.242+0000 7fbb05600640 1 -- 172.21.15.31:0/2613272478 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fbaf40bb470 con 0x7fbb080c6730 2024-09-11T00:33:35.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.243+0000 7fbb0d800640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbae8077f70 0x7fbae807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:35.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.244+0000 7fbb0d800640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbae8077f70 0x7fbae807a430 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbb080aeb90 tx=0x7fbafc005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:35.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.248+0000 7fbb05600640 1 -- 172.21.15.31:0/2613272478 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbaf4040030 con 0x7fbb080c6730 2024-09-11T00:33:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.535+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2613272478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb080ae8e0 con 0x7fbb080c6730 2024-09-11T00:33:35.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.536+0000 7fbb05600640 1 -- 172.21.15.31:0/2613272478 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5699 (secure 0 0 0) 0x7fbaf4083a40 con 0x7fbb080c6730 2024-09-11T00:33:35.538 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:35.538 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"btime":"2024-09-11T00:33:34:297550+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:34.297543+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:33:35.539 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2024-09-11T00:33:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbae8077f70 msgr2=0x7fbae807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbae8077f70 0x7fbae807a430 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbb080aeb90 tx=0x7fbafc005f90 comp rx=0 tx=0).stop 2024-09-11T00:33:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2613272478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb080c6730 msgr2=0x7fbb081e1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb080c6730 0x7fbb081e1990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4009a00 tx=0x7fbaf40052f0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2613272478 shutdown_connections 2024-09-11T00:33:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbae8077f70 0x7fbae807a430 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb08156e40 0x7fbb081e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb080c6730 0x7fbb081e1990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 --2- 172.21.15.31:0/2613272478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb080bf240 0x7fbb080c5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2613272478 >> 172.21.15.31:0/2613272478 conn(0x7fbb08068ac0 msgr2=0x7fbb08153f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2613272478 shutdown_connections 2024-09-11T00:33:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.539+0000 7fbb0ec00640 1 -- 172.21.15.31:0/2613272478 wait complete. 2024-09-11T00:33:35.558 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-11T00:33:35.558 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:33:35.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.723+0000 7fda93800640 1 -- 172.21.15.31:0/3727748319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda8c0c9b40 msgr2=0x7fda8c156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.723+0000 7fda93800640 1 --2- 172.21.15.31:0/3727748319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda8c0c9b40 0x7fda8c156ae0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fda78005fd0 tx=0x7fda7802a530 comp rx=0 tx=0).stop 2024-09-11T00:33:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.724+0000 7fda93800640 1 -- 172.21.15.31:0/3727748319 shutdown_connections 2024-09-11T00:33:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.724+0000 7fda93800640 1 --2- 172.21.15.31:0/3727748319 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda8c159650 0x7fda8c159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.724+0000 7fda93800640 1 --2- 172.21.15.31:0/3727748319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda8c0c9b40 0x7fda8c156ae0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.724+0000 7fda93800640 1 -- 172.21.15.31:0/3727748319 >> 172.21.15.31:0/3727748319 conn(0x7fda8c068f10 msgr2=0x7fda8c0cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.724+0000 7fda93800640 1 -- 172.21.15.31:0/3727748319 shutdown_connections 2024-09-11T00:33:35.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.724+0000 7fda93800640 1 -- 172.21.15.31:0/3727748319 wait complete. 2024-09-11T00:33:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.726+0000 7fda93800640 1 Processor -- start 2024-09-11T00:33:35.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.726+0000 7fda93800640 1 -- start start 2024-09-11T00:33:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda93800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8c0c9b40 0x7fda8c1ec1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda93800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda8c1534f0 0x7fda8c1eb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda93800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda8c159650 0x7fda8c1eb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda93800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c0cca60 con 0x7fda8c1534f0 2024-09-11T00:33:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda93800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c1538d0 con 0x7fda8c0c9b40 2024-09-11T00:33:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda93800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fda8c1e00d0 con 0x7fda8c159650 2024-09-11T00:33:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda8be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda8c1534f0 0x7fda8c1eb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda8be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda8c1534f0 0x7fda8c1eb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35254/0 (socket says 172.21.15.31:35254) 2024-09-11T00:33:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda8be00640 1 -- 172.21.15.31:0/1481718968 learned_addr learned my addr 172.21.15.31:0/1481718968 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:35.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda90e00640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8c0c9b40 0x7fda8c1ec1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda91800640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda8c159650 0x7fda8c1eb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda8be00640 1 -- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda8c159650 msgr2=0x7fda8c1eb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda8be00640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda8c159650 0x7fda8c1eb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda8be00640 1 -- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8c0c9b40 msgr2=0x7fda8c1ec1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda8be00640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8c0c9b40 0x7fda8c1ec1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda8be00640 1 -- 172.21.15.31:0/1481718968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c1e9ed0 con 0x7fda8c1534f0 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.727+0000 7fda90e00640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8c0c9b40 0x7fda8c1ec1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.728+0000 7fda8be00640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda8c1534f0 0x7fda8c1eb3c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fda78006010 tx=0x7fda780039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.728+0000 7fda88c00640 1 -- 172.21.15.31:0/1481718968 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7803f070 con 0x7fda8c1534f0 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.728+0000 7fda88c00640 1 -- 172.21.15.31:0/1481718968 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda78004890 con 0x7fda8c1534f0 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.728+0000 7fda88c00640 1 -- 172.21.15.31:0/1481718968 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda78035030 con 0x7fda8c1534f0 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.728+0000 7fda93800640 1 -- 172.21.15.31:0/1481718968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8c1591d0 con 0x7fda8c1534f0 2024-09-11T00:33:35.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.728+0000 7fda93800640 1 -- 172.21.15.31:0/1481718968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8c1e1d50 con 0x7fda8c1534f0 2024-09-11T00:33:35.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.729+0000 7fda93800640 1 -- 172.21.15.31:0/1481718968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda8c156ae0 con 0x7fda8c1534f0 2024-09-11T00:33:35.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.731+0000 7fda88c00640 1 -- 172.21.15.31:0/1481718968 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fda78004c80 con 0x7fda8c1534f0 2024-09-11T00:33:35.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.732+0000 7fda88c00640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda6c077fa0 0x7fda6c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:35.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.733+0000 7fda88c00640 1 -- 172.21.15.31:0/1481718968 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fda780bb4d0 con 0x7fda8c1534f0 2024-09-11T00:33:35.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.733+0000 7fda90e00640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda6c077fa0 0x7fda6c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:35.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.734+0000 7fda90e00640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda6c077fa0 0x7fda6c07a460 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fda8c151980 tx=0x7fda80009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:35.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.737+0000 7fda88c00640 1 -- 172.21.15.31:0/1481718968 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fda78083aa0 con 0x7fda8c1534f0 2024-09-11T00:33:35.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.993+0000 7fda93800640 1 -- 172.21.15.31:0/1481718968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda8c14fc90 con 0x7fda8c1534f0 2024-09-11T00:33:35.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.994+0000 7fda88c00640 1 -- 172.21.15.31:0/1481718968 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+5703 (secure 0 0 0) 0x7fda7808c1c0 con 0x7fda8c1534f0 2024-09-11T00:33:35.996 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:35.996 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":33,"btime":"2024-09-11T00:33:35:539881+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:34.621041+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:33:35.996 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 33 2024-09-11T00:33:35.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 -- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda6c077fa0 msgr2=0x7fda6c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:35.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda6c077fa0 0x7fda6c07a460 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fda8c151980 tx=0x7fda80009040 comp rx=0 tx=0).stop 2024-09-11T00:33:35.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 -- 172.21.15.31:0/1481718968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda8c1534f0 msgr2=0x7fda8c1eb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:35.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda8c1534f0 0x7fda8c1eb3c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fda78006010 tx=0x7fda780039e0 comp rx=0 tx=0).stop 2024-09-11T00:33:35.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 -- 172.21.15.31:0/1481718968 shutdown_connections 2024-09-11T00:33:35.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fda6c077fa0 0x7fda6c07a460 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:36.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fda8c159650 0x7fda8c1eb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:36.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda8c1534f0 0x7fda8c1eb3c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:36.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 --2- 172.21.15.31:0/1481718968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda8c0c9b40 0x7fda8c1ec1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:36.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 -- 172.21.15.31:0/1481718968 >> 172.21.15.31:0/1481718968 conn(0x7fda8c068f10 msgr2=0x7fda8c156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:36.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 -- 172.21.15.31:0/1481718968 shutdown_connections 2024-09-11T00:33:36.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:35.997+0000 7fda93800640 1 -- 172.21.15.31:0/1481718968 wait complete. 2024-09-11T00:33:36.018 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:33:35:539881+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 33, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5076": { "addr": "172.21.15.31:6835/4202305151", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6833", "nonce": 4202305151, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 4202305151, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5076, "incarnation": 26, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:32:58.315114+0000", "qdb_cluster": [ 5076 ], "qdb_leader": 5076, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5076 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 33, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4829": { "addr": "172.21.15.204:6835/558470224", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 558470224, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 558470224, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4829, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:33:34.621041+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4829 } } } ], "standbys": [ { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:33:38.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.211+0000 7febdfe00640 1 -- 172.21.15.31:0/378845543 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febd80abde0 msgr2=0x7febd806db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.211+0000 7febdfe00640 1 --2- 172.21.15.31:0/378845543 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febd80abde0 0x7febd806db00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7febc4009860 tx=0x7febc402a5a0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.211+0000 7febdfe00640 1 -- 172.21.15.31:0/378845543 shutdown_connections 2024-09-11T00:33:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.211+0000 7febdfe00640 1 --2- 172.21.15.31:0/378845543 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febd80abde0 0x7febd806db00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.211+0000 7febdfe00640 1 -- 172.21.15.31:0/378845543 >> 172.21.15.31:0/378845543 conn(0x7febd80c8320 msgr2=0x7febd80c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.212+0000 7febdfe00640 1 -- 172.21.15.31:0/378845543 shutdown_connections 2024-09-11T00:33:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.212+0000 7febdfe00640 1 -- 172.21.15.31:0/378845543 wait complete. 2024-09-11T00:33:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.213+0000 7febdfe00640 1 Processor -- start 2024-09-11T00:33:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.213+0000 7febdfe00640 1 -- start start 2024-09-11T00:33:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.214+0000 7febdfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7febd8074630 0x7febd814aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.214+0000 7febdfe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febd80abde0 0x7febd81e1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.214+0000 7febdfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febd80c2d20 0x7febd81ef5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.214+0000 7febdfe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7febd8153b70 con 0x7febd80c2d20 2024-09-11T00:33:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.214+0000 7febdfe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7febd8152d60 con 0x7febd8074630 2024-09-11T00:33:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.214+0000 7febdfe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7febd814ff70 con 0x7febd80abde0 2024-09-11T00:33:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.214+0000 7febdde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febd80c2d20 0x7febd81ef5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febd80c2d20 0x7febd81ef5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33684/0 (socket says 172.21.15.31:33684) 2024-09-11T00:33:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdca00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febd80abde0 0x7febd81e1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdde00640 1 -- 172.21.15.31:0/1914013181 learned_addr learned my addr 172.21.15.31:0/1914013181 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdd400640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7febd8074630 0x7febd814aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdde00640 1 -- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febd80abde0 msgr2=0x7febd81e1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdde00640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febd80abde0 0x7febd81e1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdde00640 1 -- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7febd8074630 msgr2=0x7febd814aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdde00640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7febd8074630 0x7febd814aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdde00640 1 -- 172.21.15.31:0/1914013181 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd814bf90 con 0x7febd80c2d20 2024-09-11T00:33:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdca00640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febd80abde0 0x7febd81e1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:33:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.215+0000 7febdde00640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febd80c2d20 0x7febd81ef5f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7febd000c700 tx=0x7febd000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.216+0000 7febd5600640 1 -- 172.21.15.31:0/1914013181 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd0003b70 con 0x7febd80c2d20 2024-09-11T00:33:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.216+0000 7febdfe00640 1 -- 172.21.15.31:0/1914013181 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd814cfd0 con 0x7febd80c2d20 2024-09-11T00:33:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.216+0000 7febd5600640 1 -- 172.21.15.31:0/1914013181 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febd0003d10 con 0x7febd80c2d20 2024-09-11T00:33:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.216+0000 7febd5600640 1 -- 172.21.15.31:0/1914013181 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd000aab0 con 0x7febd80c2d20 2024-09-11T00:33:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.216+0000 7febdfe00640 1 -- 172.21.15.31:0/1914013181 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd81e2340 con 0x7febd80c2d20 2024-09-11T00:33:38.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.217+0000 7febcaa00640 1 -- 172.21.15.31:0/1914013181 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd806e200 con 0x7febd80c2d20 2024-09-11T00:33:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.219+0000 7febd5600640 1 -- 172.21.15.31:0/1914013181 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7febd000ac50 con 0x7febd80c2d20 2024-09-11T00:33:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.220+0000 7febd5600640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febb8077f70 0x7febb807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.220+0000 7febd5600640 1 -- 172.21.15.31:0/1914013181 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7febd0099f60 con 0x7febd80c2d20 2024-09-11T00:33:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.220+0000 7febdd400640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febb8077f70 0x7febb807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:38.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.221+0000 7febdd400640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febb8077f70 0x7febb807a430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7febcc000c00 tx=0x7febcc007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.225+0000 7febd5600640 1 -- 172.21.15.31:0/1914013181 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7febd0061e60 con 0x7febd80c2d20 2024-09-11T00:33:38.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.514+0000 7febcaa00640 1 -- 172.21.15.31:0/1914013181 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7febd8068b40 con 0x7febd80c2d20 2024-09-11T00:33:38.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.514+0000 7febd5600640 1 -- 172.21.15.31:0/1914013181 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5707 (secure 0 0 0) 0x7febd0066890 con 0x7febd80c2d20 2024-09-11T00:33:38.516 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:38.516 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":35,"btime":"2024-09-11T00:33:37:543491+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:37.543487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-09-11T00:33:38.517 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 35 2024-09-11T00:33:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 -- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febb8077f70 msgr2=0x7febb807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febb8077f70 0x7febb807a430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7febcc000c00 tx=0x7febcc007ec0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 -- 172.21.15.31:0/1914013181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febd80c2d20 msgr2=0x7febd81ef5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febd80c2d20 0x7febd81ef5f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7febd000c700 tx=0x7febd000cbc0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 -- 172.21.15.31:0/1914013181 shutdown_connections 2024-09-11T00:33:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febb8077f70 0x7febb807a430 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febd80c2d20 0x7febd81ef5f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febd80abde0 0x7febd81e1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 --2- 172.21.15.31:0/1914013181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7febd8074630 0x7febd814aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 -- 172.21.15.31:0/1914013181 >> 172.21.15.31:0/1914013181 conn(0x7febd80c8320 msgr2=0x7febd8150b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:38.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 -- 172.21.15.31:0/1914013181 shutdown_connections 2024-09-11T00:33:38.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.517+0000 7febcaa00640 1 -- 172.21.15.31:0/1914013181 wait complete. 2024-09-11T00:33:38.537 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=4721 2024-09-11T00:33:38.537 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.b 2024-09-11T00:33:38.538 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-11T00:33:38.538 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-11T00:33:38.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-11T00:33:38.542 INFO:tasks.ceph.mds.b:Started 2024-09-11T00:33:38.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:38.678 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:33:38.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.731+0000 7f3d63e00640 1 -- 172.21.15.31:0/255904019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d640c5ea0 msgr2=0x7f3d640c7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.731+0000 7f3d63e00640 1 --2- 172.21.15.31:0/255904019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d640c5ea0 0x7f3d640c7950 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c005fd0 tx=0x7f3d5c02a980 comp rx=0 tx=0).stop 2024-09-11T00:33:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.731+0000 7f3d63e00640 1 -- 172.21.15.31:0/255904019 shutdown_connections 2024-09-11T00:33:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.731+0000 7f3d63e00640 1 --2- 172.21.15.31:0/255904019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d640c5ea0 0x7f3d640c7950 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.731+0000 7f3d63e00640 1 --2- 172.21.15.31:0/255904019 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d640c95e0 0x7f3d64153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.731+0000 7f3d63e00640 1 -- 172.21.15.31:0/255904019 >> 172.21.15.31:0/255904019 conn(0x7f3d640ab800 msgr2=0x7f3d640abbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.731+0000 7f3d63e00640 1 -- 172.21.15.31:0/255904019 shutdown_connections 2024-09-11T00:33:38.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.732+0000 7f3d63e00640 1 -- 172.21.15.31:0/255904019 wait complete. 2024-09-11T00:33:38.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.733+0000 7f3d63e00640 1 Processor -- start 2024-09-11T00:33:38.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.733+0000 7f3d63e00640 1 -- start start 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.734+0000 7f3d63e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d64074420 0x7f3d641eef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d640c5ea0 0x7f3d6414a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d640c95e0 0x7f3d6406e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64071fc0 con 0x7f3d640c95e0 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3d640751e0 con 0x7f3d64074420 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3d64074800 con 0x7f3d640c5ea0 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d640c95e0 0x7f3d6406e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d640c95e0 0x7f3d6406e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33702/0 (socket says 172.21.15.31:33702) 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63400640 1 -- 172.21.15.31:0/4234609226 learned_addr learned my addr 172.21.15.31:0/4234609226 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d62a00640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d64074420 0x7f3d641eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:38.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d62000640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d640c5ea0 0x7f3d6414a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63400640 1 -- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d640c5ea0 msgr2=0x7f3d6414a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63400640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d640c5ea0 0x7f3d6414a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63400640 1 -- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d64074420 msgr2=0x7f3d641eef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63400640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d64074420 0x7f3d641eef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.735+0000 7f3d63400640 1 -- 172.21.15.31:0/4234609226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d640aa6f0 con 0x7f3d640c95e0 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.736+0000 7f3d63400640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d640c95e0 0x7f3d6406e9b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009d90 tx=0x7f3d5c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.736+0000 7f3d5aa00640 1 -- 172.21.15.31:0/4234609226 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c034050 con 0x7f3d640c95e0 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.736+0000 7f3d5aa00640 1 -- 172.21.15.31:0/4234609226 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d5c004450 con 0x7f3d640c95e0 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.736+0000 7f3d5aa00640 1 -- 172.21.15.31:0/4234609226 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c004860 con 0x7f3d640c95e0 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.736+0000 7f3d63e00640 1 -- 172.21.15.31:0/4234609226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d640ab510 con 0x7f3d640c95e0 2024-09-11T00:33:38.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.736+0000 7f3d63e00640 1 -- 172.21.15.31:0/4234609226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d640aa130 con 0x7f3d640c95e0 2024-09-11T00:33:38.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.737+0000 7f3d63e00640 1 -- 172.21.15.31:0/4234609226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d640c7950 con 0x7f3d640c95e0 2024-09-11T00:33:38.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.739+0000 7f3d5aa00640 1 -- 172.21.15.31:0/4234609226 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3d5c041070 con 0x7f3d640c95e0 2024-09-11T00:33:38.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.740+0000 7f3d5aa00640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d44077f70 0x7f3d4407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:38.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.740+0000 7f3d5aa00640 1 -- 172.21.15.31:0/4234609226 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f3d5c0bb4a0 con 0x7f3d640c95e0 2024-09-11T00:33:38.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.741+0000 7f3d62a00640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d44077f70 0x7f3d4407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:38.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.741+0000 7f3d62a00640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d44077f70 0x7f3d4407a430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d54007fd0 tx=0x7f3d54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:38.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:38.745+0000 7f3d5aa00640 1 -- 172.21.15.31:0/4234609226 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3d5c0834c0 con 0x7f3d640c95e0 2024-09-11T00:33:39.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.028+0000 7f3d63e00640 1 -- 172.21.15.31:0/4234609226 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d64153b00 con 0x7f3d640c95e0 2024-09-11T00:33:39.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.028+0000 7f3d5aa00640 1 -- 172.21.15.31:0/4234609226 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5707 (secure 0 0 0) 0x7f3d5c087ef0 con 0x7f3d640c95e0 2024-09-11T00:33:39.030 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:39.031 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":35,"btime":"2024-09-11T00:33:37:543491+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:37.543487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-09-11T00:33:39.031 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 35 2024-09-11T00:33:39.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 -- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d44077f70 msgr2=0x7f3d4407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:39.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d44077f70 0x7f3d4407a430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d54007fd0 tx=0x7f3d54009040 comp rx=0 tx=0).stop 2024-09-11T00:33:39.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 -- 172.21.15.31:0/4234609226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d640c95e0 msgr2=0x7f3d6406e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:39.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d640c95e0 0x7f3d6406e9b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009d90 tx=0x7f3d5c003f10 comp rx=0 tx=0).stop 2024-09-11T00:33:39.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 -- 172.21.15.31:0/4234609226 shutdown_connections 2024-09-11T00:33:39.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3d44077f70 0x7f3d4407a430 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:39.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d640c95e0 0x7f3d6406e9b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:39.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3d640c5ea0 0x7f3d6414a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:39.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 --2- 172.21.15.31:0/4234609226 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d64074420 0x7f3d641eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:39.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 -- 172.21.15.31:0/4234609226 >> 172.21.15.31:0/4234609226 conn(0x7f3d640ab800 msgr2=0x7f3d64152d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:39.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 -- 172.21.15.31:0/4234609226 shutdown_connections 2024-09-11T00:33:39.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:39.031+0000 7f3d63e00640 1 -- 172.21.15.31:0/4234609226 wait complete. 2024-09-11T00:33:39.049 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-11T00:33:39.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:33:39.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:33:41.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:41.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.236+0000 7fad6be00640 1 -- 172.21.15.31:0/1105864689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad6c0c5aa0 msgr2=0x7fad6c0c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:41.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.236+0000 7fad6be00640 1 --2- 172.21.15.31:0/1105864689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad6c0c5aa0 0x7fad6c0c7150 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fad6c06df10 tx=0x7fad5c02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:33:41.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.236+0000 7fad6be00640 1 -- 172.21.15.31:0/1105864689 shutdown_connections 2024-09-11T00:33:41.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.236+0000 7fad6be00640 1 --2- 172.21.15.31:0/1105864689 >> v2:172.21.15.204:3300/0 conn(0x7fad6c0c6420 0x7fad6c153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:41.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.236+0000 7fad6be00640 1 --2- 172.21.15.31:0/1105864689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad6c0c5aa0 0x7fad6c0c7150 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:41.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.236+0000 7fad6be00640 1 -- 172.21.15.31:0/1105864689 >> 172.21.15.31:0/1105864689 conn(0x7fad6c06caf0 msgr2=0x7fad6c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:41.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.236+0000 7fad6be00640 1 -- 172.21.15.31:0/1105864689 shutdown_connections 2024-09-11T00:33:41.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.236+0000 7fad6be00640 1 -- 172.21.15.31:0/1105864689 wait complete. 2024-09-11T00:33:41.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.238+0000 7fad6be00640 1 Processor -- start 2024-09-11T00:33:41.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.238+0000 7fad6be00640 1 -- start start 2024-09-11T00:33:41.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.239+0000 7fad6be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad6c0c5aa0 0x7fad6c072450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:41.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.239+0000 7fad6be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad6c0c6420 0x7fad6c072990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.239+0000 7fad6be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad6c153850 0x7fad6c0745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.239+0000 7fad6be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c0c2e80 con 0x7fad6c153850 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.239+0000 7fad6be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c0c2ba0 con 0x7fad6c0c5aa0 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.239+0000 7fad6be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fad6c0c5e80 con 0x7fad6c0c6420 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad6c153850 0x7fad6c0745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad6c0c5aa0 0x7fad6c072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad6c0c5aa0 0x7fad6c072450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:45238/0 (socket says 172.21.15.31:45238) 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6aa00640 1 -- 172.21.15.31:0/2843905461 learned_addr learned my addr 172.21.15.31:0/2843905461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad6c153850 0x7fad6c0745b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33716/0 (socket says 172.21.15.31:33716) 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6a000640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad6c0c6420 0x7fad6c072990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6b400640 1 -- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad6c0c6420 msgr2=0x7fad6c072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6b400640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad6c0c6420 0x7fad6c072990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:41.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6b400640 1 -- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad6c0c5aa0 msgr2=0x7fad6c072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6b400640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad6c0c5aa0 0x7fad6c072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6b400640 1 -- 172.21.15.31:0/2843905461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6c0c42f0 con 0x7fad6c153850 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6aa00640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad6c0c5aa0 0x7fad6c072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.240+0000 7fad6b400640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad6c153850 0x7fad6c0745b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fad6400c760 tx=0x7fad6400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.241+0000 7fad62a00640 1 -- 172.21.15.31:0/2843905461 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad64003b70 con 0x7fad6c153850 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.241+0000 7fad62a00640 1 -- 172.21.15.31:0/2843905461 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad64003d10 con 0x7fad6c153850 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.241+0000 7fad6be00640 1 -- 172.21.15.31:0/2843905461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6c1e22d0 con 0x7fad6c153850 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.241+0000 7fad62a00640 1 -- 172.21.15.31:0/2843905461 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6400aab0 con 0x7fad6c153850 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.241+0000 7fad6be00640 1 -- 172.21.15.31:0/2843905461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6c06ede0 con 0x7fad6c153850 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.243+0000 7fad6be00640 1 -- 172.21.15.31:0/2843905461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad6c06efe0 con 0x7fad6c153850 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.243+0000 7fad62a00640 1 -- 172.21.15.31:0/2843905461 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fad6400ac50 con 0x7fad6c153850 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.245+0000 7fad62a00640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad4c077f70 0x7fad4c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.245+0000 7fad62a00640 1 -- 172.21.15.31:0/2843905461 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fad64099e70 con 0x7fad6c153850 2024-09-11T00:33:41.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.245+0000 7fad6aa00640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad4c077f70 0x7fad4c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:41.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.246+0000 7fad6aa00640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad4c077f70 0x7fad4c07a430 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fad5c0096c0 tx=0x7fad5c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:41.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.249+0000 7fad62a00640 1 -- 172.21.15.31:0/2843905461 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fad6c06efe0 con 0x7fad6c153850 2024-09-11T00:33:41.345 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:33:41.343+0000 7fc1da600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:33:41.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.511+0000 7fad6be00640 1 -- 172.21.15.31:0/2843905461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad6c072120 con 0x7fad6c153850 2024-09-11T00:33:41.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.512+0000 7fad62a00640 1 -- 172.21.15.31:0/2843905461 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6532 (secure 0 0 0) 0x7fad6c0c42f0 con 0x7fad6c153850 2024-09-11T00:33:41.514 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:41.514 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"btime":"2024-09-11T00:33:39:561976+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4898,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:37.543487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-09-11T00:33:41.514 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 36 2024-09-11T00:33:41.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.514+0000 7fad6be00640 1 -- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad4c077f70 msgr2=0x7fad4c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:41.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.514+0000 7fad6be00640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad4c077f70 0x7fad4c07a430 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fad5c0096c0 tx=0x7fad5c004730 comp rx=0 tx=0).stop 2024-09-11T00:33:41.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.514+0000 7fad6be00640 1 -- 172.21.15.31:0/2843905461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad6c153850 msgr2=0x7fad6c0745b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:41.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.514+0000 7fad6be00640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad6c153850 0x7fad6c0745b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fad6400c760 tx=0x7fad6400cc20 comp rx=0 tx=0).stop 2024-09-11T00:33:41.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.514+0000 7fad6be00640 1 -- 172.21.15.31:0/2843905461 shutdown_connections 2024-09-11T00:33:41.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.514+0000 7fad6be00640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad4c077f70 0x7fad4c07a430 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:41.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.514+0000 7fad6be00640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad6c153850 0x7fad6c0745b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:41.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.514+0000 7fad6be00640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad6c0c6420 0x7fad6c072990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:41.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.514+0000 7fad6be00640 1 --2- 172.21.15.31:0/2843905461 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad6c0c5aa0 0x7fad6c072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:41.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.514+0000 7fad6be00640 1 -- 172.21.15.31:0/2843905461 >> 172.21.15.31:0/2843905461 conn(0x7fad6c06caf0 msgr2=0x7fad6c151110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:41.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.515+0000 7fad6be00640 1 -- 172.21.15.31:0/2843905461 shutdown_connections 2024-09-11T00:33:41.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:41.515+0000 7fad6be00640 1 -- 172.21.15.31:0/2843905461 wait complete. 2024-09-11T00:33:41.535 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-11T00:33:41.535 INFO:tasks.mds_thrash.fs.[b]:waiting for 14 secs before thrashing 2024-09-11T00:33:55.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:55.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.727+0000 7f7119800640 1 -- 172.21.15.31:0/3824515895 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71140c5ef0 msgr2=0x7f7114153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.727+0000 7f7119800640 1 --2- 172.21.15.31:0/3824515895 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71140c5ef0 0x7f7114153480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7100005fd0 tx=0x7f710002a5d0 comp rx=0 tx=0).stop 2024-09-11T00:33:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.728+0000 7f7119800640 1 -- 172.21.15.31:0/3824515895 shutdown_connections 2024-09-11T00:33:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.728+0000 7f7119800640 1 --2- 172.21.15.31:0/3824515895 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71140c5ef0 0x7f7114153480 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.728+0000 7f7119800640 1 --2- 172.21.15.31:0/3824515895 >> v2:172.21.15.204:3300/0 conn(0x7f71140c5b10 0x7f7114156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.728+0000 7f7119800640 1 -- 172.21.15.31:0/3824515895 >> 172.21.15.31:0/3824515895 conn(0x7f71140a9750 msgr2=0x7f71140a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.728+0000 7f7119800640 1 -- 172.21.15.31:0/3824515895 shutdown_connections 2024-09-11T00:33:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.728+0000 7f7119800640 1 -- 172.21.15.31:0/3824515895 wait complete. 2024-09-11T00:33:55.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.729+0000 7f7119800640 1 Processor -- start 2024-09-11T00:33:55.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.729+0000 7f7119800640 1 -- start start 2024-09-11T00:33:55.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.730+0000 7f7119800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71140c5b10 0x7f7114159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:55.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.730+0000 7f7119800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71140c5ef0 0x7f71140a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:55.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.730+0000 7f7119800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71140c6310 0x7f711414cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:55.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.730+0000 7f7119800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f71141539c0 con 0x7f71140c5ef0 2024-09-11T00:33:55.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.730+0000 7f7119800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f71140c3950 con 0x7f71140c5b10 2024-09-11T00:33:55.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.730+0000 7f7119800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7114153b40 con 0x7f71140c6310 2024-09-11T00:33:55.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.730+0000 7f7112a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71140c5b10 0x7f7114159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:55.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.730+0000 7f7112000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71140c5ef0 0x7f71140a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:55.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.731+0000 7f7112a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71140c5b10 0x7f7114159040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:57690/0 (socket says 172.21.15.31:57690) 2024-09-11T00:33:55.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.731+0000 7f7112a00640 1 -- 172.21.15.31:0/2129908038 learned_addr learned my addr 172.21.15.31:0/2129908038 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:55.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.731+0000 7f7113400640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71140c6310 0x7f711414cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:55.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.731+0000 7f7112a00640 1 -- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71140c6310 msgr2=0x7f711414cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:55.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.731+0000 7f7112a00640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71140c6310 0x7f711414cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:55.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.731+0000 7f7112a00640 1 -- 172.21.15.31:0/2129908038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71140c5ef0 msgr2=0x7f71140a8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:55.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.731+0000 7f7112a00640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71140c5ef0 0x7f71140a8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:55.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.731+0000 7f7112a00640 1 -- 172.21.15.31:0/2129908038 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7114159580 con 0x7f71140c5b10 2024-09-11T00:33:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.731+0000 7f7112a00640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71140c5b10 0x7f7114159040 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f710400ea60 tx=0x7f710400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.732+0000 7f710aa00640 1 -- 172.21.15.31:0/2129908038 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710400cde0 con 0x7f71140c5b10 2024-09-11T00:33:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.732+0000 7f710aa00640 1 -- 172.21.15.31:0/2129908038 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7104004020 con 0x7f71140c5b10 2024-09-11T00:33:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.732+0000 7f710aa00640 1 -- 172.21.15.31:0/2129908038 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7104004450 con 0x7f71140c5b10 2024-09-11T00:33:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.732+0000 7f7119800640 1 -- 172.21.15.31:0/2129908038 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71141584b0 con 0x7f71140c5b10 2024-09-11T00:33:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.732+0000 7f7119800640 1 -- 172.21.15.31:0/2129908038 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7114074f50 con 0x7f71140c5b10 2024-09-11T00:33:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.733+0000 7f7119800640 1 -- 172.21.15.31:0/2129908038 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7114156ad0 con 0x7f71140c5b10 2024-09-11T00:33:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.735+0000 7f710aa00640 1 -- 172.21.15.31:0/2129908038 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f710403a970 con 0x7f71140c5b10 2024-09-11T00:33:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.736+0000 7f710aa00640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70f4077fa0 0x7f70f407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.736+0000 7f710aa00640 1 -- 172.21.15.31:0/2129908038 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f710409ab40 con 0x7f71140c5b10 2024-09-11T00:33:55.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.736+0000 7f7112000640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70f4077fa0 0x7f70f407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:55.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.737+0000 7f7112000640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70f4077fa0 0x7f70f407a460 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7100033000 tx=0x7f7100004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:55.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:55.739+0000 7f710aa00640 1 -- 172.21.15.31:0/2129908038 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7104062b40 con 0x7f71140c5b10 2024-09-11T00:33:56.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.007+0000 7f7119800640 1 -- 172.21.15.31:0/2129908038 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71140c1e70 con 0x7f71140c5b10 2024-09-11T00:33:56.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.007+0000 7f710aa00640 1 -- 172.21.15.31:0/2129908038 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6532 (secure 0 0 0) 0x7f7104067570 con 0x7f71140c5b10 2024-09-11T00:33:56.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:56.009 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"btime":"2024-09-11T00:33:39:561976+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4898,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:37.543487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-09-11T00:33:56.010 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 36 2024-09-11T00:33:56.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.010+0000 7f7119800640 1 -- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70f4077fa0 msgr2=0x7f70f407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:56.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.010+0000 7f7119800640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70f4077fa0 0x7f70f407a460 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7100033000 tx=0x7f7100004730 comp rx=0 tx=0).stop 2024-09-11T00:33:56.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.010+0000 7f7119800640 1 -- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71140c5b10 msgr2=0x7f7114159040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:56.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.010+0000 7f7119800640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71140c5b10 0x7f7114159040 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f710400ea60 tx=0x7f710400ef20 comp rx=0 tx=0).stop 2024-09-11T00:33:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.010+0000 7f7119800640 1 -- 172.21.15.31:0/2129908038 shutdown_connections 2024-09-11T00:33:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.010+0000 7f7119800640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f70f4077fa0 0x7f70f407a460 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.010+0000 7f7119800640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f71140c6310 0x7f711414cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.010+0000 7f7119800640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71140c5ef0 0x7f71140a8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.010+0000 7f7119800640 1 --2- 172.21.15.31:0/2129908038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f71140c5b10 0x7f7114159040 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.010+0000 7f7119800640 1 -- 172.21.15.31:0/2129908038 >> 172.21.15.31:0/2129908038 conn(0x7f71140a9750 msgr2=0x7f7114152560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.011+0000 7f7119800640 1 -- 172.21.15.31:0/2129908038 shutdown_connections 2024-09-11T00:33:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.011+0000 7f7119800640 1 -- 172.21.15.31:0/2129908038 wait complete. 2024-09-11T00:33:56.029 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-11T00:33:56.030 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:33:56.030 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:33:56.047 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:33:56.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.229+0000 7fc2be200640 1 -- 172.21.15.31:0/2811496858 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2b8072480 msgr2=0x7fc2b81511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.229+0000 7fc2be200640 1 --2- 172.21.15.31:0/2811496858 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2b8072480 0x7fc2b81511a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8009830 tx=0x7fc2a802a5a0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.229+0000 7fc2be200640 1 -- 172.21.15.31:0/2811496858 shutdown_connections 2024-09-11T00:33:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.229+0000 7fc2be200640 1 --2- 172.21.15.31:0/2811496858 >> v2:172.21.15.204:3300/0 conn(0x7fc2b8153f20 0x7fc2b81516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.229+0000 7fc2be200640 1 --2- 172.21.15.31:0/2811496858 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2b8072480 0x7fc2b81511a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.229+0000 7fc2be200640 1 -- 172.21.15.31:0/2811496858 >> 172.21.15.31:0/2811496858 conn(0x7fc2b80a9750 msgr2=0x7fc2b80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.229+0000 7fc2be200640 1 -- 172.21.15.31:0/2811496858 shutdown_connections 2024-09-11T00:33:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.229+0000 7fc2be200640 1 -- 172.21.15.31:0/2811496858 wait complete. 2024-09-11T00:33:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.231+0000 7fc2be200640 1 Processor -- start 2024-09-11T00:33:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.231+0000 7fc2be200640 1 -- start start 2024-09-11T00:33:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2be200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2b8072480 0x7fc2b81ece80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2be200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2b80c23a0 0x7fc2b814cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2be200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2b8153f20 0x7fc2b80c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2be200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b8074200 con 0x7fc2b8153f20 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2be200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b8074080 con 0x7fc2b8072480 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2be200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b8072860 con 0x7fc2b80c23a0 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2b7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2b8072480 0x7fc2b81ece80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2b7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2b8153f20 0x7fc2b80c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2b7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2b8072480 0x7fc2b81ece80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:57710/0 (socket says 172.21.15.31:57710) 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2b7400640 1 -- 172.21.15.31:0/4091673070 learned_addr learned my addr 172.21.15.31:0/4091673070 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.232+0000 7fc2b6a00640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2b80c23a0 0x7fc2b814cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2b7e00640 1 -- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2b80c23a0 msgr2=0x7fc2b814cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2b7e00640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2b80c23a0 0x7fc2b814cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2b7e00640 1 -- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2b8072480 msgr2=0x7fc2b81ece80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2b7e00640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2b8072480 0x7fc2b81ece80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2b7e00640 1 -- 172.21.15.31:0/4091673070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b80c7430 con 0x7fc2b8153f20 2024-09-11T00:33:56.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2b7e00640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2b8153f20 0x7fc2b80c62e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000ebc0 tx=0x7fc2b000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:56.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2af400640 1 -- 172.21.15.31:0/4091673070 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0010430 con 0x7fc2b8153f20 2024-09-11T00:33:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2af400640 1 -- 172.21.15.31:0/4091673070 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2b00105d0 con 0x7fc2b8153f20 2024-09-11T00:33:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2af400640 1 -- 172.21.15.31:0/4091673070 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0010a00 con 0x7fc2b8153f20 2024-09-11T00:33:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2be200640 1 -- 172.21.15.31:0/4091673070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b80c3260 con 0x7fc2b8153f20 2024-09-11T00:33:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.233+0000 7fc2be200640 1 -- 172.21.15.31:0/4091673070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b814d070 con 0x7fc2b8153f20 2024-09-11T00:33:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.234+0000 7fc2be200640 1 -- 172.21.15.31:0/4091673070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b81511a0 con 0x7fc2b8153f20 2024-09-11T00:33:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.236+0000 7fc2af400640 1 -- 172.21.15.31:0/4091673070 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc2b0010ba0 con 0x7fc2b8153f20 2024-09-11T00:33:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.237+0000 7fc2af400640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc298077f70 0x7fc29807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.237+0000 7fc2af400640 1 -- 172.21.15.31:0/4091673070 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fc2b009af40 con 0x7fc2b8153f20 2024-09-11T00:33:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.237+0000 7fc2b7400640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc298077f70 0x7fc29807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.238+0000 7fc2b7400640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc298077f70 0x7fc29807a430 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80095f0 tx=0x7fc2a8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.242+0000 7fc2af400640 1 -- 172.21.15.31:0/4091673070 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc2b0062ec0 con 0x7fc2b8153f20 2024-09-11T00:33:56.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.534+0000 7fc2be200640 1 -- 172.21.15.31:0/4091673070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2b80c3430 con 0x7fc2b8153f20 2024-09-11T00:33:56.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.534+0000 7fc2af400640 1 -- 172.21.15.31:0/4091673070 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6532 (secure 0 0 0) 0x7fc2b00678f0 con 0x7fc2b8153f20 2024-09-11T00:33:56.536 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:56.536 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"btime":"2024-09-11T00:33:39:561976+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4898,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:37.543487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-09-11T00:33:56.537 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 36 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 -- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc298077f70 msgr2=0x7fc29807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc298077f70 0x7fc29807a430 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80095f0 tx=0x7fc2a8004730 comp rx=0 tx=0).stop 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 -- 172.21.15.31:0/4091673070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2b8153f20 msgr2=0x7fc2b80c62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2b8153f20 0x7fc2b80c62e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000ebc0 tx=0x7fc2b000c6a0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 -- 172.21.15.31:0/4091673070 shutdown_connections 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc298077f70 0x7fc29807a430 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2b8153f20 0x7fc2b80c62e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc2b80c23a0 0x7fc2b814cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 --2- 172.21.15.31:0/4091673070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc2b8072480 0x7fc2b81ece80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 -- 172.21.15.31:0/4091673070 >> 172.21.15.31:0/4091673070 conn(0x7fc2b80a9750 msgr2=0x7fc2b8152d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 -- 172.21.15.31:0/4091673070 shutdown_connections 2024-09-11T00:33:56.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:56.537+0000 7fc2be200640 1 -- 172.21.15.31:0/4091673070 wait complete. 2024-09-11T00:33:56.557 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:33:58.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:33:58.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.733+0000 7f3388c00640 1 -- 172.21.15.31:0/2531599887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3384159b60 msgr2=0x7f33841617f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:58.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.733+0000 7f3388c00640 1 --2- 172.21.15.31:0/2531599887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3384159b60 0x7f33841617f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f337c005fd0 tx=0x7f337c02a940 comp rx=0 tx=0).stop 2024-09-11T00:33:58.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.733+0000 7f3388c00640 1 -- 172.21.15.31:0/2531599887 shutdown_connections 2024-09-11T00:33:58.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.733+0000 7f3388c00640 1 --2- 172.21.15.31:0/2531599887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3384159b60 0x7f33841617f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:58.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.733+0000 7f3388c00640 1 -- 172.21.15.31:0/2531599887 >> 172.21.15.31:0/2531599887 conn(0x7f33840adb00 msgr2=0x7f33840adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:58.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.733+0000 7f3388c00640 1 -- 172.21.15.31:0/2531599887 shutdown_connections 2024-09-11T00:33:58.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.733+0000 7f3388c00640 1 -- 172.21.15.31:0/2531599887 wait complete. 2024-09-11T00:33:58.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.735+0000 7f3388c00640 1 Processor -- start 2024-09-11T00:33:58.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.735+0000 7f3388c00640 1 -- start start 2024-09-11T00:33:58.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.736+0000 7f3388c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338406cc10 0x7f3384149520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:58.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.736+0000 7f3388c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f338414c880 0x7f33840afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:58.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.736+0000 7f3388c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3384159b60 0x7f3384153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:58.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.736+0000 7f3388c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f338414cc60 con 0x7f338406cc10 2024-09-11T00:33:58.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.736+0000 7f3388c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3384156810 con 0x7f338414c880 2024-09-11T00:33:58.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.736+0000 7f3388c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f338406edc0 con 0x7f3384159b60 2024-09-11T00:33:58.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.736+0000 7f3383400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338406cc10 0x7f3384149520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:58.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.736+0000 7f3383400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338406cc10 0x7f3384149520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36660/0 (socket says 172.21.15.31:36660) 2024-09-11T00:33:58.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.736+0000 7f3383400640 1 -- 172.21.15.31:0/830823206 learned_addr learned my addr 172.21.15.31:0/830823206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:33:58.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.737+0000 7f337be00640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f338414c880 0x7f33840afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:58.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.737+0000 7f3383e00640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3384159b60 0x7f3384153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:58.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.737+0000 7f3383400640 1 -- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3384159b60 msgr2=0x7f3384153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:58.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.737+0000 7f3383400640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3384159b60 0x7f3384153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:58.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.737+0000 7f3383400640 1 -- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f338414c880 msgr2=0x7f33840afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:58.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.737+0000 7f3383400640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f338414c880 0x7f33840afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:58.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.737+0000 7f3383400640 1 -- 172.21.15.31:0/830823206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33841ec680 con 0x7f338406cc10 2024-09-11T00:33:58.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.737+0000 7f3383400640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338406cc10 0x7f3384149520 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f337400c500 tx=0x7f337400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:58.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.737+0000 7f337b400640 1 -- 172.21.15.31:0/830823206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3374016020 con 0x7f338406cc10 2024-09-11T00:33:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.738+0000 7f337b400640 1 -- 172.21.15.31:0/830823206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3374012920 con 0x7f338406cc10 2024-09-11T00:33:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.738+0000 7f337b400640 1 -- 172.21.15.31:0/830823206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3374012d90 con 0x7f338406cc10 2024-09-11T00:33:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.738+0000 7f3388c00640 1 -- 172.21.15.31:0/830823206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33841ec900 con 0x7f338406cc10 2024-09-11T00:33:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.738+0000 7f3388c00640 1 -- 172.21.15.31:0/830823206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33840af3c0 con 0x7f338406cc10 2024-09-11T00:33:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.739+0000 7f3388c00640 1 -- 172.21.15.31:0/830823206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33840c59a0 con 0x7f338406cc10 2024-09-11T00:33:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.740+0000 7f337b400640 1 -- 172.21.15.31:0/830823206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f337401a3b0 con 0x7f338406cc10 2024-09-11T00:33:58.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.741+0000 7f337b400640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3360077fa0 0x7f336007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:33:58.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.741+0000 7f337b400640 1 -- 172.21.15.31:0/830823206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f337409a1c0 con 0x7f338406cc10 2024-09-11T00:33:58.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.741+0000 7f337be00640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3360077fa0 0x7f336007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:33:58.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.742+0000 7f337be00640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3360077fa0 0x7f336007a460 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f33841f02c0 tx=0x7f336c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:33:58.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.747+0000 7f337b400640 1 -- 172.21.15.31:0/830823206 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3374020030 con 0x7f338406cc10 2024-09-11T00:33:58.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.995+0000 7f3388c00640 1 -- 172.21.15.31:0/830823206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3384149e70 con 0x7f338406cc10 2024-09-11T00:33:58.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.995+0000 7f337b400640 1 -- 172.21.15.31:0/830823206 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6532 (secure 0 0 0) 0x7f33841ec680 con 0x7f338406cc10 2024-09-11T00:33:58.997 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:33:58.997 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"btime":"2024-09-11T00:33:39:561976+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4898,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:37.543487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-09-11T00:33:58.998 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 36 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 -- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3360077fa0 msgr2=0x7f336007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3360077fa0 0x7f336007a460 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f33841f02c0 tx=0x7f336c009040 comp rx=0 tx=0).stop 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 -- 172.21.15.31:0/830823206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338406cc10 msgr2=0x7f3384149520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338406cc10 0x7f3384149520 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f337400c500 tx=0x7f337400c9c0 comp rx=0 tx=0).stop 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 -- 172.21.15.31:0/830823206 shutdown_connections 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3360077fa0 0x7f336007a460 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3384159b60 0x7f3384153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f338414c880 0x7f33840afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 --2- 172.21.15.31:0/830823206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f338406cc10 0x7f3384149520 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 -- 172.21.15.31:0/830823206 >> 172.21.15.31:0/830823206 conn(0x7f33840adb00 msgr2=0x7f33840adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:33:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 -- 172.21.15.31:0/830823206 shutdown_connections 2024-09-11T00:33:59.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:33:58.998+0000 7f3388c00640 1 -- 172.21.15.31:0/830823206 wait complete. 2024-09-11T00:33:59.016 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:34:01.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:01.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.204+0000 7f8b8d400640 1 -- 172.21.15.31:0/2381365002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b88156960 msgr2=0x7f8b8806e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:01.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.204+0000 7f8b8d400640 1 --2- 172.21.15.31:0/2381365002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b88156960 0x7f8b8806e140 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8b88153d20 tx=0x7f8b7402a560 comp rx=0 tx=0).stop 2024-09-11T00:34:01.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.204+0000 7f8b8d400640 1 -- 172.21.15.31:0/2381365002 shutdown_connections 2024-09-11T00:34:01.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.204+0000 7f8b8d400640 1 --2- 172.21.15.31:0/2381365002 >> v2:172.21.15.204:3300/0 conn(0x7f8b880cc220 0x7f8b8806e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:01.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.204+0000 7f8b8d400640 1 --2- 172.21.15.31:0/2381365002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b88156960 0x7f8b8806e140 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.204+0000 7f8b8d400640 1 -- 172.21.15.31:0/2381365002 >> 172.21.15.31:0/2381365002 conn(0x7f8b88068f10 msgr2=0x7f8b880cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.204+0000 7f8b8d400640 1 -- 172.21.15.31:0/2381365002 shutdown_connections 2024-09-11T00:34:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.204+0000 7f8b8d400640 1 -- 172.21.15.31:0/2381365002 wait complete. 2024-09-11T00:34:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.206+0000 7f8b8d400640 1 Processor -- start 2024-09-11T00:34:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.206+0000 7f8b8d400640 1 -- start start 2024-09-11T00:34:01.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.207+0000 7f8b8d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b880cc220 0x7f8b881f01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.207+0000 7f8b8d400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b88150020 0x7f8b8814f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.207+0000 7f8b8d400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b88156960 0x7f8b880c5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.207+0000 7f8b8d400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880c6ca0 con 0x7f8b880cc220 2024-09-11T00:34:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.207+0000 7f8b8d400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880c6b20 con 0x7f8b88156960 2024-09-11T00:34:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.207+0000 7f8b8d400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8b881573d0 con 0x7f8b88150020 2024-09-11T00:34:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.208+0000 7f8b87e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b880cc220 0x7f8b881f01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.208+0000 7f8b87e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b880cc220 0x7f8b881f01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36680/0 (socket says 172.21.15.31:36680) 2024-09-11T00:34:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.208+0000 7f8b87e00640 1 -- 172.21.15.31:0/4138360683 learned_addr learned my addr 172.21.15.31:0/4138360683 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.208+0000 7f8b8ca00640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b88156960 0x7f8b880c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.208+0000 7f8b87400640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b88150020 0x7f8b8814f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.208+0000 7f8b87e00640 1 -- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b88150020 msgr2=0x7f8b8814f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.208+0000 7f8b87e00640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b88150020 0x7f8b8814f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.208+0000 7f8b87e00640 1 -- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b88156960 msgr2=0x7f8b880c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.208+0000 7f8b87e00640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b88156960 0x7f8b880c5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.208+0000 7f8b87e00640 1 -- 172.21.15.31:0/4138360683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b880c12c0 con 0x7f8b880cc220 2024-09-11T00:34:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.209+0000 7f8b87e00640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b880cc220 0x7f8b881f01f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800c410 tx=0x7f8b7800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.210+0000 7f8b7fe00640 1 -- 172.21.15.31:0/4138360683 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78017070 con 0x7f8b880cc220 2024-09-11T00:34:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.210+0000 7f8b7fe00640 1 -- 172.21.15.31:0/4138360683 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b780035f0 con 0x7f8b880cc220 2024-09-11T00:34:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.210+0000 7f8b7fe00640 1 -- 172.21.15.31:0/4138360683 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78003a80 con 0x7f8b880cc220 2024-09-11T00:34:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.210+0000 7f8b8d400640 1 -- 172.21.15.31:0/4138360683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b880c15a0 con 0x7f8b880cc220 2024-09-11T00:34:01.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.210+0000 7f8b8d400640 1 -- 172.21.15.31:0/4138360683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b880c4e00 con 0x7f8b880cc220 2024-09-11T00:34:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.211+0000 7f8b8d400640 1 -- 172.21.15.31:0/4138360683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b880ad4e0 con 0x7f8b880cc220 2024-09-11T00:34:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.213+0000 7f8b7fe00640 1 -- 172.21.15.31:0/4138360683 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8b78003c20 con 0x7f8b880cc220 2024-09-11T00:34:01.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.215+0000 7f8b7fe00640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b68077f70 0x7f8b6807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:01.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.215+0000 7f8b7fe00640 1 -- 172.21.15.31:0/4138360683 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f8b78099760 con 0x7f8b880cc220 2024-09-11T00:34:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.215+0000 7f8b87400640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b68077f70 0x7f8b6807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.216+0000 7f8b87400640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b68077f70 0x7f8b6807a430 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8b74033000 tx=0x7f8b74004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:01.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.219+0000 7f8b7fe00640 1 -- 172.21.15.31:0/4138360683 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8b78061780 con 0x7f8b880cc220 2024-09-11T00:34:01.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.523+0000 7f8b8d400640 1 -- 172.21.15.31:0/4138360683 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b88074000 con 0x7f8b880cc220 2024-09-11T00:34:01.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.524+0000 7f8b7fe00640 1 -- 172.21.15.31:0/4138360683 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6532 (secure 0 0 0) 0x7f8b780661b0 con 0x7f8b880cc220 2024-09-11T00:34:01.526 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:01.526 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"btime":"2024-09-11T00:33:39:561976+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4898,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:37.543487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-09-11T00:34:01.526 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 36 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 -- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b68077f70 msgr2=0x7f8b6807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b68077f70 0x7f8b6807a430 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8b74033000 tx=0x7f8b74004730 comp rx=0 tx=0).stop 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 -- 172.21.15.31:0/4138360683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b880cc220 msgr2=0x7f8b881f01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b880cc220 0x7f8b881f01f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800c410 tx=0x7f8b7800c8d0 comp rx=0 tx=0).stop 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 -- 172.21.15.31:0/4138360683 shutdown_connections 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b68077f70 0x7f8b6807a430 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b88156960 0x7f8b880c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b88150020 0x7f8b8814f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 --2- 172.21.15.31:0/4138360683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b880cc220 0x7f8b881f01f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 -- 172.21.15.31:0/4138360683 >> 172.21.15.31:0/4138360683 conn(0x7f8b88068f10 msgr2=0x7f8b880aca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:01.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 -- 172.21.15.31:0/4138360683 shutdown_connections 2024-09-11T00:34:01.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:01.527+0000 7f8b8d400640 1 -- 172.21.15.31:0/4138360683 wait complete. 2024-09-11T00:34:01.545 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:34:03.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:03.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.718+0000 7f4191e00640 1 -- 172.21.15.31:0/531235821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f418c0c5ef0 msgr2=0x7f418c153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:03.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.718+0000 7f4191e00640 1 --2- 172.21.15.31:0/531235821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f418c0c5ef0 0x7f418c153480 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f418c070470 tx=0x7f417802a540 comp rx=0 tx=0).stop 2024-09-11T00:34:03.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.718+0000 7f4191e00640 1 -- 172.21.15.31:0/531235821 shutdown_connections 2024-09-11T00:34:03.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.718+0000 7f4191e00640 1 --2- 172.21.15.31:0/531235821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f418c0c5ef0 0x7f418c153480 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:03.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.718+0000 7f4191e00640 1 --2- 172.21.15.31:0/531235821 >> v2:172.21.15.204:3300/0 conn(0x7f418c0c5b10 0x7f418c156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:03.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.718+0000 7f4191e00640 1 -- 172.21.15.31:0/531235821 >> 172.21.15.31:0/531235821 conn(0x7f418c0a9750 msgr2=0x7f418c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:03.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.718+0000 7f4191e00640 1 -- 172.21.15.31:0/531235821 shutdown_connections 2024-09-11T00:34:03.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.718+0000 7f4191e00640 1 -- 172.21.15.31:0/531235821 wait complete. 2024-09-11T00:34:03.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.720+0000 7f4191e00640 1 Processor -- start 2024-09-11T00:34:03.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.720+0000 7f4191e00640 1 -- start start 2024-09-11T00:34:03.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.721+0000 7f4191e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f418c0c5b10 0x7f418c151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:03.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.721+0000 7f4191e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f418c0c5ef0 0x7f418c14c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.721+0000 7f4191e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f418c0c6340 0x7f418c14cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.721+0000 7f4191e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f418c1539c0 con 0x7f418c0c5ef0 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.721+0000 7f4191e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f418c0c3950 con 0x7f418c0c5b10 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.721+0000 7f4191e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f418c153b40 con 0x7f418c0c6340 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.721+0000 7f418a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f418c0c5ef0 0x7f418c14c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.721+0000 7f418a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f418c0c5ef0 0x7f418c14c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36688/0 (socket says 172.21.15.31:36688) 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.721+0000 7f418a000640 1 -- 172.21.15.31:0/1809430099 learned_addr learned my addr 172.21.15.31:0/1809430099 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.722+0000 7f418aa00640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f418c0c5b10 0x7f418c151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.722+0000 7f418b400640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f418c0c6340 0x7f418c14cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.722+0000 7f418a000640 1 -- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f418c0c6340 msgr2=0x7f418c14cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.722+0000 7f418a000640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f418c0c6340 0x7f418c14cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:03.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.722+0000 7f418a000640 1 -- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f418c0c5b10 msgr2=0x7f418c151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:03.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.722+0000 7f418a000640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f418c0c5b10 0x7f418c151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:03.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.722+0000 7f418a000640 1 -- 172.21.15.31:0/1809430099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418c1ef810 con 0x7f418c0c5ef0 2024-09-11T00:34:03.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.722+0000 7f418a000640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f418c0c5ef0 0x7f418c14c980 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4178009920 tx=0x7f4178032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:03.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.723+0000 7f4183400640 1 -- 172.21.15.31:0/1809430099 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178040070 con 0x7f418c0c5ef0 2024-09-11T00:34:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.723+0000 7f4183400640 1 -- 172.21.15.31:0/1809430099 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4178005310 con 0x7f418c0c5ef0 2024-09-11T00:34:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.723+0000 7f4183400640 1 -- 172.21.15.31:0/1809430099 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41780047a0 con 0x7f418c0c5ef0 2024-09-11T00:34:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.723+0000 7f4191e00640 1 -- 172.21.15.31:0/1809430099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f418c074f10 con 0x7f418c0c5ef0 2024-09-11T00:34:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.723+0000 7f4191e00640 1 -- 172.21.15.31:0/1809430099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f418c074440 con 0x7f418c0c5ef0 2024-09-11T00:34:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.724+0000 7f4191e00640 1 -- 172.21.15.31:0/1809430099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f418c156ad0 con 0x7f418c0c5ef0 2024-09-11T00:34:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.725+0000 7f4183400640 1 -- 172.21.15.31:0/1809430099 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4178004b50 con 0x7f418c0c5ef0 2024-09-11T00:34:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.726+0000 7f4183400640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f416c077f70 0x7f416c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.726+0000 7f4183400640 1 -- 172.21.15.31:0/1809430099 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f41780bb9d0 con 0x7f418c0c5ef0 2024-09-11T00:34:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.727+0000 7f418aa00640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f416c077f70 0x7f416c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.727+0000 7f418aa00640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f416c077f70 0x7f416c07a430 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f417c005fd0 tx=0x7f417c00b590 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:03.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.730+0000 7f4183400640 1 -- 172.21.15.31:0/1809430099 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4178083f10 con 0x7f418c0c5ef0 2024-09-11T00:34:03.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.985+0000 7f4191e00640 1 -- 172.21.15.31:0/1809430099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f418c0c4c20 con 0x7f418c0c5ef0 2024-09-11T00:34:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.986+0000 7f4183400640 1 -- 172.21.15.31:0/1809430099 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6532 (secure 0 0 0) 0x7f4178083810 con 0x7f418c0c5ef0 2024-09-11T00:34:03.988 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:03.988 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"btime":"2024-09-11T00:33:39:561976+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4898,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:37.543487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-09-11T00:34:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 36 2024-09-11T00:34:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 -- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f416c077f70 msgr2=0x7f416c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f416c077f70 0x7f416c07a430 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f417c005fd0 tx=0x7f417c00b590 comp rx=0 tx=0).stop 2024-09-11T00:34:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 -- 172.21.15.31:0/1809430099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f418c0c5ef0 msgr2=0x7f418c14c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f418c0c5ef0 0x7f418c14c980 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4178009920 tx=0x7f4178032c10 comp rx=0 tx=0).stop 2024-09-11T00:34:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 -- 172.21.15.31:0/1809430099 shutdown_connections 2024-09-11T00:34:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f416c077f70 0x7f416c07a430 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f418c0c6340 0x7f418c14cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f418c0c5ef0 0x7f418c14c980 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 --2- 172.21.15.31:0/1809430099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f418c0c5b10 0x7f418c151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:03.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 -- 172.21.15.31:0/1809430099 >> 172.21.15.31:0/1809430099 conn(0x7f418c0a9750 msgr2=0x7f418c1520c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:03.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 -- 172.21.15.31:0/1809430099 shutdown_connections 2024-09-11T00:34:03.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:03.989+0000 7f4191e00640 1 -- 172.21.15.31:0/1809430099 wait complete. 2024-09-11T00:34:04.012 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:34:06.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:06.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.210+0000 7fe5cbe00640 1 -- 172.21.15.31:0/1621615985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5b0004800 msgr2=0x7fe5b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:06.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.210+0000 7fe5cbe00640 1 --2- 172.21.15.31:0/1621615985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5b0004800 0x7fe5b0004c80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009b50 tx=0x7fe5b0005830 comp rx=0 tx=0).stop 2024-09-11T00:34:06.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.210+0000 7fe5cbe00640 1 -- 172.21.15.31:0/1621615985 shutdown_connections 2024-09-11T00:34:06.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.210+0000 7fe5cbe00640 1 --2- 172.21.15.31:0/1621615985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5b0004800 0x7fe5b0004c80 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:06.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.210+0000 7fe5cbe00640 1 -- 172.21.15.31:0/1621615985 >> 172.21.15.31:0/1621615985 conn(0x7fe5cc06caf0 msgr2=0x7fe5cc06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:06.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.210+0000 7fe5cbe00640 1 -- 172.21.15.31:0/1621615985 shutdown_connections 2024-09-11T00:34:06.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.210+0000 7fe5cbe00640 1 -- 172.21.15.31:0/1621615985 wait complete. 2024-09-11T00:34:06.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.212+0000 7fe5cbe00640 1 Processor -- start 2024-09-11T00:34:06.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.212+0000 7fe5cbe00640 1 -- start start 2024-09-11T00:34:06.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.213+0000 7fe5cbe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5b0004800 0x7fe5cc0c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:06.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.213+0000 7fe5cbe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5cc147e50 0x7fe5cc1eb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:06.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.213+0000 7fe5cbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5cc1ebd80 0x7fe5cc072280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:06.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.213+0000 7fe5cbe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc151450 con 0x7fe5cc1ebd80 2024-09-11T00:34:06.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.213+0000 7fe5cbe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc1512d0 con 0x7fe5b0004800 2024-09-11T00:34:06.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.213+0000 7fe5cbe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe5cc0c5d20 con 0x7fe5cc147e50 2024-09-11T00:34:06.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.214+0000 7fe5cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5cc1ebd80 0x7fe5cc072280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:06.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.214+0000 7fe5cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5cc1ebd80 0x7fe5cc072280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36700/0 (socket says 172.21.15.31:36700) 2024-09-11T00:34:06.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.214+0000 7fe5cb400640 1 -- 172.21.15.31:0/3340894434 learned_addr learned my addr 172.21.15.31:0/3340894434 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:06.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.214+0000 7fe5caa00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5b0004800 0x7fe5cc0c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:06.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.214+0000 7fe5ca000640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5cc147e50 0x7fe5cc1eb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:06.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.214+0000 7fe5caa00640 1 -- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5cc147e50 msgr2=0x7fe5cc1eb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:06.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.214+0000 7fe5caa00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5cc147e50 0x7fe5cc1eb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:06.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.214+0000 7fe5caa00640 1 -- 172.21.15.31:0/3340894434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5cc1ebd80 msgr2=0x7fe5cc072280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:06.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.214+0000 7fe5caa00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5cc1ebd80 0x7fe5cc072280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:06.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.214+0000 7fe5caa00640 1 -- 172.21.15.31:0/3340894434 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc1ec210 con 0x7fe5b0004800 2024-09-11T00:34:06.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.215+0000 7fe5caa00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5b0004800 0x7fe5cc0c3750 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc001160 tx=0x7fe5bc00ea20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:06.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.215+0000 7fe5c2a00640 1 -- 172.21.15.31:0/3340894434 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc015020 con 0x7fe5b0004800 2024-09-11T00:34:06.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.215+0000 7fe5c2a00640 1 -- 172.21.15.31:0/3340894434 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5bc0109a0 con 0x7fe5b0004800 2024-09-11T00:34:06.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.215+0000 7fe5cbe00640 1 -- 172.21.15.31:0/3340894434 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5cc147b70 con 0x7fe5b0004800 2024-09-11T00:34:06.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.215+0000 7fe5c2a00640 1 -- 172.21.15.31:0/3340894434 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc010df0 con 0x7fe5b0004800 2024-09-11T00:34:06.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.215+0000 7fe5cbe00640 1 -- 172.21.15.31:0/3340894434 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5cc1ef090 con 0x7fe5b0004800 2024-09-11T00:34:06.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.216+0000 7fe5cbe00640 1 -- 172.21.15.31:0/3340894434 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5cc0ac6a0 con 0x7fe5b0004800 2024-09-11T00:34:06.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.218+0000 7fe5c2a00640 1 -- 172.21.15.31:0/3340894434 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe5bc01a400 con 0x7fe5b0004800 2024-09-11T00:34:06.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.219+0000 7fe5c2a00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe5ac077f70 0x7fe5ac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:06.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.219+0000 7fe5c2a00640 1 -- 172.21.15.31:0/3340894434 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fe5bc09aa90 con 0x7fe5b0004800 2024-09-11T00:34:06.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.219+0000 7fe5ca000640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe5ac077f70 0x7fe5ac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:06.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.220+0000 7fe5ca000640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe5ac077f70 0x7fe5ac07a430 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8035000 tx=0x7fe5b80046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:06.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.224+0000 7fe5c2a00640 1 -- 172.21.15.31:0/3340894434 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe5bc01f020 con 0x7fe5b0004800 2024-09-11T00:34:06.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.501+0000 7fe5cbe00640 1 -- 172.21.15.31:0/3340894434 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5cc0c1700 con 0x7fe5b0004800 2024-09-11T00:34:06.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.502+0000 7fe5c2a00640 1 -- 172.21.15.31:0/3340894434 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6532 (secure 0 0 0) 0x7fe5bc098e20 con 0x7fe5b0004800 2024-09-11T00:34:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:06.504 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"btime":"2024-09-11T00:33:39:561976+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4898,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:37.543487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-09-11T00:34:06.504 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 36 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 -- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe5ac077f70 msgr2=0x7fe5ac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe5ac077f70 0x7fe5ac07a430 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8035000 tx=0x7fe5b80046c0 comp rx=0 tx=0).stop 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 -- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5b0004800 msgr2=0x7fe5cc0c3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5b0004800 0x7fe5cc0c3750 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc001160 tx=0x7fe5bc00ea20 comp rx=0 tx=0).stop 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 -- 172.21.15.31:0/3340894434 shutdown_connections 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe5ac077f70 0x7fe5ac07a430 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe5cc1ebd80 0x7fe5cc072280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe5cc147e50 0x7fe5cc1eb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 --2- 172.21.15.31:0/3340894434 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5b0004800 0x7fe5cc0c3750 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 -- 172.21.15.31:0/3340894434 >> 172.21.15.31:0/3340894434 conn(0x7fe5cc06caf0 msgr2=0x7fe5cc0c0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 -- 172.21.15.31:0/3340894434 shutdown_connections 2024-09-11T00:34:06.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:06.505+0000 7fe5cbe00640 1 -- 172.21.15.31:0/3340894434 wait complete. 2024-09-11T00:34:06.525 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:34:08.526 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:08.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.700+0000 7f8611200640 1 -- 172.21.15.31:0/1331382326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85f0004040 msgr2=0x7f85f0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:08.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.700+0000 7f8611200640 1 --2- 172.21.15.31:0/1331382326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85f0004040 0x7f85f0004800 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f85f4006fb0 tx=0x7f85f0005830 comp rx=0 tx=0).stop 2024-09-11T00:34:08.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.700+0000 7f860be00640 1 -- 172.21.15.31:0/1331382326 reap_dead start 2024-09-11T00:34:08.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.700+0000 7f8611200640 1 -- 172.21.15.31:0/1331382326 shutdown_connections 2024-09-11T00:34:08.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.700+0000 7f8611200640 1 -- 172.21.15.31:0/1331382326 >> 172.21.15.31:0/1331382326 conn(0x7f860c068f10 msgr2=0x7f860c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:08.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.700+0000 7f8611200640 1 -- 172.21.15.31:0/1331382326 shutdown_connections 2024-09-11T00:34:08.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.700+0000 7f8611200640 1 -- 172.21.15.31:0/1331382326 wait complete. 2024-09-11T00:34:08.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.702+0000 7f8611200640 1 Processor -- start 2024-09-11T00:34:08.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.702+0000 7f8611200640 1 -- start start 2024-09-11T00:34:08.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f8611200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85f0004040 0x7f860c1ec9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:08.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f8611200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f860c1e1e10 0x7f860c14d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f8611200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f860c14d610 0x7f860c1e08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f8611200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f860c157550 con 0x7f860c14d610 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f8611200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f860c1573d0 con 0x7f85f0004040 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f8611200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f860c156960 con 0x7f860c1e1e10 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f860be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f860c14d610 0x7f860c1e08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f860be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f860c14d610 0x7f860c1e08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34220/0 (socket says 172.21.15.31:34220) 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f860be00640 1 -- 172.21.15.31:0/2437795093 learned_addr learned my addr 172.21.15.31:0/2437795093 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f860b400640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85f0004040 0x7f860c1ec9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.703+0000 7f8603e00640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f860c1e1e10 0x7f860c14d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.704+0000 7f860be00640 1 -- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f860c1e1e10 msgr2=0x7f860c14d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.704+0000 7f860be00640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f860c1e1e10 0x7f860c14d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.704+0000 7f860be00640 1 -- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85f0004040 msgr2=0x7f860c1ec9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:08.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.704+0000 7f860be00640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85f0004040 0x7f860c1ec9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:08.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.704+0000 7f860be00640 1 -- 172.21.15.31:0/2437795093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c1ecef0 con 0x7f860c14d610 2024-09-11T00:34:08.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.704+0000 7f860be00640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f860c14d610 0x7f860c1e08c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8604011b00 tx=0x7f8604009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:08.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.704+0000 7f8603400640 1 -- 172.21.15.31:0/2437795093 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604004b20 con 0x7f860c14d610 2024-09-11T00:34:08.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.704+0000 7f8603400640 1 -- 172.21.15.31:0/2437795093 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8604004cc0 con 0x7f860c14d610 2024-09-11T00:34:08.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.704+0000 7f8611200640 1 -- 172.21.15.31:0/2437795093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860c1df5e0 con 0x7f860c14d610 2024-09-11T00:34:08.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.704+0000 7f8603400640 1 -- 172.21.15.31:0/2437795093 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604003bd0 con 0x7f860c14d610 2024-09-11T00:34:08.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.705+0000 7f8611200640 1 -- 172.21.15.31:0/2437795093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860c1dfba0 con 0x7f860c14d610 2024-09-11T00:34:08.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.706+0000 7f8611200640 1 -- 172.21.15.31:0/2437795093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f860c0cc220 con 0x7f860c14d610 2024-09-11T00:34:08.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.707+0000 7f8603400640 1 -- 172.21.15.31:0/2437795093 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8604018070 con 0x7f860c14d610 2024-09-11T00:34:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.708+0000 7f8603400640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f85e8077fa0 0x7f85e807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.708+0000 7f8603400640 1 -- 172.21.15.31:0/2437795093 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f860409a4f0 con 0x7f860c14d610 2024-09-11T00:34:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.709+0000 7f860b400640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f85e8077fa0 0x7f85e807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:08.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.710+0000 7f860b400640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f85e8077fa0 0x7f85e807a460 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009e50 tx=0x7f85fc005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:08.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.713+0000 7f8603400640 1 -- 172.21.15.31:0/2437795093 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8604062510 con 0x7f860c14d610 2024-09-11T00:34:08.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.971+0000 7f8611200640 1 -- 172.21.15.31:0/2437795093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f860c0cc8e0 con 0x7f860c14d610 2024-09-11T00:34:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.973+0000 7f8603400640 1 -- 172.21.15.31:0/2437795093 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6532 (secure 0 0 0) 0x7f8604066f40 con 0x7f860c14d610 2024-09-11T00:34:08.975 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:08.975 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":36,"btime":"2024-09-11T00:33:39:561976+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4898,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:33:37.543487+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":12,"addr":"172.21.15.204:6835/558470224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":558470224},{"type":"v1","addr":"172.21.15.204:6835","nonce":558470224}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-09-11T00:34:08.975 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 36 2024-09-11T00:34:08.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.975+0000 7f8611200640 1 -- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f85e8077fa0 msgr2=0x7f85e807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:08.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.975+0000 7f8611200640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f85e8077fa0 0x7f85e807a460 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009e50 tx=0x7f85fc005f90 comp rx=0 tx=0).stop 2024-09-11T00:34:08.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.975+0000 7f8611200640 1 -- 172.21.15.31:0/2437795093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f860c14d610 msgr2=0x7f860c1e08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.975+0000 7f8611200640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f860c14d610 0x7f860c1e08c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8604011b00 tx=0x7f8604009200 comp rx=0 tx=0).stop 2024-09-11T00:34:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.976+0000 7f8611200640 1 -- 172.21.15.31:0/2437795093 shutdown_connections 2024-09-11T00:34:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.976+0000 7f8611200640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f85e8077fa0 0x7f85e807a460 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.976+0000 7f8611200640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f860c14d610 0x7f860c1e08c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.976+0000 7f8611200640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f860c1e1e10 0x7f860c14d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.976+0000 7f8611200640 1 --2- 172.21.15.31:0/2437795093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85f0004040 0x7f860c1ec9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.976+0000 7f8611200640 1 -- 172.21.15.31:0/2437795093 >> 172.21.15.31:0/2437795093 conn(0x7f860c068f10 msgr2=0x7f860c0acb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.976+0000 7f8611200640 1 -- 172.21.15.31:0/2437795093 shutdown_connections 2024-09-11T00:34:08.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:08.976+0000 7f8611200640 1 -- 172.21.15.31:0/2437795093 wait complete. 2024-09-11T00:34:08.995 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:34:10.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:34:10.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:34:10.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.225+0000 7fb48fe00640 1 -- 172.21.15.31:0/243386825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4900c9a10 msgr2=0x7fb490159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.225+0000 7fb48fe00640 1 --2- 172.21.15.31:0/243386825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4900c9a10 0x7fb490159670 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb488005fd0 tx=0x7fb48802a8b0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.225+0000 7fb48fe00640 1 -- 172.21.15.31:0/243386825 shutdown_connections 2024-09-11T00:34:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.225+0000 7fb48fe00640 1 --2- 172.21.15.31:0/243386825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4900c9a10 0x7fb490159670 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.225+0000 7fb48fe00640 1 --2- 172.21.15.31:0/243386825 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb49014c8e0 0x7fb490156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.225+0000 7fb48fe00640 1 -- 172.21.15.31:0/243386825 >> 172.21.15.31:0/243386825 conn(0x7fb490068ee0 msgr2=0x7fb4900cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.225+0000 7fb48fe00640 1 -- 172.21.15.31:0/243386825 shutdown_connections 2024-09-11T00:34:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.225+0000 7fb48fe00640 1 -- 172.21.15.31:0/243386825 wait complete. 2024-09-11T00:34:11.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.227+0000 7fb48fe00640 1 Processor -- start 2024-09-11T00:34:11.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.227+0000 7fb48fe00640 1 -- start start 2024-09-11T00:34:11.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.228+0000 7fb48fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4900c9a10 0x7fb490159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:11.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.228+0000 7fb48fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb49014c8e0 0x7fb4901e1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:11.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.228+0000 7fb48fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb49014ccc0 0x7fb4901e2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:11.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.228+0000 7fb48fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb4900c6c90 con 0x7fb49014c8e0 2024-09-11T00:34:11.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.228+0000 7fb48fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb4900c6490 con 0x7fb4900c9a10 2024-09-11T00:34:11.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.228+0000 7fb48fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb4900adca0 con 0x7fb49014ccc0 2024-09-11T00:34:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.228+0000 7fb48e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb49014c8e0 0x7fb4901e1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.228+0000 7fb48e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb49014c8e0 0x7fb4901e1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34234/0 (socket says 172.21.15.31:34234) 2024-09-11T00:34:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.228+0000 7fb48e000640 1 -- 172.21.15.31:0/3219683540 learned_addr learned my addr 172.21.15.31:0/3219683540 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:11.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.229+0000 7fb48f400640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb49014ccc0 0x7fb4901e2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:11.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.229+0000 7fb48ea00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4900c9a10 0x7fb490159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:11.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.229+0000 7fb48e000640 1 -- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb49014ccc0 msgr2=0x7fb4901e2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:11.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.229+0000 7fb48e000640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb49014ccc0 0x7fb4901e2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.229+0000 7fb48e000640 1 -- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4900c9a10 msgr2=0x7fb490159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:11.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.229+0000 7fb48e000640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4900c9a10 0x7fb490159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.229+0000 7fb48e000640 1 -- 172.21.15.31:0/3219683540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb490157510 con 0x7fb49014c8e0 2024-09-11T00:34:11.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.229+0000 7fb48f400640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb49014ccc0 0x7fb4901e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:34:11.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.229+0000 7fb48ea00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4900c9a10 0x7fb490159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:34:11.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.229+0000 7fb48e000640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb49014c8e0 0x7fb4901e1dc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00c3c0 tx=0x7fb47c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:11.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.230+0000 7fb486a00640 1 -- 172.21.15.31:0/3219683540 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c017070 con 0x7fb49014c8e0 2024-09-11T00:34:11.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.230+0000 7fb486a00640 1 -- 172.21.15.31:0/3219683540 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb47c0035f0 con 0x7fb49014c8e0 2024-09-11T00:34:11.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.230+0000 7fb48fe00640 1 -- 172.21.15.31:0/3219683540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb490158530 con 0x7fb49014c8e0 2024-09-11T00:34:11.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.230+0000 7fb486a00640 1 -- 172.21.15.31:0/3219683540 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c003a20 con 0x7fb49014c8e0 2024-09-11T00:34:11.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.230+0000 7fb48fe00640 1 -- 172.21.15.31:0/3219683540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4901e2840 con 0x7fb49014c8e0 2024-09-11T00:34:11.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.231+0000 7fb48fe00640 1 -- 172.21.15.31:0/3219683540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb490156b00 con 0x7fb49014c8e0 2024-09-11T00:34:11.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.232+0000 7fb486a00640 1 -- 172.21.15.31:0/3219683540 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb47c003bc0 con 0x7fb49014c8e0 2024-09-11T00:34:11.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.234+0000 7fb486a00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb470077fa0 0x7fb47007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:11.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.234+0000 7fb486a00640 1 -- 172.21.15.31:0/3219683540 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7fb47c0998f0 con 0x7fb49014c8e0 2024-09-11T00:34:11.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.234+0000 7fb48ea00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb470077fa0 0x7fb47007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:11.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.235+0000 7fb48ea00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb470077fa0 0x7fb47007a460 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb480009c30 tx=0x7fb480005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:11.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.238+0000 7fb486a00640 1 -- 172.21.15.31:0/3219683540 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb47c061d60 con 0x7fb49014c8e0 2024-09-11T00:34:11.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.530+0000 7fb48fe00640 1 -- 172.21.15.31:0/3219683540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4900cab80 con 0x7fb49014c8e0 2024-09-11T00:34:11.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.530+0000 7fb486a00640 1 -- 172.21.15.31:0/3219683540 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+5700 (secure 0 0 0) 0x7fb47c061660 con 0x7fb49014c8e0 2024-09-11T00:34:11.532 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:11.532 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":39,"btime":"2024-09-11T00:34:11:307207+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:10.310214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:34:11.533 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 39 2024-09-11T00:34:11.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 -- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb470077fa0 msgr2=0x7fb47007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:11.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb470077fa0 0x7fb47007a460 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb480009c30 tx=0x7fb480005f90 comp rx=0 tx=0).stop 2024-09-11T00:34:11.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 -- 172.21.15.31:0/3219683540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb49014c8e0 msgr2=0x7fb4901e1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:11.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb49014c8e0 0x7fb4901e1dc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00c3c0 tx=0x7fb47c00c880 comp rx=0 tx=0).stop 2024-09-11T00:34:11.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 -- 172.21.15.31:0/3219683540 shutdown_connections 2024-09-11T00:34:11.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb470077fa0 0x7fb47007a460 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb49014ccc0 0x7fb4901e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb49014c8e0 0x7fb4901e1dc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 --2- 172.21.15.31:0/3219683540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4900c9a10 0x7fb490159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 -- 172.21.15.31:0/3219683540 >> 172.21.15.31:0/3219683540 conn(0x7fb490068ee0 msgr2=0x7fb4900ab570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:11.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.533+0000 7fb48fe00640 1 -- 172.21.15.31:0/3219683540 shutdown_connections 2024-09-11T00:34:11.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.534+0000 7fb48fe00640 1 -- 172.21.15.31:0/3219683540 wait complete. 2024-09-11T00:34:11.554 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-11T00:34:11.554 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:34:11.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:11.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.738+0000 7f3948a00640 1 -- 172.21.15.31:0/1838521321 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3944150020 msgr2=0x7f39440ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:11.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.738+0000 7f3948a00640 1 --2- 172.21.15.31:0/1838521321 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3944150020 0x7f39440ad4e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3934009890 tx=0x7f393402a5d0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.738+0000 7f3948a00640 1 -- 172.21.15.31:0/1838521321 shutdown_connections 2024-09-11T00:34:11.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.738+0000 7f3948a00640 1 --2- 172.21.15.31:0/1838521321 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3944150020 0x7f39440ad4e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.738+0000 7f3948a00640 1 -- 172.21.15.31:0/1838521321 >> 172.21.15.31:0/1838521321 conn(0x7f3944068f10 msgr2=0x7f39440cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:11.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.738+0000 7f3948a00640 1 -- 172.21.15.31:0/1838521321 shutdown_connections 2024-09-11T00:34:11.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.738+0000 7f3948a00640 1 -- 172.21.15.31:0/1838521321 wait complete. 2024-09-11T00:34:11.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.740+0000 7f3948a00640 1 Processor -- start 2024-09-11T00:34:11.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.740+0000 7f3948a00640 1 -- start start 2024-09-11T00:34:11.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.741+0000 7f3948a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f394406e850 0x7f39441dff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:11.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.741+0000 7f3948a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39440ada20 0x7f3944150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:11.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.741+0000 7f3948a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3944150020 0x7f39441e1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:11.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.741+0000 7f3948a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3944157550 con 0x7f394406e850 2024-09-11T00:34:11.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.741+0000 7f3948a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f39441573d0 con 0x7f39440ada20 2024-09-11T00:34:11.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.741+0000 7f3948a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3944156960 con 0x7f3944150020 2024-09-11T00:34:11.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.741+0000 7f3942800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f394406e850 0x7f39441dff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:11.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.741+0000 7f3942800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f394406e850 0x7f39441dff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34250/0 (socket says 172.21.15.31:34250) 2024-09-11T00:34:11.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.741+0000 7f3942800640 1 -- 172.21.15.31:0/1155708277 learned_addr learned my addr 172.21.15.31:0/1155708277 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:11.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.742+0000 7f3941e00640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39440ada20 0x7f3944150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:11.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.742+0000 7f3943200640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3944150020 0x7f39441e1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:11.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.742+0000 7f3942800640 1 -- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3944150020 msgr2=0x7f39441e1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:11.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.742+0000 7f3942800640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3944150020 0x7f39441e1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.742+0000 7f3942800640 1 -- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39440ada20 msgr2=0x7f3944150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:11.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.742+0000 7f3942800640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39440ada20 0x7f3944150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:11.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.742+0000 7f3942800640 1 -- 172.21.15.31:0/1155708277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3944150940 con 0x7f394406e850 2024-09-11T00:34:11.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.743+0000 7f3942800640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f394406e850 0x7f39441dff80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f39340099b0 tx=0x7f3934032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:11.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.743+0000 7f393aa00640 1 -- 172.21.15.31:0/1155708277 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934040070 con 0x7f394406e850 2024-09-11T00:34:11.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.743+0000 7f393aa00640 1 -- 172.21.15.31:0/1155708277 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3934032e70 con 0x7f394406e850 2024-09-11T00:34:11.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.743+0000 7f393aa00640 1 -- 172.21.15.31:0/1155708277 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39340055b0 con 0x7f394406e850 2024-09-11T00:34:11.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.743+0000 7f3948a00640 1 -- 172.21.15.31:0/1155708277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39441e09c0 con 0x7f394406e850 2024-09-11T00:34:11.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.743+0000 7f3948a00640 1 -- 172.21.15.31:0/1155708277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3944150d10 con 0x7f394406e850 2024-09-11T00:34:11.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.744+0000 7f3948a00640 1 -- 172.21.15.31:0/1155708277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39440cc220 con 0x7f394406e850 2024-09-11T00:34:11.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.745+0000 7f393aa00640 1 -- 172.21.15.31:0/1155708277 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3934003800 con 0x7f394406e850 2024-09-11T00:34:11.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.746+0000 7f393aa00640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3924077fa0 0x7f392407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:11.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.747+0000 7f393aa00640 1 -- 172.21.15.31:0/1155708277 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f39340bb790 con 0x7f394406e850 2024-09-11T00:34:11.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.747+0000 7f3941e00640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3924077fa0 0x7f392407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:11.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.748+0000 7f3941e00640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3924077fa0 0x7f392407a460 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3930000f60 tx=0x7f3930009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:11.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:11.751+0000 7f393aa00640 1 -- 172.21.15.31:0/1155708277 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3934083640 con 0x7f394406e850 2024-09-11T00:34:12.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.046+0000 7f3948a00640 1 -- 172.21.15.31:0/1155708277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39440cc8e0 con 0x7f394406e850 2024-09-11T00:34:12.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.047+0000 7f393aa00640 1 -- 172.21.15.31:0/1155708277 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+5700 (secure 0 0 0) 0x7f3934088070 con 0x7f394406e850 2024-09-11T00:34:12.049 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:12.049 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":39,"btime":"2024-09-11T00:34:11:307207+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:10.310214+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:34:12.049 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 39 2024-09-11T00:34:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 -- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3924077fa0 msgr2=0x7f392407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3924077fa0 0x7f392407a460 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3930000f60 tx=0x7f3930009040 comp rx=0 tx=0).stop 2024-09-11T00:34:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 -- 172.21.15.31:0/1155708277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f394406e850 msgr2=0x7f39441dff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f394406e850 0x7f39441dff80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f39340099b0 tx=0x7f3934032c80 comp rx=0 tx=0).stop 2024-09-11T00:34:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 -- 172.21.15.31:0/1155708277 shutdown_connections 2024-09-11T00:34:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3924077fa0 0x7f392407a460 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3944150020 0x7f39441e1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39440ada20 0x7f3944150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 --2- 172.21.15.31:0/1155708277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f394406e850 0x7f39441dff80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 -- 172.21.15.31:0/1155708277 >> 172.21.15.31:0/1155708277 conn(0x7f3944068f10 msgr2=0x7f394414b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:12.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 -- 172.21.15.31:0/1155708277 shutdown_connections 2024-09-11T00:34:12.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:12.050+0000 7f3948a00640 1 -- 172.21.15.31:0/1155708277 wait complete. 2024-09-11T00:34:12.069 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:34:11:307207+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 39, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5076": { "addr": "172.21.15.31:6835/4202305151", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6833", "nonce": 4202305151, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 4202305151, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5076, "incarnation": 26, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:32:58.315114+0000", "qdb_cluster": [ 5076 ], "qdb_leader": 5076, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5076 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 39, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4898": { "addr": "172.21.15.204:6837/645869523", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 645869523, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 645869523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4898, "incarnation": 37, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:34:10.310214+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4898 } } } ], "standbys": [ { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:34:14.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:14.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.240+0000 7febc7e00640 1 -- 172.21.15.31:0/577687842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8150020 msgr2=0x7febc80ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:14.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.240+0000 7febc7e00640 1 --2- 172.21.15.31:0/577687842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8150020 0x7febc80ad4e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7febc8153d20 tx=0x7febb802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:34:14.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.240+0000 7febc7e00640 1 -- 172.21.15.31:0/577687842 shutdown_connections 2024-09-11T00:34:14.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.240+0000 7febc7e00640 1 --2- 172.21.15.31:0/577687842 >> v2:172.21.15.204:3300/0 conn(0x7febc80ada20 0x7febc806e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:14.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.240+0000 7febc7e00640 1 --2- 172.21.15.31:0/577687842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8150020 0x7febc80ad4e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:14.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.240+0000 7febc7e00640 1 -- 172.21.15.31:0/577687842 >> 172.21.15.31:0/577687842 conn(0x7febc8068f10 msgr2=0x7febc80cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:14.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.240+0000 7febc7e00640 1 -- 172.21.15.31:0/577687842 shutdown_connections 2024-09-11T00:34:14.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.240+0000 7febc7e00640 1 -- 172.21.15.31:0/577687842 wait complete. 2024-09-11T00:34:14.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.243+0000 7febc7e00640 1 Processor -- start 2024-09-11T00:34:14.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.243+0000 7febc7e00640 1 -- start start 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febc80ada20 0x7febc80c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8150020 0x7febc81ef270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7febc8156960 0x7febc81e08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7febc8157550 con 0x7febc8150020 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc7e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7febc81573d0 con 0x7febc8156960 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc7e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7febc80cc220 con 0x7febc80ada20 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febc80ada20 0x7febc80c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8150020 0x7febc81ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febc80ada20 0x7febc80c39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:60504/0 (socket says 172.21.15.31:60504) 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc6a00640 1 -- 172.21.15.31:0/2634872513 learned_addr learned my addr 172.21.15.31:0/2634872513 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:14.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.244+0000 7febc7400640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7febc8156960 0x7febc81e08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:14.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.245+0000 7febc6a00640 1 -- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7febc8156960 msgr2=0x7febc81e08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:14.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.245+0000 7febc6a00640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7febc8156960 0x7febc81e08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:14.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.245+0000 7febc6a00640 1 -- 172.21.15.31:0/2634872513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8150020 msgr2=0x7febc81ef270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:14.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.245+0000 7febc6a00640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8150020 0x7febc81ef270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:14.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.245+0000 7febc6a00640 1 -- 172.21.15.31:0/2634872513 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc814eed0 con 0x7febc80ada20 2024-09-11T00:34:14.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.245+0000 7febc6a00640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febc80ada20 0x7febc80c39a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7febb8009850 tx=0x7febb80038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:14.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.246+0000 7febbea00640 1 -- 172.21.15.31:0/2634872513 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb803f070 con 0x7febc80ada20 2024-09-11T00:34:14.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.246+0000 7febc7e00640 1 -- 172.21.15.31:0/2634872513 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc81efbb0 con 0x7febc80ada20 2024-09-11T00:34:14.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.246+0000 7febbea00640 1 -- 172.21.15.31:0/2634872513 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febb80047a0 con 0x7febc80ada20 2024-09-11T00:34:14.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.246+0000 7febbea00640 1 -- 172.21.15.31:0/2634872513 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8035030 con 0x7febc80ada20 2024-09-11T00:34:14.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.246+0000 7febc7e00640 1 -- 172.21.15.31:0/2634872513 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7febc8153760 con 0x7febc80ada20 2024-09-11T00:34:14.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.247+0000 7febc7e00640 1 -- 172.21.15.31:0/2634872513 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febc80ad4e0 con 0x7febc80ada20 2024-09-11T00:34:14.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.248+0000 7febbea00640 1 -- 172.21.15.31:0/2634872513 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7febb8004bf0 con 0x7febc80ada20 2024-09-11T00:34:14.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.249+0000 7febbea00640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feba8077f70 0x7feba807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:14.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.249+0000 7febbea00640 1 -- 172.21.15.31:0/2634872513 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7febb80bb3c0 con 0x7febc80ada20 2024-09-11T00:34:14.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.250+0000 7febc6000640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feba8077f70 0x7feba807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:14.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.250+0000 7febc6000640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feba8077f70 0x7feba807a430 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7febc80c27a0 tx=0x7febb4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:14.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.255+0000 7febbea00640 1 -- 172.21.15.31:0/2634872513 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7febb8083270 con 0x7febc80ada20 2024-09-11T00:34:14.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.581+0000 7febc7e00640 1 -- 172.21.15.31:0/2634872513 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7febc80cb8a0 con 0x7febc80ada20 2024-09-11T00:34:14.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.582+0000 7febbea00640 1 -- 172.21.15.31:0/2634872513 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5707 (secure 0 0 0) 0x7febb8087ca0 con 0x7febc80ada20 2024-09-11T00:34:14.585 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:14.585 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"btime":"2024-09-11T00:34:12:311597+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:34:14.585 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 -- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feba8077f70 msgr2=0x7feba807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feba8077f70 0x7feba807a430 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7febc80c27a0 tx=0x7febb4005f90 comp rx=0 tx=0).stop 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 -- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febc80ada20 msgr2=0x7febc80c39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febc80ada20 0x7febc80c39a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7febb8009850 tx=0x7febb80038a0 comp rx=0 tx=0).stop 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 -- 172.21.15.31:0/2634872513 shutdown_connections 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feba8077f70 0x7feba807a430 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7febc8156960 0x7febc81e08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febc8150020 0x7febc81ef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 --2- 172.21.15.31:0/2634872513 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7febc80ada20 0x7febc80c39a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 -- 172.21.15.31:0/2634872513 >> 172.21.15.31:0/2634872513 conn(0x7febc8068f10 msgr2=0x7febc81549d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 -- 172.21.15.31:0/2634872513 shutdown_connections 2024-09-11T00:34:14.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:14.585+0000 7febc7e00640 1 -- 172.21.15.31:0/2634872513 wait complete. 2024-09-11T00:34:14.604 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=4829 2024-09-11T00:34:14.604 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.d 2024-09-11T00:34:15.573 INFO:tasks.ceph.mds.b.smithi204.stderr:2024-09-11T00:34:15.570+0000 7fdb2d000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:34:18.605 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-11T00:34:18.606 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:34:18.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:34:18.611 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:34:18.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:18.762 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:34:18.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.797+0000 7f8dc7200640 1 -- 172.21.15.31:0/1201986818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc80c5a20 msgr2=0x7f8dc80bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:18.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.797+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1201986818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc80c5a20 0x7f8dc80bf600 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80c7220 tx=0x7f8db402a620 comp rx=0 tx=0).stop 2024-09-11T00:34:18.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.797+0000 7f8dc7200640 1 -- 172.21.15.31:0/1201986818 shutdown_connections 2024-09-11T00:34:18.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.797+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1201986818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc80c5a20 0x7f8dc80bf600 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:18.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.797+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1201986818 >> v2:172.21.15.204:3300/0 conn(0x7f8dc80c6760 0x7f8dc80c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:18.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.797+0000 7f8dc7200640 1 -- 172.21.15.31:0/1201986818 >> 172.21.15.31:0/1201986818 conn(0x7f8dc80694f0 msgr2=0x7f8dc80c9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:18.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.797+0000 7f8dc7200640 1 -- 172.21.15.31:0/1201986818 shutdown_connections 2024-09-11T00:34:18.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.797+0000 7f8dc7200640 1 -- 172.21.15.31:0/1201986818 wait complete. 2024-09-11T00:34:18.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.799+0000 7f8dc7200640 1 Processor -- start 2024-09-11T00:34:18.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.799+0000 7f8dc7200640 1 -- start start 2024-09-11T00:34:18.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.800+0000 7f8dc7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8dc80c5a20 0x7f8dc81ec930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:18.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.800+0000 7f8dc7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc80c6760 0x7f8dc8070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:18.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.800+0000 7f8dc7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc80c6b40 0x7f8dc81efb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:18.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.800+0000 7f8dc7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc80c9460 con 0x7f8dc80c6b40 2024-09-11T00:34:18.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.800+0000 7f8dc7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc80c92e0 con 0x7f8dc80c6760 2024-09-11T00:34:18.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.800+0000 7f8dc7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc8072ab0 con 0x7f8dc80c5a20 2024-09-11T00:34:18.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.800+0000 7f8dc6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc80c6b40 0x7f8dc81efb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:18.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc80c6760 0x7f8dc8070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:18.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc5e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8dc80c5a20 0x7f8dc81ec930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:18.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc80c6760 0x7f8dc8070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:56444/0 (socket says 172.21.15.31:56444) 2024-09-11T00:34:18.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc5400640 1 -- 172.21.15.31:0/1648983221 learned_addr learned my addr 172.21.15.31:0/1648983221 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:18.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc80c6b40 0x7f8dc81efb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54860/0 (socket says 172.21.15.31:54860) 2024-09-11T00:34:18.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc6800640 1 -- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8dc80c5a20 msgr2=0x7f8dc81ec930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc6800640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8dc80c5a20 0x7f8dc81ec930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc6800640 1 -- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc80c6760 msgr2=0x7f8dc8070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc6800640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc80c6760 0x7f8dc8070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc6800640 1 -- 172.21.15.31:0/1648983221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc81df590 con 0x7f8dc80c6b40 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.801+0000 7f8dc6800640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc80c6b40 0x7f8dc81efb70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000c730 tx=0x7f8dc000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.802+0000 7f8dbe000640 1 -- 172.21.15.31:0/1648983221 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0003b70 con 0x7f8dc80c6b40 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.802+0000 7f8dbe000640 1 -- 172.21.15.31:0/1648983221 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8dc0003d10 con 0x7f8dc80c6b40 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.802+0000 7f8dbe000640 1 -- 172.21.15.31:0/1648983221 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0005330 con 0x7f8dc80c6b40 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.802+0000 7f8dc7200640 1 -- 172.21.15.31:0/1648983221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc81df870 con 0x7f8dc80c6b40 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.802+0000 7f8dc7200640 1 -- 172.21.15.31:0/1648983221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc81ef670 con 0x7f8dc80c6b40 2024-09-11T00:34:18.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.803+0000 7f8dc7200640 1 -- 172.21.15.31:0/1648983221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dc81ef480 con 0x7f8dc80c6b40 2024-09-11T00:34:18.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.804+0000 7f8dbe000640 1 -- 172.21.15.31:0/1648983221 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8dc000a640 con 0x7f8dc80c6b40 2024-09-11T00:34:18.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.805+0000 7f8dbe000640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8da8077f70 0x7f8da807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:18.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.805+0000 7f8dbe000640 1 -- 172.21.15.31:0/1648983221 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f8dc009a130 con 0x7f8dc80c6b40 2024-09-11T00:34:18.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.806+0000 7f8dc5e00640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8da8077f70 0x7f8da807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:18.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.806+0000 7f8dc5e00640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8da8077f70 0x7f8da807a430 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009bd0 tx=0x7f8db8005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:18.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:18.811+0000 7f8dbe000640 1 -- 172.21.15.31:0/1648983221 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8dc80a8f70 con 0x7f8dc80c6b40 2024-09-11T00:34:19.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.062+0000 7f8dc7200640 1 -- 172.21.15.31:0/1648983221 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dc80bfc20 con 0x7f8dc80c6b40 2024-09-11T00:34:19.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.062+0000 7f8dbe000640 1 -- 172.21.15.31:0/1648983221 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5707 (secure 0 0 0) 0x7f8dc0061fe0 con 0x7f8dc80c6b40 2024-09-11T00:34:19.065 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:19.065 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":40,"btime":"2024-09-11T00:34:12:311597+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:34:19.066 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 40 2024-09-11T00:34:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 -- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8da8077f70 msgr2=0x7f8da807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8da8077f70 0x7f8da807a430 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009bd0 tx=0x7f8db8005f90 comp rx=0 tx=0).stop 2024-09-11T00:34:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 -- 172.21.15.31:0/1648983221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc80c6b40 msgr2=0x7f8dc81efb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc80c6b40 0x7f8dc81efb70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000c730 tx=0x7f8dc000cbf0 comp rx=0 tx=0).stop 2024-09-11T00:34:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 -- 172.21.15.31:0/1648983221 shutdown_connections 2024-09-11T00:34:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8da8077f70 0x7f8da807a430 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dc80c6b40 0x7f8dc81efb70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc80c6760 0x7f8dc8070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 --2- 172.21.15.31:0/1648983221 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8dc80c5a20 0x7f8dc81ec930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 -- 172.21.15.31:0/1648983221 >> 172.21.15.31:0/1648983221 conn(0x7f8dc80694f0 msgr2=0x7f8dc8156e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 -- 172.21.15.31:0/1648983221 shutdown_connections 2024-09-11T00:34:19.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:19.065+0000 7f8dc7200640 1 -- 172.21.15.31:0/1648983221 wait complete. 2024-09-11T00:34:19.088 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:34:21.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.263+0000 7f33c1a00640 1 -- 172.21.15.31:0/2989797568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33bc150020 msgr2=0x7f33bc0ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:21.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.263+0000 7f33c1a00640 1 --2- 172.21.15.31:0/2989797568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33bc150020 0x7f33bc0ad4e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009830 tx=0x7f33ac02a5b0 comp rx=0 tx=0).stop 2024-09-11T00:34:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.263+0000 7f33c1a00640 1 -- 172.21.15.31:0/2989797568 shutdown_connections 2024-09-11T00:34:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.263+0000 7f33c1a00640 1 --2- 172.21.15.31:0/2989797568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33bc150020 0x7f33bc0ad4e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.263+0000 7f33c1a00640 1 -- 172.21.15.31:0/2989797568 >> 172.21.15.31:0/2989797568 conn(0x7f33bc068f10 msgr2=0x7f33bc0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.263+0000 7f33c1a00640 1 -- 172.21.15.31:0/2989797568 shutdown_connections 2024-09-11T00:34:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.263+0000 7f33c1a00640 1 -- 172.21.15.31:0/2989797568 wait complete. 2024-09-11T00:34:21.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.265+0000 7f33c1a00640 1 Processor -- start 2024-09-11T00:34:21.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.265+0000 7f33c1a00640 1 -- start start 2024-09-11T00:34:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33c1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33bc06e850 0x7f33bc1e19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33c1a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33bc0ada20 0x7f33bc150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:21.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33c1a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33bc150020 0x7f33bc1efbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33c1a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc157550 con 0x7f33bc06e850 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33c1a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc1573d0 con 0x7f33bc0ada20 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33c1a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f33bc156960 con 0x7f33bc150020 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33bbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33bc06e850 0x7f33bc1e19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33bbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33bc06e850 0x7f33bc1e19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54878/0 (socket says 172.21.15.31:54878) 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33bbe00640 1 -- 172.21.15.31:0/2138059391 learned_addr learned my addr 172.21.15.31:0/2138059391 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33bb400640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33bc0ada20 0x7f33bc150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.266+0000 7f33c1000640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33bc150020 0x7f33bc1efbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.267+0000 7f33bbe00640 1 -- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33bc150020 msgr2=0x7f33bc1efbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.267+0000 7f33bbe00640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33bc150020 0x7f33bc1efbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.267+0000 7f33bbe00640 1 -- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33bc0ada20 msgr2=0x7f33bc150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:21.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.267+0000 7f33bbe00640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33bc0ada20 0x7f33bc150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.267+0000 7f33bbe00640 1 -- 172.21.15.31:0/2138059391 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33bc14b020 con 0x7f33bc06e850 2024-09-11T00:34:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.267+0000 7f33bbe00640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33bc06e850 0x7f33bc1e19a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f33ac000c00 tx=0x7f33ac033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:21.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.267+0000 7f33b3e00640 1 -- 172.21.15.31:0/2138059391 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac031d60 con 0x7f33bc06e850 2024-09-11T00:34:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.267+0000 7f33b3e00640 1 -- 172.21.15.31:0/2138059391 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33ac033070 con 0x7f33bc06e850 2024-09-11T00:34:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.267+0000 7f33b3e00640 1 -- 172.21.15.31:0/2138059391 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ac0334c0 con 0x7f33bc06e850 2024-09-11T00:34:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.267+0000 7f33c1a00640 1 -- 172.21.15.31:0/2138059391 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33bc153540 con 0x7f33bc06e850 2024-09-11T00:34:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.268+0000 7f33c1a00640 1 -- 172.21.15.31:0/2138059391 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33bc1e1ee0 con 0x7f33bc06e850 2024-09-11T00:34:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.269+0000 7f33c1a00640 1 -- 172.21.15.31:0/2138059391 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33bc153780 con 0x7f33bc06e850 2024-09-11T00:34:21.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.270+0000 7f33b3e00640 1 -- 172.21.15.31:0/2138059391 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f33ac0035f0 con 0x7f33bc06e850 2024-09-11T00:34:21.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.271+0000 7f33b3e00640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f339c077f70 0x7f339c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:21.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.271+0000 7f33b3e00640 1 -- 172.21.15.31:0/2138059391 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f33ac0bb5c0 con 0x7f33bc06e850 2024-09-11T00:34:21.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.271+0000 7f33bb400640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f339c077f70 0x7f339c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:21.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.272+0000 7f33bb400640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f339c077f70 0x7f339c07a430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f33bc1ed0d0 tx=0x7f33a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:21.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.277+0000 7f33b3e00640 1 -- 172.21.15.31:0/2138059391 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f33bc153780 con 0x7f33bc06e850 2024-09-11T00:34:21.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.542+0000 7f33c1a00640 1 -- 172.21.15.31:0/2138059391 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33bc0adef0 con 0x7f33bc06e850 2024-09-11T00:34:21.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.542+0000 7f33b3e00640 1 -- 172.21.15.31:0/2138059391 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6535 (secure 0 0 0) 0x7f33bc0adef0 con 0x7f33bc06e850 2024-09-11T00:34:21.545 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:21.545 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":41,"btime":"2024-09-11T00:34:19:339980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5302,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:34:21.546 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 41 2024-09-11T00:34:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 -- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f339c077f70 msgr2=0x7f339c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f339c077f70 0x7f339c07a430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f33bc1ed0d0 tx=0x7f33a8009040 comp rx=0 tx=0).stop 2024-09-11T00:34:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 -- 172.21.15.31:0/2138059391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33bc06e850 msgr2=0x7f33bc1e19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33bc06e850 0x7f33bc1e19a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f33ac000c00 tx=0x7f33ac033d60 comp rx=0 tx=0).stop 2024-09-11T00:34:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 -- 172.21.15.31:0/2138059391 shutdown_connections 2024-09-11T00:34:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f339c077f70 0x7f339c07a430 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:21.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33bc150020 0x7f33bc1efbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33bc0ada20 0x7f33bc150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 --2- 172.21.15.31:0/2138059391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33bc06e850 0x7f33bc1e19a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 -- 172.21.15.31:0/2138059391 >> 172.21.15.31:0/2138059391 conn(0x7f33bc068f10 msgr2=0x7f33bc14ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.546+0000 7f33c1a00640 1 -- 172.21.15.31:0/2138059391 shutdown_connections 2024-09-11T00:34:21.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:21.547+0000 7f33c1a00640 1 -- 172.21.15.31:0/2138059391 wait complete. 2024-09-11T00:34:21.569 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-11T00:34:21.569 INFO:tasks.mds_thrash.fs.[b]:waiting for 74 secs before thrashing 2024-09-11T00:34:41.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:34:41.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:34:54.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.885+0000 7ff113e00640 1 -- 172.21.15.31:0/2842343739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff114150020 msgr2=0x7ff1140ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.885+0000 7ff113e00640 1 --2- 172.21.15.31:0/2842343739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff114150020 0x7ff1140ad4e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff104009830 tx=0x7ff10402a5c0 comp rx=0 tx=0).stop 2024-09-11T00:34:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.886+0000 7ff113e00640 1 -- 172.21.15.31:0/2842343739 shutdown_connections 2024-09-11T00:34:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.886+0000 7ff113e00640 1 --2- 172.21.15.31:0/2842343739 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff114150020 0x7ff1140ad4e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.886+0000 7ff113e00640 1 -- 172.21.15.31:0/2842343739 >> 172.21.15.31:0/2842343739 conn(0x7ff114068f10 msgr2=0x7ff1140cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.886+0000 7ff113e00640 1 -- 172.21.15.31:0/2842343739 shutdown_connections 2024-09-11T00:34:54.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.886+0000 7ff113e00640 1 -- 172.21.15.31:0/2842343739 wait complete. 2024-09-11T00:34:54.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.888+0000 7ff113e00640 1 Processor -- start 2024-09-11T00:34:54.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.888+0000 7ff113e00640 1 -- start start 2024-09-11T00:34:54.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.889+0000 7ff113e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff11406e850 0x7ff1140c1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:54.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.889+0000 7ff113e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1140ada20 0x7ff1141ec710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:54.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.889+0000 7ff113e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff114150020 0x7ff11414a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:54.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.889+0000 7ff113e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff114157550 con 0x7ff114150020 2024-09-11T00:34:54.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.889+0000 7ff113e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff1141573d0 con 0x7ff11406e850 2024-09-11T00:34:54.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.889+0000 7ff113e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff114156960 con 0x7ff1140ada20 2024-09-11T00:34:54.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.889+0000 7ff113400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff114150020 0x7ff11414a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:54.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.889+0000 7ff113400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff114150020 0x7ff11414a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60284/0 (socket says 172.21.15.31:60284) 2024-09-11T00:34:54.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.889+0000 7ff113400640 1 -- 172.21.15.31:0/2973130745 learned_addr learned my addr 172.21.15.31:0/2973130745 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff113400640 1 -- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1140ada20 msgr2=0x7ff1141ec710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff113400640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1140ada20 0x7ff1141ec710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff113400640 1 -- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff11406e850 msgr2=0x7ff1140c1fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff113400640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff11406e850 0x7ff1140c1fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff113400640 1 -- 172.21.15.31:0/2973130745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1140c24f0 con 0x7ff114150020 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff113400640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff114150020 0x7ff11414a1a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00cbc0 tx=0x7ff10c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff10b400640 1 -- 172.21.15.31:0/2973130745 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c016020 con 0x7ff114150020 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff10b400640 1 -- 172.21.15.31:0/2973130745 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff10c00ad90 con 0x7ff114150020 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff10b400640 1 -- 172.21.15.31:0/2973130745 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c012680 con 0x7ff114150020 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff113e00640 1 -- 172.21.15.31:0/2973130745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1141ecc50 con 0x7ff114150020 2024-09-11T00:34:54.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.890+0000 7ff113e00640 1 -- 172.21.15.31:0/2973130745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11414ae20 con 0x7ff114150020 2024-09-11T00:34:54.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.892+0000 7ff113e00640 1 -- 172.21.15.31:0/2973130745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff11414b020 con 0x7ff114150020 2024-09-11T00:34:54.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.893+0000 7ff10b400640 1 -- 172.21.15.31:0/2973130745 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff10c012820 con 0x7ff114150020 2024-09-11T00:34:54.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.894+0000 7ff10b400640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff0f4077fa0 0x7ff0f407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:54.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.894+0000 7ff10b400640 1 -- 172.21.15.31:0/2973130745 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7ff10c09abf0 con 0x7ff114150020 2024-09-11T00:34:54.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.895+0000 7ff112a00640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff0f4077fa0 0x7ff0f407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:54.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.898+0000 7ff10b400640 1 -- 172.21.15.31:0/2973130745 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff10c020020 con 0x7ff114150020 2024-09-11T00:34:54.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:54.899+0000 7ff112a00640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff0f4077fa0 0x7ff0f407a460 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff104034000 tx=0x7ff104004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:55.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.190+0000 7ff113e00640 1 -- 172.21.15.31:0/2973130745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1140cc220 con 0x7ff114150020 2024-09-11T00:34:55.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.191+0000 7ff10b400640 1 -- 172.21.15.31:0/2973130745 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6535 (secure 0 0 0) 0x7ff10c063170 con 0x7ff114150020 2024-09-11T00:34:55.193 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:55.193 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":41,"btime":"2024-09-11T00:34:19:339980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5302,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:34:55.193 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 41 2024-09-11T00:34:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 -- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff0f4077fa0 msgr2=0x7ff0f407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff0f4077fa0 0x7ff0f407a460 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff104034000 tx=0x7ff104004730 comp rx=0 tx=0).stop 2024-09-11T00:34:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 -- 172.21.15.31:0/2973130745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff114150020 msgr2=0x7ff11414a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff114150020 0x7ff11414a1a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00cbc0 tx=0x7ff10c00a060 comp rx=0 tx=0).stop 2024-09-11T00:34:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 -- 172.21.15.31:0/2973130745 shutdown_connections 2024-09-11T00:34:55.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff0f4077fa0 0x7ff0f407a460 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff114150020 0x7ff11414a1a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1140ada20 0x7ff1141ec710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 --2- 172.21.15.31:0/2973130745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff11406e850 0x7ff1140c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 -- 172.21.15.31:0/2973130745 >> 172.21.15.31:0/2973130745 conn(0x7ff114068f10 msgr2=0x7ff11414ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 -- 172.21.15.31:0/2973130745 shutdown_connections 2024-09-11T00:34:55.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.193+0000 7ff113e00640 1 -- 172.21.15.31:0/2973130745 wait complete. 2024-09-11T00:34:55.214 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-11T00:34:55.214 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-11T00:34:55.214 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:34:55.233 INFO:tasks.ceph.mds.a:Stopped 2024-09-11T00:34:55.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:55.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.441+0000 7f0352800640 1 -- 172.21.15.31:0/3897183538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f034c0718b0 msgr2=0x7f034c073320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:55.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.441+0000 7f0352800640 1 --2- 172.21.15.31:0/3897183538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f034c0718b0 0x7f034c073320 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f034c0c9a50 tx=0x7f034402a790 comp rx=0 tx=0).stop 2024-09-11T00:34:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.441+0000 7f0352800640 1 -- 172.21.15.31:0/3897183538 shutdown_connections 2024-09-11T00:34:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.441+0000 7f0352800640 1 --2- 172.21.15.31:0/3897183538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f034c0718b0 0x7f034c073320 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.441+0000 7f0352800640 1 --2- 172.21.15.31:0/3897183538 >> v2:172.21.15.204:3300/0 conn(0x7f034c074b90 0x7f034c070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.441+0000 7f0352800640 1 -- 172.21.15.31:0/3897183538 >> 172.21.15.31:0/3897183538 conn(0x7f034c06dc00 msgr2=0x7f034c06e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.441+0000 7f0352800640 1 -- 172.21.15.31:0/3897183538 shutdown_connections 2024-09-11T00:34:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.441+0000 7f0352800640 1 -- 172.21.15.31:0/3897183538 wait complete. 2024-09-11T00:34:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.443+0000 7f0352800640 1 Processor -- start 2024-09-11T00:34:55.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.443+0000 7f0352800640 1 -- start start 2024-09-11T00:34:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.444+0000 7f0352800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f034c0718b0 0x7f034c0c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.444+0000 7f0352800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f034c074b90 0x7f034c1ed760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.444+0000 7f0352800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f034c0c8cb0 0x7f034c1edca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.444+0000 7f0352800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f034c0c9090 con 0x7f034c074b90 2024-09-11T00:34:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.444+0000 7f0352800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f034c153980 con 0x7f034c0718b0 2024-09-11T00:34:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.444+0000 7f0352800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f034c0c9210 con 0x7f034c0c8cb0 2024-09-11T00:34:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.444+0000 7f034aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f034c074b90 0x7f034c1ed760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.444+0000 7f034aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f034c074b90 0x7f034c1ed760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60302/0 (socket says 172.21.15.31:60302) 2024-09-11T00:34:55.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.444+0000 7f034aa00640 1 -- 172.21.15.31:0/3375867862 learned_addr learned my addr 172.21.15.31:0/3375867862 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.444+0000 7f034be00640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f034c0c8cb0 0x7f034c1edca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.445+0000 7f034aa00640 1 -- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f034c0c8cb0 msgr2=0x7f034c1edca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.445+0000 7f034aa00640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f034c0c8cb0 0x7f034c1edca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.445+0000 7f034aa00640 1 -- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f034c0718b0 msgr2=0x7f034c0c47b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.445+0000 7f034aa00640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f034c0718b0 0x7f034c0c47b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.445+0000 7f034aa00640 1 -- 172.21.15.31:0/3375867862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c0c4cf0 con 0x7f034c074b90 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.445+0000 7f034aa00640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f034c074b90 0x7f034c1ed760 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f033800e9f0 tx=0x7f033800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.445+0000 7f0343400640 1 -- 172.21.15.31:0/3375867862 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033800cd80 con 0x7f034c074b90 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.445+0000 7f0343400640 1 -- 172.21.15.31:0/3375867862 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0338004030 con 0x7f034c074b90 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.445+0000 7f0343400640 1 -- 172.21.15.31:0/3375867862 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0338004480 con 0x7f034c074b90 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.445+0000 7f0352800640 1 -- 172.21.15.31:0/3375867862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034c075d90 con 0x7f034c074b90 2024-09-11T00:34:55.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.446+0000 7f0352800640 1 -- 172.21.15.31:0/3375867862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034c06d8c0 con 0x7f034c074b90 2024-09-11T00:34:55.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.447+0000 7f0352800640 1 -- 172.21.15.31:0/3375867862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034c0750c0 con 0x7f034c074b90 2024-09-11T00:34:55.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.448+0000 7f0343400640 1 -- 172.21.15.31:0/3375867862 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f03380035f0 con 0x7f034c074b90 2024-09-11T00:34:55.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.449+0000 7f0343400640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f032c077f70 0x7f032c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:55.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.450+0000 7f0343400640 1 -- 172.21.15.31:0/3375867862 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f033809add0 con 0x7f034c074b90 2024-09-11T00:34:55.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.450+0000 7f034b400640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f032c077f70 0x7f032c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:55.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.451+0000 7f034b400640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f032c077f70 0x7f032c07a430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f033c0014a0 tx=0x7f033c007930 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:55.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.454+0000 7f0343400640 1 -- 172.21.15.31:0/3375867862 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f03380632c0 con 0x7f034c074b90 2024-09-11T00:34:55.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.765+0000 7f0352800640 1 -- 172.21.15.31:0/3375867862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f034c073300 con 0x7f034c074b90 2024-09-11T00:34:55.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.765+0000 7f0343400640 1 -- 172.21.15.31:0/3375867862 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6535 (secure 0 0 0) 0x7f0338062bc0 con 0x7f034c074b90 2024-09-11T00:34:55.767 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:55.767 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":41,"btime":"2024-09-11T00:34:19:339980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5302,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:34:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 41 2024-09-11T00:34:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 -- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f032c077f70 msgr2=0x7f032c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f032c077f70 0x7f032c07a430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f033c0014a0 tx=0x7f033c007930 comp rx=0 tx=0).stop 2024-09-11T00:34:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 -- 172.21.15.31:0/3375867862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f034c074b90 msgr2=0x7f034c1ed760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f034c074b90 0x7f034c1ed760 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f033800e9f0 tx=0x7f033800eeb0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 -- 172.21.15.31:0/3375867862 shutdown_connections 2024-09-11T00:34:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f032c077f70 0x7f032c07a430 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f034c0c8cb0 0x7f034c1edca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f034c074b90 0x7f034c1ed760 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 --2- 172.21.15.31:0/3375867862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f034c0718b0 0x7f034c0c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 -- 172.21.15.31:0/3375867862 >> 172.21.15.31:0/3375867862 conn(0x7f034c06dc00 msgr2=0x7f034c151c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.768+0000 7f0352800640 1 -- 172.21.15.31:0/3375867862 shutdown_connections 2024-09-11T00:34:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:55.769+0000 7f0352800640 1 -- 172.21.15.31:0/3375867862 wait complete. 2024-09-11T00:34:55.789 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:34:57.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:34:57.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.953+0000 7f94d2600640 1 -- 172.21.15.31:0/3035848641 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94cc0c64b0 msgr2=0x7f94cc06dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:57.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.953+0000 7f94d2600640 1 --2- 172.21.15.31:0/3035848641 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94cc0c64b0 0x7f94cc06dc60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f94b4009860 tx=0x7f94b402a4b0 comp rx=0 tx=0).stop 2024-09-11T00:34:57.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.953+0000 7f94d2600640 1 -- 172.21.15.31:0/3035848641 shutdown_connections 2024-09-11T00:34:57.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.953+0000 7f94d2600640 1 --2- 172.21.15.31:0/3035848641 >> v2:172.21.15.204:3300/0 conn(0x7f94cc0acaf0 0x7f94cc0c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:57.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.953+0000 7f94d2600640 1 --2- 172.21.15.31:0/3035848641 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94cc0c64b0 0x7f94cc06dc60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:57.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.953+0000 7f94d2600640 1 -- 172.21.15.31:0/3035848641 >> 172.21.15.31:0/3035848641 conn(0x7f94cc06caf0 msgr2=0x7f94cc06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:57.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.953+0000 7f94d2600640 1 -- 172.21.15.31:0/3035848641 shutdown_connections 2024-09-11T00:34:57.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.953+0000 7f94d2600640 1 -- 172.21.15.31:0/3035848641 wait complete. 2024-09-11T00:34:57.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.954+0000 7f94d2600640 1 Processor -- start 2024-09-11T00:34:57.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.954+0000 7f94d2600640 1 -- start start 2024-09-11T00:34:57.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.955+0000 7f94d2600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94cc0acaf0 0x7f94cc0c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:57.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.955+0000 7f94d2600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94cc0c64b0 0x7f94cc074090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:57.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.955+0000 7f94d2600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94cc0c7240 0x7f94cc0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:57.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.955+0000 7f94d2600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f94cc155b90 con 0x7f94cc0c7240 2024-09-11T00:34:57.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.955+0000 7f94d2600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f94cc0700f0 con 0x7f94cc0c64b0 2024-09-11T00:34:57.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.955+0000 7f94d2600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f94cc0c7620 con 0x7f94cc0acaf0 2024-09-11T00:34:57.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94d1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94cc0c7240 0x7f94cc0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:57.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94d1200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94cc0acaf0 0x7f94cc0c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:57.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94d1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94cc0c7240 0x7f94cc0745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39154/0 (socket says 172.21.15.31:39154) 2024-09-11T00:34:57.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94d1c00640 1 -- 172.21.15.31:0/4261738922 learned_addr learned my addr 172.21.15.31:0/4261738922 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:34:57.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94c3e00640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94cc0c64b0 0x7f94cc074090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:57.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94d1c00640 1 -- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94cc0acaf0 msgr2=0x7f94cc0c0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:57.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94d1c00640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94cc0acaf0 0x7f94cc0c0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:57.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94d1c00640 1 -- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94cc0c64b0 msgr2=0x7f94cc074090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:57.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94d1c00640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94cc0c64b0 0x7f94cc074090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:57.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94d1c00640 1 -- 172.21.15.31:0/4261738922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94cc074b10 con 0x7f94cc0c7240 2024-09-11T00:34:57.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.956+0000 7f94d1c00640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94cc0c7240 0x7f94cc0745d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f94c400cce0 tx=0x7f94c400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:57.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.957+0000 7f94c9600640 1 -- 172.21.15.31:0/4261738922 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c4003970 con 0x7f94cc0c7240 2024-09-11T00:34:57.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.957+0000 7f94c9600640 1 -- 172.21.15.31:0/4261738922 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94c4003c00 con 0x7f94cc0c7240 2024-09-11T00:34:57.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.957+0000 7f94c9600640 1 -- 172.21.15.31:0/4261738922 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c4004030 con 0x7f94cc0c7240 2024-09-11T00:34:57.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.957+0000 7f94d2600640 1 -- 172.21.15.31:0/4261738922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94cc1ec280 con 0x7f94cc0c7240 2024-09-11T00:34:57.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.957+0000 7f94d2600640 1 -- 172.21.15.31:0/4261738922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94cc151520 con 0x7f94cc0c7240 2024-09-11T00:34:57.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.958+0000 7f94d2600640 1 -- 172.21.15.31:0/4261738922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94cc154a10 con 0x7f94cc0c7240 2024-09-11T00:34:57.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.959+0000 7f94c9600640 1 -- 172.21.15.31:0/4261738922 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f94c40041d0 con 0x7f94cc0c7240 2024-09-11T00:34:57.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.960+0000 7f94c9600640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f94a8077f70 0x7f94a807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:34:57.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.961+0000 7f94c9600640 1 -- 172.21.15.31:0/4261738922 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f94c409aa90 con 0x7f94cc0c7240 2024-09-11T00:34:57.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.961+0000 7f94d1200640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f94a8077f70 0x7f94a807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:34:57.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.962+0000 7f94d1200640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f94a8077f70 0x7f94a807a430 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f94bc007d60 tx=0x7f94bc007cf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:34:57.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:57.964+0000 7f94c9600640 1 -- 172.21.15.31:0/4261738922 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f94c4014070 con 0x7f94cc0c7240 2024-09-11T00:34:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.202+0000 7f94d2600640 1 -- 172.21.15.31:0/4261738922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94cc14a890 con 0x7f94cc0c7240 2024-09-11T00:34:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.202+0000 7f94c9600640 1 -- 172.21.15.31:0/4261738922 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6535 (secure 0 0 0) 0x7f94c40629c0 con 0x7f94cc0c7240 2024-09-11T00:34:58.204 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:34:58.204 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":41,"btime":"2024-09-11T00:34:19:339980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5302,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:34:58.205 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 41 2024-09-11T00:34:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 -- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f94a8077f70 msgr2=0x7f94a807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f94a8077f70 0x7f94a807a430 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f94bc007d60 tx=0x7f94bc007cf0 comp rx=0 tx=0).stop 2024-09-11T00:34:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 -- 172.21.15.31:0/4261738922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94cc0c7240 msgr2=0x7f94cc0745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:34:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94cc0c7240 0x7f94cc0745d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f94c400cce0 tx=0x7f94c400a620 comp rx=0 tx=0).stop 2024-09-11T00:34:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 -- 172.21.15.31:0/4261738922 shutdown_connections 2024-09-11T00:34:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f94a8077f70 0x7f94a807a430 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94cc0c7240 0x7f94cc0745d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94cc0c64b0 0x7f94cc074090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 --2- 172.21.15.31:0/4261738922 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94cc0acaf0 0x7f94cc0c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:34:58.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 -- 172.21.15.31:0/4261738922 >> 172.21.15.31:0/4261738922 conn(0x7f94cc06caf0 msgr2=0x7f94cc0c43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:34:58.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 -- 172.21.15.31:0/4261738922 shutdown_connections 2024-09-11T00:34:58.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:34:58.205+0000 7f94d2600640 1 -- 172.21.15.31:0/4261738922 wait complete. 2024-09-11T00:34:58.225 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:35:00.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.395+0000 7f58de000640 1 -- 172.21.15.31:0/3667571612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58d8153f20 msgr2=0x7f58d81516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.395+0000 7f58de000640 1 --2- 172.21.15.31:0/3667571612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58d8153f20 0x7f58d81516e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f58d0005fd0 tx=0x7f58d002a960 comp rx=0 tx=0).stop 2024-09-11T00:35:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.396+0000 7f58de000640 1 -- 172.21.15.31:0/3667571612 shutdown_connections 2024-09-11T00:35:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.396+0000 7f58de000640 1 --2- 172.21.15.31:0/3667571612 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58d8153f20 0x7f58d81516e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.396+0000 7f58de000640 1 --2- 172.21.15.31:0/3667571612 >> v2:172.21.15.204:3300/0 conn(0x7f58d8072480 0x7f58d81511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.396+0000 7f58de000640 1 -- 172.21.15.31:0/3667571612 >> 172.21.15.31:0/3667571612 conn(0x7f58d80a9750 msgr2=0x7f58d80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:00.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.396+0000 7f58de000640 1 -- 172.21.15.31:0/3667571612 shutdown_connections 2024-09-11T00:35:00.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.396+0000 7f58de000640 1 -- 172.21.15.31:0/3667571612 wait complete. 2024-09-11T00:35:00.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.397+0000 7f58de000640 1 Processor -- start 2024-09-11T00:35:00.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.398+0000 7f58de000640 1 -- start start 2024-09-11T00:35:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.398+0000 7f58de000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d8072480 0x7f58d8155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.398+0000 7f58de000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58d80c23a0 0x7f58d80c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:00.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.398+0000 7f58de000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58d8153f20 0x7f58d81ea8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.398+0000 7f58de000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f58d8074200 con 0x7f58d8072480 2024-09-11T00:35:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.398+0000 7f58de000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f58d8074080 con 0x7f58d80c23a0 2024-09-11T00:35:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.398+0000 7f58de000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f58d8072860 con 0x7f58d8153f20 2024-09-11T00:35:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.399+0000 7f58d7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d8072480 0x7f58d8155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.399+0000 7f58d7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d8072480 0x7f58d8155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39164/0 (socket says 172.21.15.31:39164) 2024-09-11T00:35:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.399+0000 7f58d7400640 1 -- 172.21.15.31:0/53655868 learned_addr learned my addr 172.21.15.31:0/53655868 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:00.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.399+0000 7f58d7400640 1 -- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58d8153f20 msgr2=0x7f58d81ea8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:00.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.399+0000 7f58d7400640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58d8153f20 0x7f58d81ea8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:00.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.399+0000 7f58d7400640 1 -- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58d80c23a0 msgr2=0x7f58d80c3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:00.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.399+0000 7f58d7400640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58d80c23a0 0x7f58d80c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:00.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.399+0000 7f58d7400640 1 -- 172.21.15.31:0/53655868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d81eae10 con 0x7f58d8072480 2024-09-11T00:35:00.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.399+0000 7f58d7400640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d8072480 0x7f58d8155840 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f58c800b450 tx=0x7f58c800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:00.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.400+0000 7f58cfe00640 1 -- 172.21.15.31:0/53655868 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c8015020 con 0x7f58d8072480 2024-09-11T00:35:00.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.400+0000 7f58cfe00640 1 -- 172.21.15.31:0/53655868 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58c80119a0 con 0x7f58d8072480 2024-09-11T00:35:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.400+0000 7f58cfe00640 1 -- 172.21.15.31:0/53655868 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c8011df0 con 0x7f58d8072480 2024-09-11T00:35:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.400+0000 7f58de000640 1 -- 172.21.15.31:0/53655868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d80c4e20 con 0x7f58d8072480 2024-09-11T00:35:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.400+0000 7f58de000640 1 -- 172.21.15.31:0/53655868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d80c3440 con 0x7f58d8072480 2024-09-11T00:35:00.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.401+0000 7f58de000640 1 -- 172.21.15.31:0/53655868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d81511a0 con 0x7f58d8072480 2024-09-11T00:35:00.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.402+0000 7f58cfe00640 1 -- 172.21.15.31:0/53655868 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f58c801a400 con 0x7f58d8072480 2024-09-11T00:35:00.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.403+0000 7f58cfe00640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f58b4077f70 0x7f58b407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:00.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.403+0000 7f58cfe00640 1 -- 172.21.15.31:0/53655868 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f58c809a7b0 con 0x7f58d8072480 2024-09-11T00:35:00.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.403+0000 7f58cea00640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f58b4077f70 0x7f58b407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.404+0000 7f58cea00640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f58b4077f70 0x7f58b407a430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f58c0007fd0 tx=0x7f58c0007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:00.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.407+0000 7f58cfe00640 1 -- 172.21.15.31:0/53655868 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f58c801f020 con 0x7f58d8072480 2024-09-11T00:35:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.647+0000 7f58de000640 1 -- 172.21.15.31:0/53655868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58d81583b0 con 0x7f58d8072480 2024-09-11T00:35:00.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.647+0000 7f58cfe00640 1 -- 172.21.15.31:0/53655868 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6535 (secure 0 0 0) 0x7f58d81eae10 con 0x7f58d8072480 2024-09-11T00:35:00.649 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:00.650 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":41,"btime":"2024-09-11T00:34:19:339980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5302,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:00.650 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 41 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 -- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f58b4077f70 msgr2=0x7f58b407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f58b4077f70 0x7f58b407a430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f58c0007fd0 tx=0x7f58c0007900 comp rx=0 tx=0).stop 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 -- 172.21.15.31:0/53655868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d8072480 msgr2=0x7f58d8155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d8072480 0x7f58d8155840 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f58c800b450 tx=0x7f58c800b910 comp rx=0 tx=0).stop 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 -- 172.21.15.31:0/53655868 shutdown_connections 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f58b4077f70 0x7f58b407a430 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f58d8153f20 0x7f58d81ea8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58d80c23a0 0x7f58d80c3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 --2- 172.21.15.31:0/53655868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d8072480 0x7f58d8155840 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 -- 172.21.15.31:0/53655868 >> 172.21.15.31:0/53655868 conn(0x7f58d80a9750 msgr2=0x7f58d814ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 -- 172.21.15.31:0/53655868 shutdown_connections 2024-09-11T00:35:00.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:00.650+0000 7f58de000640 1 -- 172.21.15.31:0/53655868 wait complete. 2024-09-11T00:35:00.667 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:35:02.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:02.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.838+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1926652148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d541518b0 msgr2=0x7f9d540c37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:02.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.838+0000 7f9d5aa00640 1 --2- 172.21.15.31:0/1926652148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d541518b0 0x7f9d540c37d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c005fd0 tx=0x7f9d4c02a9d0 comp rx=0 tx=0).stop 2024-09-11T00:35:02.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.838+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1926652148 shutdown_connections 2024-09-11T00:35:02.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.838+0000 7f9d5aa00640 1 --2- 172.21.15.31:0/1926652148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d541518b0 0x7f9d540c37d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:02.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.838+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1926652148 >> 172.21.15.31:0/1926652148 conn(0x7f9d540a9750 msgr2=0x7f9d540a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:02.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.838+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1926652148 shutdown_connections 2024-09-11T00:35:02.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.838+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1926652148 wait complete. 2024-09-11T00:35:02.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.839+0000 7f9d5aa00640 1 Processor -- start 2024-09-11T00:35:02.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.840+0000 7f9d5aa00640 1 -- start start 2024-09-11T00:35:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.840+0000 7f9d5aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d54072480 0x7f9d541dfa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.840+0000 7f9d5aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d540c23a0 0x7f9d541436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.840+0000 7f9d5aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d541518b0 0x7f9d54143c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.840+0000 7f9d5aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9d54074080 con 0x7f9d541518b0 2024-09-11T00:35:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.840+0000 7f9d5aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d540c3d50 con 0x7f9d540c23a0 2024-09-11T00:35:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.840+0000 7f9d5aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9d54074200 con 0x7f9d54072480 2024-09-11T00:35:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.841+0000 7f9d58a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d541518b0 0x7f9d54143c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.841+0000 7f9d58a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d541518b0 0x7f9d54143c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39176/0 (socket says 172.21.15.31:39176) 2024-09-11T00:35:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.841+0000 7f9d58a00640 1 -- 172.21.15.31:0/1676061341 learned_addr learned my addr 172.21.15.31:0/1676061341 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.841+0000 7f9d4be00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d540c23a0 0x7f9d541436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.841+0000 7f9d58a00640 1 -- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d54072480 msgr2=0x7f9d541dfa40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:35:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.841+0000 7f9d58a00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d54072480 0x7f9d541dfa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.841+0000 7f9d58a00640 1 -- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d540c23a0 msgr2=0x7f9d541436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.841+0000 7f9d58a00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d540c23a0 0x7f9d541436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.841+0000 7f9d58a00640 1 -- 172.21.15.31:0/1676061341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d541edd80 con 0x7f9d541518b0 2024-09-11T00:35:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.841+0000 7f9d58a00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d541518b0 0x7f9d54143c00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c006010 tx=0x7f9d4c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:02.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.842+0000 7f9d50c00640 1 -- 172.21.15.31:0/1676061341 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c047070 con 0x7f9d541518b0 2024-09-11T00:35:02.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.842+0000 7f9d50c00640 1 -- 172.21.15.31:0/1676061341 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d4c003cf0 con 0x7f9d541518b0 2024-09-11T00:35:02.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.842+0000 7f9d50c00640 1 -- 172.21.15.31:0/1676061341 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c0042a0 con 0x7f9d541518b0 2024-09-11T00:35:02.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.842+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1676061341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d541ee060 con 0x7f9d541518b0 2024-09-11T00:35:02.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.842+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1676061341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d541eb7c0 con 0x7f9d541518b0 2024-09-11T00:35:02.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.843+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1676061341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d54153f20 con 0x7f9d541518b0 2024-09-11T00:35:02.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.844+0000 7f9d50c00640 1 -- 172.21.15.31:0/1676061341 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9d4c004440 con 0x7f9d541518b0 2024-09-11T00:35:02.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.845+0000 7f9d50c00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9d30077f70 0x7f9d3007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:02.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.846+0000 7f9d50c00640 1 -- 172.21.15.31:0/1676061341 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f9d4c0bb3e0 con 0x7f9d541518b0 2024-09-11T00:35:02.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.846+0000 7f9d53e00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9d30077f70 0x7f9d3007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:02.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.847+0000 7f9d53e00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9d30077f70 0x7f9d3007a430 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9d44001480 tx=0x7f9d440079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:02.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:02.850+0000 7f9d50c00640 1 -- 172.21.15.31:0/1676061341 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9d4c033020 con 0x7f9d541518b0 2024-09-11T00:35:03.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.088+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1676061341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d540a9530 con 0x7f9d541518b0 2024-09-11T00:35:03.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.089+0000 7f9d50c00640 1 -- 172.21.15.31:0/1676061341 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6535 (secure 0 0 0) 0x7f9d4c0838d0 con 0x7f9d541518b0 2024-09-11T00:35:03.091 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:03.091 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":41,"btime":"2024-09-11T00:34:19:339980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5302,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:03.091 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 41 2024-09-11T00:35:03.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.091+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9d30077f70 msgr2=0x7f9d3007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:03.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.091+0000 7f9d5aa00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9d30077f70 0x7f9d3007a430 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9d44001480 tx=0x7f9d440079b0 comp rx=0 tx=0).stop 2024-09-11T00:35:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.091+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1676061341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d541518b0 msgr2=0x7f9d54143c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.091+0000 7f9d5aa00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d541518b0 0x7f9d54143c00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c006010 tx=0x7f9d4c003970 comp rx=0 tx=0).stop 2024-09-11T00:35:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.092+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1676061341 shutdown_connections 2024-09-11T00:35:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.092+0000 7f9d5aa00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9d30077f70 0x7f9d3007a430 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.092+0000 7f9d5aa00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d541518b0 0x7f9d54143c00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.092+0000 7f9d5aa00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d540c23a0 0x7f9d541436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.092+0000 7f9d5aa00640 1 --2- 172.21.15.31:0/1676061341 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d54072480 0x7f9d541dfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.092+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1676061341 >> 172.21.15.31:0/1676061341 conn(0x7f9d540a9750 msgr2=0x7f9d5414a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.092+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1676061341 shutdown_connections 2024-09-11T00:35:03.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:03.092+0000 7f9d5aa00640 1 -- 172.21.15.31:0/1676061341 wait complete. 2024-09-11T00:35:03.109 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:35:05.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:05.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.285+0000 7ff36d000640 1 -- 172.21.15.31:0/1200437621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff368153f20 msgr2=0x7ff3681516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:05.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.285+0000 7ff36d000640 1 --2- 172.21.15.31:0/1200437621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff368153f20 0x7ff3681516e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff360005fd0 tx=0x7ff36002ab20 comp rx=0 tx=0).stop 2024-09-11T00:35:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.285+0000 7ff36d000640 1 -- 172.21.15.31:0/1200437621 shutdown_connections 2024-09-11T00:35:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.285+0000 7ff36d000640 1 --2- 172.21.15.31:0/1200437621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff368153f20 0x7ff3681516e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.285+0000 7ff36d000640 1 --2- 172.21.15.31:0/1200437621 >> v2:172.21.15.204:3300/0 conn(0x7ff368072480 0x7ff3681511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.285+0000 7ff36d000640 1 -- 172.21.15.31:0/1200437621 >> 172.21.15.31:0/1200437621 conn(0x7ff3680a9750 msgr2=0x7ff3680a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.285+0000 7ff36d000640 1 -- 172.21.15.31:0/1200437621 shutdown_connections 2024-09-11T00:35:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.285+0000 7ff36d000640 1 -- 172.21.15.31:0/1200437621 wait complete. 2024-09-11T00:35:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.287+0000 7ff36d000640 1 Processor -- start 2024-09-11T00:35:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.287+0000 7ff36d000640 1 -- start start 2024-09-11T00:35:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.287+0000 7ff36d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff368072480 0x7ff3680c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.287+0000 7ff36d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3680c23a0 0x7ff3681496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff36d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff368153f20 0x7ff368146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff36d000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff368074200 con 0x7ff3680c23a0 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff36d000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff368074080 con 0x7ff368153f20 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff36d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff368072860 con 0x7ff368072480 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff365400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3680c23a0 0x7ff3681496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff365e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff368072480 0x7ff3680c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff366800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff368153f20 0x7ff368146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff365e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff368072480 0x7ff3680c4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:60864/0 (socket says 172.21.15.31:60864) 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff365e00640 1 -- 172.21.15.31:0/1792485204 learned_addr learned my addr 172.21.15.31:0/1792485204 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff365400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3680c23a0 0x7ff3681496e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39190/0 (socket says 172.21.15.31:39190) 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff365e00640 1 -- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff368153f20 msgr2=0x7ff368146660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff365e00640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff368153f20 0x7ff368146660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff365e00640 1 -- 172.21.15.31:0/1792485204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3680c23a0 msgr2=0x7ff3681496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff365e00640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3680c23a0 0x7ff3681496e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.288+0000 7ff365e00640 1 -- 172.21.15.31:0/1792485204 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3681ea140 con 0x7ff368072480 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.289+0000 7ff365e00640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff368072480 0x7ff3680c4d30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff35800ea20 tx=0x7ff35800eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.289+0000 7ff35e000640 1 -- 172.21.15.31:0/1792485204 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35800cd80 con 0x7ff368072480 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.289+0000 7ff35e000640 1 -- 172.21.15.31:0/1792485204 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff358010430 con 0x7ff368072480 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.289+0000 7ff36d000640 1 -- 172.21.15.31:0/1792485204 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3681ea420 con 0x7ff368072480 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.289+0000 7ff35e000640 1 -- 172.21.15.31:0/1792485204 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff358010840 con 0x7ff368072480 2024-09-11T00:35:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.289+0000 7ff36d000640 1 -- 172.21.15.31:0/1792485204 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3681ea9e0 con 0x7ff368072480 2024-09-11T00:35:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.290+0000 7ff36d000640 1 -- 172.21.15.31:0/1792485204 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff36806d910 con 0x7ff368072480 2024-09-11T00:35:05.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.292+0000 7ff35e000640 1 -- 172.21.15.31:0/1792485204 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff358029080 con 0x7ff368072480 2024-09-11T00:35:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.293+0000 7ff35e000640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff348077f70 0x7ff34807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.293+0000 7ff35e000640 1 -- 172.21.15.31:0/1792485204 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7ff35809b330 con 0x7ff368072480 2024-09-11T00:35:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.294+0000 7ff365400640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff348077f70 0x7ff34807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.294+0000 7ff365400640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff348077f70 0x7ff34807a430 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff354007fd0 tx=0x7ff354009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:05.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.297+0000 7ff35e000640 1 -- 172.21.15.31:0/1792485204 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff36806d910 con 0x7ff368072480 2024-09-11T00:35:05.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.528+0000 7ff36d000640 1 -- 172.21.15.31:0/1792485204 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3681511a0 con 0x7ff368072480 2024-09-11T00:35:05.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.529+0000 7ff35e000640 1 -- 172.21.15.31:0/1792485204 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6535 (secure 0 0 0) 0x7ff358063250 con 0x7ff368072480 2024-09-11T00:35:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:05.531 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":41,"btime":"2024-09-11T00:34:19:339980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5302,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 41 2024-09-11T00:35:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 -- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff348077f70 msgr2=0x7ff34807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff348077f70 0x7ff34807a430 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff354007fd0 tx=0x7ff354009040 comp rx=0 tx=0).stop 2024-09-11T00:35:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 -- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff368072480 msgr2=0x7ff3680c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff368072480 0x7ff3680c4d30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff35800ea20 tx=0x7ff35800eee0 comp rx=0 tx=0).stop 2024-09-11T00:35:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 -- 172.21.15.31:0/1792485204 shutdown_connections 2024-09-11T00:35:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff348077f70 0x7ff34807a430 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff368153f20 0x7ff368146660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3680c23a0 0x7ff3681496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 --2- 172.21.15.31:0/1792485204 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff368072480 0x7ff3680c4d30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:05.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 -- 172.21.15.31:0/1792485204 >> 172.21.15.31:0/1792485204 conn(0x7ff3680a9750 msgr2=0x7ff3680a8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:05.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 -- 172.21.15.31:0/1792485204 shutdown_connections 2024-09-11T00:35:05.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:05.532+0000 7ff36d000640 1 -- 172.21.15.31:0/1792485204 wait complete. 2024-09-11T00:35:05.549 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:35:07.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:07.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.719+0000 7fe036800640 1 -- 172.21.15.31:0/3824809688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0301518b0 msgr2=0x7fe0300c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:07.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.719+0000 7fe036800640 1 --2- 172.21.15.31:0/3824809688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0301518b0 0x7fe0300c3600 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe028009be0 tx=0x7fe02802a960 comp rx=0 tx=0).stop 2024-09-11T00:35:07.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.719+0000 7fe036800640 1 -- 172.21.15.31:0/3824809688 shutdown_connections 2024-09-11T00:35:07.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.719+0000 7fe036800640 1 --2- 172.21.15.31:0/3824809688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0301518b0 0x7fe0300c3600 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:07.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.719+0000 7fe036800640 1 --2- 172.21.15.31:0/3824809688 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0300c23a0 0x7fe0300c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:07.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.719+0000 7fe036800640 1 -- 172.21.15.31:0/3824809688 >> 172.21.15.31:0/3824809688 conn(0x7fe0300a9750 msgr2=0x7fe0300a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:07.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.719+0000 7fe036800640 1 -- 172.21.15.31:0/3824809688 shutdown_connections 2024-09-11T00:35:07.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.719+0000 7fe036800640 1 -- 172.21.15.31:0/3824809688 wait complete. 2024-09-11T00:35:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.720+0000 7fe036800640 1 Processor -- start 2024-09-11T00:35:07.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.721+0000 7fe036800640 1 -- start start 2024-09-11T00:35:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.721+0000 7fe036800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe030072480 0x7fe030146d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.721+0000 7fe036800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0300c23a0 0x7fe030150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.721+0000 7fe036800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0301518b0 0x7fe0301def40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe036800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe030074290 con 0x7fe030072480 2024-09-11T00:35:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe036800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe030072860 con 0x7fe0300c23a0 2024-09-11T00:35:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe036800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe0300729e0 con 0x7fe0301518b0 2024-09-11T00:35:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe02f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe030072480 0x7fe030146d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:07.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe02f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe030072480 0x7fe030146d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39206/0 (socket says 172.21.15.31:39206) 2024-09-11T00:35:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe02f400640 1 -- 172.21.15.31:0/3416822440 learned_addr learned my addr 172.21.15.31:0/3416822440 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe02fe00640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0301518b0 0x7fe0301def40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe027e00640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0300c23a0 0x7fe030150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe02f400640 1 -- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0301518b0 msgr2=0x7fe0301def40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe02f400640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0301518b0 0x7fe0301def40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe02f400640 1 -- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0300c23a0 msgr2=0x7fe030150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe02f400640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0300c23a0 0x7fe030150260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe02f400640 1 -- 172.21.15.31:0/3416822440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0301dda50 con 0x7fe030072480 2024-09-11T00:35:07.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.722+0000 7fe02f400640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe030072480 0x7fe030146d20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe02000c490 tx=0x7fe02000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:07.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.723+0000 7fe027400640 1 -- 172.21.15.31:0/3416822440 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe020016020 con 0x7fe030072480 2024-09-11T00:35:07.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.723+0000 7fe027400640 1 -- 172.21.15.31:0/3416822440 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe020012940 con 0x7fe030072480 2024-09-11T00:35:07.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.723+0000 7fe027400640 1 -- 172.21.15.31:0/3416822440 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe020012d90 con 0x7fe030072480 2024-09-11T00:35:07.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.723+0000 7fe036800640 1 -- 172.21.15.31:0/3416822440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe030149450 con 0x7fe030072480 2024-09-11T00:35:07.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.723+0000 7fe036800640 1 -- 172.21.15.31:0/3416822440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe03014a7b0 con 0x7fe030072480 2024-09-11T00:35:07.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.724+0000 7fe036800640 1 -- 172.21.15.31:0/3416822440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe03014ebb0 con 0x7fe030072480 2024-09-11T00:35:07.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.725+0000 7fe027400640 1 -- 172.21.15.31:0/3416822440 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe02001a3b0 con 0x7fe030072480 2024-09-11T00:35:07.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.726+0000 7fe027400640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe00c077fa0 0x7fe00c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:07.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.726+0000 7fe027400640 1 -- 172.21.15.31:0/3416822440 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7fe02009a110 con 0x7fe030072480 2024-09-11T00:35:07.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.726+0000 7fe027e00640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe00c077fa0 0x7fe00c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:07.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.727+0000 7fe027e00640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe00c077fa0 0x7fe00c07a460 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe0300a92a0 tx=0x7fe018005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:07.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.730+0000 7fe027400640 1 -- 172.21.15.31:0/3416822440 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe020020030 con 0x7fe030072480 2024-09-11T00:35:07.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.963+0000 7fe036800640 1 -- 172.21.15.31:0/3416822440 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe030153f20 con 0x7fe030072480 2024-09-11T00:35:07.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.963+0000 7fe027400640 1 -- 172.21.15.31:0/3416822440 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6535 (secure 0 0 0) 0x7fe020062040 con 0x7fe030072480 2024-09-11T00:35:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:07.966 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":41,"btime":"2024-09-11T00:34:19:339980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5302,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:32:58.315114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5076":{"gid":5076,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/4202305151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4202305151},{"type":"v1","addr":"172.21.15.31:6835","nonce":4202305151}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5076,"qdb_cluster":[5076]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:07.966 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 41 2024-09-11T00:35:07.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 -- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe00c077fa0 msgr2=0x7fe00c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:07.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe00c077fa0 0x7fe00c07a460 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe0300a92a0 tx=0x7fe018005f90 comp rx=0 tx=0).stop 2024-09-11T00:35:07.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 -- 172.21.15.31:0/3416822440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe030072480 msgr2=0x7fe030146d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:07.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe030072480 0x7fe030146d20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe02000c490 tx=0x7fe02000c950 comp rx=0 tx=0).stop 2024-09-11T00:35:07.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 -- 172.21.15.31:0/3416822440 shutdown_connections 2024-09-11T00:35:07.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe00c077fa0 0x7fe00c07a460 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:07.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0301518b0 0x7fe0301def40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:07.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0300c23a0 0x7fe030150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:07.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 --2- 172.21.15.31:0/3416822440 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe030072480 0x7fe030146d20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:07.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 -- 172.21.15.31:0/3416822440 >> 172.21.15.31:0/3416822440 conn(0x7fe0300a9750 msgr2=0x7fe03014ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:07.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 -- 172.21.15.31:0/3416822440 shutdown_connections 2024-09-11T00:35:07.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:07.966+0000 7fe036800640 1 -- 172.21.15.31:0/3416822440 wait complete. 2024-09-11T00:35:07.983 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:35:09.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:10.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.145+0000 7fbba7600640 1 -- 172.21.15.31:0/1260126302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba01518b0 msgr2=0x7fbba00c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:10.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.145+0000 7fbba7600640 1 --2- 172.21.15.31:0/1260126302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba01518b0 0x7fbba00c3600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009be0 tx=0x7fbb9802a960 comp rx=0 tx=0).stop 2024-09-11T00:35:10.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.145+0000 7fbba7600640 1 -- 172.21.15.31:0/1260126302 shutdown_connections 2024-09-11T00:35:10.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.145+0000 7fbba7600640 1 --2- 172.21.15.31:0/1260126302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba01518b0 0x7fbba00c3600 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.145+0000 7fbba7600640 1 --2- 172.21.15.31:0/1260126302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba00c23a0 0x7fbba00c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.145+0000 7fbba7600640 1 -- 172.21.15.31:0/1260126302 >> 172.21.15.31:0/1260126302 conn(0x7fbba00a9750 msgr2=0x7fbba00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:10.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.145+0000 7fbba7600640 1 -- 172.21.15.31:0/1260126302 shutdown_connections 2024-09-11T00:35:10.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.145+0000 7fbba7600640 1 -- 172.21.15.31:0/1260126302 wait complete. 2024-09-11T00:35:10.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.147+0000 7fbba7600640 1 Processor -- start 2024-09-11T00:35:10.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.147+0000 7fbba7600640 1 -- start start 2024-09-11T00:35:10.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba7600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba0072480 0x7fbba00c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:10.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba7600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba00c23a0 0x7fbba00c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:10.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba7600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba01518b0 0x7fbba01df8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:10.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba7600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbba0074290 con 0x7fbba0072480 2024-09-11T00:35:10.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba7600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbba0072860 con 0x7fbba00c23a0 2024-09-11T00:35:10.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba7600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbba00729e0 con 0x7fbba01518b0 2024-09-11T00:35:10.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba0072480 0x7fbba00c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:10.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbb9fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba00c23a0 0x7fbba00c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:10.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba0072480 0x7fbba00c6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42292/0 (socket says 172.21.15.31:42292) 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba4c00640 1 -- 172.21.15.31:0/4118734272 learned_addr learned my addr 172.21.15.31:0/4118734272 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba5600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba01518b0 0x7fbba01df8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbb9fe00640 1 -- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba01518b0 msgr2=0x7fbba01df8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbb9fe00640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba01518b0 0x7fbba01df8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbb9fe00640 1 -- 172.21.15.31:0/4118734272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba0072480 msgr2=0x7fbba00c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbb9fe00640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba0072480 0x7fbba00c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbb9fe00640 1 -- 172.21.15.31:0/4118734272 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba00c68a0 con 0x7fbba00c23a0 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.148+0000 7fbba4c00640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba0072480 0x7fbba00c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.149+0000 7fbb9fe00640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba00c23a0 0x7fbba00c6af0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00e9c0 tx=0x7fbb8c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.149+0000 7fbb9cc00640 1 -- 172.21.15.31:0/4118734272 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c00ccc0 con 0x7fbba00c23a0 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.149+0000 7fbb9cc00640 1 -- 172.21.15.31:0/4118734272 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb8c00ce60 con 0x7fbba00c23a0 2024-09-11T00:35:10.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.149+0000 7fbb9cc00640 1 -- 172.21.15.31:0/4118734272 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c004280 con 0x7fbba00c23a0 2024-09-11T00:35:10.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.149+0000 7fbba7600640 1 -- 172.21.15.31:0/4118734272 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba00c7030 con 0x7fbba00c23a0 2024-09-11T00:35:10.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.149+0000 7fbba7600640 1 -- 172.21.15.31:0/4118734272 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba0153810 con 0x7fbba00c23a0 2024-09-11T00:35:10.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.150+0000 7fbba7600640 1 -- 172.21.15.31:0/4118734272 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba0153630 con 0x7fbba00c23a0 2024-09-11T00:35:10.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.152+0000 7fbb9cc00640 1 -- 172.21.15.31:0/4118734272 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbb8c0035f0 con 0x7fbba00c23a0 2024-09-11T00:35:10.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.153+0000 7fbb9cc00640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb80077fa0 0x7fbb8007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:10.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.153+0000 7fbb9cc00640 1 -- 172.21.15.31:0/4118734272 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7fbb8c09abc0 con 0x7fbba00c23a0 2024-09-11T00:35:10.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.153+0000 7fbba4c00640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb80077fa0 0x7fbb8007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:10.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.154+0000 7fbba4c00640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb80077fa0 0x7fbb8007a460 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbb94007b10 tx=0x7fbb94007a60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.157+0000 7fbb9cc00640 1 -- 172.21.15.31:0/4118734272 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbba0153630 con 0x7fbba00c23a0 2024-09-11T00:35:10.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.391+0000 7fbba7600640 1 -- 172.21.15.31:0/4118734272 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbba0153f20 con 0x7fbba00c23a0 2024-09-11T00:35:10.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.392+0000 7fbb9cc00640 1 -- 172.21.15.31:0/4118734272 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5702 (secure 0 0 0) 0x7fbb8c062aa0 con 0x7fbba00c23a0 2024-09-11T00:35:10.394 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:10.394 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":42,"btime":"2024-09-11T00:35:09:312345+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:09.312339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 42 2024-09-11T00:35:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 -- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb80077fa0 msgr2=0x7fbb8007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb80077fa0 0x7fbb8007a460 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbb94007b10 tx=0x7fbb94007a60 comp rx=0 tx=0).stop 2024-09-11T00:35:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 -- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba00c23a0 msgr2=0x7fbba00c6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba00c23a0 0x7fbba00c6af0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00e9c0 tx=0x7fbb8c00ee80 comp rx=0 tx=0).stop 2024-09-11T00:35:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 -- 172.21.15.31:0/4118734272 shutdown_connections 2024-09-11T00:35:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb80077fa0 0x7fbb8007a460 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba01518b0 0x7fbba01df8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba00c23a0 0x7fbba00c6af0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 --2- 172.21.15.31:0/4118734272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba0072480 0x7fbba00c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 -- 172.21.15.31:0/4118734272 >> 172.21.15.31:0/4118734272 conn(0x7fbba00a9750 msgr2=0x7fbba014f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 -- 172.21.15.31:0/4118734272 shutdown_connections 2024-09-11T00:35:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.394+0000 7fbba7600640 1 -- 172.21.15.31:0/4118734272 wait complete. 2024-09-11T00:35:10.413 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-11T00:35:10.413 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:35:10.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:10.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.573+0000 7f4347800640 1 -- 172.21.15.31:0/2426022014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43400c23a0 msgr2=0x7f43400c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:10.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.573+0000 7f4347800640 1 --2- 172.21.15.31:0/2426022014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43400c23a0 0x7f43400c30c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4328009830 tx=0x7f432802a5a0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.573+0000 7f4347800640 1 -- 172.21.15.31:0/2426022014 shutdown_connections 2024-09-11T00:35:10.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.573+0000 7f4347800640 1 --2- 172.21.15.31:0/2426022014 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43401518b0 0x7f43400c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.573+0000 7f4347800640 1 --2- 172.21.15.31:0/2426022014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43400c23a0 0x7f43400c30c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.573+0000 7f4347800640 1 -- 172.21.15.31:0/2426022014 >> 172.21.15.31:0/2426022014 conn(0x7f43400a9750 msgr2=0x7f43400a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:10.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.573+0000 7f4347800640 1 -- 172.21.15.31:0/2426022014 shutdown_connections 2024-09-11T00:35:10.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.573+0000 7f4347800640 1 -- 172.21.15.31:0/2426022014 wait complete. 2024-09-11T00:35:10.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.575+0000 7f4347800640 1 Processor -- start 2024-09-11T00:35:10.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.575+0000 7f4347800640 1 -- start start 2024-09-11T00:35:10.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4347800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4340072480 0x7f4340148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:10.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4347800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43400c23a0 0x7f434014aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:10.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4347800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43401518b0 0x7f43401ed020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:10.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4347800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4340074200 con 0x7f43400c23a0 2024-09-11T00:35:10.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4347800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4340074080 con 0x7f4340072480 2024-09-11T00:35:10.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4347800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4340072860 con 0x7f43401518b0 2024-09-11T00:35:10.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4337e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43400c23a0 0x7f434014aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:10.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4337e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43400c23a0 0x7f434014aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42320/0 (socket says 172.21.15.31:42320) 2024-09-11T00:35:10.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4337e00640 1 -- 172.21.15.31:0/443742234 learned_addr learned my addr 172.21.15.31:0/443742234 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:10.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4344e00640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4340072480 0x7f4340148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:10.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4345800640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43401518b0 0x7f43401ed020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:10.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4337e00640 1 -- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43401518b0 msgr2=0x7f43401ed020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:10.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4337e00640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43401518b0 0x7f43401ed020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4337e00640 1 -- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4340072480 msgr2=0x7f4340148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:10.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4337e00640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4340072480 0x7f4340148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.576+0000 7f4337e00640 1 -- 172.21.15.31:0/443742234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4340148d00 con 0x7f43400c23a0 2024-09-11T00:35:10.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.577+0000 7f4337e00640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43400c23a0 0x7f434014aed0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4328000c00 tx=0x7f4328032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:10.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.577+0000 7f433d600640 1 -- 172.21.15.31:0/443742234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4328040070 con 0x7f43400c23a0 2024-09-11T00:35:10.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.577+0000 7f433d600640 1 -- 172.21.15.31:0/443742234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43280038e0 con 0x7f43400c23a0 2024-09-11T00:35:10.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.577+0000 7f433d600640 1 -- 172.21.15.31:0/443742234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4328003d30 con 0x7f43400c23a0 2024-09-11T00:35:10.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.577+0000 7f4347800640 1 -- 172.21.15.31:0/443742234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43401dff50 con 0x7f43400c23a0 2024-09-11T00:35:10.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.577+0000 7f4347800640 1 -- 172.21.15.31:0/443742234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43400aa4b0 con 0x7f43400c23a0 2024-09-11T00:35:10.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.578+0000 7f4347800640 1 -- 172.21.15.31:0/443742234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43400c3b90 con 0x7f43400c23a0 2024-09-11T00:35:10.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.579+0000 7f433d600640 1 -- 172.21.15.31:0/443742234 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f432804d030 con 0x7f43400c23a0 2024-09-11T00:35:10.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.580+0000 7f433d600640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f431c077f70 0x7f431c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:10.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.580+0000 7f433d600640 1 -- 172.21.15.31:0/443742234 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f43280bb600 con 0x7f43400c23a0 2024-09-11T00:35:10.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.582+0000 7f4344e00640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f431c077f70 0x7f431c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:10.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.582+0000 7f4344e00640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f431c077f70 0x7f431c07a430 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4340154aa0 tx=0x7f4330007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:10.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.587+0000 7f433d600640 1 -- 172.21.15.31:0/443742234 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4328083580 con 0x7f43400c23a0 2024-09-11T00:35:10.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.826+0000 7f4347800640 1 -- 172.21.15.31:0/443742234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43401dfd20 con 0x7f43400c23a0 2024-09-11T00:35:10.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.826+0000 7f433d600640 1 -- 172.21.15.31:0/443742234 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+5706 (secure 0 0 0) 0x7f4328087fb0 con 0x7f43400c23a0 2024-09-11T00:35:10.828 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:10.829 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":43,"btime":"2024-09-11T00:35:10:588522+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:09.634906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:10.829 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 43 2024-09-11T00:35:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 -- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f431c077f70 msgr2=0x7f431c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f431c077f70 0x7f431c07a430 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4340154aa0 tx=0x7f4330007e90 comp rx=0 tx=0).stop 2024-09-11T00:35:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 -- 172.21.15.31:0/443742234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43400c23a0 msgr2=0x7f434014aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43400c23a0 0x7f434014aed0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4328000c00 tx=0x7f4328032c10 comp rx=0 tx=0).stop 2024-09-11T00:35:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 -- 172.21.15.31:0/443742234 shutdown_connections 2024-09-11T00:35:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f431c077f70 0x7f431c07a430 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43401518b0 0x7f43401ed020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43400c23a0 0x7f434014aed0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 --2- 172.21.15.31:0/443742234 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4340072480 0x7f4340148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 -- 172.21.15.31:0/443742234 >> 172.21.15.31:0/443742234 conn(0x7f43400a9750 msgr2=0x7f434014fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 -- 172.21.15.31:0/443742234 shutdown_connections 2024-09-11T00:35:10.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:10.829+0000 7f4347800640 1 -- 172.21.15.31:0/443742234 wait complete. 2024-09-11T00:35:10.848 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:35:10:588522+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 43, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 43, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5302": { "addr": "172.21.15.204:6835/3444890210", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 3444890210, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 3444890210, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5302, "incarnation": 42, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:35:09.634906+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5302 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4898": { "addr": "172.21.15.204:6837/645869523", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 645869523, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 645869523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4898, "incarnation": 37, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:34:12.311591+0000", "qdb_cluster": [ 4898 ], "qdb_leader": 4898, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4898 } } } ], "standbys": [ { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:35:11.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:35:11.190 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:35:12.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:13.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.046+0000 7faeed600640 1 -- 172.21.15.31:0/876528171 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faee80c6420 msgr2=0x7faee8153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:13.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.046+0000 7faeed600640 1 --2- 172.21.15.31:0/876528171 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faee80c6420 0x7faee8153310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faed0009860 tx=0x7faed002a5a0 comp rx=0 tx=0).stop 2024-09-11T00:35:13.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.047+0000 7faeed600640 1 -- 172.21.15.31:0/876528171 shutdown_connections 2024-09-11T00:35:13.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.047+0000 7faeed600640 1 --2- 172.21.15.31:0/876528171 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faee80c6420 0x7faee8153310 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:13.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.047+0000 7faeed600640 1 -- 172.21.15.31:0/876528171 >> 172.21.15.31:0/876528171 conn(0x7faee806caf0 msgr2=0x7faee806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:13.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.047+0000 7faeed600640 1 -- 172.21.15.31:0/876528171 shutdown_connections 2024-09-11T00:35:13.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.047+0000 7faeed600640 1 -- 172.21.15.31:0/876528171 wait complete. 2024-09-11T00:35:13.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.048+0000 7faeed600640 1 Processor -- start 2024-09-11T00:35:13.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.048+0000 7faeed600640 1 -- start start 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.049+0000 7faeed600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faee80c5aa0 0x7faee80aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.049+0000 7faeed600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faee80c6420 0x7faee8153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.049+0000 7faeed600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8153850 0x7faee81553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.049+0000 7faeed600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faee80c5e80 con 0x7faee8153850 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.049+0000 7faeed600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faee80c3160 con 0x7faee80c6420 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.049+0000 7faeed600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faee80c6000 con 0x7faee80c5aa0 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.050+0000 7faeecc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8153850 0x7faee81553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.050+0000 7faeecc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8153850 0x7faee81553f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42322/0 (socket says 172.21.15.31:42322) 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.050+0000 7faeecc00640 1 -- 172.21.15.31:0/3805858419 learned_addr learned my addr 172.21.15.31:0/3805858419 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.050+0000 7faee7e00640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faee80c5aa0 0x7faee80aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:13.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.050+0000 7faedfe00640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faee80c6420 0x7faee8153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:13.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.050+0000 7faeecc00640 1 -- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faee80c5aa0 msgr2=0x7faee80aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:13.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.050+0000 7faeecc00640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faee80c5aa0 0x7faee80aabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:13.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.050+0000 7faeecc00640 1 -- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faee80c6420 msgr2=0x7faee8153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:13.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.050+0000 7faeecc00640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faee80c6420 0x7faee8153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:13.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.050+0000 7faeecc00640 1 -- 172.21.15.31:0/3805858419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee80ac7c0 con 0x7faee8153850 2024-09-11T00:35:13.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.051+0000 7faeecc00640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8153850 0x7faee81553f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7faee000c680 tx=0x7faee000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:13.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.051+0000 7faee4c00640 1 -- 172.21.15.31:0/3805858419 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0003b70 con 0x7faee8153850 2024-09-11T00:35:13.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.051+0000 7faee4c00640 1 -- 172.21.15.31:0/3805858419 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faee0003d30 con 0x7faee8153850 2024-09-11T00:35:13.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.051+0000 7faee4c00640 1 -- 172.21.15.31:0/3805858419 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0005330 con 0x7faee8153850 2024-09-11T00:35:13.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.051+0000 7faeed600640 1 -- 172.21.15.31:0/3805858419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee8154980 con 0x7faee8153850 2024-09-11T00:35:13.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.051+0000 7faeed600640 1 -- 172.21.15.31:0/3805858419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee80acd60 con 0x7faee8153850 2024-09-11T00:35:13.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.052+0000 7faeed600640 1 -- 172.21.15.31:0/3805858419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faee8150da0 con 0x7faee8153850 2024-09-11T00:35:13.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.053+0000 7faee4c00640 1 -- 172.21.15.31:0/3805858419 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7faee00054d0 con 0x7faee8153850 2024-09-11T00:35:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.055+0000 7faee4c00640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faec4077f70 0x7faec407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.055+0000 7faee4c00640 1 -- 172.21.15.31:0/3805858419 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7faee0099ec0 con 0x7faee8153850 2024-09-11T00:35:13.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.055+0000 7faee7e00640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faec4077f70 0x7faec407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:13.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.056+0000 7faee7e00640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faec4077f70 0x7faec407a430 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faed8000c00 tx=0x7faed8007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:13.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.059+0000 7faee4c00640 1 -- 172.21.15.31:0/3805858419 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7faee8150da0 con 0x7faee8153850 2024-09-11T00:35:13.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.319+0000 7faeed600640 1 -- 172.21.15.31:0/3805858419 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faee8153310 con 0x7faee8153850 2024-09-11T00:35:13.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.320+0000 7faee4c00640 1 -- 172.21.15.31:0/3805858419 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5710 (secure 0 0 0) 0x7faee00057f0 con 0x7faee8153850 2024-09-11T00:35:13.322 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:13.322 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"btime":"2024-09-11T00:35:12:604116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:13.322 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 45 2024-09-11T00:35:13.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.322+0000 7faeed600640 1 -- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faec4077f70 msgr2=0x7faec407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:13.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.322+0000 7faeed600640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faec4077f70 0x7faec407a430 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faed8000c00 tx=0x7faed8007eb0 comp rx=0 tx=0).stop 2024-09-11T00:35:13.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.322+0000 7faeed600640 1 -- 172.21.15.31:0/3805858419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8153850 msgr2=0x7faee81553f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:13.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.322+0000 7faeed600640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8153850 0x7faee81553f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7faee000c680 tx=0x7faee000cb40 comp rx=0 tx=0).stop 2024-09-11T00:35:13.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.322+0000 7faeed600640 1 -- 172.21.15.31:0/3805858419 shutdown_connections 2024-09-11T00:35:13.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.322+0000 7faeed600640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faec4077f70 0x7faec407a430 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:13.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.322+0000 7faeed600640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8153850 0x7faee81553f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:13.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.322+0000 7faeed600640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faee80c6420 0x7faee8153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:13.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.322+0000 7faeed600640 1 --2- 172.21.15.31:0/3805858419 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faee80c5aa0 0x7faee80aabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:13.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.322+0000 7faeed600640 1 -- 172.21.15.31:0/3805858419 >> 172.21.15.31:0/3805858419 conn(0x7faee806caf0 msgr2=0x7faee81562c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:13.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.323+0000 7faeed600640 1 -- 172.21.15.31:0/3805858419 shutdown_connections 2024-09-11T00:35:13.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:13.323+0000 7faeed600640 1 -- 172.21.15.31:0/3805858419 wait complete. 2024-09-11T00:35:13.343 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=5076 2024-09-11T00:35:13.343 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.a 2024-09-11T00:35:14.344 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-11T00:35:14.344 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-11T00:35:14.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-11T00:35:14.350 INFO:tasks.ceph.mds.a:Started 2024-09-11T00:35:14.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:14.355 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:35:14.353+0000 7f4e71200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:35:14.495 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-11T00:35:14.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.538+0000 7f6670400640 1 -- 172.21.15.31:0/2559565538 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66680c71c0 msgr2=0x7f66680c5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:14.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.538+0000 7f6670400640 1 --2- 172.21.15.31:0/2559565538 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66680c71c0 0x7f66680c5920 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f66680cab30 tx=0x7f665402a4a0 comp rx=0 tx=0).stop 2024-09-11T00:35:14.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.538+0000 7f6670400640 1 -- 172.21.15.31:0/2559565538 shutdown_connections 2024-09-11T00:35:14.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.538+0000 7f6670400640 1 --2- 172.21.15.31:0/2559565538 >> v2:172.21.15.31:3300/0 conn(0x7f66680c8ed0 0x7f66680c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:14.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.538+0000 7f6670400640 1 --2- 172.21.15.31:0/2559565538 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66680c71c0 0x7f66680c5920 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:14.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.538+0000 7f6670400640 1 -- 172.21.15.31:0/2559565538 >> 172.21.15.31:0/2559565538 conn(0x7f66680688e0 msgr2=0x7f6668068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:14.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.538+0000 7f6670400640 1 -- 172.21.15.31:0/2559565538 shutdown_connections 2024-09-11T00:35:14.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.538+0000 7f6670400640 1 -- 172.21.15.31:0/2559565538 wait complete. 2024-09-11T00:35:14.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.540+0000 7f6670400640 1 Processor -- start 2024-09-11T00:35:14.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.540+0000 7f6670400640 1 -- start start 2024-09-11T00:35:14.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.541+0000 7f6670400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66680c71c0 0x7f66681ef6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:14.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.541+0000 7f6670400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66680c8ed0 0x7f66680a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:14.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.541+0000 7f6670400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6668158260 0x7f66680aa000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:14.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.541+0000 7f6670400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6668158640 con 0x7f66680c8ed0 2024-09-11T00:35:14.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.541+0000 7f6670400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f66680c44c0 con 0x7f66680c71c0 2024-09-11T00:35:14.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.541+0000 7f6670400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f66680c98a0 con 0x7f6668158260 2024-09-11T00:35:14.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66680c8ed0 0x7f66680a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:14.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66680c8ed0 0x7f66680a9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42358/0 (socket says 172.21.15.31:42358) 2024-09-11T00:35:14.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666d000640 1 -- 172.21.15.31:0/3764490734 learned_addr learned my addr 172.21.15.31:0/3764490734 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:14.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666da00640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66680c71c0 0x7f66681ef6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:14.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666e400640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6668158260 0x7f66680aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:14.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666d000640 1 -- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6668158260 msgr2=0x7f66680aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:14.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666d000640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6668158260 0x7f66680aa000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:14.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666d000640 1 -- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66680c71c0 msgr2=0x7f66681ef6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:14.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666d000640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66680c71c0 0x7f66681ef6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:14.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666d000640 1 -- 172.21.15.31:0/3764490734 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66680ac040 con 0x7f66680c8ed0 2024-09-11T00:35:14.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.542+0000 7f666d000640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66680c8ed0 0x7f66680a9ac0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6654009780 tx=0x7f6654032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:14.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.543+0000 7f6665600640 1 -- 172.21.15.31:0/3764490734 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6654040070 con 0x7f66680c8ed0 2024-09-11T00:35:14.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.543+0000 7f6665600640 1 -- 172.21.15.31:0/3764490734 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6654005310 con 0x7f66680c8ed0 2024-09-11T00:35:14.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.543+0000 7f6665600640 1 -- 172.21.15.31:0/3764490734 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66540047a0 con 0x7f66680c8ed0 2024-09-11T00:35:14.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.543+0000 7f6670400640 1 -- 172.21.15.31:0/3764490734 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66681f0170 con 0x7f66680c8ed0 2024-09-11T00:35:14.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.543+0000 7f6670400640 1 -- 172.21.15.31:0/3764490734 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66680c2ec0 con 0x7f66680c8ed0 2024-09-11T00:35:14.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.544+0000 7f6670400640 1 -- 172.21.15.31:0/3764490734 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66680c9360 con 0x7f66680c8ed0 2024-09-11T00:35:14.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.545+0000 7f6665600640 1 -- 172.21.15.31:0/3764490734 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6654004940 con 0x7f66680c8ed0 2024-09-11T00:35:14.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.547+0000 7f6665600640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6648077f70 0x7f664807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:14.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.547+0000 7f6665600640 1 -- 172.21.15.31:0/3764490734 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f66540bb5f0 con 0x7f66680c8ed0 2024-09-11T00:35:14.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.547+0000 7f666da00640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6648077f70 0x7f664807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:14.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.548+0000 7f666da00640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6648077f70 0x7f664807a430 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f665c000f60 tx=0x7f665c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:14.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.551+0000 7f6665600640 1 -- 172.21.15.31:0/3764490734 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f66680ac040 con 0x7f66680c8ed0 2024-09-11T00:35:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.832+0000 7f6670400640 1 -- 172.21.15.31:0/3764490734 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66680c8310 con 0x7f66680c8ed0 2024-09-11T00:35:14.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.833+0000 7f6665600640 1 -- 172.21.15.31:0/3764490734 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6535 (secure 0 0 0) 0x7f66540834d0 con 0x7f66680c8ed0 2024-09-11T00:35:14.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:14.836 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"btime":"2024-09-11T00:35:14:607653+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5374,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:14.836 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2024-09-11T00:35:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 -- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6648077f70 msgr2=0x7f664807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6648077f70 0x7f664807a430 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f665c000f60 tx=0x7f665c009040 comp rx=0 tx=0).stop 2024-09-11T00:35:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 -- 172.21.15.31:0/3764490734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66680c8ed0 msgr2=0x7f66680a9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66680c8ed0 0x7f66680a9ac0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6654009780 tx=0x7f6654032c10 comp rx=0 tx=0).stop 2024-09-11T00:35:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 -- 172.21.15.31:0/3764490734 shutdown_connections 2024-09-11T00:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6648077f70 0x7f664807a430 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6668158260 0x7f66680aa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66680c8ed0 0x7f66680a9ac0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 --2- 172.21.15.31:0/3764490734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66680c71c0 0x7f66681ef6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 -- 172.21.15.31:0/3764490734 >> 172.21.15.31:0/3764490734 conn(0x7f66680688e0 msgr2=0x7f666806e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 -- 172.21.15.31:0/3764490734 shutdown_connections 2024-09-11T00:35:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:14.836+0000 7f6670400640 1 -- 172.21.15.31:0/3764490734 wait complete. 2024-09-11T00:35:14.857 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-11T00:35:14.858 INFO:tasks.mds_thrash.fs.[a]:waiting for 105 secs before thrashing 2024-09-11T00:35:35.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:35.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.758+0000 7f75b4c00640 1 -- 172.21.15.31:0/2336313177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7594003a00 msgr2=0x7f7594003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:35.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.758+0000 7f75b4c00640 1 --2- 172.21.15.31:0/2336313177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7594003a00 0x7f7594003e20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f75b00c3e40 tx=0x7f7594005830 comp rx=0 tx=0).stop 2024-09-11T00:35:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.758+0000 7f75afe00640 1 -- 172.21.15.31:0/2336313177 reap_dead start 2024-09-11T00:35:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.758+0000 7f75b4c00640 1 -- 172.21.15.31:0/2336313177 shutdown_connections 2024-09-11T00:35:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.758+0000 7f75b4c00640 1 -- 172.21.15.31:0/2336313177 >> 172.21.15.31:0/2336313177 conn(0x7f75b00aa6f0 msgr2=0x7f75b00aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.758+0000 7f75b4c00640 1 -- 172.21.15.31:0/2336313177 shutdown_connections 2024-09-11T00:35:35.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.758+0000 7f75b4c00640 1 -- 172.21.15.31:0/2336313177 wait complete. 2024-09-11T00:35:35.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.760+0000 7f75b4c00640 1 Processor -- start 2024-09-11T00:35:35.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.760+0000 7f75b4c00640 1 -- start start 2024-09-11T00:35:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.761+0000 7f75b4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b01513f0 0x7f75b01e9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.761+0000 7f75b4c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75b00a8df0 0x7f75b00a9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.761+0000 7f75b4c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b01479a0 0x7f75b0147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.761+0000 7f75b4c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f75b0153440 con 0x7f75b01513f0 2024-09-11T00:35:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.761+0000 7f75b4c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f75b0153160 con 0x7f75b01479a0 2024-09-11T00:35:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.761+0000 7f75b4c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f75b00c4070 con 0x7f75b00a8df0 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75af400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b01513f0 0x7f75b01e9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75af400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b01513f0 0x7f75b01e9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46748/0 (socket says 172.21.15.31:46748) 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75af400640 1 -- 172.21.15.31:0/1119927459 learned_addr learned my addr 172.21.15.31:0/1119927459 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75afe00640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b01479a0 0x7f75b0147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75aea00640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75b00a8df0 0x7f75b00a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75af400640 1 -- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75b00a8df0 msgr2=0x7f75b00a9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75af400640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75b00a8df0 0x7f75b00a9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75af400640 1 -- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b01479a0 msgr2=0x7f75b0147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75af400640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b01479a0 0x7f75b0147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75af400640 1 -- 172.21.15.31:0/1119927459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b01dd050 con 0x7f75b01513f0 2024-09-11T00:35:35.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.762+0000 7f75af400640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b01513f0 0x7f75b01e9820 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f75a002b130 tx=0x7f75a0000f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:35.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.763+0000 7f75a7400640 1 -- 172.21.15.31:0/1119927459 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0044070 con 0x7f75b01513f0 2024-09-11T00:35:35.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.763+0000 7f75a7400640 1 -- 172.21.15.31:0/1119927459 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75a00036d0 con 0x7f75b01513f0 2024-09-11T00:35:35.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.763+0000 7f75a7400640 1 -- 172.21.15.31:0/1119927459 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0031070 con 0x7f75b01513f0 2024-09-11T00:35:35.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.763+0000 7f75b4c00640 1 -- 172.21.15.31:0/1119927459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b00ae6e0 con 0x7f75b01513f0 2024-09-11T00:35:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.763+0000 7f75b4c00640 1 -- 172.21.15.31:0/1119927459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b0150e20 con 0x7f75b01513f0 2024-09-11T00:35:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.764+0000 7f75b4c00640 1 -- 172.21.15.31:0/1119927459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b0077030 con 0x7f75b01513f0 2024-09-11T00:35:35.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.765+0000 7f75a7400640 1 -- 172.21.15.31:0/1119927459 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f75a0003b00 con 0x7f75b01513f0 2024-09-11T00:35:35.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.767+0000 7f75a7400640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7590077fa0 0x7f759007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:35.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.767+0000 7f75a7400640 1 -- 172.21.15.31:0/1119927459 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f75a00ba190 con 0x7f75b01513f0 2024-09-11T00:35:35.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.767+0000 7f75aea00640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7590077fa0 0x7f759007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:35.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.768+0000 7f75aea00640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7590077fa0 0x7f759007a460 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f75b00c2f90 tx=0x7f759c00b260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:35.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:35.771+0000 7f75a7400640 1 -- 172.21.15.31:0/1119927459 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f75b0077030 con 0x7f75b01513f0 2024-09-11T00:35:36.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.027+0000 7f75b4c00640 1 -- 172.21.15.31:0/1119927459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75b00740e0 con 0x7f75b01513f0 2024-09-11T00:35:36.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.028+0000 7f75a7400640 1 -- 172.21.15.31:0/1119927459 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6535 (secure 0 0 0) 0x7f75a0003e20 con 0x7f75b01513f0 2024-09-11T00:35:36.030 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:36.030 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"btime":"2024-09-11T00:35:14:607653+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5374,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:36.030 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2024-09-11T00:35:36.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.030+0000 7f75b4c00640 1 -- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7590077fa0 msgr2=0x7f759007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:36.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.030+0000 7f75b4c00640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7590077fa0 0x7f759007a460 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f75b00c2f90 tx=0x7f759c00b260 comp rx=0 tx=0).stop 2024-09-11T00:35:36.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.030+0000 7f75b4c00640 1 -- 172.21.15.31:0/1119927459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b01513f0 msgr2=0x7f75b01e9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:36.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.030+0000 7f75b4c00640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b01513f0 0x7f75b01e9820 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f75a002b130 tx=0x7f75a0000f60 comp rx=0 tx=0).stop 2024-09-11T00:35:36.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.031+0000 7f75b4c00640 1 -- 172.21.15.31:0/1119927459 shutdown_connections 2024-09-11T00:35:36.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.031+0000 7f75b4c00640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7590077fa0 0x7f759007a460 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.031+0000 7f75b4c00640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b01479a0 0x7f75b0147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.031+0000 7f75b4c00640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75b00a8df0 0x7f75b00a9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.031+0000 7f75b4c00640 1 --2- 172.21.15.31:0/1119927459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b01513f0 0x7f75b01e9820 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.031+0000 7f75b4c00640 1 -- 172.21.15.31:0/1119927459 >> 172.21.15.31:0/1119927459 conn(0x7f75b00aa6f0 msgr2=0x7f75b0149b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:36.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.031+0000 7f75b4c00640 1 -- 172.21.15.31:0/1119927459 shutdown_connections 2024-09-11T00:35:36.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.031+0000 7f75b4c00640 1 -- 172.21.15.31:0/1119927459 wait complete. 2024-09-11T00:35:36.049 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-11T00:35:36.050 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-11T00:35:36.050 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:35:36.069 INFO:tasks.ceph.mds.b:Stopped 2024-09-11T00:35:36.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:36.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.245+0000 7f1570c00640 1 -- 172.21.15.31:0/1146811815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c0c23a0 msgr2=0x7f156c1516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:36.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.245+0000 7f1570c00640 1 --2- 172.21.15.31:0/1146811815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c0c23a0 0x7f156c1516e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1558009830 tx=0x7f155802a5a0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.245+0000 7f1570c00640 1 -- 172.21.15.31:0/1146811815 shutdown_connections 2024-09-11T00:35:36.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.245+0000 7f1570c00640 1 --2- 172.21.15.31:0/1146811815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c0c23a0 0x7f156c1516e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.245+0000 7f1570c00640 1 --2- 172.21.15.31:0/1146811815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f156c072480 0x7f156c1511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.245+0000 7f1570c00640 1 -- 172.21.15.31:0/1146811815 >> 172.21.15.31:0/1146811815 conn(0x7f156c0a9750 msgr2=0x7f156c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:36.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.245+0000 7f1570c00640 1 -- 172.21.15.31:0/1146811815 shutdown_connections 2024-09-11T00:35:36.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.246+0000 7f1570c00640 1 -- 172.21.15.31:0/1146811815 wait complete. 2024-09-11T00:35:36.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.247+0000 7f1570c00640 1 Processor -- start 2024-09-11T00:35:36.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.248+0000 7f1570c00640 1 -- start start 2024-09-11T00:35:36.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f1570c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f156c072480 0x7f156c157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:36.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f1570c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c0c23a0 0x7f156c14cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:36.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f1570c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f156c153f20 0x7f156c14d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:36.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f1570c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f156c074200 con 0x7f156c0c23a0 2024-09-11T00:35:36.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f1570c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f156c074080 con 0x7f156c072480 2024-09-11T00:35:36.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f1570c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f156c072860 con 0x7f156c153f20 2024-09-11T00:35:36.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f1569400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c0c23a0 0x7f156c14cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:36.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f1569400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c0c23a0 0x7f156c14cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46764/0 (socket says 172.21.15.31:46764) 2024-09-11T00:35:36.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f1569e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f156c072480 0x7f156c157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:36.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f1569400640 1 -- 172.21.15.31:0/985555835 learned_addr learned my addr 172.21.15.31:0/985555835 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:36.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.249+0000 7f156a800640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f156c153f20 0x7f156c14d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.250+0000 7f1569400640 1 -- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f156c153f20 msgr2=0x7f156c14d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.250+0000 7f1569400640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f156c153f20 0x7f156c14d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.250+0000 7f1569400640 1 -- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f156c072480 msgr2=0x7f156c157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.250+0000 7f1569400640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f156c072480 0x7f156c157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.250+0000 7f1569400640 1 -- 172.21.15.31:0/985555835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156c0c14a0 con 0x7f156c0c23a0 2024-09-11T00:35:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.250+0000 7f1569400640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c0c23a0 0x7f156c14cca0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f155802ab90 tx=0x7f1558032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.250+0000 7f1562000640 1 -- 172.21.15.31:0/985555835 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558040070 con 0x7f156c0c23a0 2024-09-11T00:35:36.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.250+0000 7f1562000640 1 -- 172.21.15.31:0/985555835 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1558005310 con 0x7f156c0c23a0 2024-09-11T00:35:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.250+0000 7f1562000640 1 -- 172.21.15.31:0/985555835 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15580047a0 con 0x7f156c0c23a0 2024-09-11T00:35:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.250+0000 7f1570c00640 1 -- 172.21.15.31:0/985555835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156c0ad810 con 0x7f156c0c23a0 2024-09-11T00:35:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.251+0000 7f1570c00640 1 -- 172.21.15.31:0/985555835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f156c0c11a0 con 0x7f156c0c23a0 2024-09-11T00:35:36.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.252+0000 7f1570c00640 1 -- 172.21.15.31:0/985555835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f156c1eaa90 con 0x7f156c0c23a0 2024-09-11T00:35:36.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.253+0000 7f1562000640 1 -- 172.21.15.31:0/985555835 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1558004b50 con 0x7f156c0c23a0 2024-09-11T00:35:36.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.254+0000 7f1562000640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f154c077f70 0x7f154c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:36.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.255+0000 7f1562000640 1 -- 172.21.15.31:0/985555835 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f15580bb760 con 0x7f156c0c23a0 2024-09-11T00:35:36.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.255+0000 7f1569e00640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f154c077f70 0x7f154c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:36.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.256+0000 7f1569e00640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f154c077f70 0x7f154c07a430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f155c009bd0 tx=0x7f155c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:36.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.259+0000 7f1562000640 1 -- 172.21.15.31:0/985555835 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f156c1eaa90 con 0x7f156c0c23a0 2024-09-11T00:35:36.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.562+0000 7f1570c00640 1 -- 172.21.15.31:0/985555835 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f156c1500c0 con 0x7f156c0c23a0 2024-09-11T00:35:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.562+0000 7f1562000640 1 -- 172.21.15.31:0/985555835 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6535 (secure 0 0 0) 0x7f1558083640 con 0x7f156c0c23a0 2024-09-11T00:35:36.564 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:36.564 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"btime":"2024-09-11T00:35:14:607653+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5374,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2024-09-11T00:35:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 -- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f154c077f70 msgr2=0x7f154c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f154c077f70 0x7f154c07a430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f155c009bd0 tx=0x7f155c005f90 comp rx=0 tx=0).stop 2024-09-11T00:35:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 -- 172.21.15.31:0/985555835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c0c23a0 msgr2=0x7f156c14cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c0c23a0 0x7f156c14cca0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f155802ab90 tx=0x7f1558032c10 comp rx=0 tx=0).stop 2024-09-11T00:35:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 -- 172.21.15.31:0/985555835 shutdown_connections 2024-09-11T00:35:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f154c077f70 0x7f154c07a430 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f156c153f20 0x7f156c14d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f156c0c23a0 0x7f156c14cca0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 --2- 172.21.15.31:0/985555835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f156c072480 0x7f156c157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:36.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 -- 172.21.15.31:0/985555835 >> 172.21.15.31:0/985555835 conn(0x7f156c0a9750 msgr2=0x7f156c0c57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:36.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 -- 172.21.15.31:0/985555835 shutdown_connections 2024-09-11T00:35:36.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:36.565+0000 7f1570c00640 1 -- 172.21.15.31:0/985555835 wait complete. 2024-09-11T00:35:36.586 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:35:38.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.763+0000 7f9a2d800640 1 -- 172.21.15.31:0/419117000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a28150020 msgr2=0x7f9a280ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.763+0000 7f9a2d800640 1 --2- 172.21.15.31:0/419117000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a28150020 0x7f9a280ad4e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9a2806ed70 tx=0x7f9a1802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:35:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.763+0000 7f9a2d800640 1 -- 172.21.15.31:0/419117000 shutdown_connections 2024-09-11T00:35:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.763+0000 7f9a2d800640 1 --2- 172.21.15.31:0/419117000 >> v2:172.21.15.204:3300/0 conn(0x7f9a2806e850 0x7f9a28160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:38.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.763+0000 7f9a2d800640 1 --2- 172.21.15.31:0/419117000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a28150020 0x7f9a280ad4e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.763+0000 7f9a2d800640 1 -- 172.21.15.31:0/419117000 >> 172.21.15.31:0/419117000 conn(0x7f9a28068f10 msgr2=0x7f9a280cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.763+0000 7f9a2d800640 1 -- 172.21.15.31:0/419117000 shutdown_connections 2024-09-11T00:35:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.763+0000 7f9a2d800640 1 -- 172.21.15.31:0/419117000 wait complete. 2024-09-11T00:35:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.764+0000 7f9a2d800640 1 Processor -- start 2024-09-11T00:35:38.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.765+0000 7f9a2d800640 1 -- start start 2024-09-11T00:35:38.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.765+0000 7f9a2d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a2806e850 0x7f9a280b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:38.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2d800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9a280ada20 0x7f9a280b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:38.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a28150020 0x7f9a281e0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:38.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2d800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a28157550 con 0x7f9a28150020 2024-09-11T00:35:38.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2d800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9a281573d0 con 0x7f9a2806e850 2024-09-11T00:35:38.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2d800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9a28156960 con 0x7f9a280ada20 2024-09-11T00:35:38.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a27e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a2806e850 0x7f9a280b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:38.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a27e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a2806e850 0x7f9a280b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:36836/0 (socket says 172.21.15.31:36836) 2024-09-11T00:35:38.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a27e00640 1 -- 172.21.15.31:0/572009160 learned_addr learned my addr 172.21.15.31:0/572009160 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:38.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2ce00640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a28150020 0x7f9a281e0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:38.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a27400640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9a280ada20 0x7f9a280b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:38.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2ce00640 1 -- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9a280ada20 msgr2=0x7f9a280b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:38.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2ce00640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9a280ada20 0x7f9a280b0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:38.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2ce00640 1 -- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a2806e850 msgr2=0x7f9a280b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:38.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2ce00640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a2806e850 0x7f9a280b0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:38.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.766+0000 7f9a2ce00640 1 -- 172.21.15.31:0/572009160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a280af1c0 con 0x7f9a28150020 2024-09-11T00:35:38.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.767+0000 7f9a2ce00640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a28150020 0x7f9a281e0e50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000e960 tx=0x7f9a2000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.767+0000 7f9a1fe00640 1 -- 172.21.15.31:0/572009160 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20004d20 con 0x7f9a28150020 2024-09-11T00:35:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.767+0000 7f9a1fe00640 1 -- 172.21.15.31:0/572009160 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a2000ca60 con 0x7f9a28150020 2024-09-11T00:35:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.767+0000 7f9a1fe00640 1 -- 172.21.15.31:0/572009160 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20005020 con 0x7f9a28150020 2024-09-11T00:35:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.767+0000 7f9a2d800640 1 -- 172.21.15.31:0/572009160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a280aea70 con 0x7f9a28150020 2024-09-11T00:35:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.767+0000 7f9a2d800640 1 -- 172.21.15.31:0/572009160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a281df9a0 con 0x7f9a28150020 2024-09-11T00:35:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.768+0000 7f9a2d800640 1 -- 172.21.15.31:0/572009160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a280ae890 con 0x7f9a28150020 2024-09-11T00:35:38.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.770+0000 7f9a1fe00640 1 -- 172.21.15.31:0/572009160 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9a2000f040 con 0x7f9a28150020 2024-09-11T00:35:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.771+0000 7f9a1fe00640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9a08077f70 0x7f9a0807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:38.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.771+0000 7f9a1fe00640 1 -- 172.21.15.31:0/572009160 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f9a2009a700 con 0x7f9a28150020 2024-09-11T00:35:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.771+0000 7f9a27e00640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9a08077f70 0x7f9a0807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:38.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.772+0000 7f9a27e00640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9a08077f70 0x7f9a0807a430 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9a281e04f0 tx=0x7f9a18004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:38.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:38.776+0000 7f9a1fe00640 1 -- 172.21.15.31:0/572009160 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9a280ae890 con 0x7f9a28150020 2024-09-11T00:35:39.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.038+0000 7f9a2d800640 1 -- 172.21.15.31:0/572009160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a280cc220 con 0x7f9a28150020 2024-09-11T00:35:39.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.039+0000 7f9a1fe00640 1 -- 172.21.15.31:0/572009160 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6535 (secure 0 0 0) 0x7f9a20016030 con 0x7f9a28150020 2024-09-11T00:35:39.065 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:39.066 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"btime":"2024-09-11T00:35:14:607653+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5374,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2024-09-11T00:35:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.041+0000 7f9a2d800640 1 -- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9a08077f70 msgr2=0x7f9a0807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.041+0000 7f9a2d800640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9a08077f70 0x7f9a0807a430 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9a281e04f0 tx=0x7f9a18004730 comp rx=0 tx=0).stop 2024-09-11T00:35:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.041+0000 7f9a2d800640 1 -- 172.21.15.31:0/572009160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a28150020 msgr2=0x7f9a281e0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.041+0000 7f9a2d800640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a28150020 0x7f9a281e0e50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000e960 tx=0x7f9a2000ed60 comp rx=0 tx=0).stop 2024-09-11T00:35:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.042+0000 7f9a2d800640 1 -- 172.21.15.31:0/572009160 shutdown_connections 2024-09-11T00:35:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.042+0000 7f9a2d800640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9a08077f70 0x7f9a0807a430 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:39.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.042+0000 7f9a2d800640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a28150020 0x7f9a281e0e50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:39.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.042+0000 7f9a2d800640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9a280ada20 0x7f9a280b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:39.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.042+0000 7f9a2d800640 1 --2- 172.21.15.31:0/572009160 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a2806e850 0x7f9a280b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:39.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.042+0000 7f9a2d800640 1 -- 172.21.15.31:0/572009160 >> 172.21.15.31:0/572009160 conn(0x7f9a28068f10 msgr2=0x7f9a2814ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:39.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.042+0000 7f9a2d800640 1 -- 172.21.15.31:0/572009160 shutdown_connections 2024-09-11T00:35:39.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:39.042+0000 7f9a2d800640 1 -- 172.21.15.31:0/572009160 wait complete. 2024-09-11T00:35:39.068 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:35:41.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.241+0000 7f0e0e600640 1 -- 172.21.15.31:0/2883615190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e080ac2c0 msgr2=0x7f0e080ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.241+0000 7f0e0e600640 1 --2- 172.21.15.31:0/2883615190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e080ac2c0 0x7f0e080ac6a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0e080c5ea0 tx=0x7f0dfc02a440 comp rx=0 tx=0).stop 2024-09-11T00:35:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.241+0000 7f0e0e600640 1 -- 172.21.15.31:0/2883615190 shutdown_connections 2024-09-11T00:35:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.241+0000 7f0e0e600640 1 --2- 172.21.15.31:0/2883615190 >> v2:172.21.15.204:3300/0 conn(0x7f0e080a6e20 0x7f0e081510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.241+0000 7f0e0e600640 1 --2- 172.21.15.31:0/2883615190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e080ac2c0 0x7f0e080ac6a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:41.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.241+0000 7f0e0e600640 1 -- 172.21.15.31:0/2883615190 >> 172.21.15.31:0/2883615190 conn(0x7f0e0806caf0 msgr2=0x7f0e0806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:41.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.241+0000 7f0e0e600640 1 -- 172.21.15.31:0/2883615190 shutdown_connections 2024-09-11T00:35:41.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.241+0000 7f0e0e600640 1 -- 172.21.15.31:0/2883615190 wait complete. 2024-09-11T00:35:41.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.242+0000 7f0e0e600640 1 Processor -- start 2024-09-11T00:35:41.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.243+0000 7f0e0e600640 1 -- start start 2024-09-11T00:35:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.243+0000 7f0e0e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e080a6e20 0x7f0e081ec950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.243+0000 7f0e0e600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0e080ac2c0 0x7f0e080c3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0e600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e080acbe0 0x7f0e081ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0e600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e08151780 con 0x7f0e080a6e20 2024-09-11T00:35:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0e600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0e08151600 con 0x7f0e080acbe0 2024-09-11T00:35:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0e600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0e08151900 con 0x7f0e080ac2c0 2024-09-11T00:35:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e080a6e20 0x7f0e081ec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e080a6e20 0x7f0e081ec950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34324/0 (socket says 172.21.15.31:34324) 2024-09-11T00:35:41.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0d200640 1 -- 172.21.15.31:0/317757679 learned_addr learned my addr 172.21.15.31:0/317757679 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:41.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0dc00640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e080acbe0 0x7f0e081ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:41.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e07e00640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0e080ac2c0 0x7f0e080c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:41.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0d200640 1 -- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0e080ac2c0 msgr2=0x7f0e080c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:41.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0d200640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0e080ac2c0 0x7f0e080c3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:41.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0d200640 1 -- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e080acbe0 msgr2=0x7f0e081ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:41.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0d200640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e080acbe0 0x7f0e081ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:41.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e0d200640 1 -- 172.21.15.31:0/317757679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e081ed340 con 0x7f0e080a6e20 2024-09-11T00:35:41.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.244+0000 7f0e07e00640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0e080ac2c0 0x7f0e080c3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:35:41.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.245+0000 7f0e0d200640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e080a6e20 0x7f0e081ec950 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc005fd0 tx=0x7f0dfc0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:41.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.245+0000 7f0e04c00640 1 -- 172.21.15.31:0/317757679 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc03f070 con 0x7f0e080a6e20 2024-09-11T00:35:41.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.245+0000 7f0e04c00640 1 -- 172.21.15.31:0/317757679 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0dfc004890 con 0x7f0e080a6e20 2024-09-11T00:35:41.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.245+0000 7f0e04c00640 1 -- 172.21.15.31:0/317757679 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc035030 con 0x7f0e080a6e20 2024-09-11T00:35:41.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.245+0000 7f0e0e600640 1 -- 172.21.15.31:0/317757679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e081dea00 con 0x7f0e080a6e20 2024-09-11T00:35:41.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.245+0000 7f0e0e600640 1 -- 172.21.15.31:0/317757679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e081dd4a0 con 0x7f0e080a6e20 2024-09-11T00:35:41.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.247+0000 7f0e0e600640 1 -- 172.21.15.31:0/317757679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e081dd6a0 con 0x7f0e080a6e20 2024-09-11T00:35:41.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.248+0000 7f0e04c00640 1 -- 172.21.15.31:0/317757679 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0dfc004cc0 con 0x7f0e080a6e20 2024-09-11T00:35:41.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.249+0000 7f0e04c00640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0de8077fa0 0x7f0de807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:41.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.249+0000 7f0e04c00640 1 -- 172.21.15.31:0/317757679 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f0dfc0bb5b0 con 0x7f0e080a6e20 2024-09-11T00:35:41.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.251+0000 7f0e07e00640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0de8077fa0 0x7f0de807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:41.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.254+0000 7f0e04c00640 1 -- 172.21.15.31:0/317757679 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0e081dd6a0 con 0x7f0e080a6e20 2024-09-11T00:35:41.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.254+0000 7f0e07e00640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0de8077fa0 0x7f0de807a460 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e080c4130 tx=0x7f0df4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.583+0000 7f0e0e600640 1 -- 172.21.15.31:0/317757679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e081510c0 con 0x7f0e080a6e20 2024-09-11T00:35:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.583+0000 7f0e04c00640 1 -- 172.21.15.31:0/317757679 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6535 (secure 0 0 0) 0x7f0dfc08c030 con 0x7f0e080a6e20 2024-09-11T00:35:41.585 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:41.586 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"btime":"2024-09-11T00:35:14:607653+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5374,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:41.586 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2024-09-11T00:35:41.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.586+0000 7f0e0e600640 1 -- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0de8077fa0 msgr2=0x7f0de807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.586+0000 7f0e0e600640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0de8077fa0 0x7f0de807a460 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0e080c4130 tx=0x7f0df4008040 comp rx=0 tx=0).stop 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.586+0000 7f0e0e600640 1 -- 172.21.15.31:0/317757679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e080a6e20 msgr2=0x7f0e081ec950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.586+0000 7f0e0e600640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e080a6e20 0x7f0e081ec950 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc005fd0 tx=0x7f0dfc0039e0 comp rx=0 tx=0).stop 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.586+0000 7f0e0e600640 1 -- 172.21.15.31:0/317757679 shutdown_connections 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.586+0000 7f0e0e600640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0de8077fa0 0x7f0de807a460 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.586+0000 7f0e0e600640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e080acbe0 0x7f0e081ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.586+0000 7f0e0e600640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0e080ac2c0 0x7f0e080c3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.586+0000 7f0e0e600640 1 --2- 172.21.15.31:0/317757679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e080a6e20 0x7f0e081ec950 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.586+0000 7f0e0e600640 1 -- 172.21.15.31:0/317757679 >> 172.21.15.31:0/317757679 conn(0x7f0e0806caf0 msgr2=0x7f0e0814eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.587+0000 7f0e0e600640 1 -- 172.21.15.31:0/317757679 shutdown_connections 2024-09-11T00:35:41.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:41.587+0000 7f0e0e600640 1 -- 172.21.15.31:0/317757679 wait complete. 2024-09-11T00:35:41.605 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:35:42.609 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:35:42.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:35:43.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.790+0000 7f2993400640 1 -- 172.21.15.31:0/171700538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f298c0c44f0 msgr2=0x7f298c0c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.790+0000 7f2993400640 1 --2- 172.21.15.31:0/171700538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f298c0c44f0 0x7f298c0c48d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f298c1585c0 tx=0x7f298002a4d0 comp rx=0 tx=0).stop 2024-09-11T00:35:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.790+0000 7f2993400640 1 -- 172.21.15.31:0/171700538 shutdown_connections 2024-09-11T00:35:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.790+0000 7f2993400640 1 --2- 172.21.15.31:0/171700538 >> v2:172.21.15.204:3300/0 conn(0x7f298c157c40 0x7f298c15e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.790+0000 7f2993400640 1 --2- 172.21.15.31:0/171700538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f298c0c44f0 0x7f298c0c48d0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.790+0000 7f2993400640 1 -- 172.21.15.31:0/171700538 >> 172.21.15.31:0/171700538 conn(0x7f298c0ab8a0 msgr2=0x7f298c0abcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.790+0000 7f2993400640 1 -- 172.21.15.31:0/171700538 shutdown_connections 2024-09-11T00:35:43.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.790+0000 7f2993400640 1 -- 172.21.15.31:0/171700538 wait complete. 2024-09-11T00:35:43.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.792+0000 7f2993400640 1 Processor -- start 2024-09-11T00:35:43.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.792+0000 7f2993400640 1 -- start start 2024-09-11T00:35:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.793+0000 7f2993400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f298c0c44f0 0x7f298c1f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.793+0000 7f2993400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f298c0c4e10 0x7f298c0aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.793+0000 7f2993400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f298c157c40 0x7f298c0ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.793+0000 7f2993400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f298c151af0 con 0x7f298c0c44f0 2024-09-11T00:35:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.793+0000 7f2993400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f298c0c54e0 con 0x7f298c0c4e10 2024-09-11T00:35:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.793+0000 7f2993400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f298c0701b0 con 0x7f298c157c40 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f2990a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f298c0c44f0 0x7f298c1f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f2990a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f298c0c44f0 0x7f298c1f4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34342/0 (socket says 172.21.15.31:34342) 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f2990a00640 1 -- 172.21.15.31:0/3298755320 learned_addr learned my addr 172.21.15.31:0/3298755320 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f298be00640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f298c0c4e10 0x7f298c0aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f2991400640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f298c157c40 0x7f298c0ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f2990a00640 1 -- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f298c157c40 msgr2=0x7f298c0ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f2990a00640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f298c157c40 0x7f298c0ad9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f2990a00640 1 -- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f298c0c4e10 msgr2=0x7f298c0aaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f2990a00640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f298c0c4e10 0x7f298c0aaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f2990a00640 1 -- 172.21.15.31:0/3298755320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298c1f47c0 con 0x7f298c0c44f0 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.794+0000 7f298be00640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f298c0c4e10 0x7f298c0aaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:35:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.795+0000 7f2990a00640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f298c0c44f0 0x7f298c1f4280 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2980000c00 tx=0x7f2980003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.795+0000 7f2988c00640 1 -- 172.21.15.31:0/3298755320 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298003f070 con 0x7f298c0c44f0 2024-09-11T00:35:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.795+0000 7f2988c00640 1 -- 172.21.15.31:0/3298755320 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29800047a0 con 0x7f298c0c44f0 2024-09-11T00:35:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.795+0000 7f2988c00640 1 -- 172.21.15.31:0/3298755320 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980035030 con 0x7f298c0c44f0 2024-09-11T00:35:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.795+0000 7f2993400640 1 -- 172.21.15.31:0/3298755320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f298c06ed50 con 0x7f298c0c44f0 2024-09-11T00:35:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.795+0000 7f2993400640 1 -- 172.21.15.31:0/3298755320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f298c1f3a80 con 0x7f298c0c44f0 2024-09-11T00:35:43.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.796+0000 7f2993400640 1 -- 172.21.15.31:0/3298755320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f298c0a9380 con 0x7f298c0c44f0 2024-09-11T00:35:43.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.797+0000 7f2988c00640 1 -- 172.21.15.31:0/3298755320 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2980004bb0 con 0x7f298c0c44f0 2024-09-11T00:35:43.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.798+0000 7f2988c00640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f296c077f70 0x7f296c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:43.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.799+0000 7f2988c00640 1 -- 172.21.15.31:0/3298755320 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f29800bb430 con 0x7f298c0c44f0 2024-09-11T00:35:43.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.799+0000 7f298be00640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f296c077f70 0x7f296c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:43.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.800+0000 7f298be00640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f296c077f70 0x7f296c07a430 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f298c06a670 tx=0x7f2978009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:43.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:43.803+0000 7f2988c00640 1 -- 172.21.15.31:0/3298755320 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2980083a00 con 0x7f298c0c44f0 2024-09-11T00:35:44.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.086+0000 7f2993400640 1 -- 172.21.15.31:0/3298755320 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f298c0c8ff0 con 0x7f298c0c44f0 2024-09-11T00:35:44.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.086+0000 7f2988c00640 1 -- 172.21.15.31:0/3298755320 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6535 (secure 0 0 0) 0x7f298008c120 con 0x7f298c0c44f0 2024-09-11T00:35:44.088 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:44.089 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"btime":"2024-09-11T00:35:14:607653+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5374,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2024-09-11T00:35:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 -- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f296c077f70 msgr2=0x7f296c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f296c077f70 0x7f296c07a430 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f298c06a670 tx=0x7f2978009040 comp rx=0 tx=0).stop 2024-09-11T00:35:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 -- 172.21.15.31:0/3298755320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f298c0c44f0 msgr2=0x7f298c1f4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f298c0c44f0 0x7f298c1f4280 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2980000c00 tx=0x7f2980003910 comp rx=0 tx=0).stop 2024-09-11T00:35:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 -- 172.21.15.31:0/3298755320 shutdown_connections 2024-09-11T00:35:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f296c077f70 0x7f296c07a430 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f298c157c40 0x7f298c0ad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f298c0c4e10 0x7f298c0aaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 --2- 172.21.15.31:0/3298755320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f298c0c44f0 0x7f298c1f4280 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:44.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 -- 172.21.15.31:0/3298755320 >> 172.21.15.31:0/3298755320 conn(0x7f298c0ab8a0 msgr2=0x7f298c152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 -- 172.21.15.31:0/3298755320 shutdown_connections 2024-09-11T00:35:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:44.089+0000 7f2993400640 1 -- 172.21.15.31:0/3298755320 wait complete. 2024-09-11T00:35:44.109 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:35:46.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:46.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.282+0000 7fec2fe00640 1 -- 172.21.15.31:0/3856378363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec300a6e20 msgr2=0x7fec301510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:46.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.282+0000 7fec2fe00640 1 --2- 172.21.15.31:0/3856378363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec300a6e20 0x7fec301510c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fec18009830 tx=0x7fec1802a4a0 comp rx=0 tx=0).stop 2024-09-11T00:35:46.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.282+0000 7fec2fe00640 1 -- 172.21.15.31:0/3856378363 shutdown_connections 2024-09-11T00:35:46.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.282+0000 7fec2fe00640 1 --2- 172.21.15.31:0/3856378363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec300a6e20 0x7fec301510c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:46.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.282+0000 7fec2fe00640 1 -- 172.21.15.31:0/3856378363 >> 172.21.15.31:0/3856378363 conn(0x7fec3006caf0 msgr2=0x7fec3006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:46.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.283+0000 7fec2fe00640 1 -- 172.21.15.31:0/3856378363 shutdown_connections 2024-09-11T00:35:46.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.283+0000 7fec2fe00640 1 -- 172.21.15.31:0/3856378363 wait complete. 2024-09-11T00:35:46.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.284+0000 7fec2fe00640 1 Processor -- start 2024-09-11T00:35:46.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.284+0000 7fec2fe00640 1 -- start start 2024-09-11T00:35:46.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.285+0000 7fec2fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec300a6e20 0x7fec3014c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:46.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.285+0000 7fec2fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec300ac2c0 0x7fec30146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:46.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.285+0000 7fec2fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec30151600 0x7fec30146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:46.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.286+0000 7fec2fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec300c68c0 con 0x7fec300ac2c0 2024-09-11T00:35:46.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.286+0000 7fec2fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fec300c5d20 con 0x7fec300a6e20 2024-09-11T00:35:46.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.286+0000 7fec2fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fec300c6a40 con 0x7fec30151600 2024-09-11T00:35:46.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.286+0000 7fec27e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec300ac2c0 0x7fec30146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:46.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.286+0000 7fec2ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec300a6e20 0x7fec3014c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:46.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.286+0000 7fec27e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec300ac2c0 0x7fec30146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34354/0 (socket says 172.21.15.31:34354) 2024-09-11T00:35:46.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.286+0000 7fec27e00640 1 -- 172.21.15.31:0/1075133124 learned_addr learned my addr 172.21.15.31:0/1075133124 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:46.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.286+0000 7fec2f400640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec30151600 0x7fec30146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:46.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.287+0000 7fec27e00640 1 -- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec30151600 msgr2=0x7fec30146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:46.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.287+0000 7fec27e00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec30151600 0x7fec30146880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:46.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.287+0000 7fec27e00640 1 -- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec300a6e20 msgr2=0x7fec3014c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:46.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.287+0000 7fec27e00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec300a6e20 0x7fec3014c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:46.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.287+0000 7fec27e00640 1 -- 172.21.15.31:0/1075133124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec30149a60 con 0x7fec300ac2c0 2024-09-11T00:35:46.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.287+0000 7fec27e00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec300ac2c0 0x7fec30146340 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fec18000c00 tx=0x7fec18032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:46.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.287+0000 7fec26a00640 1 -- 172.21.15.31:0/1075133124 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec18040070 con 0x7fec300ac2c0 2024-09-11T00:35:46.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.287+0000 7fec26a00640 1 -- 172.21.15.31:0/1075133124 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec18003af0 con 0x7fec300ac2c0 2024-09-11T00:35:46.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.287+0000 7fec26a00640 1 -- 172.21.15.31:0/1075133124 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec18045460 con 0x7fec300ac2c0 2024-09-11T00:35:46.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.288+0000 7fec2fe00640 1 -- 172.21.15.31:0/1075133124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec30149280 con 0x7fec300ac2c0 2024-09-11T00:35:46.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.288+0000 7fec2fe00640 1 -- 172.21.15.31:0/1075133124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec30148d70 con 0x7fec300ac2c0 2024-09-11T00:35:46.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.289+0000 7fec2fe00640 1 -- 172.21.15.31:0/1075133124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec300c6e30 con 0x7fec300ac2c0 2024-09-11T00:35:46.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.291+0000 7fec26a00640 1 -- 172.21.15.31:0/1075133124 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fec18045600 con 0x7fec300ac2c0 2024-09-11T00:35:46.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.292+0000 7fec26a00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec0c077f70 0x7fec0c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:46.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.293+0000 7fec26a00640 1 -- 172.21.15.31:0/1075133124 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7fec180bbc30 con 0x7fec300ac2c0 2024-09-11T00:35:46.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.293+0000 7fec2ea00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec0c077f70 0x7fec0c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:46.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.294+0000 7fec2ea00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec0c077f70 0x7fec0c07a430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec200017c0 tx=0x7fec20009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:46.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.297+0000 7fec26a00640 1 -- 172.21.15.31:0/1075133124 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fec18083a90 con 0x7fec300ac2c0 2024-09-11T00:35:46.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.579+0000 7fec2fe00640 1 -- 172.21.15.31:0/1075133124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec300c6450 con 0x7fec300ac2c0 2024-09-11T00:35:46.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.580+0000 7fec26a00640 1 -- 172.21.15.31:0/1075133124 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6535 (secure 0 0 0) 0x7fec180884c0 con 0x7fec300ac2c0 2024-09-11T00:35:46.582 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:46.582 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"btime":"2024-09-11T00:35:14:607653+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5374,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:46.582 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2024-09-11T00:35:46.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.582+0000 7fec2fe00640 1 -- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec0c077f70 msgr2=0x7fec0c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:46.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.582+0000 7fec2fe00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec0c077f70 0x7fec0c07a430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec200017c0 tx=0x7fec20009040 comp rx=0 tx=0).stop 2024-09-11T00:35:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.582+0000 7fec2fe00640 1 -- 172.21.15.31:0/1075133124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec300ac2c0 msgr2=0x7fec30146340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.582+0000 7fec2fe00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec300ac2c0 0x7fec30146340 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fec18000c00 tx=0x7fec18032d50 comp rx=0 tx=0).stop 2024-09-11T00:35:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.582+0000 7fec2fe00640 1 -- 172.21.15.31:0/1075133124 shutdown_connections 2024-09-11T00:35:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.582+0000 7fec2fe00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec0c077f70 0x7fec0c07a430 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.582+0000 7fec2fe00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec30151600 0x7fec30146880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.582+0000 7fec2fe00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec300ac2c0 0x7fec30146340 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.582+0000 7fec2fe00640 1 --2- 172.21.15.31:0/1075133124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec300a6e20 0x7fec3014c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.582+0000 7fec2fe00640 1 -- 172.21.15.31:0/1075133124 >> 172.21.15.31:0/1075133124 conn(0x7fec3006caf0 msgr2=0x7fec3014eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.583+0000 7fec2fe00640 1 -- 172.21.15.31:0/1075133124 shutdown_connections 2024-09-11T00:35:46.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:46.583+0000 7fec2fe00640 1 -- 172.21.15.31:0/1075133124 wait complete. 2024-09-11T00:35:46.604 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:35:48.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:48.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.776+0000 7fce3a000640 1 -- 172.21.15.31:0/1890323075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce34074f50 msgr2=0x7fce3406db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.776+0000 7fce3a000640 1 --2- 172.21.15.31:0/1890323075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce34074f50 0x7fce3406db00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fce340adaa0 tx=0x7fce2402a490 comp rx=0 tx=0).stop 2024-09-11T00:35:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.776+0000 7fce3a000640 1 -- 172.21.15.31:0/1890323075 shutdown_connections 2024-09-11T00:35:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.776+0000 7fce3a000640 1 --2- 172.21.15.31:0/1890323075 >> v2:172.21.15.31:3300/0 conn(0x7fce340abde0 0x7fce34156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.776+0000 7fce3a000640 1 --2- 172.21.15.31:0/1890323075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce34074f50 0x7fce3406db00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.776+0000 7fce3a000640 1 --2- 172.21.15.31:0/1890323075 >> v2:172.21.15.204:3300/0 conn(0x7fce34074630 0x7fce34074a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.776+0000 7fce3a000640 1 -- 172.21.15.31:0/1890323075 >> 172.21.15.31:0/1890323075 conn(0x7fce340c8320 msgr2=0x7fce340c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.776+0000 7fce3a000640 1 -- 172.21.15.31:0/1890323075 shutdown_connections 2024-09-11T00:35:48.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.776+0000 7fce3a000640 1 -- 172.21.15.31:0/1890323075 wait complete. 2024-09-11T00:35:48.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.778+0000 7fce3a000640 1 Processor -- start 2024-09-11T00:35:48.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.778+0000 7fce3a000640 1 -- start start 2024-09-11T00:35:48.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.779+0000 7fce3a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce34074630 0x7fce341dfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.779+0000 7fce3a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce34074f50 0x7fce34146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.779+0000 7fce3a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce340abde0 0x7fce3414b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.779+0000 7fce3a000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fce34153b70 con 0x7fce34074630 2024-09-11T00:35:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.779+0000 7fce3a000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fce340c2d20 con 0x7fce340abde0 2024-09-11T00:35:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.779+0000 7fce3a000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fce3414c9f0 con 0x7fce34074f50 2024-09-11T00:35:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.779+0000 7fce33400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce34074630 0x7fce341dfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.779+0000 7fce33400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce34074630 0x7fce341dfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51154/0 (socket says 172.21.15.31:51154) 2024-09-11T00:35:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.780+0000 7fce32a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce34074f50 0x7fce34146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.780+0000 7fce33400640 1 -- 172.21.15.31:0/1346193762 learned_addr learned my addr 172.21.15.31:0/1346193762 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.780+0000 7fce33e00640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce340abde0 0x7fce3414b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.780+0000 7fce33400640 1 -- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce34074f50 msgr2=0x7fce34146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.780+0000 7fce33400640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce34074f50 0x7fce34146bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.780+0000 7fce33400640 1 -- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce340abde0 msgr2=0x7fce3414b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.780+0000 7fce33400640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce340abde0 0x7fce3414b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.780+0000 7fce33400640 1 -- 172.21.15.31:0/1346193762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3414a1f0 con 0x7fce34074630 2024-09-11T00:35:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.780+0000 7fce33400640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce34074630 0x7fce341dfb20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fce2c00eb20 tx=0x7fce2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.781+0000 7fce2b400640 1 -- 172.21.15.31:0/1346193762 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2c004b20 con 0x7fce34074630 2024-09-11T00:35:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.781+0000 7fce2b400640 1 -- 172.21.15.31:0/1346193762 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce2c004cc0 con 0x7fce34074630 2024-09-11T00:35:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.781+0000 7fce2b400640 1 -- 172.21.15.31:0/1346193762 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2c005280 con 0x7fce34074630 2024-09-11T00:35:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.781+0000 7fce3a000640 1 -- 172.21.15.31:0/1346193762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce340c3440 con 0x7fce34074630 2024-09-11T00:35:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.781+0000 7fce3a000640 1 -- 172.21.15.31:0/1346193762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce341e0060 con 0x7fce34074630 2024-09-11T00:35:48.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.782+0000 7fce3a000640 1 -- 172.21.15.31:0/1346193762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce34074a10 con 0x7fce34074630 2024-09-11T00:35:48.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.783+0000 7fce2b400640 1 -- 172.21.15.31:0/1346193762 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fce2c020030 con 0x7fce34074630 2024-09-11T00:35:48.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.785+0000 7fce2b400640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fce14077fa0 0x7fce1407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:48.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.785+0000 7fce2b400640 1 -- 172.21.15.31:0/1346193762 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7fce2c09ae40 con 0x7fce34074630 2024-09-11T00:35:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.785+0000 7fce32a00640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fce14077fa0 0x7fce1407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:48.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.786+0000 7fce32a00640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fce14077fa0 0x7fce1407a460 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fce24034000 tx=0x7fce24004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:48.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:48.789+0000 7fce2b400640 1 -- 172.21.15.31:0/1346193762 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fce2c062c90 con 0x7fce34074630 2024-09-11T00:35:49.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.089+0000 7fce3a000640 1 -- 172.21.15.31:0/1346193762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce34153cf0 con 0x7fce34074630 2024-09-11T00:35:49.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.089+0000 7fce2b400640 1 -- 172.21.15.31:0/1346193762 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6535 (secure 0 0 0) 0x7fce2c0676c0 con 0x7fce34074630 2024-09-11T00:35:49.092 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:49.092 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"btime":"2024-09-11T00:35:14:607653+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5374,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:49.093 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2024-09-11T00:35:49.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 -- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fce14077fa0 msgr2=0x7fce1407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:49.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fce14077fa0 0x7fce1407a460 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fce24034000 tx=0x7fce24004730 comp rx=0 tx=0).stop 2024-09-11T00:35:49.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 -- 172.21.15.31:0/1346193762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce34074630 msgr2=0x7fce341dfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:49.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce34074630 0x7fce341dfb20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fce2c00eb20 tx=0x7fce2c00c2d0 comp rx=0 tx=0).stop 2024-09-11T00:35:49.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 -- 172.21.15.31:0/1346193762 shutdown_connections 2024-09-11T00:35:49.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fce14077fa0 0x7fce1407a460 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:49.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce340abde0 0x7fce3414b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:49.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fce34074f50 0x7fce34146bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:49.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 --2- 172.21.15.31:0/1346193762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce34074630 0x7fce341dfb20 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:49.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 -- 172.21.15.31:0/1346193762 >> 172.21.15.31:0/1346193762 conn(0x7fce340c8320 msgr2=0x7fce34148de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:49.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 -- 172.21.15.31:0/1346193762 shutdown_connections 2024-09-11T00:35:49.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:49.092+0000 7fce3a000640 1 -- 172.21.15.31:0/1346193762 wait complete. 2024-09-11T00:35:49.113 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:35:51.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.300+0000 7f7879e00640 1 -- 172.21.15.31:0/523338910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78740acbe0 msgr2=0x7f7874151600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.300+0000 7f7879e00640 1 --2- 172.21.15.31:0/523338910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78740acbe0 0x7f7874151600 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7874071a50 tx=0x7f786c02a8f0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.301+0000 7f7879e00640 1 -- 172.21.15.31:0/523338910 shutdown_connections 2024-09-11T00:35:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.301+0000 7f7879e00640 1 --2- 172.21.15.31:0/523338910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78740acbe0 0x7f7874151600 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.301+0000 7f7879e00640 1 --2- 172.21.15.31:0/523338910 >> v2:172.21.15.204:3300/0 conn(0x7f78740a6e20 0x7f78741510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.301+0000 7f7879e00640 1 --2- 172.21.15.31:0/523338910 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78740ac2c0 0x7f78740ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.301+0000 7f7879e00640 1 -- 172.21.15.31:0/523338910 >> 172.21.15.31:0/523338910 conn(0x7f787406caf0 msgr2=0x7f787406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.301+0000 7f7879e00640 1 -- 172.21.15.31:0/523338910 shutdown_connections 2024-09-11T00:35:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.301+0000 7f7879e00640 1 -- 172.21.15.31:0/523338910 wait complete. 2024-09-11T00:35:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.302+0000 7f7879e00640 1 Processor -- start 2024-09-11T00:35:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.303+0000 7f7879e00640 1 -- start start 2024-09-11T00:35:51.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.303+0000 7f7879e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78740a6e20 0x7f7874155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:51.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.303+0000 7f7879e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78740ac2c0 0x7f7874073a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:51.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.303+0000 7f7879e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78740acbe0 0x7f7874072d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:51.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.303+0000 7f7879e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f78740c6330 con 0x7f78740a6e20 2024-09-11T00:35:51.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.303+0000 7f7879e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78740c61b0 con 0x7f78740acbe0 2024-09-11T00:35:51.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.303+0000 7f7879e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f78740c64b0 con 0x7f78740ac2c0 2024-09-11T00:35:51.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f7878a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78740a6e20 0x7f7874155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:51.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f7878a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78740a6e20 0x7f7874155120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51172/0 (socket says 172.21.15.31:51172) 2024-09-11T00:35:51.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f7878a00640 1 -- 172.21.15.31:0/2569543334 learned_addr learned my addr 172.21.15.31:0/2569543334 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:51.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f786be00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78740ac2c0 0x7f7874073a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:51.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f7879400640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78740acbe0 0x7f7874072d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:51.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f7878a00640 1 -- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78740ac2c0 msgr2=0x7f7874073a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:51.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f7878a00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78740ac2c0 0x7f7874073a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f7878a00640 1 -- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78740acbe0 msgr2=0x7f7874072d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:51.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f7878a00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78740acbe0 0x7f7874072d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f7878a00640 1 -- 172.21.15.31:0/2569543334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7874072520 con 0x7f78740a6e20 2024-09-11T00:35:51.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.304+0000 7f7878a00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78740a6e20 0x7f7874155120 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f786400e9f0 tx=0x7f786400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:51.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.305+0000 7f7871600640 1 -- 172.21.15.31:0/2569543334 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786400cd80 con 0x7f78740a6e20 2024-09-11T00:35:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.305+0000 7f7871600640 1 -- 172.21.15.31:0/2569543334 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7864005020 con 0x7f78740a6e20 2024-09-11T00:35:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.305+0000 7f7879e00640 1 -- 172.21.15.31:0/2569543334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f787414ca40 con 0x7f78740a6e20 2024-09-11T00:35:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.305+0000 7f7871600640 1 -- 172.21.15.31:0/2569543334 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864005450 con 0x7f78740a6e20 2024-09-11T00:35:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.305+0000 7f7879e00640 1 -- 172.21.15.31:0/2569543334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78741e10f0 con 0x7f78740a6e20 2024-09-11T00:35:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.306+0000 7f7879e00640 1 -- 172.21.15.31:0/2569543334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78740ac6a0 con 0x7f78740a6e20 2024-09-11T00:35:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.307+0000 7f7871600640 1 -- 172.21.15.31:0/2569543334 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f78640047a0 con 0x7f78740a6e20 2024-09-11T00:35:51.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.308+0000 7f7871600640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7850077fc0 0x7f785007a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:51.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.309+0000 7f7871600640 1 -- 172.21.15.31:0/2569543334 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f786409ac60 con 0x7f78740a6e20 2024-09-11T00:35:51.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.309+0000 7f786be00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7850077fc0 0x7f785007a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:51.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.310+0000 7f786be00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7850077fc0 0x7f785007a480 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f785c005d80 tx=0x7f785c005c50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:51.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.314+0000 7f7871600640 1 -- 172.21.15.31:0/2569543334 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7864063340 con 0x7f78740a6e20 2024-09-11T00:35:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.609+0000 7f7879e00640 1 -- 172.21.15.31:0/2569543334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f787414e610 con 0x7f78740a6e20 2024-09-11T00:35:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.610+0000 7f7871600640 1 -- 172.21.15.31:0/2569543334 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6535 (secure 0 0 0) 0x7f7864062c40 con 0x7f78740a6e20 2024-09-11T00:35:51.612 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:51.612 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"btime":"2024-09-11T00:35:14:607653+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5374,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:51.613 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2024-09-11T00:35:51.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 -- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7850077fc0 msgr2=0x7f785007a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:51.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7850077fc0 0x7f785007a480 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f785c005d80 tx=0x7f785c005c50 comp rx=0 tx=0).stop 2024-09-11T00:35:51.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 -- 172.21.15.31:0/2569543334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78740a6e20 msgr2=0x7f7874155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:51.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78740a6e20 0x7f7874155120 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f786400e9f0 tx=0x7f786400edf0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 -- 172.21.15.31:0/2569543334 shutdown_connections 2024-09-11T00:35:51.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7850077fc0 0x7f785007a480 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78740acbe0 0x7f7874072d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78740ac2c0 0x7f7874073a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 --2- 172.21.15.31:0/2569543334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78740a6e20 0x7f7874155120 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:51.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 -- 172.21.15.31:0/2569543334 >> 172.21.15.31:0/2569543334 conn(0x7f787406caf0 msgr2=0x7f787414d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:51.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 -- 172.21.15.31:0/2569543334 shutdown_connections 2024-09-11T00:35:51.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:51.613+0000 7f7879e00640 1 -- 172.21.15.31:0/2569543334 wait complete. 2024-09-11T00:35:51.633 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:35:53.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:53.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.816+0000 7f18f6a00640 1 -- 172.21.15.31:0/4179738461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f00ada20 msgr2=0x7f18f006e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:53.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.816+0000 7f18f6a00640 1 --2- 172.21.15.31:0/4179738461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f00ada20 0x7f18f006e140 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f18f008d7a0 tx=0x7f18dc02a4d0 comp rx=0 tx=0).stop 2024-09-11T00:35:53.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.816+0000 7f18f6a00640 1 -- 172.21.15.31:0/4179738461 shutdown_connections 2024-09-11T00:35:53.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.816+0000 7f18f6a00640 1 --2- 172.21.15.31:0/4179738461 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18f0156960 0x7f18f006e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.816+0000 7f18f6a00640 1 --2- 172.21.15.31:0/4179738461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f00ada20 0x7f18f006e140 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.816+0000 7f18f6a00640 1 --2- 172.21.15.31:0/4179738461 >> v2:172.21.15.204:3300/0 conn(0x7f18f0150020 0x7f18f00ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.816+0000 7f18f6a00640 1 -- 172.21.15.31:0/4179738461 >> 172.21.15.31:0/4179738461 conn(0x7f18f0068f10 msgr2=0x7f18f00cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.816+0000 7f18f6a00640 1 -- 172.21.15.31:0/4179738461 shutdown_connections 2024-09-11T00:35:53.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.816+0000 7f18f6a00640 1 -- 172.21.15.31:0/4179738461 wait complete. 2024-09-11T00:35:53.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.818+0000 7f18f6a00640 1 Processor -- start 2024-09-11T00:35:53.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.818+0000 7f18f6a00640 1 -- start start 2024-09-11T00:35:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.819+0000 7f18f6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18f00ada20 0x7f18f014ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.819+0000 7f18f6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18f0150020 0x7f18f014e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.819+0000 7f18f6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0156960 0x7f18f00c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.819+0000 7f18f6a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f18f0157550 con 0x7f18f0156960 2024-09-11T00:35:53.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.819+0000 7f18f6a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18f01573d0 con 0x7f18f0150020 2024-09-11T00:35:53.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.819+0000 7f18f6a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f18f00cc220 con 0x7f18f00ada20 2024-09-11T00:35:53.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.819+0000 7f18f6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0156960 0x7f18f00c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:53.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.819+0000 7f18f6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0156960 0x7f18f00c2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51180/0 (socket says 172.21.15.31:51180) 2024-09-11T00:35:53.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.819+0000 7f18f6000640 1 -- 172.21.15.31:0/484558369 learned_addr learned my addr 172.21.15.31:0/484558369 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:53.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.820+0000 7f18f4c00640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18f0150020 0x7f18f014e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:53.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.820+0000 7f18f5600640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18f00ada20 0x7f18f014ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:53.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.820+0000 7f18f6000640 1 -- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18f00ada20 msgr2=0x7f18f014ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:53.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.820+0000 7f18f6000640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18f00ada20 0x7f18f014ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:53.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.820+0000 7f18f6000640 1 -- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18f0150020 msgr2=0x7f18f014e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:53.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.820+0000 7f18f6000640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18f0150020 0x7f18f014e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:53.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.820+0000 7f18f6000640 1 -- 172.21.15.31:0/484558369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f014ea70 con 0x7f18f0156960 2024-09-11T00:35:53.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.820+0000 7f18f6000640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0156960 0x7f18f00c2aa0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f18e800eaa0 tx=0x7f18e800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:53.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.821+0000 7f18ed600640 1 -- 172.21.15.31:0/484558369 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e800cdb0 con 0x7f18f0156960 2024-09-11T00:35:53.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.821+0000 7f18ed600640 1 -- 172.21.15.31:0/484558369 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18e8004c10 con 0x7f18f0156960 2024-09-11T00:35:53.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.821+0000 7f18f6a00640 1 -- 172.21.15.31:0/484558369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f01efcd0 con 0x7f18f0156960 2024-09-11T00:35:53.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.821+0000 7f18ed600640 1 -- 172.21.15.31:0/484558369 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e8005020 con 0x7f18f0156960 2024-09-11T00:35:53.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.821+0000 7f18f6a00640 1 -- 172.21.15.31:0/484558369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f00c42f0 con 0x7f18f0156960 2024-09-11T00:35:53.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.822+0000 7f18f6a00640 1 -- 172.21.15.31:0/484558369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18f00ab280 con 0x7f18f0156960 2024-09-11T00:35:53.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.824+0000 7f18ed600640 1 -- 172.21.15.31:0/484558369 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f18e800f040 con 0x7f18f0156960 2024-09-11T00:35:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.825+0000 7f18ed600640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18d0077f70 0x7f18d007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.825+0000 7f18ed600640 1 -- 172.21.15.31:0/484558369 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f18e809a980 con 0x7f18f0156960 2024-09-11T00:35:53.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.826+0000 7f18f5600640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18d0077f70 0x7f18d007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:53.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.827+0000 7f18f5600640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18d0077f70 0x7f18d007a430 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f18e4005fd0 tx=0x7f18e400a4e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:53.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:53.831+0000 7f18ed600640 1 -- 172.21.15.31:0/484558369 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f18f00ab280 con 0x7f18f0156960 2024-09-11T00:35:54.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.075+0000 7f18f6a00640 1 -- 172.21.15.31:0/484558369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18f006ef00 con 0x7f18f0156960 2024-09-11T00:35:54.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.076+0000 7f18ed600640 1 -- 172.21.15.31:0/484558369 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6535 (secure 0 0 0) 0x7f18f006ef00 con 0x7f18f0156960 2024-09-11T00:35:54.078 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:54.078 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"btime":"2024-09-11T00:35:14:607653+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5374,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:34:12.311591+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4898},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4898":{"gid":4898,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6837/645869523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":645869523},{"type":"v1","addr":"172.21.15.204:6837","nonce":645869523}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4898,"qdb_cluster":[4898]},"id":2}]} 2024-09-11T00:35:54.078 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2024-09-11T00:35:54.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 -- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18d0077f70 msgr2=0x7f18d007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:54.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18d0077f70 0x7f18d007a430 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f18e4005fd0 tx=0x7f18e400a4e0 comp rx=0 tx=0).stop 2024-09-11T00:35:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 -- 172.21.15.31:0/484558369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0156960 msgr2=0x7f18f00c2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0156960 0x7f18f00c2aa0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f18e800eaa0 tx=0x7f18e800ef60 comp rx=0 tx=0).stop 2024-09-11T00:35:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 -- 172.21.15.31:0/484558369 shutdown_connections 2024-09-11T00:35:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18d0077f70 0x7f18d007a430 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0156960 0x7f18f00c2aa0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18f0150020 0x7f18f014e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 --2- 172.21.15.31:0/484558369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18f00ada20 0x7f18f014ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 -- 172.21.15.31:0/484558369 >> 172.21.15.31:0/484558369 conn(0x7f18f0068f10 msgr2=0x7f18f00acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 -- 172.21.15.31:0/484558369 shutdown_connections 2024-09-11T00:35:54.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:54.078+0000 7f18f6a00640 1 -- 172.21.15.31:0/484558369 wait complete. 2024-09-11T00:35:54.100 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:35:56.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:56.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.275+0000 7fb9ef200640 1 -- 172.21.15.31:0/3200392555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9cc004800 msgr2=0x7fb9cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.275+0000 7fb9ef200640 1 --2- 172.21.15.31:0/3200392555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9cc004800 0x7fb9cc004c80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8154fd0 tx=0x7fb9cc005830 comp rx=0 tx=0).stop 2024-09-11T00:35:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.275+0000 7fb9ed200640 1 -- 172.21.15.31:0/3200392555 reap_dead start 2024-09-11T00:35:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.275+0000 7fb9ef200640 1 -- 172.21.15.31:0/3200392555 shutdown_connections 2024-09-11T00:35:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.275+0000 7fb9ef200640 1 -- 172.21.15.31:0/3200392555 >> 172.21.15.31:0/3200392555 conn(0x7fb9e80a9750 msgr2=0x7fb9e80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.276+0000 7fb9ef200640 1 -- 172.21.15.31:0/3200392555 shutdown_connections 2024-09-11T00:35:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.276+0000 7fb9ef200640 1 -- 172.21.15.31:0/3200392555 wait complete. 2024-09-11T00:35:56.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.278+0000 7fb9ef200640 1 Processor -- start 2024-09-11T00:35:56.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.278+0000 7fb9ef200640 1 -- start start 2024-09-11T00:35:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9ef200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9cc004800 0x7fb9e814aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9ef200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e8153870 0x7fb9e81e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9ef200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9e81def70 0x7fb9e81df3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9ef200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e8074200 con 0x7fb9e8153870 2024-09-11T00:35:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9ef200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e8074080 con 0x7fb9e81def70 2024-09-11T00:35:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9ef200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e8072860 con 0x7fb9cc004800 2024-09-11T00:35:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9e7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e8153870 0x7fb9e81e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9e7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e8153870 0x7fb9e81e1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51192/0 (socket says 172.21.15.31:51192) 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9e7400640 1 -- 172.21.15.31:0/2968087284 learned_addr learned my addr 172.21.15.31:0/2968087284 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9e7e00640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9cc004800 0x7fb9e814aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9ed200640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9e81def70 0x7fb9e81df3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.279+0000 7fb9e7400640 1 -- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9cc004800 msgr2=0x7fb9e814aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.280+0000 7fb9e7400640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9cc004800 0x7fb9e814aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.280+0000 7fb9e7400640 1 -- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9e81def70 msgr2=0x7fb9e81df3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.280+0000 7fb9e7400640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9e81def70 0x7fb9e81df3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.280+0000 7fb9e7400640 1 -- 172.21.15.31:0/2968087284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e814f3e0 con 0x7fb9e8153870 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.280+0000 7fb9e7400640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e8153870 0x7fb9e81e1ce0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009a70 tx=0x7fb9d4033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.280+0000 7fb9dfe00640 1 -- 172.21.15.31:0/2968087284 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4038040 con 0x7fb9e8153870 2024-09-11T00:35:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.280+0000 7fb9dfe00640 1 -- 172.21.15.31:0/2968087284 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9d4043070 con 0x7fb9e8153870 2024-09-11T00:35:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.280+0000 7fb9ef200640 1 -- 172.21.15.31:0/2968087284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e81527b0 con 0x7fb9e8153870 2024-09-11T00:35:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.280+0000 7fb9dfe00640 1 -- 172.21.15.31:0/2968087284 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4033cd0 con 0x7fb9e8153870 2024-09-11T00:35:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.280+0000 7fb9ef200640 1 -- 172.21.15.31:0/2968087284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e81dd740 con 0x7fb9e8153870 2024-09-11T00:35:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.282+0000 7fb9ef200640 1 -- 172.21.15.31:0/2968087284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e8077240 con 0x7fb9e8153870 2024-09-11T00:35:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.283+0000 7fb9dfe00640 1 -- 172.21.15.31:0/2968087284 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb9d403b030 con 0x7fb9e8153870 2024-09-11T00:35:56.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.284+0000 7fb9dfe00640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9c8077fa0 0x7fb9c807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:56.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.284+0000 7fb9dfe00640 1 -- 172.21.15.31:0/2968087284 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7fb9d40ba7a0 con 0x7fb9e8153870 2024-09-11T00:35:56.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.284+0000 7fb9e7e00640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9c8077fa0 0x7fb9c807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:56.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.285+0000 7fb9e7e00640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9c8077fa0 0x7fb9c807a460 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8001eb0 tx=0x7fb9d8001e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:56.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.289+0000 7fb9dfe00640 1 -- 172.21.15.31:0/2968087284 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb9e8077240 con 0x7fb9e8153870 2024-09-11T00:35:56.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.581+0000 7fb9ef200640 1 -- 172.21.15.31:0/2968087284 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9e8154000 con 0x7fb9e8153870 2024-09-11T00:35:56.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.583+0000 7fb9dfe00640 1 -- 172.21.15.31:0/2968087284 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5703 (secure 0 0 0) 0x7fb9d404c020 con 0x7fb9e8153870 2024-09-11T00:35:56.585 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:56.585 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":49,"btime":"2024-09-11T00:35:56:327973+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:55.331551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:35:56.585 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 49 2024-09-11T00:35:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 -- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9c8077fa0 msgr2=0x7fb9c807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9c8077fa0 0x7fb9c807a460 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8001eb0 tx=0x7fb9d8001e00 comp rx=0 tx=0).stop 2024-09-11T00:35:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 -- 172.21.15.31:0/2968087284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e8153870 msgr2=0x7fb9e81e1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e8153870 0x7fb9e81e1ce0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009a70 tx=0x7fb9d4033670 comp rx=0 tx=0).stop 2024-09-11T00:35:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 -- 172.21.15.31:0/2968087284 shutdown_connections 2024-09-11T00:35:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9c8077fa0 0x7fb9c807a460 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9e81def70 0x7fb9e81df3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9e8153870 0x7fb9e81e1ce0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 --2- 172.21.15.31:0/2968087284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9cc004800 0x7fb9e814aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 -- 172.21.15.31:0/2968087284 >> 172.21.15.31:0/2968087284 conn(0x7fb9e80a9750 msgr2=0x7fb9e80c0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 -- 172.21.15.31:0/2968087284 shutdown_connections 2024-09-11T00:35:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.586+0000 7fb9ef200640 1 -- 172.21.15.31:0/2968087284 wait complete. 2024-09-11T00:35:56.607 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-11T00:35:56.607 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:35:56.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:56.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.792+0000 7f15c2e00640 1 -- 172.21.15.31:0/2585244945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15bc06f4c0 msgr2=0x7f15bc0c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.792+0000 7f15c2e00640 1 --2- 172.21.15.31:0/2585244945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15bc06f4c0 0x7f15bc0c8280 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f15a8005fd0 tx=0x7f15a802a5c0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.792+0000 7f15c2e00640 1 -- 172.21.15.31:0/2585244945 shutdown_connections 2024-09-11T00:35:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.792+0000 7f15c2e00640 1 --2- 172.21.15.31:0/2585244945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f15bc159890 0x7f15bc159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.792+0000 7f15c2e00640 1 --2- 172.21.15.31:0/2585244945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15bc06f4c0 0x7f15bc0c8280 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.792+0000 7f15c2e00640 1 -- 172.21.15.31:0/2585244945 >> 172.21.15.31:0/2585244945 conn(0x7f15bc0c52f0 msgr2=0x7f15bc0c56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.792+0000 7f15c2e00640 1 -- 172.21.15.31:0/2585244945 shutdown_connections 2024-09-11T00:35:56.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.792+0000 7f15c2e00640 1 -- 172.21.15.31:0/2585244945 wait complete. 2024-09-11T00:35:56.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.794+0000 7f15c2e00640 1 Processor -- start 2024-09-11T00:35:56.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.794+0000 7f15c2e00640 1 -- start start 2024-09-11T00:35:56.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.795+0000 7f15c2e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15bc06f4c0 0x7f15bc1e1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:56.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.795+0000 7f15c2e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f15bc1594b0 0x7f15bc0c3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:56.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.795+0000 7f15c2e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f15bc159890 0x7f15bc0c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:56.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.795+0000 7f15c2e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc0c24b0 con 0x7f15bc06f4c0 2024-09-11T00:35:56.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.795+0000 7f15c2e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc0c2330 con 0x7f15bc159890 2024-09-11T00:35:56.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.795+0000 7f15c2e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f15bc0c4c10 con 0x7f15bc1594b0 2024-09-11T00:35:56.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15bc06f4c0 0x7f15bc1e1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:56.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15bc06f4c0 0x7f15bc1e1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51214/0 (socket says 172.21.15.31:51214) 2024-09-11T00:35:56.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1a00640 1 -- 172.21.15.31:0/837339956 learned_addr learned my addr 172.21.15.31:0/837339956 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:56.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1000640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f15bc1594b0 0x7f15bc0c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:56.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c2400640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f15bc159890 0x7f15bc0c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:56.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1a00640 1 -- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f15bc1594b0 msgr2=0x7f15bc0c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:56.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1a00640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f15bc1594b0 0x7f15bc0c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1a00640 1 -- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f15bc159890 msgr2=0x7f15bc0c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:56.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1a00640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f15bc159890 0x7f15bc0c37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:56.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1a00640 1 -- 172.21.15.31:0/837339956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15bc1581c0 con 0x7f15bc06f4c0 2024-09-11T00:35:56.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1000640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f15bc1594b0 0x7f15bc0c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:35:56.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.796+0000 7f15c1a00640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15bc06f4c0 0x7f15bc1e1040 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f15b000c530 tx=0x7f15b000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:56.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.797+0000 7f15b9600640 1 -- 172.21.15.31:0/837339956 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b0017070 con 0x7f15bc06f4c0 2024-09-11T00:35:56.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.797+0000 7f15b9600640 1 -- 172.21.15.31:0/837339956 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15b00035f0 con 0x7f15bc06f4c0 2024-09-11T00:35:56.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.797+0000 7f15b9600640 1 -- 172.21.15.31:0/837339956 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b0003a40 con 0x7f15bc06f4c0 2024-09-11T00:35:56.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.797+0000 7f15c2e00640 1 -- 172.21.15.31:0/837339956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15bc0c3d00 con 0x7f15bc06f4c0 2024-09-11T00:35:56.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.797+0000 7f15c2e00640 1 -- 172.21.15.31:0/837339956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15bc1ec480 con 0x7f15bc06f4c0 2024-09-11T00:35:56.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.798+0000 7f15c2e00640 1 -- 172.21.15.31:0/837339956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15bc0c8280 con 0x7f15bc06f4c0 2024-09-11T00:35:56.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.799+0000 7f15b9600640 1 -- 172.21.15.31:0/837339956 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f15b0003be0 con 0x7f15bc06f4c0 2024-09-11T00:35:56.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.800+0000 7f15b9600640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f159c077f70 0x7f159c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:56.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.801+0000 7f15b9600640 1 -- 172.21.15.31:0/837339956 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f15b0099b10 con 0x7f15bc06f4c0 2024-09-11T00:35:56.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.801+0000 7f15c1000640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f159c077f70 0x7f159c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:56.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.802+0000 7f15c1000640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f159c077f70 0x7f159c07a430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f15a8033000 tx=0x7f15a8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:56.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:56.804+0000 7f15b9600640 1 -- 172.21.15.31:0/837339956 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f15b0062080 con 0x7f15bc06f4c0 2024-09-11T00:35:57.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.091+0000 7f15c2e00640 1 -- 172.21.15.31:0/837339956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15bc0c6160 con 0x7f15bc06f4c0 2024-09-11T00:35:57.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.091+0000 7f15b9600640 1 -- 172.21.15.31:0/837339956 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5703 (secure 0 0 0) 0x7f15b0061980 con 0x7f15bc06f4c0 2024-09-11T00:35:57.093 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:57.093 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":49,"btime":"2024-09-11T00:35:56:327973+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:55.331551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:35:57.094 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 49 2024-09-11T00:35:57.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 -- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f159c077f70 msgr2=0x7f159c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:57.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f159c077f70 0x7f159c07a430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f15a8033000 tx=0x7f15a8004730 comp rx=0 tx=0).stop 2024-09-11T00:35:57.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 -- 172.21.15.31:0/837339956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15bc06f4c0 msgr2=0x7f15bc1e1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:57.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15bc06f4c0 0x7f15bc1e1040 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f15b000c530 tx=0x7f15b000c9f0 comp rx=0 tx=0).stop 2024-09-11T00:35:57.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 -- 172.21.15.31:0/837339956 shutdown_connections 2024-09-11T00:35:57.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f159c077f70 0x7f159c07a430 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:57.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f15bc159890 0x7f15bc0c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:57.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f15bc1594b0 0x7f15bc0c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:57.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 --2- 172.21.15.31:0/837339956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15bc06f4c0 0x7f15bc1e1040 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:57.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 -- 172.21.15.31:0/837339956 >> 172.21.15.31:0/837339956 conn(0x7f15bc0c52f0 msgr2=0x7f15bc158780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:57.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 -- 172.21.15.31:0/837339956 shutdown_connections 2024-09-11T00:35:57.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:57.094+0000 7f15c2e00640 1 -- 172.21.15.31:0/837339956 wait complete. 2024-09-11T00:35:57.114 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:35:56:327973+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5302": { "addr": "172.21.15.204:6835/3444890210", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 3444890210, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 3444890210, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5302, "incarnation": 42, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:35:12.604104+0000", "qdb_cluster": [ 5302 ], "qdb_leader": 5302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5302 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5374": { "addr": "172.21.15.31:6835/4011734390", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6833", "nonce": 4011734390, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 4011734390, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5374, "incarnation": 47, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:35:55.331551+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5374 } } } ], "standbys": [ { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:35:59.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:35:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.291+0000 7fad36000640 1 -- 172.21.15.31:0/3493902042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad300cc220 msgr2=0x7fad3006e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.291+0000 7fad36000640 1 --2- 172.21.15.31:0/3493902042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad300cc220 0x7fad3006e680 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fad28005fd0 tx=0x7fad2802a820 comp rx=0 tx=0).stop 2024-09-11T00:35:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.291+0000 7fad36000640 1 -- 172.21.15.31:0/3493902042 shutdown_connections 2024-09-11T00:35:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.291+0000 7fad36000640 1 --2- 172.21.15.31:0/3493902042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad300cc220 0x7fad3006e680 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.291+0000 7fad36000640 1 --2- 172.21.15.31:0/3493902042 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad30156960 0x7fad3006e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.291+0000 7fad36000640 1 -- 172.21.15.31:0/3493902042 >> 172.21.15.31:0/3493902042 conn(0x7fad30068f10 msgr2=0x7fad300cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:59.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.291+0000 7fad36000640 1 -- 172.21.15.31:0/3493902042 shutdown_connections 2024-09-11T00:35:59.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.291+0000 7fad36000640 1 -- 172.21.15.31:0/3493902042 wait complete. 2024-09-11T00:35:59.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.292+0000 7fad36000640 1 Processor -- start 2024-09-11T00:35:59.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.293+0000 7fad36000640 1 -- start start 2024-09-11T00:35:59.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad36000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad300cc220 0x7fad30150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:59.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad36000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad30150020 0x7fad301ef4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:59.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad36000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad30156960 0x7fad301e1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:59.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad36000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fad300c6ca0 con 0x7fad300cc220 2024-09-11T00:35:59.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad36000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fad300c6b20 con 0x7fad30150020 2024-09-11T00:35:59.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad36000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fad301573d0 con 0x7fad30156960 2024-09-11T00:35:59.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad34c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad300cc220 0x7fad30150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:59.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad2fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad30150020 0x7fad301ef4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:59.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad2fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad30150020 0x7fad301ef4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:41678/0 (socket says 172.21.15.31:41678) 2024-09-11T00:35:59.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad2fe00640 1 -- 172.21.15.31:0/2376781488 learned_addr learned my addr 172.21.15.31:0/2376781488 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:35:59.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad35600640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad30156960 0x7fad301e1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad2fe00640 1 -- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad30156960 msgr2=0x7fad301e1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.294+0000 7fad2fe00640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad30156960 0x7fad301e1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.295+0000 7fad2fe00640 1 -- 172.21.15.31:0/2376781488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad300cc220 msgr2=0x7fad30150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.295+0000 7fad2fe00640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad300cc220 0x7fad30150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.295+0000 7fad2fe00640 1 -- 172.21.15.31:0/2376781488 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad301ef9e0 con 0x7fad30150020 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.295+0000 7fad2fe00640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad30150020 0x7fad301ef4a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00e7c0 tx=0x7fad1c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.295+0000 7fad2cc00640 1 -- 172.21.15.31:0/2376781488 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c00ccc0 con 0x7fad30150020 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.295+0000 7fad2cc00640 1 -- 172.21.15.31:0/2376781488 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad1c010430 con 0x7fad30150020 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.295+0000 7fad36000640 1 -- 172.21.15.31:0/2376781488 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad3014f3e0 con 0x7fad30150020 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.295+0000 7fad2cc00640 1 -- 172.21.15.31:0/2376781488 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c010860 con 0x7fad30150020 2024-09-11T00:35:59.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.295+0000 7fad36000640 1 -- 172.21.15.31:0/2376781488 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad3014f7b0 con 0x7fad30150020 2024-09-11T00:35:59.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.297+0000 7fad36000640 1 -- 172.21.15.31:0/2376781488 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad3014f160 con 0x7fad30150020 2024-09-11T00:35:59.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.298+0000 7fad2cc00640 1 -- 172.21.15.31:0/2376781488 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fad1c03eb10 con 0x7fad30150020 2024-09-11T00:35:59.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.299+0000 7fad2cc00640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad10077f70 0x7fad1007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:35:59.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.299+0000 7fad2cc00640 1 -- 172.21.15.31:0/2376781488 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7fad1c09aec0 con 0x7fad30150020 2024-09-11T00:35:59.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.299+0000 7fad34c00640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad10077f70 0x7fad1007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:35:59.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.300+0000 7fad34c00640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad10077f70 0x7fad1007a430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fad24007fd0 tx=0x7fad24001410 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:35:59.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.304+0000 7fad2cc00640 1 -- 172.21.15.31:0/2376781488 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fad3014f160 con 0x7fad30150020 2024-09-11T00:35:59.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.586+0000 7fad36000640 1 -- 172.21.15.31:0/2376781488 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad3014ed10 con 0x7fad30150020 2024-09-11T00:35:59.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.586+0000 7fad2cc00640 1 -- 172.21.15.31:0/2376781488 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5710 (secure 0 0 0) 0x7fad1c062dc0 con 0x7fad30150020 2024-09-11T00:35:59.595 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:35:59.595 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":50,"btime":"2024-09-11T00:35:57:333736+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:35:59.596 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 50 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 -- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad10077f70 msgr2=0x7fad1007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad10077f70 0x7fad1007a430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fad24007fd0 tx=0x7fad24001410 comp rx=0 tx=0).stop 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 -- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad30150020 msgr2=0x7fad301ef4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad30150020 0x7fad301ef4a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00e7c0 tx=0x7fad1c00ec80 comp rx=0 tx=0).stop 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 -- 172.21.15.31:0/2376781488 shutdown_connections 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad10077f70 0x7fad1007a430 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad30156960 0x7fad301e1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad30150020 0x7fad301ef4a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 --2- 172.21.15.31:0/2376781488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad300cc220 0x7fad30150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 -- 172.21.15.31:0/2376781488 >> 172.21.15.31:0/2376781488 conn(0x7fad30068f10 msgr2=0x7fad301554f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 -- 172.21.15.31:0/2376781488 shutdown_connections 2024-09-11T00:35:59.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:35:59.589+0000 7fad36000640 1 -- 172.21.15.31:0/2376781488 wait complete. 2024-09-11T00:35:59.611 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=4898 2024-09-11T00:35:59.611 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.b 2024-09-11T00:35:59.623 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-11T00:35:59.621+0000 7fe80ec00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:36:05.612 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-11T00:36:05.612 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-11T00:36:05.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-11T00:36:05.617 INFO:tasks.ceph.mds.b:Started 2024-09-11T00:36:05.617 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:36:05.756 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:36:05.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.800+0000 7f49d0e00640 1 -- 172.21.15.31:0/966473526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49cc14ccc0 msgr2=0x7f49cc0ad590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:05.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.800+0000 7f49d0e00640 1 --2- 172.21.15.31:0/966473526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49cc14ccc0 0x7f49cc0ad590 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f49cc069830 tx=0x7f49b802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:36:05.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.801+0000 7f49d0e00640 1 -- 172.21.15.31:0/966473526 shutdown_connections 2024-09-11T00:36:05.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.801+0000 7f49d0e00640 1 --2- 172.21.15.31:0/966473526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49cc14ccc0 0x7f49cc0ad590 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:05.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.801+0000 7f49d0e00640 1 --2- 172.21.15.31:0/966473526 >> v2:172.21.15.204:3300/0 conn(0x7f49cc14c8e0 0x7f49cc156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:05.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.801+0000 7f49d0e00640 1 -- 172.21.15.31:0/966473526 >> 172.21.15.31:0/966473526 conn(0x7f49cc068ee0 msgr2=0x7f49cc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:05.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.801+0000 7f49d0e00640 1 -- 172.21.15.31:0/966473526 shutdown_connections 2024-09-11T00:36:05.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.801+0000 7f49d0e00640 1 -- 172.21.15.31:0/966473526 wait complete. 2024-09-11T00:36:05.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.802+0000 7f49d0e00640 1 Processor -- start 2024-09-11T00:36:05.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.803+0000 7f49d0e00640 1 -- start start 2024-09-11T00:36:05.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.803+0000 7f49d0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49cc0c9a10 0x7f49cc0c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:05.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49d0e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49cc14c8e0 0x7f49cc1e0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:05.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49d0e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49cc14ccc0 0x7f49cc0bfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:05.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49d0e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49cc0c6490 con 0x7f49cc0c9a10 2024-09-11T00:36:05.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49d0e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f49cc0c6800 con 0x7f49cc14ccc0 2024-09-11T00:36:05.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49d0e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f49cc0c6c90 con 0x7f49cc14c8e0 2024-09-11T00:36:05.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49cc0c9a10 0x7f49cc0c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:05.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49cc0c9a10 0x7f49cc0c4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32912/0 (socket says 172.21.15.31:32912) 2024-09-11T00:36:05.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49cb400640 1 -- 172.21.15.31:0/489998247 learned_addr learned my addr 172.21.15.31:0/489998247 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:36:05.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49cbe00640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49cc14ccc0 0x7f49cc0bfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:05.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49caa00640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49cc14c8e0 0x7f49cc1e0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:05.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49cb400640 1 -- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49cc14c8e0 msgr2=0x7f49cc1e0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:05.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49cb400640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49cc14c8e0 0x7f49cc1e0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:05.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49cb400640 1 -- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49cc14ccc0 msgr2=0x7f49cc0bfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:05.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49cb400640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49cc14ccc0 0x7f49cc0bfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:05.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.804+0000 7f49cb400640 1 -- 172.21.15.31:0/489998247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49cc1dfee0 con 0x7f49cc0c9a10 2024-09-11T00:36:05.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.805+0000 7f49cb400640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49cc0c9a10 0x7f49cc0c4c60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00e9f0 tx=0x7f49bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:05.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.805+0000 7f49c3400640 1 -- 172.21.15.31:0/489998247 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc00cd80 con 0x7f49cc0c9a10 2024-09-11T00:36:05.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.805+0000 7f49c3400640 1 -- 172.21.15.31:0/489998247 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49bc010430 con 0x7f49cc0c9a10 2024-09-11T00:36:05.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.805+0000 7f49c3400640 1 -- 172.21.15.31:0/489998247 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc010840 con 0x7f49cc0c9a10 2024-09-11T00:36:05.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.805+0000 7f49d0e00640 1 -- 172.21.15.31:0/489998247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49cc1e01c0 con 0x7f49cc0c9a10 2024-09-11T00:36:05.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.805+0000 7f49d0e00640 1 -- 172.21.15.31:0/489998247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49cc0bf630 con 0x7f49cc0c9a10 2024-09-11T00:36:05.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.806+0000 7f49d0e00640 1 -- 172.21.15.31:0/489998247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49cc0adc90 con 0x7f49cc0c9a10 2024-09-11T00:36:05.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.808+0000 7f49c3400640 1 -- 172.21.15.31:0/489998247 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f49bc0109e0 con 0x7f49cc0c9a10 2024-09-11T00:36:05.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.809+0000 7f49c3400640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49ac077f70 0x7f49ac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:05.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.809+0000 7f49c3400640 1 -- 172.21.15.31:0/489998247 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f49bc09ae50 con 0x7f49cc0c9a10 2024-09-11T00:36:05.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.809+0000 7f49caa00640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49ac077f70 0x7f49ac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:05.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.810+0000 7f49caa00640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49ac077f70 0x7f49ac07a430 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f49b8033000 tx=0x7f49b8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:05.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:05.815+0000 7f49c3400640 1 -- 172.21.15.31:0/489998247 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f49bc062d90 con 0x7f49cc0c9a10 2024-09-11T00:36:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.123+0000 7f49d0e00640 1 -- 172.21.15.31:0/489998247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49cc0ca930 con 0x7f49cc0c9a10 2024-09-11T00:36:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.124+0000 7f49c3400640 1 -- 172.21.15.31:0/489998247 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5710 (secure 0 0 0) 0x7f49bc0677c0 con 0x7f49cc0c9a10 2024-09-11T00:36:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:36:06.126 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":50,"btime":"2024-09-11T00:35:57:333736+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:36:06.126 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 50 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.126+0000 7f49d0e00640 1 -- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49ac077f70 msgr2=0x7f49ac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.126+0000 7f49d0e00640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49ac077f70 0x7f49ac07a430 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f49b8033000 tx=0x7f49b8004730 comp rx=0 tx=0).stop 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.126+0000 7f49d0e00640 1 -- 172.21.15.31:0/489998247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49cc0c9a10 msgr2=0x7f49cc0c4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.126+0000 7f49d0e00640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49cc0c9a10 0x7f49cc0c4c60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00e9f0 tx=0x7f49bc00eeb0 comp rx=0 tx=0).stop 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.127+0000 7f49d0e00640 1 -- 172.21.15.31:0/489998247 shutdown_connections 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.127+0000 7f49d0e00640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49ac077f70 0x7f49ac07a430 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.127+0000 7f49d0e00640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49cc14ccc0 0x7f49cc0bfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.127+0000 7f49d0e00640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49cc14c8e0 0x7f49cc1e0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.127+0000 7f49d0e00640 1 --2- 172.21.15.31:0/489998247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49cc0c9a10 0x7f49cc0c4c60 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.127+0000 7f49d0e00640 1 -- 172.21.15.31:0/489998247 >> 172.21.15.31:0/489998247 conn(0x7f49cc068ee0 msgr2=0x7f49cc159480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.127+0000 7f49d0e00640 1 -- 172.21.15.31:0/489998247 shutdown_connections 2024-09-11T00:36:06.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:06.127+0000 7f49d0e00640 1 -- 172.21.15.31:0/489998247 wait complete. 2024-09-11T00:36:06.146 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-11T00:36:08.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:36:08.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.343+0000 7fc473a00640 1 -- 172.21.15.31:0/716209540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc46c0c5b10 msgr2=0x7fc46c156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:08.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.343+0000 7fc473a00640 1 --2- 172.21.15.31:0/716209540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc46c0c5b10 0x7fc46c156ad0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc460009860 tx=0x7fc46002a4f0 comp rx=0 tx=0).stop 2024-09-11T00:36:08.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.343+0000 7fc473a00640 1 -- 172.21.15.31:0/716209540 shutdown_connections 2024-09-11T00:36:08.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.343+0000 7fc473a00640 1 --2- 172.21.15.31:0/716209540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc46c0c5b10 0x7fc46c156ad0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:08.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.343+0000 7fc473a00640 1 -- 172.21.15.31:0/716209540 >> 172.21.15.31:0/716209540 conn(0x7fc46c0a9750 msgr2=0x7fc46c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:08.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.343+0000 7fc473a00640 1 -- 172.21.15.31:0/716209540 shutdown_connections 2024-09-11T00:36:08.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.343+0000 7fc473a00640 1 -- 172.21.15.31:0/716209540 wait complete. 2024-09-11T00:36:08.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.345+0000 7fc473a00640 1 Processor -- start 2024-09-11T00:36:08.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.345+0000 7fc473a00640 1 -- start start 2024-09-11T00:36:08.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.345+0000 7fc473a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc46c0c5b10 0x7fc46c1ecd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:08.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.345+0000 7fc473a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc46c0c5ef0 0x7fc46c1ef900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:08.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc473a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc46c0c6310 0x7fc46c0accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:08.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc473a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c153b90 con 0x7fc46c0c5b10 2024-09-11T00:36:08.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc473a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c0c7320 con 0x7fc46c0c5ef0 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc473a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc46c153d10 con 0x7fc46c0c6310 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc471000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc46c0c5b10 0x7fc46c1ecd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc471000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc46c0c5b10 0x7fc46c1ecd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49846/0 (socket says 172.21.15.31:49846) 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc471000640 1 -- 172.21.15.31:0/1545427803 learned_addr learned my addr 172.21.15.31:0/1545427803 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc46be00640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc46c0c5ef0 0x7fc46c1ef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc471a00640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc46c0c6310 0x7fc46c0accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc471000640 1 -- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc46c0c6310 msgr2=0x7fc46c0accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc471000640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc46c0c6310 0x7fc46c0accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc471000640 1 -- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc46c0c5ef0 msgr2=0x7fc46c1ef900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc471000640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc46c0c5ef0 0x7fc46c1ef900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:08.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.346+0000 7fc471000640 1 -- 172.21.15.31:0/1545427803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46c1ed260 con 0x7fc46c0c5b10 2024-09-11T00:36:08.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.347+0000 7fc471000640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc46c0c5b10 0x7fc46c1ecd20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc460009980 tx=0x7fc460032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:08.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.347+0000 7fc468c00640 1 -- 172.21.15.31:0/1545427803 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc460040070 con 0x7fc46c0c5b10 2024-09-11T00:36:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.347+0000 7fc468c00640 1 -- 172.21.15.31:0/1545427803 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc460005310 con 0x7fc46c0c5b10 2024-09-11T00:36:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.347+0000 7fc468c00640 1 -- 172.21.15.31:0/1545427803 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4600047a0 con 0x7fc46c0c5b10 2024-09-11T00:36:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.347+0000 7fc473a00640 1 -- 172.21.15.31:0/1545427803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc46c154d60 con 0x7fc46c0c5b10 2024-09-11T00:36:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.347+0000 7fc473a00640 1 -- 172.21.15.31:0/1545427803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc46c1ec520 con 0x7fc46c0c5b10 2024-09-11T00:36:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.349+0000 7fc45e000640 1 -- 172.21.15.31:0/1545427803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc440005290 con 0x7fc46c0c5b10 2024-09-11T00:36:08.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.349+0000 7fc468c00640 1 -- 172.21.15.31:0/1545427803 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc46004c030 con 0x7fc46c0c5b10 2024-09-11T00:36:08.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.351+0000 7fc468c00640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc44c077f70 0x7fc44c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:08.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.351+0000 7fc468c00640 1 -- 172.21.15.31:0/1545427803 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7fc4600bb5a0 con 0x7fc46c0c5b10 2024-09-11T00:36:08.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.351+0000 7fc46be00640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc44c077f70 0x7fc44c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:08.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.352+0000 7fc46be00640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc44c077f70 0x7fc44c07a430 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc46c1e9f00 tx=0x7fc458007e30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:08.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.356+0000 7fc468c00640 1 -- 172.21.15.31:0/1545427803 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc460083390 con 0x7fc46c0c5b10 2024-09-11T00:36:08.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.649+0000 7fc45e000640 1 -- 172.21.15.31:0/1545427803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc440005580 con 0x7fc46c0c5b10 2024-09-11T00:36:08.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.649+0000 7fc468c00640 1 -- 172.21.15.31:0/1545427803 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7fc46008c100 con 0x7fc46c0c5b10 2024-09-11T00:36:08.651 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:36:08.652 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:36:08.652 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:36:08.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 -- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc44c077f70 msgr2=0x7fc44c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:08.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc44c077f70 0x7fc44c07a430 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc46c1e9f00 tx=0x7fc458007e30 comp rx=0 tx=0).stop 2024-09-11T00:36:08.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 -- 172.21.15.31:0/1545427803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc46c0c5b10 msgr2=0x7fc46c1ecd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:08.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc46c0c5b10 0x7fc46c1ecd20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc460009980 tx=0x7fc460032c10 comp rx=0 tx=0).stop 2024-09-11T00:36:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 -- 172.21.15.31:0/1545427803 shutdown_connections 2024-09-11T00:36:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc44c077f70 0x7fc44c07a430 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc46c0c6310 0x7fc46c0accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc46c0c5ef0 0x7fc46c1ef900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 --2- 172.21.15.31:0/1545427803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc46c0c5b10 0x7fc46c1ecd20 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 -- 172.21.15.31:0/1545427803 >> 172.21.15.31:0/1545427803 conn(0x7fc46c0a9750 msgr2=0x7fc46c0c3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 -- 172.21.15.31:0/1545427803 shutdown_connections 2024-09-11T00:36:08.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:08.652+0000 7fc45e000640 1 -- 172.21.15.31:0/1545427803 wait complete. 2024-09-11T00:36:08.672 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-11T00:36:08.672 INFO:tasks.mds_thrash.fs.[b]:waiting for 44 secs before thrashing 2024-09-11T00:36:12.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:36:12.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:36:42.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:36:42.838 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:36:52.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:36:52.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.847+0000 7f8a77e00640 1 -- 172.21.15.31:0/3525493606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a780ac2c0 msgr2=0x7f8a780ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:52.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.847+0000 7f8a77e00640 1 --2- 172.21.15.31:0/3525493606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a780ac2c0 0x7f8a780ac6a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009830 tx=0x7f8a6802a4a0 comp rx=0 tx=0).stop 2024-09-11T00:36:52.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.847+0000 7f8a77e00640 1 -- 172.21.15.31:0/3525493606 shutdown_connections 2024-09-11T00:36:52.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.847+0000 7f8a77e00640 1 --2- 172.21.15.31:0/3525493606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a780ac2c0 0x7f8a780ac6a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:52.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.847+0000 7f8a77e00640 1 -- 172.21.15.31:0/3525493606 >> 172.21.15.31:0/3525493606 conn(0x7f8a7806caf0 msgr2=0x7f8a7806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:52.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.847+0000 7f8a77e00640 1 -- 172.21.15.31:0/3525493606 shutdown_connections 2024-09-11T00:36:52.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.848+0000 7f8a77e00640 1 -- 172.21.15.31:0/3525493606 wait complete. 2024-09-11T00:36:52.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.849+0000 7f8a77e00640 1 Processor -- start 2024-09-11T00:36:52.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.850+0000 7f8a77e00640 1 -- start start 2024-09-11T00:36:52.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a780a6e20 0x7f8a7806f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:52.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a780ac2c0 0x7f8a78072070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:52.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a780acbe0 0x7f8a781e9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:52.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a780c5d20 con 0x7f8a780acbe0 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8a781519e0 con 0x7f8a780a6e20 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8a780c68c0 con 0x7f8a780ac2c0 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a780acbe0 0x7f8a781e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a76a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a780a6e20 0x7f8a7806f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a780acbe0 0x7f8a781e9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57440/0 (socket says 172.21.15.31:57440) 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77400640 1 -- 172.21.15.31:0/1699397362 learned_addr learned my addr 172.21.15.31:0/1699397362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a76a00640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a780a6e20 0x7f8a7806f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:48100/0 (socket says 172.21.15.31:48100) 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a76000640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a780ac2c0 0x7f8a78072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77400640 1 -- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a780ac2c0 msgr2=0x7f8a78072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77400640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a780ac2c0 0x7f8a78072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77400640 1 -- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a780a6e20 msgr2=0x7f8a7806f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77400640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a780a6e20 0x7f8a7806f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:52.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.851+0000 7f8a77400640 1 -- 172.21.15.31:0/1699397362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a781569f0 con 0x7f8a780acbe0 2024-09-11T00:36:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.852+0000 7f8a77400640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a780acbe0 0x7f8a781e9930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000c700 tx=0x7f8a7000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.852+0000 7f8a6ea00640 1 -- 172.21.15.31:0/1699397362 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70003b70 con 0x7f8a780acbe0 2024-09-11T00:36:52.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.852+0000 7f8a6ea00640 1 -- 172.21.15.31:0/1699397362 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a70003d10 con 0x7f8a780acbe0 2024-09-11T00:36:52.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.852+0000 7f8a6ea00640 1 -- 172.21.15.31:0/1699397362 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7000aa50 con 0x7f8a780acbe0 2024-09-11T00:36:52.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.852+0000 7f8a77e00640 1 -- 172.21.15.31:0/1699397362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a78156cd0 con 0x7f8a780acbe0 2024-09-11T00:36:52.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.852+0000 7f8a77e00640 1 -- 172.21.15.31:0/1699397362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a78157290 con 0x7f8a780acbe0 2024-09-11T00:36:52.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.853+0000 7f8a77e00640 1 -- 172.21.15.31:0/1699397362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a780c7450 con 0x7f8a780acbe0 2024-09-11T00:36:52.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.855+0000 7f8a6ea00640 1 -- 172.21.15.31:0/1699397362 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8a7000abf0 con 0x7f8a780acbe0 2024-09-11T00:36:52.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.856+0000 7f8a6ea00640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a58077f70 0x7f8a5807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:52.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.857+0000 7f8a6ea00640 1 -- 172.21.15.31:0/1699397362 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f8a70099fe0 con 0x7f8a780acbe0 2024-09-11T00:36:52.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.857+0000 7f8a76a00640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a58077f70 0x7f8a5807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:52.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.858+0000 7f8a76a00640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a58077f70 0x7f8a5807a430 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8a68034000 tx=0x7f8a68004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:52.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:52.862+0000 7f8a6ea00640 1 -- 172.21.15.31:0/1699397362 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8a70061ea0 con 0x7f8a780acbe0 2024-09-11T00:36:53.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.123+0000 7f8a77e00640 1 -- 172.21.15.31:0/1699397362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a7806fc00 con 0x7f8a780acbe0 2024-09-11T00:36:53.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.123+0000 7f8a6ea00640 1 -- 172.21.15.31:0/1699397362 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7f8a700668d0 con 0x7f8a780acbe0 2024-09-11T00:36:53.126 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:36:53.126 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:36:53.126 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:36:53.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 -- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a58077f70 msgr2=0x7f8a5807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:53.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a58077f70 0x7f8a5807a430 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8a68034000 tx=0x7f8a68004730 comp rx=0 tx=0).stop 2024-09-11T00:36:53.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 -- 172.21.15.31:0/1699397362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a780acbe0 msgr2=0x7f8a781e9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:53.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a780acbe0 0x7f8a781e9930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000c700 tx=0x7f8a7000cbc0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 -- 172.21.15.31:0/1699397362 shutdown_connections 2024-09-11T00:36:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a58077f70 0x7f8a5807a430 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a780acbe0 0x7f8a781e9930 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8a780ac2c0 0x7f8a78072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 --2- 172.21.15.31:0/1699397362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8a780a6e20 0x7f8a7806f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 -- 172.21.15.31:0/1699397362 >> 172.21.15.31:0/1699397362 conn(0x7f8a7806caf0 msgr2=0x7f8a7814fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 -- 172.21.15.31:0/1699397362 shutdown_connections 2024-09-11T00:36:53.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.126+0000 7f8a77e00640 1 -- 172.21.15.31:0/1699397362 wait complete. 2024-09-11T00:36:53.145 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-11T00:36:53.145 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-11T00:36:53.145 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:36:53.164 INFO:tasks.ceph.mds.a:Stopped 2024-09-11T00:36:53.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:36:53.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.382+0000 7f239c000640 1 -- 172.21.15.31:0/3535787726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23940c34e0 msgr2=0x7f23941511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:53.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.382+0000 7f239c000640 1 --2- 172.21.15.31:0/3535787726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23940c34e0 0x7f23941511c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f238c005fd0 tx=0x7f238c02a820 comp rx=0 tx=0).stop 2024-09-11T00:36:53.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.382+0000 7f239c000640 1 -- 172.21.15.31:0/3535787726 shutdown_connections 2024-09-11T00:36:53.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.382+0000 7f239c000640 1 --2- 172.21.15.31:0/3535787726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23940c34e0 0x7f23941511c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.382+0000 7f239c000640 1 --2- 172.21.15.31:0/3535787726 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2394154b60 0x7f23940c2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.382+0000 7f239c000640 1 -- 172.21.15.31:0/3535787726 >> 172.21.15.31:0/3535787726 conn(0x7f23940a9750 msgr2=0x7f23940a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:53.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.383+0000 7f239c000640 1 -- 172.21.15.31:0/3535787726 shutdown_connections 2024-09-11T00:36:53.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.383+0000 7f239c000640 1 -- 172.21.15.31:0/3535787726 wait complete. 2024-09-11T00:36:53.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.384+0000 7f239c000640 1 Processor -- start 2024-09-11T00:36:53.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.384+0000 7f239c000640 1 -- start start 2024-09-11T00:36:53.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.385+0000 7f239c000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2394072480 0x7f239406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:53.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.385+0000 7f239c000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23940c34e0 0x7f23941ec350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:53.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.385+0000 7f239c000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2394154b60 0x7f23941ec890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:53.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.385+0000 7f239c000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f23940a7140 con 0x7f2394154b60 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.385+0000 7f239c000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2394072860 con 0x7f2394072480 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.385+0000 7f239c000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f23940729e0 con 0x7f23940c34e0 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f2398c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23940c34e0 0x7f23941ec350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.385+0000 7f2399600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2394072480 0x7f239406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f239a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2394154b60 0x7f23941ec890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f2398c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23940c34e0 0x7f23941ec350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:44178/0 (socket says 172.21.15.31:44178) 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f2399600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2394072480 0x7f239406ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:48132/0 (socket says 172.21.15.31:48132) 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f2398c00640 1 -- 172.21.15.31:0/2923628566 learned_addr learned my addr 172.21.15.31:0/2923628566 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f239a000640 1 -- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23940c34e0 msgr2=0x7f23941ec350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f239a000640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23940c34e0 0x7f23941ec350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f239a000640 1 -- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2394072480 msgr2=0x7f239406ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:53.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f239a000640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2394072480 0x7f239406ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f239a000640 1 -- 172.21.15.31:0/2923628566 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23940c7480 con 0x7f2394154b60 2024-09-11T00:36:53.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.386+0000 7f239a000640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2394154b60 0x7f23941ec890 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f238c009000 tx=0x7f238c032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:53.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.387+0000 7f2391600640 1 -- 172.21.15.31:0/2923628566 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c037040 con 0x7f2394154b60 2024-09-11T00:36:53.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.387+0000 7f2391600640 1 -- 172.21.15.31:0/2923628566 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f238c042070 con 0x7f2394154b60 2024-09-11T00:36:53.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.387+0000 7f239c000640 1 -- 172.21.15.31:0/2923628566 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23941efa70 con 0x7f2394154b60 2024-09-11T00:36:53.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.387+0000 7f2391600640 1 -- 172.21.15.31:0/2923628566 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c0042a0 con 0x7f2394154b60 2024-09-11T00:36:53.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.387+0000 7f239c000640 1 -- 172.21.15.31:0/2923628566 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23941f0030 con 0x7f2394154b60 2024-09-11T00:36:53.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.388+0000 7f239c000640 1 -- 172.21.15.31:0/2923628566 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2394151870 con 0x7f2394154b60 2024-09-11T00:36:53.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.390+0000 7f2391600640 1 -- 172.21.15.31:0/2923628566 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f238c03a030 con 0x7f2394154b60 2024-09-11T00:36:53.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.391+0000 7f2391600640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2374077fa0 0x7f237407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:53.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.391+0000 7f2391600640 1 -- 172.21.15.31:0/2923628566 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f238c0bac00 con 0x7f2394154b60 2024-09-11T00:36:53.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.391+0000 7f2399600640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2374077fa0 0x7f237407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:53.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.392+0000 7f2399600640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2374077fa0 0x7f237407a460 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2388001720 tx=0x7f2388007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:53.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.396+0000 7f2391600640 1 -- 172.21.15.31:0/2923628566 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f238c082960 con 0x7f2394154b60 2024-09-11T00:36:53.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.682+0000 7f239c000640 1 -- 172.21.15.31:0/2923628566 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23940a6f20 con 0x7f2394154b60 2024-09-11T00:36:53.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.683+0000 7f2391600640 1 -- 172.21.15.31:0/2923628566 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7f238c087390 con 0x7f2394154b60 2024-09-11T00:36:53.685 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:36:53.685 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:36:53.686 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:36:53.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.685+0000 7f239c000640 1 -- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2374077fa0 msgr2=0x7f237407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.685+0000 7f239c000640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2374077fa0 0x7f237407a460 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2388001720 tx=0x7f2388007e90 comp rx=0 tx=0).stop 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.685+0000 7f239c000640 1 -- 172.21.15.31:0/2923628566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2394154b60 msgr2=0x7f23941ec890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.685+0000 7f239c000640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2394154b60 0x7f23941ec890 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f238c009000 tx=0x7f238c032c90 comp rx=0 tx=0).stop 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.685+0000 7f239c000640 1 -- 172.21.15.31:0/2923628566 shutdown_connections 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.685+0000 7f239c000640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2374077fa0 0x7f237407a460 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.685+0000 7f239c000640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2394154b60 0x7f23941ec890 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.685+0000 7f239c000640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23940c34e0 0x7f23941ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.685+0000 7f239c000640 1 --2- 172.21.15.31:0/2923628566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2394072480 0x7f239406ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.685+0000 7f239c000640 1 -- 172.21.15.31:0/2923628566 >> 172.21.15.31:0/2923628566 conn(0x7f23940a9750 msgr2=0x7f23940747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.686+0000 7f239c000640 1 -- 172.21.15.31:0/2923628566 shutdown_connections 2024-09-11T00:36:53.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:53.686+0000 7f239c000640 1 -- 172.21.15.31:0/2923628566 wait complete. 2024-09-11T00:36:53.704 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:36:55.705 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:36:55.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.870+0000 7f9236400640 1 -- 172.21.15.31:0/1746139136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9218003c60 msgr2=0x7f9218004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:55.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.870+0000 7f9236400640 1 --2- 172.21.15.31:0/1746139136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9218003c60 0x7f9218004060 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9224001830 tx=0x7f9218005830 comp rx=0 tx=0).stop 2024-09-11T00:36:55.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.870+0000 7f9235a00640 1 -- 172.21.15.31:0/1746139136 reap_dead start 2024-09-11T00:36:55.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.870+0000 7f9236400640 1 -- 172.21.15.31:0/1746139136 shutdown_connections 2024-09-11T00:36:55.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.870+0000 7f9236400640 1 -- 172.21.15.31:0/1746139136 >> 172.21.15.31:0/1746139136 conn(0x7f9230068f10 msgr2=0x7f92300cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:55.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.870+0000 7f9236400640 1 -- 172.21.15.31:0/1746139136 shutdown_connections 2024-09-11T00:36:55.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.870+0000 7f9236400640 1 -- 172.21.15.31:0/1746139136 wait complete. 2024-09-11T00:36:55.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.872+0000 7f9236400640 1 Processor -- start 2024-09-11T00:36:55.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.872+0000 7f9236400640 1 -- start start 2024-09-11T00:36:55.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.873+0000 7f9236400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9218003c60 0x7f92300c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:55.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.873+0000 7f9236400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92300c5630 0x7f92300c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.873+0000 7f9236400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f92300c2fe0 0x7f92300af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.873+0000 7f9236400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f92300c6ca0 con 0x7f92300c5630 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.873+0000 7f9236400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f92300c6b20 con 0x7f9218003c60 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.873+0000 7f9236400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f92301573d0 con 0x7f92300c2fe0 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.873+0000 7f922fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92300c5630 0x7f92300c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.873+0000 7f922fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92300c5630 0x7f92300c2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57462/0 (socket says 172.21.15.31:57462) 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.873+0000 7f922fe00640 1 -- 172.21.15.31:0/2406378083 learned_addr learned my addr 172.21.15.31:0/2406378083 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.874+0000 7f9235a00640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f92300c2fe0 0x7f92300af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.874+0000 7f9235000640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9218003c60 0x7f92300c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.874+0000 7f922fe00640 1 -- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f92300c2fe0 msgr2=0x7f92300af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.874+0000 7f922fe00640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f92300c2fe0 0x7f92300af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:55.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.874+0000 7f922fe00640 1 -- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9218003c60 msgr2=0x7f92300c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:55.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.874+0000 7f922fe00640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9218003c60 0x7f92300c41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:55.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.874+0000 7f922fe00640 1 -- 172.21.15.31:0/2406378083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92300b0860 con 0x7f92300c5630 2024-09-11T00:36:55.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.874+0000 7f9235000640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9218003c60 0x7f92300c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:36:55.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.874+0000 7f922fe00640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92300c5630 0x7f92300c2aa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f921c011b10 tx=0x7f921c009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:55.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.875+0000 7f922cc00640 1 -- 172.21.15.31:0/2406378083 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c01e070 con 0x7f92300c5630 2024-09-11T00:36:55.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.875+0000 7f922cc00640 1 -- 172.21.15.31:0/2406378083 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f921c004050 con 0x7f92300c5630 2024-09-11T00:36:55.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.875+0000 7f922cc00640 1 -- 172.21.15.31:0/2406378083 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c00fea0 con 0x7f92300c5630 2024-09-11T00:36:55.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.875+0000 7f9236400640 1 -- 172.21.15.31:0/2406378083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92300b0b40 con 0x7f92300c5630 2024-09-11T00:36:55.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.875+0000 7f9236400640 1 -- 172.21.15.31:0/2406378083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92300c46f0 con 0x7f92300c5630 2024-09-11T00:36:55.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.876+0000 7f9236400640 1 -- 172.21.15.31:0/2406378083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92300ad4e0 con 0x7f92300c5630 2024-09-11T00:36:55.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.877+0000 7f922cc00640 1 -- 172.21.15.31:0/2406378083 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f921c0035f0 con 0x7f92300c5630 2024-09-11T00:36:55.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.878+0000 7f922cc00640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9210077fa0 0x7f921007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:55.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.878+0000 7f922cc00640 1 -- 172.21.15.31:0/2406378083 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f921c09b0a0 con 0x7f92300c5630 2024-09-11T00:36:55.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.879+0000 7f9235000640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9210077fa0 0x7f921007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:55.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.880+0000 7f9235000640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9210077fa0 0x7f921007a460 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9224032000 tx=0x7f9224001360 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:55.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:55.884+0000 7f922cc00640 1 -- 172.21.15.31:0/2406378083 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f921c0044a0 con 0x7f92300c5630 2024-09-11T00:36:56.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.171+0000 7f9236400640 1 -- 172.21.15.31:0/2406378083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92300cc8e0 con 0x7f92300c5630 2024-09-11T00:36:56.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.171+0000 7f922cc00640 1 -- 172.21.15.31:0/2406378083 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7f921c062fe0 con 0x7f92300c5630 2024-09-11T00:36:56.173 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:36:56.173 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:36:56.174 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:36:56.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 -- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9210077fa0 msgr2=0x7f921007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9210077fa0 0x7f921007a460 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9224032000 tx=0x7f9224001360 comp rx=0 tx=0).stop 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 -- 172.21.15.31:0/2406378083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92300c5630 msgr2=0x7f92300c2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92300c5630 0x7f92300c2aa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f921c011b10 tx=0x7f921c009e90 comp rx=0 tx=0).stop 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 -- 172.21.15.31:0/2406378083 shutdown_connections 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9210077fa0 0x7f921007a460 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f92300c2fe0 0x7f92300af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f92300c5630 0x7f92300c2aa0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 --2- 172.21.15.31:0/2406378083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9218003c60 0x7f92300c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 -- 172.21.15.31:0/2406378083 >> 172.21.15.31:0/2406378083 conn(0x7f9230068f10 msgr2=0x7f9230155320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 -- 172.21.15.31:0/2406378083 shutdown_connections 2024-09-11T00:36:56.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:56.174+0000 7f9236400640 1 -- 172.21.15.31:0/2406378083 wait complete. 2024-09-11T00:36:56.194 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:36:58.195 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:36:58.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.375+0000 7efcdea00640 1 -- 172.21.15.31:0/4037798262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcd8150020 msgr2=0x7efcd80ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:58.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.375+0000 7efcdea00640 1 --2- 172.21.15.31:0/4037798262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcd8150020 0x7efcd80ad4e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009830 tx=0x7efcc802a510 comp rx=0 tx=0).stop 2024-09-11T00:36:58.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.375+0000 7efcdea00640 1 -- 172.21.15.31:0/4037798262 shutdown_connections 2024-09-11T00:36:58.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.375+0000 7efcdea00640 1 --2- 172.21.15.31:0/4037798262 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efcd80ada20 0x7efcd806e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:58.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.375+0000 7efcdea00640 1 --2- 172.21.15.31:0/4037798262 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcd8150020 0x7efcd80ad4e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:58.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.375+0000 7efcdea00640 1 -- 172.21.15.31:0/4037798262 >> 172.21.15.31:0/4037798262 conn(0x7efcd8068f10 msgr2=0x7efcd80cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:58.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.375+0000 7efcdea00640 1 -- 172.21.15.31:0/4037798262 shutdown_connections 2024-09-11T00:36:58.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.375+0000 7efcdea00640 1 -- 172.21.15.31:0/4037798262 wait complete. 2024-09-11T00:36:58.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.377+0000 7efcdea00640 1 Processor -- start 2024-09-11T00:36:58.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.377+0000 7efcdea00640 1 -- start start 2024-09-11T00:36:58.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcdea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcd80ada20 0x7efcd814efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:58.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcdea00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efcd8150020 0x7efcd81f02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:58.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcdea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcd8156960 0x7efcd81f0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:58.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcdea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efcd8157550 con 0x7efcd8156960 2024-09-11T00:36:58.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcdea00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efcd81573d0 con 0x7efcd80ada20 2024-09-11T00:36:58.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcdea00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efcd80cc220 con 0x7efcd8150020 2024-09-11T00:36:58.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcde000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcd8156960 0x7efcd81f0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:58.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcdcc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efcd8150020 0x7efcd81f02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:58.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcdd600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcd80ada20 0x7efcd814efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:58.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcde000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcd8156960 0x7efcd81f0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46576/0 (socket says 172.21.15.31:46576) 2024-09-11T00:36:58.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcde000640 1 -- 172.21.15.31:0/3018066316 learned_addr learned my addr 172.21.15.31:0/3018066316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:36:58.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcde000640 1 -- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efcd8150020 msgr2=0x7efcd81f02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:58.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcde000640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efcd8150020 0x7efcd81f02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:58.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcde000640 1 -- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcd80ada20 msgr2=0x7efcd814efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:58.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcde000640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcd80ada20 0x7efcd814efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:58.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.379+0000 7efcde000640 1 -- 172.21.15.31:0/3018066316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd81ed690 con 0x7efcd8156960 2024-09-11T00:36:58.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.380+0000 7efcde000640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcd8156960 0x7efcd81f0820 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7efcd000cb10 tx=0x7efcd000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:58.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.380+0000 7efcd5600640 1 -- 172.21.15.31:0/3018066316 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0016020 con 0x7efcd8156960 2024-09-11T00:36:58.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.380+0000 7efcd5600640 1 -- 172.21.15.31:0/3018066316 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcd000ad80 con 0x7efcd8156960 2024-09-11T00:36:58.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.380+0000 7efcd5600640 1 -- 172.21.15.31:0/3018066316 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0012680 con 0x7efcd8156960 2024-09-11T00:36:58.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.380+0000 7efcdea00640 1 -- 172.21.15.31:0/3018066316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd80c21d0 con 0x7efcd8156960 2024-09-11T00:36:58.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.380+0000 7efcdea00640 1 -- 172.21.15.31:0/3018066316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd81ed3b0 con 0x7efcd8156960 2024-09-11T00:36:58.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.382+0000 7efcdea00640 1 -- 172.21.15.31:0/3018066316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcd80c6e40 con 0x7efcd8156960 2024-09-11T00:36:58.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.383+0000 7efcd5600640 1 -- 172.21.15.31:0/3018066316 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7efcd0012820 con 0x7efcd8156960 2024-09-11T00:36:58.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.384+0000 7efcd5600640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcb8077fa0 0x7efcb807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:36:58.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.384+0000 7efcd5600640 1 -- 172.21.15.31:0/3018066316 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7efcd009ab40 con 0x7efcd8156960 2024-09-11T00:36:58.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.384+0000 7efcdd600640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcb8077fa0 0x7efcb807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:36:58.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.385+0000 7efcdd600640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcb8077fa0 0x7efcb807a460 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efcc8034000 tx=0x7efcc8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:36:58.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.388+0000 7efcd5600640 1 -- 172.21.15.31:0/3018066316 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efcd80c6e40 con 0x7efcd8156960 2024-09-11T00:36:58.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.660+0000 7efcdea00640 1 -- 172.21.15.31:0/3018066316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcd80ad4e0 con 0x7efcd8156960 2024-09-11T00:36:58.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.660+0000 7efcd5600640 1 -- 172.21.15.31:0/3018066316 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7efcd0020020 con 0x7efcd8156960 2024-09-11T00:36:58.662 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:36:58.662 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:36:58.663 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:36:58.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 -- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcb8077fa0 msgr2=0x7efcb807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:58.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcb8077fa0 0x7efcb807a460 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efcc8034000 tx=0x7efcc8004730 comp rx=0 tx=0).stop 2024-09-11T00:36:58.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 -- 172.21.15.31:0/3018066316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcd8156960 msgr2=0x7efcd81f0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:36:58.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcd8156960 0x7efcd81f0820 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7efcd000cb10 tx=0x7efcd000a060 comp rx=0 tx=0).stop 2024-09-11T00:36:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 -- 172.21.15.31:0/3018066316 shutdown_connections 2024-09-11T00:36:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcb8077fa0 0x7efcb807a460 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efcd8156960 0x7efcd81f0820 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efcd8150020 0x7efcd81f02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 --2- 172.21.15.31:0/3018066316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efcd80ada20 0x7efcd814efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:36:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 -- 172.21.15.31:0/3018066316 >> 172.21.15.31:0/3018066316 conn(0x7efcd8068f10 msgr2=0x7efcd80acce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:36:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.663+0000 7efcdea00640 1 -- 172.21.15.31:0/3018066316 shutdown_connections 2024-09-11T00:36:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:36:58.664+0000 7efcdea00640 1 -- 172.21.15.31:0/3018066316 wait complete. 2024-09-11T00:36:58.682 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:36:59.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:00.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.037+0000 7f6666600640 1 -- 172.21.15.31:0/3590626186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66600c9a10 msgr2=0x7f6660159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.037+0000 7f6666600640 1 --2- 172.21.15.31:0/3590626186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66600c9a10 0x7f6660159670 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6658005fd0 tx=0x7f665802a8b0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.037+0000 7f6666600640 1 -- 172.21.15.31:0/3590626186 shutdown_connections 2024-09-11T00:37:00.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.037+0000 7f6666600640 1 --2- 172.21.15.31:0/3590626186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66600c9a10 0x7f6660159670 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.037+0000 7f6666600640 1 --2- 172.21.15.31:0/3590626186 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f666014ccc0 0x7f6660156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.037+0000 7f6666600640 1 -- 172.21.15.31:0/3590626186 >> 172.21.15.31:0/3590626186 conn(0x7f6660068ee0 msgr2=0x7f66600cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:00.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.037+0000 7f6666600640 1 -- 172.21.15.31:0/3590626186 shutdown_connections 2024-09-11T00:37:00.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.037+0000 7f6666600640 1 -- 172.21.15.31:0/3590626186 wait complete. 2024-09-11T00:37:00.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.039+0000 7f6666600640 1 Processor -- start 2024-09-11T00:37:00.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.039+0000 7f6666600640 1 -- start start 2024-09-11T00:37:00.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.040+0000 7f6666600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66600c9a10 0x7f66601e1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.040+0000 7f6666600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f666014c8e0 0x7f666014e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.040+0000 7f6666600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666014ccc0 0x7f66600c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.040+0000 7f6666600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66600c6c90 con 0x7f666014ccc0 2024-09-11T00:37:00.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.040+0000 7f6666600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f66600c6490 con 0x7f666014c8e0 2024-09-11T00:37:00.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.040+0000 7f6666600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f66600adca0 con 0x7f66600c9a10 2024-09-11T00:37:00.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.041+0000 7f6665c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666014ccc0 0x7f66600c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.041+0000 7f6665c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666014ccc0 0x7f66600c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46592/0 (socket says 172.21.15.31:46592) 2024-09-11T00:37:00.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.041+0000 7f6665c00640 1 -- 172.21.15.31:0/187237894 learned_addr learned my addr 172.21.15.31:0/187237894 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:00.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.041+0000 7f665fe00640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f666014c8e0 0x7f666014e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.041+0000 7f6665200640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66600c9a10 0x7f66601e1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.041+0000 7f6665c00640 1 -- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66600c9a10 msgr2=0x7f66601e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.041+0000 7f6665c00640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66600c9a10 0x7f66601e1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.041+0000 7f6665c00640 1 -- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f666014c8e0 msgr2=0x7f666014e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.041+0000 7f6665c00640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f666014c8e0 0x7f666014e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.041+0000 7f6665c00640 1 -- 172.21.15.31:0/187237894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f666014d360 con 0x7f666014ccc0 2024-09-11T00:37:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.042+0000 7f6665c00640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666014ccc0 0x7f66600c51f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6658009cd0 tx=0x7f6658031760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.042+0000 7f665cc00640 1 -- 172.21.15.31:0/187237894 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6658040070 con 0x7f666014ccc0 2024-09-11T00:37:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.042+0000 7f665cc00640 1 -- 172.21.15.31:0/187237894 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6658004040 con 0x7f666014ccc0 2024-09-11T00:37:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.042+0000 7f665cc00640 1 -- 172.21.15.31:0/187237894 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6658004450 con 0x7f666014ccc0 2024-09-11T00:37:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.042+0000 7f6666600640 1 -- 172.21.15.31:0/187237894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f666014d640 con 0x7f666014ccc0 2024-09-11T00:37:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.042+0000 7f6666600640 1 -- 172.21.15.31:0/187237894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66601df6e0 con 0x7f666014ccc0 2024-09-11T00:37:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.043+0000 7f6666600640 1 -- 172.21.15.31:0/187237894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6660156b00 con 0x7f666014ccc0 2024-09-11T00:37:00.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.045+0000 7f665cc00640 1 -- 172.21.15.31:0/187237894 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f66580045f0 con 0x7f666014ccc0 2024-09-11T00:37:00.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.046+0000 7f665cc00640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6640077fa0 0x7f664007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.046+0000 7f665cc00640 1 -- 172.21.15.31:0/187237894 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f66580bbb00 con 0x7f666014ccc0 2024-09-11T00:37:00.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.046+0000 7f6665200640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6640077fa0 0x7f664007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.047+0000 7f6665200640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6640077fa0 0x7f664007a460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f666014c290 tx=0x7f6654009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:00.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.051+0000 7f665cc00640 1 -- 172.21.15.31:0/187237894 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6658083980 con 0x7f666014ccc0 2024-09-11T00:37:00.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.294+0000 7f6666600640 1 -- 172.21.15.31:0/187237894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f666014eb50 con 0x7f666014ccc0 2024-09-11T00:37:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.295+0000 7f665cc00640 1 -- 172.21.15.31:0/187237894 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7f66580883b0 con 0x7f666014ccc0 2024-09-11T00:37:00.297 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:00.297 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:37:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:37:00.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 -- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6640077fa0 msgr2=0x7f664007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6640077fa0 0x7f664007a460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f666014c290 tx=0x7f6654009040 comp rx=0 tx=0).stop 2024-09-11T00:37:00.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 -- 172.21.15.31:0/187237894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666014ccc0 msgr2=0x7f66600c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666014ccc0 0x7f66600c51f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6658009cd0 tx=0x7f6658031760 comp rx=0 tx=0).stop 2024-09-11T00:37:00.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 -- 172.21.15.31:0/187237894 shutdown_connections 2024-09-11T00:37:00.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6640077fa0 0x7f664007a460 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f666014ccc0 0x7f66600c51f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f666014c8e0 0x7f666014e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 --2- 172.21.15.31:0/187237894 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66600c9a10 0x7f66601e1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 -- 172.21.15.31:0/187237894 >> 172.21.15.31:0/187237894 conn(0x7f6660068ee0 msgr2=0x7f66601552e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:00.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 -- 172.21.15.31:0/187237894 shutdown_connections 2024-09-11T00:37:00.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.298+0000 7f6666600640 1 -- 172.21.15.31:0/187237894 wait complete. 2024-09-11T00:37:00.315 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-11T00:37:00.316 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:37:00.316 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:37:00.337 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:37:00.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:00.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.496+0000 7fa1c7200640 1 -- 172.21.15.31:0/1510011137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c80ac2c0 msgr2=0x7fa1c80ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.496+0000 7fa1c7200640 1 --2- 172.21.15.31:0/1510011137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c80ac2c0 0x7fa1c80ac6a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8005fd0 tx=0x7fa1b802a4b0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.496+0000 7fa1c7200640 1 -- 172.21.15.31:0/1510011137 shutdown_connections 2024-09-11T00:37:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.496+0000 7fa1c7200640 1 --2- 172.21.15.31:0/1510011137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c80ac2c0 0x7fa1c80ac6a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.496+0000 7fa1c7200640 1 -- 172.21.15.31:0/1510011137 >> 172.21.15.31:0/1510011137 conn(0x7fa1c806caf0 msgr2=0x7fa1c806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.496+0000 7fa1c7200640 1 -- 172.21.15.31:0/1510011137 shutdown_connections 2024-09-11T00:37:00.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.497+0000 7fa1c7200640 1 -- 172.21.15.31:0/1510011137 wait complete. 2024-09-11T00:37:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.498+0000 7fa1c7200640 1 Processor -- start 2024-09-11T00:37:00.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.498+0000 7fa1c7200640 1 -- start start 2024-09-11T00:37:00.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.499+0000 7fa1c7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1c80a6e20 0x7fa1c80c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.499+0000 7fa1c7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1c80ac2c0 0x7fa1c806e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.499+0000 7fa1c7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c80acbe0 0x7fa1c806e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.499+0000 7fa1c7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c80c5d20 con 0x7fa1c80acbe0 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.499+0000 7fa1c7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c81519e0 con 0x7fa1c80ac2c0 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.499+0000 7fa1c7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c80c68c0 con 0x7fa1c80a6e20 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.499+0000 7fa1c6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c80acbe0 0x7fa1c806e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.499+0000 7fa1c6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c80acbe0 0x7fa1c806e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46608/0 (socket says 172.21.15.31:46608) 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.499+0000 7fa1c6800640 1 -- 172.21.15.31:0/2008687792 learned_addr learned my addr 172.21.15.31:0/2008687792 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1c5e00640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1c80a6e20 0x7fa1c80c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1bfe00640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1c80ac2c0 0x7fa1c806e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1c6800640 1 -- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1c80a6e20 msgr2=0x7fa1c80c0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1c6800640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1c80a6e20 0x7fa1c80c0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1c6800640 1 -- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1c80ac2c0 msgr2=0x7fa1c806e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1c6800640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1c80ac2c0 0x7fa1c806e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1c6800640 1 -- 172.21.15.31:0/2008687792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c806fdb0 con 0x7fa1c80acbe0 2024-09-11T00:37:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1c5e00640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1c80a6e20 0x7fa1c80c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:37:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1c6800640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c80acbe0 0x7fa1c806e710 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa1c000b9c0 tx=0x7fa1c000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1be000640 1 -- 172.21.15.31:0/2008687792 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0019070 con 0x7fa1c80acbe0 2024-09-11T00:37:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1be000640 1 -- 172.21.15.31:0/2008687792 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1c000cc90 con 0x7fa1c80acbe0 2024-09-11T00:37:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1be000640 1 -- 172.21.15.31:0/2008687792 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c00146a0 con 0x7fa1c80acbe0 2024-09-11T00:37:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.500+0000 7fa1c7200640 1 -- 172.21.15.31:0/2008687792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c81ec0b0 con 0x7fa1c80acbe0 2024-09-11T00:37:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.501+0000 7fa1c7200640 1 -- 172.21.15.31:0/2008687792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c81ec670 con 0x7fa1c80acbe0 2024-09-11T00:37:00.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.502+0000 7fa1c7200640 1 -- 172.21.15.31:0/2008687792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c80c6cd0 con 0x7fa1c80acbe0 2024-09-11T00:37:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.503+0000 7fa1be000640 1 -- 172.21.15.31:0/2008687792 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa1c001d3b0 con 0x7fa1c80acbe0 2024-09-11T00:37:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.504+0000 7fa1be000640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa1a4077f70 0x7fa1a407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.504+0000 7fa1be000640 1 -- 172.21.15.31:0/2008687792 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7fa1c009a400 con 0x7fa1c80acbe0 2024-09-11T00:37:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.504+0000 7fa1c5e00640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa1a4077f70 0x7fa1a407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.505+0000 7fa1c5e00640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa1a4077f70 0x7fa1a407a430 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8033000 tx=0x7fa1b8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:00.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.508+0000 7fa1be000640 1 -- 172.21.15.31:0/2008687792 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa1c00628d0 con 0x7fa1c80acbe0 2024-09-11T00:37:00.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:00.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.739+0000 7fa1c7200640 1 -- 172.21.15.31:0/2008687792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1c806d120 con 0x7fa1c80acbe0 2024-09-11T00:37:00.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.740+0000 7fa1be000640 1 -- 172.21.15.31:0/2008687792 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7fa1c0066cb0 con 0x7fa1c80acbe0 2024-09-11T00:37:00.742 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:00.742 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:37:00.742 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:37:00.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.742+0000 7fa1c7200640 1 -- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa1a4077f70 msgr2=0x7fa1a407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.742+0000 7fa1c7200640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa1a4077f70 0x7fa1a407a430 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8033000 tx=0x7fa1b8004730 comp rx=0 tx=0).stop 2024-09-11T00:37:00.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.742+0000 7fa1c7200640 1 -- 172.21.15.31:0/2008687792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c80acbe0 msgr2=0x7fa1c806e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.742+0000 7fa1c7200640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c80acbe0 0x7fa1c806e710 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa1c000b9c0 tx=0x7fa1c000be80 comp rx=0 tx=0).stop 2024-09-11T00:37:00.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.742+0000 7fa1c7200640 1 -- 172.21.15.31:0/2008687792 shutdown_connections 2024-09-11T00:37:00.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.742+0000 7fa1c7200640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa1a4077f70 0x7fa1a407a430 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.742+0000 7fa1c7200640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c80acbe0 0x7fa1c806e710 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.742+0000 7fa1c7200640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1c80ac2c0 0x7fa1c806e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.742+0000 7fa1c7200640 1 --2- 172.21.15.31:0/2008687792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa1c80a6e20 0x7fa1c80c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.742+0000 7fa1c7200640 1 -- 172.21.15.31:0/2008687792 >> 172.21.15.31:0/2008687792 conn(0x7fa1c806caf0 msgr2=0x7fa1c814f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:00.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.743+0000 7fa1c7200640 1 -- 172.21.15.31:0/2008687792 shutdown_connections 2024-09-11T00:37:00.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.743+0000 7fa1c7200640 1 -- 172.21.15.31:0/2008687792 wait complete. 2024-09-11T00:37:00.760 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:37:00.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.844+0000 7f5f6b800640 1 -- 172.21.15.31:0/2926546333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c004800 msgr2=0x7f5f4c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.844+0000 7f5f6b800640 1 --2- 172.21.15.31:0/2926546333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c004800 0x7f5f4c004c80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009b20 tx=0x7f5f4c005830 comp rx=0 tx=0).stop 2024-09-11T00:37:00.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.844+0000 7f5f6b800640 1 -- 172.21.15.31:0/2926546333 shutdown_connections 2024-09-11T00:37:00.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.844+0000 7f5f6b800640 1 --2- 172.21.15.31:0/2926546333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c004800 0x7f5f4c004c80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.844+0000 7f5f6b800640 1 -- 172.21.15.31:0/2926546333 >> 172.21.15.31:0/2926546333 conn(0x7f5f640a9750 msgr2=0x7f5f640a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:00.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.844+0000 7f5f6b800640 1 -- 172.21.15.31:0/2926546333 shutdown_connections 2024-09-11T00:37:00.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.844+0000 7f5f6b800640 1 -- 172.21.15.31:0/2926546333 wait complete. 2024-09-11T00:37:00.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.846+0000 7f5f6b800640 1 Processor -- start 2024-09-11T00:37:00.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.846+0000 7f5f6b800640 1 -- start start 2024-09-11T00:37:00.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.846+0000 7f5f6b800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5f4c004800 0x7f5f641df100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.846+0000 7f5f6b800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f640c6520 0x7f5f640a8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f6b800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f641482a0 0x7f5f641dd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f6b800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f64074200 con 0x7f5f641482a0 2024-09-11T00:37:00.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f6b800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5f64074080 con 0x7f5f640c6520 2024-09-11T00:37:00.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f6b800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5f64072860 con 0x7f5f4c004800 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f69800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f641482a0 0x7f5f641dd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f69800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f641482a0 0x7f5f641dd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46632/0 (socket says 172.21.15.31:46632) 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f69800640 1 -- 172.21.15.31:0/2859430460 learned_addr learned my addr 172.21.15.31:0/2859430460 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f68e00640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5f4c004800 0x7f5f641df100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f63e00640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f640c6520 0x7f5f640a8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f69800640 1 -- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5f4c004800 msgr2=0x7f5f641df100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f69800640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5f4c004800 0x7f5f641df100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f69800640 1 -- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f640c6520 msgr2=0x7f5f640a8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f69800640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f640c6520 0x7f5f640a8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.847+0000 7f5f69800640 1 -- 172.21.15.31:0/2859430460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f640c69a0 con 0x7f5f641482a0 2024-09-11T00:37:00.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.848+0000 7f5f69800640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f641482a0 0x7f5f641dd760 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5f640731e0 tx=0x7f5f5c006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:00.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.848+0000 7f5f60c00640 1 -- 172.21.15.31:0/2859430460 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c015070 con 0x7f5f641482a0 2024-09-11T00:37:00.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.848+0000 7f5f60c00640 1 -- 172.21.15.31:0/2859430460 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f5c00f980 con 0x7f5f641482a0 2024-09-11T00:37:00.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.848+0000 7f5f60c00640 1 -- 172.21.15.31:0/2859430460 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c00fdd0 con 0x7f5f641482a0 2024-09-11T00:37:00.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.848+0000 7f5f6b800640 1 -- 172.21.15.31:0/2859430460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f641de0a0 con 0x7f5f641482a0 2024-09-11T00:37:00.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.848+0000 7f5f6b800640 1 -- 172.21.15.31:0/2859430460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f640c5a40 con 0x7f5f641482a0 2024-09-11T00:37:00.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.849+0000 7f5f6b800640 1 -- 172.21.15.31:0/2859430460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f640c6210 con 0x7f5f641482a0 2024-09-11T00:37:00.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.850+0000 7f5f60c00640 1 -- 172.21.15.31:0/2859430460 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5f5c011430 con 0x7f5f641482a0 2024-09-11T00:37:00.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.851+0000 7f5f60c00640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f44077fa0 0x7f5f4407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:00.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.851+0000 7f5f60c00640 1 -- 172.21.15.31:0/2859430460 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f5f5c09af70 con 0x7f5f641482a0 2024-09-11T00:37:00.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.852+0000 7f5f68e00640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f44077fa0 0x7f5f4407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:00.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.852+0000 7f5f68e00640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f44077fa0 0x7f5f4407a460 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5f640adab0 tx=0x7f5f58006b10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:00.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:00.855+0000 7f5f60c00640 1 -- 172.21.15.31:0/2859430460 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5f640c69a0 con 0x7f5f641482a0 2024-09-11T00:37:01.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.089+0000 7f5f6b800640 1 -- 172.21.15.31:0/2859430460 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f6414d580 con 0x7f5f641482a0 2024-09-11T00:37:01.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.090+0000 7f5f60c00640 1 -- 172.21.15.31:0/2859430460 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7f5f5c062df0 con 0x7f5f641482a0 2024-09-11T00:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:01.092 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:37:01.092 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.092+0000 7f5f6b800640 1 -- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f44077fa0 msgr2=0x7f5f4407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.092+0000 7f5f6b800640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f44077fa0 0x7f5f4407a460 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5f640adab0 tx=0x7f5f58006b10 comp rx=0 tx=0).stop 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.092+0000 7f5f6b800640 1 -- 172.21.15.31:0/2859430460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f641482a0 msgr2=0x7f5f641dd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.092+0000 7f5f6b800640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f641482a0 0x7f5f641dd760 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5f640731e0 tx=0x7f5f5c006660 comp rx=0 tx=0).stop 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.093+0000 7f5f6b800640 1 -- 172.21.15.31:0/2859430460 shutdown_connections 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.093+0000 7f5f6b800640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5f44077fa0 0x7f5f4407a460 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.093+0000 7f5f6b800640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f641482a0 0x7f5f641dd760 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.093+0000 7f5f6b800640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f640c6520 0x7f5f640a8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.093+0000 7f5f6b800640 1 --2- 172.21.15.31:0/2859430460 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5f4c004800 0x7f5f641df100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.093+0000 7f5f6b800640 1 -- 172.21.15.31:0/2859430460 >> 172.21.15.31:0/2859430460 conn(0x7f5f640a9750 msgr2=0x7f5f6414ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:01.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.093+0000 7f5f6b800640 1 -- 172.21.15.31:0/2859430460 shutdown_connections 2024-09-11T00:37:01.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:01.093+0000 7f5f6b800640 1 -- 172.21.15.31:0/2859430460 wait complete. 2024-09-11T00:37:01.110 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:37:02.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.931+0000 7f5ea9600640 1 -- 172.21.15.31:0/1825893041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea414ccc0 msgr2=0x7f5ea40ad590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.931+0000 7f5ea9600640 1 --2- 172.21.15.31:0/1825893041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea414ccc0 0x7f5ea40ad590 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009830 tx=0x7f5e8c02a530 comp rx=0 tx=0).stop 2024-09-11T00:37:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.932+0000 7f5ea9600640 1 -- 172.21.15.31:0/1825893041 shutdown_connections 2024-09-11T00:37:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.932+0000 7f5ea9600640 1 --2- 172.21.15.31:0/1825893041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea414ccc0 0x7f5ea40ad590 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.932+0000 7f5ea9600640 1 --2- 172.21.15.31:0/1825893041 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ea414c8e0 0x7f5ea4156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.932+0000 7f5ea9600640 1 -- 172.21.15.31:0/1825893041 >> 172.21.15.31:0/1825893041 conn(0x7f5ea4068ee0 msgr2=0x7f5ea40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.932+0000 7f5ea9600640 1 -- 172.21.15.31:0/1825893041 shutdown_connections 2024-09-11T00:37:02.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.932+0000 7f5ea9600640 1 -- 172.21.15.31:0/1825893041 wait complete. 2024-09-11T00:37:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.933+0000 7f5ea9600640 1 Processor -- start 2024-09-11T00:37:02.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.933+0000 7f5ea9600640 1 -- start start 2024-09-11T00:37:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.934+0000 7f5ea9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea40c9a10 0x7f5ea40ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.934+0000 7f5ea9600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ea414c8e0 0x7f5ea414bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.934+0000 7f5ea9600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea414ccc0 0x7f5ea414c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.934+0000 7f5ea9600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea40c6c90 con 0x7f5ea40c9a10 2024-09-11T00:37:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.934+0000 7f5ea9600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea40c6490 con 0x7f5ea414ccc0 2024-09-11T00:37:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.934+0000 7f5ea9600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5ea40adad0 con 0x7f5ea414c8e0 2024-09-11T00:37:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.934+0000 7f5ea3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea40c9a10 0x7f5ea40ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.934+0000 7f5ea3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea40c9a10 0x7f5ea40ca0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46640/0 (socket says 172.21.15.31:46640) 2024-09-11T00:37:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.934+0000 7f5ea3e00640 1 -- 172.21.15.31:0/1147409045 learned_addr learned my addr 172.21.15.31:0/1147409045 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:02.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.935+0000 7f5ea3e00640 1 -- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ea414c8e0 msgr2=0x7f5ea414bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:02.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.935+0000 7f5ea3e00640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ea414c8e0 0x7f5ea414bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:02.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.935+0000 7f5ea3e00640 1 -- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea414ccc0 msgr2=0x7f5ea414c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.935+0000 7f5ea3e00640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea414ccc0 0x7f5ea414c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.935+0000 7f5ea3e00640 1 -- 172.21.15.31:0/1147409045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea414f110 con 0x7f5ea40c9a10 2024-09-11T00:37:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.935+0000 7f5ea3e00640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea40c9a10 0x7f5ea40ca0f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400e9f0 tx=0x7f5e9400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.935+0000 7f5ea0c00640 1 -- 172.21.15.31:0/1147409045 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9400cd80 con 0x7f5ea40c9a10 2024-09-11T00:37:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.935+0000 7f5ea0c00640 1 -- 172.21.15.31:0/1147409045 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e94010430 con 0x7f5ea40c9a10 2024-09-11T00:37:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.935+0000 7f5ea9600640 1 -- 172.21.15.31:0/1147409045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ea414f780 con 0x7f5ea40c9a10 2024-09-11T00:37:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.936+0000 7f5ea0c00640 1 -- 172.21.15.31:0/1147409045 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94010840 con 0x7f5ea40c9a10 2024-09-11T00:37:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.936+0000 7f5ea9600640 1 -- 172.21.15.31:0/1147409045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ea414fd40 con 0x7f5ea40c9a10 2024-09-11T00:37:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.937+0000 7f5ea9600640 1 -- 172.21.15.31:0/1147409045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ea4156b00 con 0x7f5ea40c9a10 2024-09-11T00:37:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.938+0000 7f5ea0c00640 1 -- 172.21.15.31:0/1147409045 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5e940109e0 con 0x7f5ea40c9a10 2024-09-11T00:37:02.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.939+0000 7f5ea0c00640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5e80077f70 0x7f5e8007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:02.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.939+0000 7f5ea0c00640 1 -- 172.21.15.31:0/1147409045 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f5e9409afd0 con 0x7f5ea40c9a10 2024-09-11T00:37:02.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.939+0000 7f5e9b400640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5e80077f70 0x7f5e8007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:02.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.940+0000 7f5e9b400640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5e80077f70 0x7f5e8007a430 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c034000 tx=0x7f5e8c0046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:02.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:02.944+0000 7f5ea0c00640 1 -- 172.21.15.31:0/1147409045 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5e940634e0 con 0x7f5ea40c9a10 2024-09-11T00:37:03.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.186+0000 7f5ea9600640 1 -- 172.21.15.31:0/1147409045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ea40c5160 con 0x7f5ea40c9a10 2024-09-11T00:37:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.187+0000 7f5ea0c00640 1 -- 172.21.15.31:0/1147409045 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7f5e94062de0 con 0x7f5ea40c9a10 2024-09-11T00:37:03.189 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:03.189 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:37:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:37:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.189+0000 7f5ea9600640 1 -- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5e80077f70 msgr2=0x7f5e8007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.189+0000 7f5ea9600640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5e80077f70 0x7f5e8007a430 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c034000 tx=0x7f5e8c0046c0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.189+0000 7f5ea9600640 1 -- 172.21.15.31:0/1147409045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea40c9a10 msgr2=0x7f5ea40ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.189+0000 7f5ea9600640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea40c9a10 0x7f5ea40ca0f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400e9f0 tx=0x7f5e9400eeb0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.190+0000 7f5ea9600640 1 -- 172.21.15.31:0/1147409045 shutdown_connections 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.190+0000 7f5ea9600640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5e80077f70 0x7f5e8007a430 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.190+0000 7f5ea9600640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea414ccc0 0x7f5ea414c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.190+0000 7f5ea9600640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ea414c8e0 0x7f5ea414bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.190+0000 7f5ea9600640 1 --2- 172.21.15.31:0/1147409045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea40c9a10 0x7f5ea40ca0f0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.190+0000 7f5ea9600640 1 -- 172.21.15.31:0/1147409045 >> 172.21.15.31:0/1147409045 conn(0x7f5ea4068ee0 msgr2=0x7f5ea4159480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.190+0000 7f5ea9600640 1 -- 172.21.15.31:0/1147409045 shutdown_connections 2024-09-11T00:37:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.190+0000 7f5ea9600640 1 -- 172.21.15.31:0/1147409045 wait complete. 2024-09-11T00:37:03.209 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:37:03.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.276+0000 7f3e98e00640 1 -- 172.21.15.31:0/2890167842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e780051c0 msgr2=0x7f3e7800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:03.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.276+0000 7f3e98e00640 1 --2- 172.21.15.31:0/2890167842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e780051c0 0x7f3e7800b9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3e94151c90 tx=0x7f3e78005830 comp rx=0 tx=0).stop 2024-09-11T00:37:03.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.276+0000 7f3e92800640 1 -- 172.21.15.31:0/2890167842 reap_dead start 2024-09-11T00:37:03.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.276+0000 7f3e98e00640 1 -- 172.21.15.31:0/2890167842 shutdown_connections 2024-09-11T00:37:03.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.276+0000 7f3e98e00640 1 -- 172.21.15.31:0/2890167842 >> 172.21.15.31:0/2890167842 conn(0x7f3e940a9750 msgr2=0x7f3e940a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:03.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.276+0000 7f3e98e00640 1 -- 172.21.15.31:0/2890167842 shutdown_connections 2024-09-11T00:37:03.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.276+0000 7f3e98e00640 1 -- 172.21.15.31:0/2890167842 wait complete. 2024-09-11T00:37:03.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.277+0000 7f3e98e00640 1 Processor -- start 2024-09-11T00:37:03.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.277+0000 7f3e98e00640 1 -- start start 2024-09-11T00:37:03.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.278+0000 7f3e98e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780051c0 0x7f3e940c7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:03.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.278+0000 7f3e98e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940c4c10 0x7f3e940c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:03.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.278+0000 7f3e98e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e940c3600 0x7f3e94146c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:03.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.278+0000 7f3e98e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3e94074200 con 0x7f3e940c3600 2024-09-11T00:37:03.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.278+0000 7f3e98e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3e94074080 con 0x7f3e780051c0 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.278+0000 7f3e98e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3e94072860 con 0x7f3e940c4c10 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e92800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e940c3600 0x7f3e94146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e92800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e940c3600 0x7f3e94146c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46664/0 (socket says 172.21.15.31:46664) 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e92800640 1 -- 172.21.15.31:0/2449152149 learned_addr learned my addr 172.21.15.31:0/2449152149 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e91e00640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780051c0 0x7f3e940c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e91400640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940c4c10 0x7f3e940c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e92800640 1 -- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940c4c10 msgr2=0x7f3e940c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e92800640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940c4c10 0x7f3e940c30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e92800640 1 -- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780051c0 msgr2=0x7f3e940c7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e92800640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780051c0 0x7f3e940c7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e92800640 1 -- 172.21.15.31:0/2449152149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e940ada30 con 0x7f3e940c3600 2024-09-11T00:37:03.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e91e00640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780051c0 0x7f3e940c7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:37:03.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e92800640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e940c3600 0x7f3e94146c40 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c005fd0 tx=0x7f3e8c02bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:03.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e8a000640 1 -- 172.21.15.31:0/2449152149 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c031070 con 0x7f3e940c3600 2024-09-11T00:37:03.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e8a000640 1 -- 172.21.15.31:0/2449152149 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e8c035620 con 0x7f3e940c3600 2024-09-11T00:37:03.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e8a000640 1 -- 172.21.15.31:0/2449152149 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c035a50 con 0x7f3e940c3600 2024-09-11T00:37:03.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.279+0000 7f3e98e00640 1 -- 172.21.15.31:0/2449152149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e941ed220 con 0x7f3e940c3600 2024-09-11T00:37:03.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.280+0000 7f3e98e00640 1 -- 172.21.15.31:0/2449152149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e941ed4e0 con 0x7f3e940c3600 2024-09-11T00:37:03.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.281+0000 7f3e98e00640 1 -- 172.21.15.31:0/2449152149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e94073630 con 0x7f3e940c3600 2024-09-11T00:37:03.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.282+0000 7f3e8a000640 1 -- 172.21.15.31:0/2449152149 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3e8c035bf0 con 0x7f3e940c3600 2024-09-11T00:37:03.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.283+0000 7f3e8a000640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e74077f70 0x7f3e7407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:03.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.283+0000 7f3e8a000640 1 -- 172.21.15.31:0/2449152149 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f3e8c0ba5f0 con 0x7f3e940c3600 2024-09-11T00:37:03.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.283+0000 7f3e91e00640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e74077f70 0x7f3e7407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:03.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.284+0000 7f3e91e00640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e74077f70 0x7f3e7407a430 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3e840048c0 tx=0x7f3e840015a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:03.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.287+0000 7f3e8a000640 1 -- 172.21.15.31:0/2449152149 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3e94073630 con 0x7f3e940c3600 2024-09-11T00:37:03.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.535+0000 7f3e98e00640 1 -- 172.21.15.31:0/2449152149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e94154000 con 0x7f3e940c3600 2024-09-11T00:37:03.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.536+0000 7f3e8a000640 1 -- 172.21.15.31:0/2449152149 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7f3e8c08b150 con 0x7f3e940c3600 2024-09-11T00:37:03.538 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:03.538 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:37:03.538 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:37:03.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.538+0000 7f3e98e00640 1 -- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e74077f70 msgr2=0x7f3e7407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.538+0000 7f3e98e00640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e74077f70 0x7f3e7407a430 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3e840048c0 tx=0x7f3e840015a0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.538+0000 7f3e98e00640 1 -- 172.21.15.31:0/2449152149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e940c3600 msgr2=0x7f3e94146c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.538+0000 7f3e98e00640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e940c3600 0x7f3e94146c40 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c005fd0 tx=0x7f3e8c02bd60 comp rx=0 tx=0).stop 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.538+0000 7f3e98e00640 1 -- 172.21.15.31:0/2449152149 shutdown_connections 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.538+0000 7f3e98e00640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e74077f70 0x7f3e7407a430 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.538+0000 7f3e98e00640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e940c3600 0x7f3e94146c40 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.538+0000 7f3e98e00640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e940c4c10 0x7f3e940c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.538+0000 7f3e98e00640 1 --2- 172.21.15.31:0/2449152149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780051c0 0x7f3e940c7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.538+0000 7f3e98e00640 1 -- 172.21.15.31:0/2449152149 >> 172.21.15.31:0/2449152149 conn(0x7f3e940a9750 msgr2=0x7f3e9414ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.539+0000 7f3e98e00640 1 -- 172.21.15.31:0/2449152149 shutdown_connections 2024-09-11T00:37:03.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:03.539+0000 7f3e98e00640 1 -- 172.21.15.31:0/2449152149 wait complete. 2024-09-11T00:37:03.556 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:37:05.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:05.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.381+0000 7f8e79400640 1 -- 172.21.15.31:0/923606828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e58004020 msgr2=0x7f8e58004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:05.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.381+0000 7f8e79400640 1 --2- 172.21.15.31:0/923606828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e58004020 0x7f8e58004800 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8e74154fd0 tx=0x7f8e58005830 comp rx=0 tx=0).stop 2024-09-11T00:37:05.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.381+0000 7f8e73200640 1 -- 172.21.15.31:0/923606828 reap_dead start 2024-09-11T00:37:05.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.381+0000 7f8e79400640 1 -- 172.21.15.31:0/923606828 shutdown_connections 2024-09-11T00:37:05.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.381+0000 7f8e79400640 1 -- 172.21.15.31:0/923606828 >> 172.21.15.31:0/923606828 conn(0x7f8e740a9750 msgr2=0x7f8e740a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:05.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.381+0000 7f8e79400640 1 -- 172.21.15.31:0/923606828 shutdown_connections 2024-09-11T00:37:05.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.381+0000 7f8e79400640 1 -- 172.21.15.31:0/923606828 wait complete. 2024-09-11T00:37:05.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.382+0000 7f8e79400640 1 Processor -- start 2024-09-11T00:37:05.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.383+0000 7f8e79400640 1 -- start start 2024-09-11T00:37:05.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.383+0000 7f8e79400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e58004020 0x7f8e740c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:05.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.383+0000 7f8e79400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8e740c3030 0x7f8e741ea470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:05.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e79400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e740c3c30 0x7f8e741ea9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:05.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e79400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8e74074200 con 0x7f8e740c3c30 2024-09-11T00:37:05.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e79400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8e74074080 con 0x7f8e58004020 2024-09-11T00:37:05.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e79400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8e74072860 con 0x7f8e740c3030 2024-09-11T00:37:05.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e73200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e740c3c30 0x7f8e741ea9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e73200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e740c3c30 0x7f8e741ea9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46676/0 (socket says 172.21.15.31:46676) 2024-09-11T00:37:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e73200640 1 -- 172.21.15.31:0/1321142785 learned_addr learned my addr 172.21.15.31:0/1321142785 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e6be00640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8e740c3030 0x7f8e741ea470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e72800640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e58004020 0x7f8e740c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e73200640 1 -- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8e740c3030 msgr2=0x7f8e741ea470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e73200640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8e740c3030 0x7f8e741ea470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e73200640 1 -- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e58004020 msgr2=0x7f8e740c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e73200640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e58004020 0x7f8e740c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e73200640 1 -- 172.21.15.31:0/1321142785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e740c37a0 con 0x7f8e740c3c30 2024-09-11T00:37:05.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.384+0000 7f8e73200640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e740c3c30 0x7f8e741ea9b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8e740c5500 tx=0x7f8e6c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:05.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.385+0000 7f8e6aa00640 1 -- 172.21.15.31:0/1321142785 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c00c9a0 con 0x7f8e740c3c30 2024-09-11T00:37:05.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.385+0000 7f8e6aa00640 1 -- 172.21.15.31:0/1321142785 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e6c017070 con 0x7f8e740c3c30 2024-09-11T00:37:05.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.385+0000 7f8e6aa00640 1 -- 172.21.15.31:0/1321142785 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c011460 con 0x7f8e740c3c30 2024-09-11T00:37:05.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.385+0000 7f8e79400640 1 -- 172.21.15.31:0/1321142785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e740c4320 con 0x7f8e740c3c30 2024-09-11T00:37:05.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.385+0000 7f8e79400640 1 -- 172.21.15.31:0/1321142785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e740c5260 con 0x7f8e740c3c30 2024-09-11T00:37:05.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.386+0000 7f8e79400640 1 -- 172.21.15.31:0/1321142785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e741511a0 con 0x7f8e740c3c30 2024-09-11T00:37:05.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.387+0000 7f8e6aa00640 1 -- 172.21.15.31:0/1321142785 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8e6c011600 con 0x7f8e740c3c30 2024-09-11T00:37:05.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.388+0000 7f8e6aa00640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e50077f70 0x7f8e5007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:05.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.389+0000 7f8e6aa00640 1 -- 172.21.15.31:0/1321142785 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f8e6c09a8b0 con 0x7f8e740c3c30 2024-09-11T00:37:05.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.389+0000 7f8e72800640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e50077f70 0x7f8e5007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:05.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.390+0000 7f8e72800640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e50077f70 0x7f8e5007a430 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8e741df260 tx=0x7f8e64005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:05.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.393+0000 7f8e6aa00640 1 -- 172.21.15.31:0/1321142785 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8e6c062cf0 con 0x7f8e740c3c30 2024-09-11T00:37:05.557 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:05.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.626+0000 7f8e79400640 1 -- 172.21.15.31:0/1321142785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e7414bea0 con 0x7f8e740c3c30 2024-09-11T00:37:05.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.629+0000 7f8e6aa00640 1 -- 172.21.15.31:0/1321142785 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7f8e6c06b410 con 0x7f8e740c3c30 2024-09-11T00:37:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:05.631 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:37:05.632 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:37:05.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.631+0000 7f8e79400640 1 -- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e50077f70 msgr2=0x7f8e5007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:05.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.631+0000 7f8e79400640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e50077f70 0x7f8e5007a430 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8e741df260 tx=0x7f8e64005f90 comp rx=0 tx=0).stop 2024-09-11T00:37:05.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.631+0000 7f8e79400640 1 -- 172.21.15.31:0/1321142785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e740c3c30 msgr2=0x7f8e741ea9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:05.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.631+0000 7f8e79400640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e740c3c30 0x7f8e741ea9b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8e740c5500 tx=0x7f8e6c00ed10 comp rx=0 tx=0).stop 2024-09-11T00:37:05.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.631+0000 7f8e79400640 1 -- 172.21.15.31:0/1321142785 shutdown_connections 2024-09-11T00:37:05.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.631+0000 7f8e79400640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e50077f70 0x7f8e5007a430 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.631+0000 7f8e79400640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e740c3c30 0x7f8e741ea9b0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.631+0000 7f8e79400640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8e740c3030 0x7f8e741ea470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.632+0000 7f8e79400640 1 --2- 172.21.15.31:0/1321142785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e58004020 0x7f8e740c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.632+0000 7f8e79400640 1 -- 172.21.15.31:0/1321142785 >> 172.21.15.31:0/1321142785 conn(0x7f8e740a9750 msgr2=0x7f8e74152f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:05.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.632+0000 7f8e79400640 1 -- 172.21.15.31:0/1321142785 shutdown_connections 2024-09-11T00:37:05.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.632+0000 7f8e79400640 1 -- 172.21.15.31:0/1321142785 wait complete. 2024-09-11T00:37:05.650 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:37:05.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.717+0000 7f8891800640 1 -- 172.21.15.31:0/3327431407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f888c074080 msgr2=0x7f888c151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:05.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.717+0000 7f8891800640 1 --2- 172.21.15.31:0/3327431407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f888c074080 0x7f888c151700 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f888c070700 tx=0x7f888402a940 comp rx=0 tx=0).stop 2024-09-11T00:37:05.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.717+0000 7f8891800640 1 -- 172.21.15.31:0/3327431407 shutdown_connections 2024-09-11T00:37:05.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.717+0000 7f8891800640 1 --2- 172.21.15.31:0/3327431407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f888c074080 0x7f888c151700 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.717+0000 7f8891800640 1 --2- 172.21.15.31:0/3327431407 >> v2:172.21.15.204:3300/0 conn(0x7f888c072480 0x7f888c1511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.717+0000 7f8891800640 1 -- 172.21.15.31:0/3327431407 >> 172.21.15.31:0/3327431407 conn(0x7f888c0a9750 msgr2=0x7f888c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:05.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.717+0000 7f8891800640 1 -- 172.21.15.31:0/3327431407 shutdown_connections 2024-09-11T00:37:05.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.717+0000 7f8891800640 1 -- 172.21.15.31:0/3327431407 wait complete. 2024-09-11T00:37:05.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.718+0000 7f8891800640 1 Processor -- start 2024-09-11T00:37:05.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.719+0000 7f8891800640 1 -- start start 2024-09-11T00:37:05.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.719+0000 7f8891800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f888c072480 0x7f888c1edca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:05.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.719+0000 7f8891800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f888c074080 0x7f888c14eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:05.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.719+0000 7f8891800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f888c0c2410 0x7f888c14fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:05.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.719+0000 7f8891800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f888c0729e0 con 0x7f888c074080 2024-09-11T00:37:05.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.719+0000 7f8891800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f888c072860 con 0x7f888c0c2410 2024-09-11T00:37:05.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.719+0000 7f8891800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f888c153f20 con 0x7f888c072480 2024-09-11T00:37:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.720+0000 7f8883e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f888c074080 0x7f888c14eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.720+0000 7f8883e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f888c074080 0x7f888c14eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46696/0 (socket says 172.21.15.31:46696) 2024-09-11T00:37:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.720+0000 7f8883e00640 1 -- 172.21.15.31:0/2745391602 learned_addr learned my addr 172.21.15.31:0/2745391602 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.720+0000 7f8883e00640 1 -- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f888c072480 msgr2=0x7f888c1edca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.720+0000 7f8883e00640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f888c072480 0x7f888c1edca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.720+0000 7f8883e00640 1 -- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f888c0c2410 msgr2=0x7f888c14fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.720+0000 7f8883e00640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f888c0c2410 0x7f888c14fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.720+0000 7f8883e00640 1 -- 172.21.15.31:0/2745391602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888c0ada10 con 0x7f888c074080 2024-09-11T00:37:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.720+0000 7f8883e00640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f888c074080 0x7f888c14eec0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f887400c3c0 tx=0x7f887400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.721+0000 7f8882a00640 1 -- 172.21.15.31:0/2745391602 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874017070 con 0x7f888c074080 2024-09-11T00:37:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.721+0000 7f8882a00640 1 -- 172.21.15.31:0/2745391602 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88740035f0 con 0x7f888c074080 2024-09-11T00:37:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.721+0000 7f8882a00640 1 -- 172.21.15.31:0/2745391602 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874003a80 con 0x7f888c074080 2024-09-11T00:37:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.721+0000 7f8891800640 1 -- 172.21.15.31:0/2745391602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f888c0ad770 con 0x7f888c074080 2024-09-11T00:37:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.721+0000 7f8891800640 1 -- 172.21.15.31:0/2745391602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f888c1ed780 con 0x7f888c074080 2024-09-11T00:37:05.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.722+0000 7f8891800640 1 -- 172.21.15.31:0/2745391602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f888c0a7f20 con 0x7f888c074080 2024-09-11T00:37:05.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.723+0000 7f8882a00640 1 -- 172.21.15.31:0/2745391602 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8874003c20 con 0x7f888c074080 2024-09-11T00:37:05.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.724+0000 7f8882a00640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8868077fa0 0x7f886807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:05.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.724+0000 7f8882a00640 1 -- 172.21.15.31:0/2745391602 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f8874099990 con 0x7f888c074080 2024-09-11T00:37:05.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.724+0000 7f888aa00640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8868077fa0 0x7f886807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:05.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.725+0000 7f888aa00640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8868077fa0 0x7f886807a460 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f887c005fd0 tx=0x7f887c005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:05.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.728+0000 7f8882a00640 1 -- 172.21.15.31:0/2745391602 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f888c0a7f20 con 0x7f888c074080 2024-09-11T00:37:05.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.960+0000 7f8891800640 1 -- 172.21.15.31:0/2745391602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f888c1511c0 con 0x7f888c074080 2024-09-11T00:37:05.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.960+0000 7f8882a00640 1 -- 172.21.15.31:0/2745391602 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7f8874061e60 con 0x7f888c074080 2024-09-11T00:37:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:05.963 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:37:05.963 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:37:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 -- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8868077fa0 msgr2=0x7f886807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:05.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8868077fa0 0x7f886807a460 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f887c005fd0 tx=0x7f887c005850 comp rx=0 tx=0).stop 2024-09-11T00:37:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 -- 172.21.15.31:0/2745391602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f888c074080 msgr2=0x7f888c14eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f888c074080 0x7f888c14eec0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f887400c3c0 tx=0x7f887400c880 comp rx=0 tx=0).stop 2024-09-11T00:37:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 -- 172.21.15.31:0/2745391602 shutdown_connections 2024-09-11T00:37:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8868077fa0 0x7f886807a460 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f888c0c2410 0x7f888c14fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f888c074080 0x7f888c14eec0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 --2- 172.21.15.31:0/2745391602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f888c072480 0x7f888c1edca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 -- 172.21.15.31:0/2745391602 >> 172.21.15.31:0/2745391602 conn(0x7f888c0a9750 msgr2=0x7f888c154350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 -- 172.21.15.31:0/2745391602 shutdown_connections 2024-09-11T00:37:05.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:05.963+0000 7f8891800640 1 -- 172.21.15.31:0/2745391602 wait complete. 2024-09-11T00:37:05.981 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:37:07.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:07.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.819+0000 7fa078200640 1 -- 172.21.15.31:0/967481477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070159670 msgr2=0x7fa070159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:07.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.819+0000 7fa078200640 1 --2- 172.21.15.31:0/967481477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070159670 0x7fa070159aa0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa068005fd0 tx=0x7fa06802a8b0 comp rx=0 tx=0).stop 2024-09-11T00:37:07.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.819+0000 7fa078200640 1 -- 172.21.15.31:0/967481477 shutdown_connections 2024-09-11T00:37:07.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.819+0000 7fa078200640 1 --2- 172.21.15.31:0/967481477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070159670 0x7fa070159aa0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:07.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.819+0000 7fa078200640 1 -- 172.21.15.31:0/967481477 >> 172.21.15.31:0/967481477 conn(0x7fa0700a9750 msgr2=0x7fa0700a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:07.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.819+0000 7fa078200640 1 -- 172.21.15.31:0/967481477 shutdown_connections 2024-09-11T00:37:07.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.820+0000 7fa078200640 1 -- 172.21.15.31:0/967481477 wait complete. 2024-09-11T00:37:07.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.821+0000 7fa078200640 1 Processor -- start 2024-09-11T00:37:07.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.821+0000 7fa078200640 1 -- start start 2024-09-11T00:37:07.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa078200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0700c5b10 0x7fa0700c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:07.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa078200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa070156ad0 0x7fa070151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:07.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa078200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070159670 0x7fa07014abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:07.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa078200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa0700c7320 con 0x7fa070159670 2024-09-11T00:37:07.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa078200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa0700c3950 con 0x7fa070156ad0 2024-09-11T00:37:07.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa078200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa070153b90 con 0x7fa0700c5b10 2024-09-11T00:37:07.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa076200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070159670 0x7fa07014abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:07.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa076200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070159670 0x7fa07014abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46700/0 (socket says 172.21.15.31:46700) 2024-09-11T00:37:07.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa076200640 1 -- 172.21.15.31:0/3463203289 learned_addr learned my addr 172.21.15.31:0/3463203289 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:07.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa075800640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0700c5b10 0x7fa0700c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:07.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.822+0000 7fa074e00640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa070156ad0 0x7fa070151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:07.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.823+0000 7fa076200640 1 -- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0700c5b10 msgr2=0x7fa0700c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:07.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.823+0000 7fa076200640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0700c5b10 0x7fa0700c45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:07.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.823+0000 7fa076200640 1 -- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa070156ad0 msgr2=0x7fa070151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:07.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.823+0000 7fa076200640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa070156ad0 0x7fa070151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:07.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.823+0000 7fa076200640 1 -- 172.21.15.31:0/3463203289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07014cff0 con 0x7fa070159670 2024-09-11T00:37:07.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.823+0000 7fa076200640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070159670 0x7fa07014abf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa068009b30 tx=0x7fa068003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:07.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.823+0000 7fa06d600640 1 -- 172.21.15.31:0/3463203289 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068031b90 con 0x7fa070159670 2024-09-11T00:37:07.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.823+0000 7fa06d600640 1 -- 172.21.15.31:0/3463203289 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa068031e20 con 0x7fa070159670 2024-09-11T00:37:07.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.823+0000 7fa06d600640 1 -- 172.21.15.31:0/3463203289 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068033330 con 0x7fa070159670 2024-09-11T00:37:07.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.823+0000 7fa078200640 1 -- 172.21.15.31:0/3463203289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0701515d0 con 0x7fa070159670 2024-09-11T00:37:07.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.824+0000 7fa078200640 1 -- 172.21.15.31:0/3463203289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0701f0180 con 0x7fa070159670 2024-09-11T00:37:07.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.825+0000 7fa078200640 1 -- 172.21.15.31:0/3463203289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa070077240 con 0x7fa070159670 2024-09-11T00:37:07.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.826+0000 7fa06d600640 1 -- 172.21.15.31:0/3463203289 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa0680334d0 con 0x7fa070159670 2024-09-11T00:37:07.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.827+0000 7fa06d600640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa050077fa0 0x7fa05007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:07.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.827+0000 7fa06d600640 1 -- 172.21.15.31:0/3463203289 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7fa0680bb7a0 con 0x7fa070159670 2024-09-11T00:37:07.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.827+0000 7fa075800640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa050077fa0 0x7fa05007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:07.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.828+0000 7fa075800640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa050077fa0 0x7fa05007a460 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa064004f60 tx=0x7fa064007e30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:07.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:07.831+0000 7fa06d600640 1 -- 172.21.15.31:0/3463203289 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa070077240 con 0x7fa070159670 2024-09-11T00:37:07.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:08.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.064+0000 7fa078200640 1 -- 172.21.15.31:0/3463203289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0700c5ef0 con 0x7fa070159670 2024-09-11T00:37:08.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.065+0000 7fa06d600640 1 -- 172.21.15.31:0/3463203289 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7fa06803f030 con 0x7fa070159670 2024-09-11T00:37:08.067 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:08.067 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:37:08.067 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:37:08.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 -- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa050077fa0 msgr2=0x7fa05007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:08.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa050077fa0 0x7fa05007a460 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa064004f60 tx=0x7fa064007e30 comp rx=0 tx=0).stop 2024-09-11T00:37:08.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 -- 172.21.15.31:0/3463203289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070159670 msgr2=0x7fa07014abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:08.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070159670 0x7fa07014abf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa068009b30 tx=0x7fa068003ba0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 -- 172.21.15.31:0/3463203289 shutdown_connections 2024-09-11T00:37:08.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa050077fa0 0x7fa05007a460 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa070159670 0x7fa07014abf0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa070156ad0 0x7fa070151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 --2- 172.21.15.31:0/3463203289 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa0700c5b10 0x7fa0700c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 -- 172.21.15.31:0/3463203289 >> 172.21.15.31:0/3463203289 conn(0x7fa0700a9750 msgr2=0x7fa07006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:08.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 -- 172.21.15.31:0/3463203289 shutdown_connections 2024-09-11T00:37:08.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.067+0000 7fa078200640 1 -- 172.21.15.31:0/3463203289 wait complete. 2024-09-11T00:37:08.085 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:37:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.143+0000 7fbacde00640 1 -- 172.21.15.31:0/3280400838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbab0004800 msgr2=0x7fbab0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.143+0000 7fbacde00640 1 --2- 172.21.15.31:0/3280400838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbab0004800 0x7fbab0004c80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbab4006f60 tx=0x7fbab0005830 comp rx=0 tx=0).stop 2024-09-11T00:37:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.143+0000 7fbacde00640 1 -- 172.21.15.31:0/3280400838 shutdown_connections 2024-09-11T00:37:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.143+0000 7fbacde00640 1 --2- 172.21.15.31:0/3280400838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbab0004800 0x7fbab0004c80 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.143+0000 7fbacde00640 1 -- 172.21.15.31:0/3280400838 >> 172.21.15.31:0/3280400838 conn(0x7fbac806caf0 msgr2=0x7fbac806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:08.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.143+0000 7fbacde00640 1 -- 172.21.15.31:0/3280400838 shutdown_connections 2024-09-11T00:37:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.143+0000 7fbacde00640 1 -- 172.21.15.31:0/3280400838 wait complete. 2024-09-11T00:37:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.144+0000 7fbacde00640 1 Processor -- start 2024-09-11T00:37:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.145+0000 7fbacde00640 1 -- start start 2024-09-11T00:37:08.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.145+0000 7fbacde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbab0004800 0x7fbac81e1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:08.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.145+0000 7fbacde00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbac8073fd0 0x7fbac81eb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.145+0000 7fbacde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbac8074840 0x7fbac80aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.145+0000 7fbacde00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbac80c5e80 con 0x7fbac8074840 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.145+0000 7fbacde00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbac80c3160 con 0x7fbab0004800 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.145+0000 7fbacde00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbac80c6000 con 0x7fbac8073fd0 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbacd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbac8074840 0x7fbac80aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbacd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbac8074840 0x7fbac80aaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51894/0 (socket says 172.21.15.31:51894) 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbacd400640 1 -- 172.21.15.31:0/1748456392 learned_addr learned my addr 172.21.15.31:0/1748456392 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbac7e00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbac8073fd0 0x7fbac81eb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbacca00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbab0004800 0x7fbac81e1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbacd400640 1 -- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbac8073fd0 msgr2=0x7fbac81eb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbacd400640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbac8073fd0 0x7fbac81eb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbacd400640 1 -- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbab0004800 msgr2=0x7fbac81e1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:08.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbacd400640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbab0004800 0x7fbac81e1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbacd400640 1 -- 172.21.15.31:0/1748456392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac80abba0 con 0x7fbac8074840 2024-09-11T00:37:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.146+0000 7fbacd400640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbac8074840 0x7fbac80aaea0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fbac80c3870 tx=0x7fbac000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.147+0000 7fbac4c00640 1 -- 172.21.15.31:0/1748456392 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0003d80 con 0x7fbac8074840 2024-09-11T00:37:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.147+0000 7fbac4c00640 1 -- 172.21.15.31:0/1748456392 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbac0004b20 con 0x7fbac8074840 2024-09-11T00:37:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.147+0000 7fbac4c00640 1 -- 172.21.15.31:0/1748456392 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0004050 con 0x7fbac8074840 2024-09-11T00:37:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.147+0000 7fbacde00640 1 -- 172.21.15.31:0/1748456392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac80ac7b0 con 0x7fbac8074840 2024-09-11T00:37:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.147+0000 7fbacde00640 1 -- 172.21.15.31:0/1748456392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac8155a30 con 0x7fbac8074840 2024-09-11T00:37:08.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.148+0000 7fbacde00640 1 -- 172.21.15.31:0/1748456392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac80c7150 con 0x7fbac8074840 2024-09-11T00:37:08.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.149+0000 7fbac4c00640 1 -- 172.21.15.31:0/1748456392 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbac0018070 con 0x7fbac8074840 2024-09-11T00:37:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.150+0000 7fbac4c00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbaa8077fa0 0x7fbaa807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.150+0000 7fbac4c00640 1 -- 172.21.15.31:0/1748456392 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7fbac009b290 con 0x7fbac8074840 2024-09-11T00:37:08.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.151+0000 7fbacca00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbaa8077fa0 0x7fbaa807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:08.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.151+0000 7fbacca00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbaa8077fa0 0x7fbaa807a460 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbabc000c00 tx=0x7fbabc006b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:08.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.155+0000 7fbac4c00640 1 -- 172.21.15.31:0/1748456392 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbac0063760 con 0x7fbac8074840 2024-09-11T00:37:08.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.385+0000 7fbacde00640 1 -- 172.21.15.31:0/1748456392 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbac81e23e0 con 0x7fbac8074840 2024-09-11T00:37:08.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.385+0000 7fbac4c00640 1 -- 172.21.15.31:0/1748456392 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6538 (secure 0 0 0) 0x7fbac0063060 con 0x7fbac8074840 2024-09-11T00:37:08.387 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:08.387 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":51,"btime":"2024-09-11T00:36:06:350514+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5449,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:35:57.333729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/4011734390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4011734390},{"type":"v1","addr":"172.21.15.31:6835","nonce":4011734390}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-09-11T00:37:08.388 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 51 2024-09-11T00:37:08.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 -- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbaa8077fa0 msgr2=0x7fbaa807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:08.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbaa8077fa0 0x7fbaa807a460 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbabc000c00 tx=0x7fbabc006b70 comp rx=0 tx=0).stop 2024-09-11T00:37:08.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 -- 172.21.15.31:0/1748456392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbac8074840 msgr2=0x7fbac80aaea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:08.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbac8074840 0x7fbac80aaea0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fbac80c3870 tx=0x7fbac000f790 comp rx=0 tx=0).stop 2024-09-11T00:37:08.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 -- 172.21.15.31:0/1748456392 shutdown_connections 2024-09-11T00:37:08.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbaa8077fa0 0x7fbaa807a460 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbac8074840 0x7fbac80aaea0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbac8073fd0 0x7fbac81eb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 --2- 172.21.15.31:0/1748456392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbab0004800 0x7fbac81e1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:08.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 -- 172.21.15.31:0/1748456392 >> 172.21.15.31:0/1748456392 conn(0x7fbac806caf0 msgr2=0x7fbac8152060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:08.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 -- 172.21.15.31:0/1748456392 shutdown_connections 2024-09-11T00:37:08.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:08.388+0000 7fbacde00640 1 -- 172.21.15.31:0/1748456392 wait complete. 2024-09-11T00:37:08.406 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:37:10.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:10.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.256+0000 7fe918800640 1 -- 172.21.15.31:0/658853689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe910072480 msgr2=0x7fe9101511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.256+0000 7fe918800640 1 --2- 172.21.15.31:0/658853689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe910072480 0x7fe9101511a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe910075260 tx=0x7fe90802a920 comp rx=0 tx=0).stop 2024-09-11T00:37:10.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.256+0000 7fe918800640 1 -- 172.21.15.31:0/658853689 shutdown_connections 2024-09-11T00:37:10.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.256+0000 7fe918800640 1 --2- 172.21.15.31:0/658853689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe910072480 0x7fe9101511a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.256+0000 7fe918800640 1 -- 172.21.15.31:0/658853689 >> 172.21.15.31:0/658853689 conn(0x7fe9100a9750 msgr2=0x7fe9100a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:10.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.256+0000 7fe918800640 1 -- 172.21.15.31:0/658853689 shutdown_connections 2024-09-11T00:37:10.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.256+0000 7fe918800640 1 -- 172.21.15.31:0/658853689 wait complete. 2024-09-11T00:37:10.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.258+0000 7fe918800640 1 Processor -- start 2024-09-11T00:37:10.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.258+0000 7fe918800640 1 -- start start 2024-09-11T00:37:10.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe918800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910072480 0x7fe9101efce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe918800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9100c23a0 0x7fe910157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe918800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe910153f20 0x7fe910148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe918800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe910074200 con 0x7fe910072480 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe918800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe910074080 con 0x7fe910153f20 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe918800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe910072860 con 0x7fe9100c23a0 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe915e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910072480 0x7fe9101efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe915e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910072480 0x7fe9101efce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51900/0 (socket says 172.21.15.31:51900) 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe915e00640 1 -- 172.21.15.31:0/859375720 learned_addr learned my addr 172.21.15.31:0/859375720 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe915400640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9100c23a0 0x7fe910157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.259+0000 7fe916800640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe910153f20 0x7fe910148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe915e00640 1 -- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9100c23a0 msgr2=0x7fe910157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe915e00640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9100c23a0 0x7fe910157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe915e00640 1 -- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe910153f20 msgr2=0x7fe910148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe915e00640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe910153f20 0x7fe910148b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe915e00640 1 -- 172.21.15.31:0/859375720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9101558c0 con 0x7fe910072480 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe916800640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe910153f20 0x7fe910148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe915e00640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910072480 0x7fe9101efce0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe90802af10 tx=0x7fe908003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe906000640 1 -- 172.21.15.31:0/859375720 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908004240 con 0x7fe910072480 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe906000640 1 -- 172.21.15.31:0/859375720 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9080043e0 con 0x7fe910072480 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe918800640 1 -- 172.21.15.31:0/859375720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe910155b60 con 0x7fe910072480 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe906000640 1 -- 172.21.15.31:0/859375720 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908004830 con 0x7fe910072480 2024-09-11T00:37:10.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.260+0000 7fe918800640 1 -- 172.21.15.31:0/859375720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe910156120 con 0x7fe910072480 2024-09-11T00:37:10.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.261+0000 7fe918800640 1 -- 172.21.15.31:0/859375720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe910151960 con 0x7fe910072480 2024-09-11T00:37:10.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.263+0000 7fe906000640 1 -- 172.21.15.31:0/859375720 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe90803a450 con 0x7fe910072480 2024-09-11T00:37:10.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.263+0000 7fe906000640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe8f4077f70 0x7fe8f407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.264+0000 7fe906000640 1 -- 172.21.15.31:0/859375720 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7fe9080bc040 con 0x7fe910072480 2024-09-11T00:37:10.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.264+0000 7fe915400640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe8f4077f70 0x7fe8f407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.264+0000 7fe915400640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe8f4077f70 0x7fe8f407a430 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe900007a50 tx=0x7fe900009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.268+0000 7fe906000640 1 -- 172.21.15.31:0/859375720 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe90802d070 con 0x7fe910072480 2024-09-11T00:37:10.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:10.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.499+0000 7fe918800640 1 -- 172.21.15.31:0/859375720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe91006a170 con 0x7fe910072480 2024-09-11T00:37:10.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.500+0000 7fe906000640 1 -- 172.21.15.31:0/859375720 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5709 (secure 0 0 0) 0x7fe908084560 con 0x7fe910072480 2024-09-11T00:37:10.502 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:10.502 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"btime":"2024-09-11T00:37:10:333389+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:09.586466+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:37:10.502 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.502+0000 7fe918800640 1 -- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe8f4077f70 msgr2=0x7fe8f407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.502+0000 7fe918800640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe8f4077f70 0x7fe8f407a430 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe900007a50 tx=0x7fe900009040 comp rx=0 tx=0).stop 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.502+0000 7fe918800640 1 -- 172.21.15.31:0/859375720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910072480 msgr2=0x7fe9101efce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.502+0000 7fe918800640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910072480 0x7fe9101efce0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe90802af10 tx=0x7fe908003c20 comp rx=0 tx=0).stop 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.503+0000 7fe918800640 1 -- 172.21.15.31:0/859375720 shutdown_connections 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.503+0000 7fe918800640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe8f4077f70 0x7fe8f407a430 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.503+0000 7fe918800640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe910153f20 0x7fe910148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.503+0000 7fe918800640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9100c23a0 0x7fe910157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.503+0000 7fe918800640 1 --2- 172.21.15.31:0/859375720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe910072480 0x7fe9101efce0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.503+0000 7fe918800640 1 -- 172.21.15.31:0/859375720 >> 172.21.15.31:0/859375720 conn(0x7fe9100a9750 msgr2=0x7fe91014afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:10.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.503+0000 7fe918800640 1 -- 172.21.15.31:0/859375720 shutdown_connections 2024-09-11T00:37:10.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.503+0000 7fe918800640 1 -- 172.21.15.31:0/859375720 wait complete. 2024-09-11T00:37:10.520 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:37:10.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.567+0000 7f31ebe00640 1 -- 172.21.15.31:0/3692149228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31ec1567f0 msgr2=0x7f31ec156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.567+0000 7f31ebe00640 1 --2- 172.21.15.31:0/3692149228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31ec1567f0 0x7f31ec156c60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f31ec06dad0 tx=0x7f31e402a790 comp rx=0 tx=0).stop 2024-09-11T00:37:10.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.567+0000 7f31ebe00640 1 -- 172.21.15.31:0/3692149228 shutdown_connections 2024-09-11T00:37:10.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.567+0000 7f31ebe00640 1 --2- 172.21.15.31:0/3692149228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31ec1567f0 0x7f31ec156c60 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.567+0000 7f31ebe00640 1 --2- 172.21.15.31:0/3692149228 >> v2:172.21.15.204:3300/0 conn(0x7f31ec0c6bb0 0x7f31ec06e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.567+0000 7f31ebe00640 1 -- 172.21.15.31:0/3692149228 >> 172.21.15.31:0/3692149228 conn(0x7f31ec06caf0 msgr2=0x7f31ec06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:10.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.567+0000 7f31ebe00640 1 -- 172.21.15.31:0/3692149228 shutdown_connections 2024-09-11T00:37:10.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.567+0000 7f31ebe00640 1 -- 172.21.15.31:0/3692149228 wait complete. 2024-09-11T00:37:10.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.568+0000 7f31ebe00640 1 Processor -- start 2024-09-11T00:37:10.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.569+0000 7f31ebe00640 1 -- start start 2024-09-11T00:37:10.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.569+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31ec0c3280 0x7f31ec153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.569+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31ec0c6bb0 0x7f31ec1ddb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31ebe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31ec1567f0 0x7f31ec1de070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31ebe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f31ec1571a0 con 0x7f31ec0c3280 2024-09-11T00:37:10.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31ebe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f31ec06e5b0 con 0x7f31ec0c6bb0 2024-09-11T00:37:10.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31ebe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f31ec152d80 con 0x7f31ec1567f0 2024-09-11T00:37:10.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31eaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31ec0c3280 0x7f31ec153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31eaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31ec0c3280 0x7f31ec153590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51908/0 (socket says 172.21.15.31:51908) 2024-09-11T00:37:10.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31eaa00640 1 -- 172.21.15.31:0/1606891643 learned_addr learned my addr 172.21.15.31:0/1606891643 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:10.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31e2000640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31ec0c6bb0 0x7f31ec1ddb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31eb400640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31ec1567f0 0x7f31ec1de070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31eaa00640 1 -- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31ec1567f0 msgr2=0x7f31ec1de070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31eaa00640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31ec1567f0 0x7f31ec1de070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31eaa00640 1 -- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31ec0c6bb0 msgr2=0x7f31ec1ddb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31eaa00640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31ec0c6bb0 0x7f31ec1ddb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.570+0000 7f31eaa00640 1 -- 172.21.15.31:0/1606891643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31ec1ec500 con 0x7f31ec0c3280 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.571+0000 7f31eaa00640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31ec0c3280 0x7f31ec153590 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f31dc00e9f0 tx=0x7f31dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.571+0000 7f31e3400640 1 -- 172.21.15.31:0/1606891643 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc00cd80 con 0x7f31ec0c3280 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.571+0000 7f31e3400640 1 -- 172.21.15.31:0/1606891643 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31dc010430 con 0x7f31ec0c3280 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.571+0000 7f31e3400640 1 -- 172.21.15.31:0/1606891643 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc010840 con 0x7f31ec0c3280 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.571+0000 7f31ebe00640 1 -- 172.21.15.31:0/1606891643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31ec1ec7a0 con 0x7f31ec0c3280 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.571+0000 7f31ebe00640 1 -- 172.21.15.31:0/1606891643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31ec1e0e40 con 0x7f31ec0c3280 2024-09-11T00:37:10.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.572+0000 7f31ebe00640 1 -- 172.21.15.31:0/1606891643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31ec0c6f90 con 0x7f31ec0c3280 2024-09-11T00:37:10.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.573+0000 7f31e3400640 1 -- 172.21.15.31:0/1606891643 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f31dc0109e0 con 0x7f31ec0c3280 2024-09-11T00:37:10.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.574+0000 7f31e3400640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31c8077f70 0x7f31c807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.574+0000 7f31e3400640 1 -- 172.21.15.31:0/1606891643 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f31dc09af20 con 0x7f31ec0c3280 2024-09-11T00:37:10.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.574+0000 7f31e2000640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31c8077f70 0x7f31c807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.575+0000 7f31e2000640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31c8077f70 0x7f31c807a430 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f31d4007fd0 tx=0x7f31d4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:10.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.578+0000 7f31e3400640 1 -- 172.21.15.31:0/1606891643 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f31dc062d90 con 0x7f31ec0c3280 2024-09-11T00:37:10.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.810+0000 7f31ebe00640 1 -- 172.21.15.31:0/1606891643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31ec0c4150 con 0x7f31ec0c3280 2024-09-11T00:37:10.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.811+0000 7f31e3400640 1 -- 172.21.15.31:0/1606891643 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5709 (secure 0 0 0) 0x7f31dc0677c0 con 0x7f31ec0c3280 2024-09-11T00:37:10.813 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:10.813 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"btime":"2024-09-11T00:37:10:333389+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:09.586466+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:37:10.813 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2024-09-11T00:37:10.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.813+0000 7f31ebe00640 1 -- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31c8077f70 msgr2=0x7f31c807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.813+0000 7f31ebe00640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31c8077f70 0x7f31c807a430 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f31d4007fd0 tx=0x7f31d4009040 comp rx=0 tx=0).stop 2024-09-11T00:37:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.813+0000 7f31ebe00640 1 -- 172.21.15.31:0/1606891643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31ec0c3280 msgr2=0x7f31ec153590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.813+0000 7f31ebe00640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31ec0c3280 0x7f31ec153590 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f31dc00e9f0 tx=0x7f31dc00eeb0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.813+0000 7f31ebe00640 1 -- 172.21.15.31:0/1606891643 shutdown_connections 2024-09-11T00:37:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.813+0000 7f31ebe00640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31c8077f70 0x7f31c807a430 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.813+0000 7f31ebe00640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f31ec1567f0 0x7f31ec1de070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.813+0000 7f31ebe00640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31ec0c6bb0 0x7f31ec1ddb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.813+0000 7f31ebe00640 1 --2- 172.21.15.31:0/1606891643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31ec0c3280 0x7f31ec153590 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.813+0000 7f31ebe00640 1 -- 172.21.15.31:0/1606891643 >> 172.21.15.31:0/1606891643 conn(0x7f31ec06caf0 msgr2=0x7f31ec0ab9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.814+0000 7f31ebe00640 1 -- 172.21.15.31:0/1606891643 shutdown_connections 2024-09-11T00:37:10.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.814+0000 7f31ebe00640 1 -- 172.21.15.31:0/1606891643 wait complete. 2024-09-11T00:37:10.831 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-11T00:37:10.832 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:37:10.832 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.990+0000 7f4b39000640 1 -- 172.21.15.31:0/2046248875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b340c5aa0 msgr2=0x7f4b340c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.990+0000 7f4b39000640 1 --2- 172.21.15.31:0/2046248875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b340c5aa0 0x7f4b340c7150 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009830 tx=0x7f4b2402a510 comp rx=0 tx=0).stop 2024-09-11T00:37:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.990+0000 7f4b39000640 1 -- 172.21.15.31:0/2046248875 shutdown_connections 2024-09-11T00:37:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.990+0000 7f4b39000640 1 --2- 172.21.15.31:0/2046248875 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b34156350 0x7f4b341567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.990+0000 7f4b39000640 1 --2- 172.21.15.31:0/2046248875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b340c5aa0 0x7f4b340c7150 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.990+0000 7f4b39000640 1 -- 172.21.15.31:0/2046248875 >> 172.21.15.31:0/2046248875 conn(0x7f4b3406caf0 msgr2=0x7f4b3406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.991+0000 7f4b39000640 1 -- 172.21.15.31:0/2046248875 shutdown_connections 2024-09-11T00:37:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.991+0000 7f4b39000640 1 -- 172.21.15.31:0/2046248875 wait complete. 2024-09-11T00:37:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.992+0000 7f4b39000640 1 Processor -- start 2024-09-11T00:37:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.993+0000 7f4b39000640 1 -- start start 2024-09-11T00:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b39000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b340c5aa0 0x7f4b341e09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b39000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b340c6420 0x7f4b341ef100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b39000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b34156350 0x7f4b341ef640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b39000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b340c2e80 con 0x7f4b340c5aa0 2024-09-11T00:37:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b39000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4b340c2ba0 con 0x7f4b34156350 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b39000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4b34156d20 con 0x7f4b340c6420 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b33400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b340c5aa0 0x7f4b341e09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b33400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b340c5aa0 0x7f4b341e09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51920/0 (socket says 172.21.15.31:51920) 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b33400640 1 -- 172.21.15.31:0/2617317231 learned_addr learned my addr 172.21.15.31:0/2617317231 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b32a00640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b340c6420 0x7f4b341ef100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b33e00640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b34156350 0x7f4b341ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b33400640 1 -- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b340c6420 msgr2=0x7f4b341ef100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b33400640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b340c6420 0x7f4b341ef100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b33400640 1 -- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b34156350 msgr2=0x7f4b341ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b33400640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b34156350 0x7f4b341ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b33400640 1 -- 172.21.15.31:0/2617317231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b340728f0 con 0x7f4b340c5aa0 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.994+0000 7f4b33e00640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b34156350 0x7f4b341ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:37:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.995+0000 7f4b33400640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b340c5aa0 0x7f4b341e09c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009950 tx=0x7f4b24032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.995+0000 7f4b2b400640 1 -- 172.21.15.31:0/2617317231 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b24040070 con 0x7f4b340c5aa0 2024-09-11T00:37:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.995+0000 7f4b2b400640 1 -- 172.21.15.31:0/2617317231 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b24003af0 con 0x7f4b340c5aa0 2024-09-11T00:37:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.995+0000 7f4b2b400640 1 -- 172.21.15.31:0/2617317231 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b24045460 con 0x7f4b340c5aa0 2024-09-11T00:37:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.995+0000 7f4b39000640 1 -- 172.21.15.31:0/2617317231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b34072bd0 con 0x7f4b340c5aa0 2024-09-11T00:37:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.996+0000 7f4b39000640 1 -- 172.21.15.31:0/2617317231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b340745e0 con 0x7f4b340c5aa0 2024-09-11T00:37:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.997+0000 7f4b39000640 1 -- 172.21.15.31:0/2617317231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b340c5e80 con 0x7f4b340c5aa0 2024-09-11T00:37:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.998+0000 7f4b2b400640 1 -- 172.21.15.31:0/2617317231 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4b24045600 con 0x7f4b340c5aa0 2024-09-11T00:37:11.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.999+0000 7f4b2b400640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4b14077f70 0x7f4b1407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:11.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:10.999+0000 7f4b2b400640 1 -- 172.21.15.31:0/2617317231 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f4b240bbd10 con 0x7f4b340c5aa0 2024-09-11T00:37:11.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.000+0000 7f4b32a00640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4b14077f70 0x7f4b1407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:11.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.000+0000 7f4b32a00640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4b14077f70 0x7f4b1407a430 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4b20007fd0 tx=0x7f4b20009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:11.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.004+0000 7f4b2b400640 1 -- 172.21.15.31:0/2617317231 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4b24084230 con 0x7f4b340c5aa0 2024-09-11T00:37:11.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.233+0000 7f4b39000640 1 -- 172.21.15.31:0/2617317231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b34151c60 con 0x7f4b340c5aa0 2024-09-11T00:37:11.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.234+0000 7f4b2b400640 1 -- 172.21.15.31:0/2617317231 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5709 (secure 0 0 0) 0x7f4b24083b30 con 0x7f4b340c5aa0 2024-09-11T00:37:11.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:11.237 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":53,"btime":"2024-09-11T00:37:10:333389+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:09.586466+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:37:11.237 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 53 2024-09-11T00:37:11.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.236+0000 7f4b39000640 1 -- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4b14077f70 msgr2=0x7f4b1407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:11.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.236+0000 7f4b39000640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4b14077f70 0x7f4b1407a430 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4b20007fd0 tx=0x7f4b20009040 comp rx=0 tx=0).stop 2024-09-11T00:37:11.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.236+0000 7f4b39000640 1 -- 172.21.15.31:0/2617317231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b340c5aa0 msgr2=0x7f4b341e09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:11.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.236+0000 7f4b39000640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b340c5aa0 0x7f4b341e09c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009950 tx=0x7f4b24032d50 comp rx=0 tx=0).stop 2024-09-11T00:37:11.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.236+0000 7f4b39000640 1 -- 172.21.15.31:0/2617317231 shutdown_connections 2024-09-11T00:37:11.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.236+0000 7f4b39000640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4b14077f70 0x7f4b1407a430 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:11.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.236+0000 7f4b39000640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b34156350 0x7f4b341ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:11.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.236+0000 7f4b39000640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4b340c6420 0x7f4b341ef100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:11.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.236+0000 7f4b39000640 1 --2- 172.21.15.31:0/2617317231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b340c5aa0 0x7f4b341e09c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:11.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.236+0000 7f4b39000640 1 -- 172.21.15.31:0/2617317231 >> 172.21.15.31:0/2617317231 conn(0x7f4b3406caf0 msgr2=0x7f4b340c6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:11.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.237+0000 7f4b39000640 1 -- 172.21.15.31:0/2617317231 shutdown_connections 2024-09-11T00:37:11.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:11.237+0000 7f4b39000640 1 -- 172.21.15.31:0/2617317231 wait complete. 2024-09-11T00:37:11.255 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:37:10:333389+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 53, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5302": { "addr": "172.21.15.204:6835/3444890210", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 3444890210, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 3444890210, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5302, "incarnation": 42, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:35:12.604104+0000", "qdb_cluster": [ 5302 ], "qdb_leader": 5302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5302 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 53, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5449": { "addr": "172.21.15.204:6837/1278135080", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 1278135080, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 1278135080, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5449, "incarnation": 52, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:37:09.586466+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5449 } } } ], "standbys": [ { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:37:12.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:12.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.691+0000 7f0059200640 1 -- 172.21.15.31:0/3835498086 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f005414ccc0 msgr2=0x7f0054156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:12.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.691+0000 7f0059200640 1 --2- 172.21.15.31:0/3835498086 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f005414ccc0 0x7f0054156b00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f003c009860 tx=0x7f003c02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:37:12.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.691+0000 7f0059200640 1 -- 172.21.15.31:0/3835498086 shutdown_connections 2024-09-11T00:37:12.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.691+0000 7f0059200640 1 --2- 172.21.15.31:0/3835498086 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f005414ccc0 0x7f0054156b00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:12.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.691+0000 7f0059200640 1 -- 172.21.15.31:0/3835498086 >> 172.21.15.31:0/3835498086 conn(0x7f0054068ee0 msgr2=0x7f00540cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:12.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.691+0000 7f0059200640 1 -- 172.21.15.31:0/3835498086 shutdown_connections 2024-09-11T00:37:12.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.691+0000 7f0059200640 1 -- 172.21.15.31:0/3835498086 wait complete. 2024-09-11T00:37:12.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.692+0000 7f0059200640 1 Processor -- start 2024-09-11T00:37:12.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.693+0000 7f0059200640 1 -- start start 2024-09-11T00:37:12.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.693+0000 7f0059200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00540c9a10 0x7f00540c18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:12.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.693+0000 7f0059200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f005414c8e0 0x7f00541e1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.693+0000 7f0059200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005414ccc0 0x7f00541e1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.693+0000 7f0059200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f00540c6c90 con 0x7f005414ccc0 2024-09-11T00:37:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.693+0000 7f0059200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f00540c6800 con 0x7f00540c9a10 2024-09-11T00:37:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.693+0000 7f0059200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f00540adca0 con 0x7f005414c8e0 2024-09-11T00:37:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.694+0000 7f0053e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005414ccc0 0x7f00541e1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.694+0000 7f004be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f005414c8e0 0x7f00541e1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.694+0000 7f0053400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00540c9a10 0x7f00540c18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.694+0000 7f004be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f005414c8e0 0x7f00541e1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:38970/0 (socket says 172.21.15.31:38970) 2024-09-11T00:37:12.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.694+0000 7f004be00640 1 -- 172.21.15.31:0/2063255287 learned_addr learned my addr 172.21.15.31:0/2063255287 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.694+0000 7f0053e00640 1 -- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f005414c8e0 msgr2=0x7f00541e1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.694+0000 7f0053e00640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f005414c8e0 0x7f00541e1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.694+0000 7f0053e00640 1 -- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00540c9a10 msgr2=0x7f00540c18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:12.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.694+0000 7f0053e00640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00540c9a10 0x7f00540c18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.694+0000 7f0053e00640 1 -- 172.21.15.31:0/2063255287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005414c370 con 0x7f005414ccc0 2024-09-11T00:37:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.695+0000 7f0053e00640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005414ccc0 0x7f00541e1eb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f004c00c690 tx=0x7f004c00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.695+0000 7f004b400640 1 -- 172.21.15.31:0/2063255287 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c003b70 con 0x7f005414ccc0 2024-09-11T00:37:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.695+0000 7f004b400640 1 -- 172.21.15.31:0/2063255287 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f004c003d30 con 0x7f005414ccc0 2024-09-11T00:37:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.695+0000 7f004b400640 1 -- 172.21.15.31:0/2063255287 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c005330 con 0x7f005414ccc0 2024-09-11T00:37:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.695+0000 7f0059200640 1 -- 172.21.15.31:0/2063255287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005414bbe0 con 0x7f005414ccc0 2024-09-11T00:37:12.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.695+0000 7f0059200640 1 -- 172.21.15.31:0/2063255287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00540abd90 con 0x7f005414ccc0 2024-09-11T00:37:12.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.696+0000 7f0059200640 1 -- 172.21.15.31:0/2063255287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f005414b940 con 0x7f005414ccc0 2024-09-11T00:37:12.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.697+0000 7f004b400640 1 -- 172.21.15.31:0/2063255287 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f004c0054d0 con 0x7f005414ccc0 2024-09-11T00:37:12.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.699+0000 7f004b400640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0030077f70 0x7f003007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:12.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.699+0000 7f004b400640 1 -- 172.21.15.31:0/2063255287 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f004c099f80 con 0x7f005414ccc0 2024-09-11T00:37:12.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.699+0000 7f0053400640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0030077f70 0x7f003007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:12.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.700+0000 7f0053400640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0030077f70 0x7f003007a430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0044000c00 tx=0x7f0044007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:12.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.703+0000 7f004b400640 1 -- 172.21.15.31:0/2063255287 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f005414b940 con 0x7f005414ccc0 2024-09-11T00:37:12.934 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:37:12.939 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:37:12.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.946+0000 7f0059200640 1 -- 172.21.15.31:0/2063255287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00540ad6c0 con 0x7f005414ccc0 2024-09-11T00:37:12.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.946+0000 7f004b400640 1 -- 172.21.15.31:0/2063255287 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5713 (secure 0 0 0) 0x7f004c0057f0 con 0x7f005414ccc0 2024-09-11T00:37:12.948 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:12.948 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":55,"btime":"2024-09-11T00:37:12:342196+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:12.949 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 55 2024-09-11T00:37:12.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 -- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0030077f70 msgr2=0x7f003007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:12.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0030077f70 0x7f003007a430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0044000c00 tx=0x7f0044007ec0 comp rx=0 tx=0).stop 2024-09-11T00:37:12.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 -- 172.21.15.31:0/2063255287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005414ccc0 msgr2=0x7f00541e1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:12.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005414ccc0 0x7f00541e1eb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f004c00c690 tx=0x7f004c00cb50 comp rx=0 tx=0).stop 2024-09-11T00:37:12.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 -- 172.21.15.31:0/2063255287 shutdown_connections 2024-09-11T00:37:12.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0030077f70 0x7f003007a430 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:12.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f005414ccc0 0x7f00541e1eb0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:12.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f005414c8e0 0x7f00541e1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:12.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 --2- 172.21.15.31:0/2063255287 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00540c9a10 0x7f00540c18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:12.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 -- 172.21.15.31:0/2063255287 >> 172.21.15.31:0/2063255287 conn(0x7f0054068ee0 msgr2=0x7f005406dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:12.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 -- 172.21.15.31:0/2063255287 shutdown_connections 2024-09-11T00:37:12.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:12.949+0000 7f0059200640 1 -- 172.21.15.31:0/2063255287 wait complete. 2024-09-11T00:37:12.968 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:37:13.255 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:13.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.418+0000 7fd8cfc00640 1 -- 172.21.15.31:0/396286985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8c80c23a0 msgr2=0x7fd8c80c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:13.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.418+0000 7fd8cfc00640 1 --2- 172.21.15.31:0/396286985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8c80c23a0 0x7fd8c80c30c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4005fd0 tx=0x7fd8b402a720 comp rx=0 tx=0).stop 2024-09-11T00:37:13.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.419+0000 7fd8cfc00640 1 -- 172.21.15.31:0/396286985 shutdown_connections 2024-09-11T00:37:13.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.419+0000 7fd8cfc00640 1 --2- 172.21.15.31:0/396286985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8c80c23a0 0x7fd8c80c30c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:13.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.419+0000 7fd8cfc00640 1 -- 172.21.15.31:0/396286985 >> 172.21.15.31:0/396286985 conn(0x7fd8c80a9750 msgr2=0x7fd8c80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:13.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.419+0000 7fd8cfc00640 1 -- 172.21.15.31:0/396286985 shutdown_connections 2024-09-11T00:37:13.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.419+0000 7fd8cfc00640 1 -- 172.21.15.31:0/396286985 wait complete. 2024-09-11T00:37:13.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.420+0000 7fd8cfc00640 1 Processor -- start 2024-09-11T00:37:13.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.420+0000 7fd8cfc00640 1 -- start start 2024-09-11T00:37:13.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.421+0000 7fd8cfc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8c8072480 0x7fd8c81ed010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:13.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.421+0000 7fd8cfc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8c80c23a0 0x7fd8c8149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:13.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.421+0000 7fd8cfc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8c81518b0 0x7fd8c80c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:13.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.421+0000 7fd8cfc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c8074080 con 0x7fd8c80c23a0 2024-09-11T00:37:13.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.421+0000 7fd8cfc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c80c3d50 con 0x7fd8c8072480 2024-09-11T00:37:13.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.421+0000 7fd8cfc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c8074200 con 0x7fd8c81518b0 2024-09-11T00:37:13.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8c7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8c80c23a0 0x7fd8c8149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:13.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8c7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8c80c23a0 0x7fd8c8149a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51940/0 (socket says 172.21.15.31:51940) 2024-09-11T00:37:13.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8c7e00640 1 -- 172.21.15.31:0/964201627 learned_addr learned my addr 172.21.15.31:0/964201627 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:13.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8cd200640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8c8072480 0x7fd8c81ed010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:13.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8cdc00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8c81518b0 0x7fd8c80c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:13.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8c7e00640 1 -- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8c81518b0 msgr2=0x7fd8c80c5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:13.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8c7e00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8c81518b0 0x7fd8c80c5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:13.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8c7e00640 1 -- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8c8072480 msgr2=0x7fd8c81ed010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:13.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8c7e00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8c8072480 0x7fd8c81ed010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:13.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8c7e00640 1 -- 172.21.15.31:0/964201627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c81dd640 con 0x7fd8c80c23a0 2024-09-11T00:37:13.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.422+0000 7fd8c7e00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8c80c23a0 0x7fd8c8149a70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009850 tx=0x7fd8b40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:13.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.423+0000 7fd8c4c00640 1 -- 172.21.15.31:0/964201627 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b403f070 con 0x7fd8c80c23a0 2024-09-11T00:37:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.423+0000 7fd8c4c00640 1 -- 172.21.15.31:0/964201627 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8b4004890 con 0x7fd8c80c23a0 2024-09-11T00:37:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.423+0000 7fd8c4c00640 1 -- 172.21.15.31:0/964201627 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4035030 con 0x7fd8c80c23a0 2024-09-11T00:37:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.423+0000 7fd8cfc00640 1 -- 172.21.15.31:0/964201627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c80c6c70 con 0x7fd8c80c23a0 2024-09-11T00:37:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.423+0000 7fd8cfc00640 1 -- 172.21.15.31:0/964201627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c80c7230 con 0x7fd8c80c23a0 2024-09-11T00:37:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.424+0000 7fd8cfc00640 1 -- 172.21.15.31:0/964201627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c80c7470 con 0x7fd8c80c23a0 2024-09-11T00:37:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.425+0000 7fd8c4c00640 1 -- 172.21.15.31:0/964201627 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd8b4004c60 con 0x7fd8c80c23a0 2024-09-11T00:37:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.426+0000 7fd8c4c00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd8a8077fa0 0x7fd8a807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.426+0000 7fd8c4c00640 1 -- 172.21.15.31:0/964201627 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7fd8b40bb400 con 0x7fd8c80c23a0 2024-09-11T00:37:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.427+0000 7fd8cd200640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd8a8077fa0 0x7fd8a807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:13.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.427+0000 7fd8cd200640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd8a8077fa0 0x7fd8a807a460 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc000f60 tx=0x7fd8bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:13.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.431+0000 7fd8c4c00640 1 -- 172.21.15.31:0/964201627 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd8c80c7470 con 0x7fd8c80c23a0 2024-09-11T00:37:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.665+0000 7fd8cfc00640 1 -- 172.21.15.31:0/964201627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8c8072d70 con 0x7fd8c80c23a0 2024-09-11T00:37:13.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.666+0000 7fd8c4c00640 1 -- 172.21.15.31:0/964201627 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5713 (secure 0 0 0) 0x7fd8c8072d70 con 0x7fd8c80c23a0 2024-09-11T00:37:13.668 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:13.668 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":55,"btime":"2024-09-11T00:37:12:342196+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:13.668 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 55 2024-09-11T00:37:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.668+0000 7fd8cfc00640 1 -- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd8a8077fa0 msgr2=0x7fd8a807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:13.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.668+0000 7fd8cfc00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd8a8077fa0 0x7fd8a807a460 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc000f60 tx=0x7fd8bc009040 comp rx=0 tx=0).stop 2024-09-11T00:37:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.668+0000 7fd8cfc00640 1 -- 172.21.15.31:0/964201627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8c80c23a0 msgr2=0x7fd8c8149a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.668+0000 7fd8cfc00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8c80c23a0 0x7fd8c8149a70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009850 tx=0x7fd8b40039e0 comp rx=0 tx=0).stop 2024-09-11T00:37:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.668+0000 7fd8cfc00640 1 -- 172.21.15.31:0/964201627 shutdown_connections 2024-09-11T00:37:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.668+0000 7fd8cfc00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd8a8077fa0 0x7fd8a807a460 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.668+0000 7fd8cfc00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8c81518b0 0x7fd8c80c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.668+0000 7fd8cfc00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8c80c23a0 0x7fd8c8149a70 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.668+0000 7fd8cfc00640 1 --2- 172.21.15.31:0/964201627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8c8072480 0x7fd8c81ed010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.668+0000 7fd8cfc00640 1 -- 172.21.15.31:0/964201627 >> 172.21.15.31:0/964201627 conn(0x7fd8c80a9750 msgr2=0x7fd8c8150280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.669+0000 7fd8cfc00640 1 -- 172.21.15.31:0/964201627 shutdown_connections 2024-09-11T00:37:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:13.669+0000 7fd8cfc00640 1 -- 172.21.15.31:0/964201627 wait complete. 2024-09-11T00:37:13.686 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=5374 2024-09-11T00:37:13.687 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.a 2024-09-11T00:37:14.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:15.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.142+0000 7fec22600640 1 -- 172.21.15.31:0/2078882403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1c0c6310 msgr2=0x7fec1c1539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.142+0000 7fec22600640 1 --2- 172.21.15.31:0/2078882403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1c0c6310 0x7fec1c1539c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fec1c070470 tx=0x7fec1402aa30 comp rx=0 tx=0).stop 2024-09-11T00:37:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.142+0000 7fec22600640 1 -- 172.21.15.31:0/2078882403 shutdown_connections 2024-09-11T00:37:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.142+0000 7fec22600640 1 --2- 172.21.15.31:0/2078882403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1c0c6310 0x7fec1c1539c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.142+0000 7fec22600640 1 --2- 172.21.15.31:0/2078882403 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec1c0c5ef0 0x7fec1c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.142+0000 7fec22600640 1 --2- 172.21.15.31:0/2078882403 >> v2:172.21.15.204:3300/0 conn(0x7fec1c0c5b10 0x7fec1c156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.142+0000 7fec22600640 1 -- 172.21.15.31:0/2078882403 >> 172.21.15.31:0/2078882403 conn(0x7fec1c0a9750 msgr2=0x7fec1c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.142+0000 7fec22600640 1 -- 172.21.15.31:0/2078882403 shutdown_connections 2024-09-11T00:37:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.142+0000 7fec22600640 1 -- 172.21.15.31:0/2078882403 wait complete. 2024-09-11T00:37:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.144+0000 7fec22600640 1 Processor -- start 2024-09-11T00:37:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.144+0000 7fec22600640 1 -- start start 2024-09-11T00:37:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec22600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec1c0c5b10 0x7fec1c14af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec22600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1c0c5ef0 0x7fec1c154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec22600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec1c0c6310 0x7fec1c151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec22600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c1597f0 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec22600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c159670 con 0x7fec1c0c6310 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec22600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fec1c159970 con 0x7fec1c0c5b10 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec12a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1c0c5ef0 0x7fec1c154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec12a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1c0c5ef0 0x7fec1c154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51960/0 (socket says 172.21.15.31:51960) 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec12a00640 1 -- 172.21.15.31:0/2864179518 learned_addr learned my addr 172.21.15.31:0/2864179518 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec1b400640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec1c0c5b10 0x7fec1c14af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec1be00640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec1c0c6310 0x7fec1c151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec12a00640 1 -- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec1c0c5b10 msgr2=0x7fec1c14af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec12a00640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec1c0c5b10 0x7fec1c14af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec12a00640 1 -- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec1c0c6310 msgr2=0x7fec1c151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec12a00640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec1c0c6310 0x7fec1c151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.145+0000 7fec12a00640 1 -- 172.21.15.31:0/2864179518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1c154d40 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.146+0000 7fec12a00640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1c0c5ef0 0x7fec1c154800 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fec0400e9c0 tx=0x7fec0400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.146+0000 7fec13e00640 1 -- 172.21.15.31:0/2864179518 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0400cd80 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.146+0000 7fec13e00640 1 -- 172.21.15.31:0/2864179518 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec04004020 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.146+0000 7fec13e00640 1 -- 172.21.15.31:0/2864179518 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04004450 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.146+0000 7fec22600640 1 -- 172.21.15.31:0/2864179518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec1c0c6980 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.146+0000 7fec22600640 1 -- 172.21.15.31:0/2864179518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec1c1f0130 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.147+0000 7fec22600640 1 -- 172.21.15.31:0/2864179518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec1c156ad0 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.148+0000 7fec13e00640 1 -- 172.21.15.31:0/2864179518 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fec040035f0 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.149+0000 7fec13e00640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febf8077f70 0x7febf807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.149+0000 7fec13e00640 1 -- 172.21.15.31:0/2864179518 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7fec0409ad90 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.150+0000 7fec1b400640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febf8077f70 0x7febf807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:15.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.150+0000 7fec1b400640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febf8077f70 0x7febf807a430 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fec0c0095f0 tx=0x7fec0c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:15.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.153+0000 7fec13e00640 1 -- 172.21.15.31:0/2864179518 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fec04062b30 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.384+0000 7fec22600640 1 -- 172.21.15.31:0/2864179518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec1c0c1e70 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.384+0000 7fec13e00640 1 -- 172.21.15.31:0/2864179518 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5713 (secure 0 0 0) 0x7fec04067560 con 0x7fec1c0c5ef0 2024-09-11T00:37:15.387 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:15.387 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":55,"btime":"2024-09-11T00:37:12:342196+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:15.387 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 55 2024-09-11T00:37:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 -- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febf8077f70 msgr2=0x7febf807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febf8077f70 0x7febf807a430 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fec0c0095f0 tx=0x7fec0c005f90 comp rx=0 tx=0).stop 2024-09-11T00:37:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 -- 172.21.15.31:0/2864179518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1c0c5ef0 msgr2=0x7fec1c154800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1c0c5ef0 0x7fec1c154800 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fec0400e9c0 tx=0x7fec0400ee80 comp rx=0 tx=0).stop 2024-09-11T00:37:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 -- 172.21.15.31:0/2864179518 shutdown_connections 2024-09-11T00:37:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7febf8077f70 0x7febf807a430 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec1c0c6310 0x7fec1c151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1c0c5ef0 0x7fec1c154800 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 --2- 172.21.15.31:0/2864179518 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec1c0c5b10 0x7fec1c14af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 -- 172.21.15.31:0/2864179518 >> 172.21.15.31:0/2864179518 conn(0x7fec1c0a9750 msgr2=0x7fec1c0c4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 -- 172.21.15.31:0/2864179518 shutdown_connections 2024-09-11T00:37:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.387+0000 7fec22600640 1 -- 172.21.15.31:0/2864179518 wait complete. 2024-09-11T00:37:15.404 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:37:15.687 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-11T00:37:15.688 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-11T00:37:15.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-11T00:37:15.692 INFO:tasks.ceph.mds.a:Started 2024-09-11T00:37:15.693 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:15.831 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-11T00:37:15.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.866+0000 7f33cfe00640 1 -- 172.21.15.31:0/1835889944 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33d0150020 msgr2=0x7f33d00ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:15.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.866+0000 7f33cfe00640 1 --2- 172.21.15.31:0/1835889944 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33d0150020 0x7f33d00ad4e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f33d0153d20 tx=0x7f33c002a530 comp rx=0 tx=0).stop 2024-09-11T00:37:15.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.866+0000 7f33cfe00640 1 -- 172.21.15.31:0/1835889944 shutdown_connections 2024-09-11T00:37:15.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.866+0000 7f33cfe00640 1 --2- 172.21.15.31:0/1835889944 >> v2:172.21.15.204:3300/0 conn(0x7f33d006e850 0x7f33d0160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.866+0000 7f33cfe00640 1 --2- 172.21.15.31:0/1835889944 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33d0150020 0x7f33d00ad4e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.866+0000 7f33cfe00640 1 -- 172.21.15.31:0/1835889944 >> 172.21.15.31:0/1835889944 conn(0x7f33d0068f10 msgr2=0x7f33d00cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:15.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.866+0000 7f33cfe00640 1 -- 172.21.15.31:0/1835889944 shutdown_connections 2024-09-11T00:37:15.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.866+0000 7f33cfe00640 1 -- 172.21.15.31:0/1835889944 wait complete. 2024-09-11T00:37:15.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.867+0000 7f33cfe00640 1 Processor -- start 2024-09-11T00:37:15.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.868+0000 7f33cfe00640 1 -- start start 2024-09-11T00:37:15.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.868+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33d006e850 0x7f33d014dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.868+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33d00ada20 0x7f33d014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.868+0000 7f33cfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33d0150020 0x7f33d00c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.868+0000 7f33cfe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33d0157550 con 0x7f33d0150020 2024-09-11T00:37:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.868+0000 7f33cfe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f33d01573d0 con 0x7f33d006e850 2024-09-11T00:37:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.868+0000 7f33cfe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f33d0156960 con 0x7f33d00ada20 2024-09-11T00:37:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.869+0000 7f33cf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33d0150020 0x7f33d00c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.869+0000 7f33ce000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33d00ada20 0x7f33d014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.869+0000 7f33cea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33d006e850 0x7f33d014dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.869+0000 7f33cf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33d0150020 0x7f33d00c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51988/0 (socket says 172.21.15.31:51988) 2024-09-11T00:37:15.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.869+0000 7f33cf400640 1 -- 172.21.15.31:0/155192198 learned_addr learned my addr 172.21.15.31:0/155192198 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:15.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.869+0000 7f33cf400640 1 -- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33d00ada20 msgr2=0x7f33d014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:15.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.869+0000 7f33cf400640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33d00ada20 0x7f33d014e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.869+0000 7f33cf400640 1 -- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33d006e850 msgr2=0x7f33d014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:15.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.869+0000 7f33cf400640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33d006e850 0x7f33d014dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:15.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.869+0000 7f33cf400640 1 -- 172.21.15.31:0/155192198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d01ec8b0 con 0x7f33d0150020 2024-09-11T00:37:15.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.870+0000 7f33cf400640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33d0150020 0x7f33d00c49a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f33c800eb40 tx=0x7f33c800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:15.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.870+0000 7f33c6a00640 1 -- 172.21.15.31:0/155192198 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8004b20 con 0x7f33d0150020 2024-09-11T00:37:15.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.870+0000 7f33c6a00640 1 -- 172.21.15.31:0/155192198 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33c8004db0 con 0x7f33d0150020 2024-09-11T00:37:15.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.870+0000 7f33cfe00640 1 -- 172.21.15.31:0/155192198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d01ecb90 con 0x7f33d0150020 2024-09-11T00:37:15.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.870+0000 7f33c6a00640 1 -- 172.21.15.31:0/155192198 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8005280 con 0x7f33d0150020 2024-09-11T00:37:15.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.870+0000 7f33cfe00640 1 -- 172.21.15.31:0/155192198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d00c2d80 con 0x7f33d0150020 2024-09-11T00:37:15.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.871+0000 7f33cfe00640 1 -- 172.21.15.31:0/155192198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33d00c2f80 con 0x7f33d0150020 2024-09-11T00:37:15.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.872+0000 7f33c6a00640 1 -- 172.21.15.31:0/155192198 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f33c8020030 con 0x7f33d0150020 2024-09-11T00:37:15.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.873+0000 7f33c6a00640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33b0077f70 0x7f33b007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:15.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.873+0000 7f33c6a00640 1 -- 172.21.15.31:0/155192198 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f33c809aec0 con 0x7f33d0150020 2024-09-11T00:37:15.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.873+0000 7f33cea00640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33b0077f70 0x7f33b007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:15.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.874+0000 7f33cea00640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33b0077f70 0x7f33b007a430 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f33c0034000 tx=0x7f33c0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:15.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:15.877+0000 7f33c6a00640 1 -- 172.21.15.31:0/155192198 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f33d00c2f80 con 0x7f33d0150020 2024-09-11T00:37:16.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.110+0000 7f33cfe00640 1 -- 172.21.15.31:0/155192198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33d00cc220 con 0x7f33d0150020 2024-09-11T00:37:16.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.111+0000 7f33c6a00640 1 -- 172.21.15.31:0/155192198 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5713 (secure 0 0 0) 0x7f33c8062cf0 con 0x7f33d0150020 2024-09-11T00:37:16.113 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:16.113 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":55,"btime":"2024-09-11T00:37:12:342196+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:35:12.604104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5302":{"gid":5302,"name":"d","rank":0,"incarnation":42,"state":"up:active","state_seq":16,"addr":"172.21.15.204:6835/3444890210","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3444890210},{"type":"v1","addr":"172.21.15.204:6835","nonce":3444890210}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5302,"qdb_cluster":[5302]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:16.113 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 55 2024-09-11T00:37:16.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.113+0000 7f33cfe00640 1 -- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33b0077f70 msgr2=0x7f33b007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.113+0000 7f33cfe00640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33b0077f70 0x7f33b007a430 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f33c0034000 tx=0x7f33c0004730 comp rx=0 tx=0).stop 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.113+0000 7f33cfe00640 1 -- 172.21.15.31:0/155192198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33d0150020 msgr2=0x7f33d00c49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.113+0000 7f33cfe00640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33d0150020 0x7f33d00c49a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f33c800eb40 tx=0x7f33c800c2d0 comp rx=0 tx=0).stop 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.113+0000 7f33cfe00640 1 -- 172.21.15.31:0/155192198 shutdown_connections 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.113+0000 7f33cfe00640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33b0077f70 0x7f33b007a430 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.113+0000 7f33cfe00640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33d0150020 0x7f33d00c49a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.113+0000 7f33cfe00640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f33d00ada20 0x7f33d014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.113+0000 7f33cfe00640 1 --2- 172.21.15.31:0/155192198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33d006e850 0x7f33d014dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.114+0000 7f33cfe00640 1 -- 172.21.15.31:0/155192198 >> 172.21.15.31:0/155192198 conn(0x7f33d0068f10 msgr2=0x7f33d01576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.114+0000 7f33cfe00640 1 -- 172.21.15.31:0/155192198 shutdown_connections 2024-09-11T00:37:16.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:16.114+0000 7f33cfe00640 1 -- 172.21.15.31:0/155192198 wait complete. 2024-09-11T00:37:16.131 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-11T00:37:16.368 INFO:tasks.ceph.mds.b.smithi204.stderr:2024-09-11T00:37:16.366+0000 7f8649200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:37:17.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.579+0000 7f1d65c00640 1 -- 172.21.15.31:0/2192007505 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d600c23a0 msgr2=0x7f1d600c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.579+0000 7f1d65c00640 1 --2- 172.21.15.31:0/2192007505 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d600c23a0 0x7f1d600c30c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009860 tx=0x7f1d4802a560 comp rx=0 tx=0).stop 2024-09-11T00:37:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.579+0000 7f1d65c00640 1 -- 172.21.15.31:0/2192007505 shutdown_connections 2024-09-11T00:37:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.579+0000 7f1d65c00640 1 --2- 172.21.15.31:0/2192007505 >> v2:172.21.15.204:3300/0 conn(0x7f1d601518b0 0x7f1d600c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.579+0000 7f1d65c00640 1 --2- 172.21.15.31:0/2192007505 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d600c23a0 0x7f1d600c30c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.579+0000 7f1d65c00640 1 -- 172.21.15.31:0/2192007505 >> 172.21.15.31:0/2192007505 conn(0x7f1d600a9750 msgr2=0x7f1d600a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.579+0000 7f1d65c00640 1 -- 172.21.15.31:0/2192007505 shutdown_connections 2024-09-11T00:37:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.579+0000 7f1d65c00640 1 -- 172.21.15.31:0/2192007505 wait complete. 2024-09-11T00:37:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.580+0000 7f1d65c00640 1 Processor -- start 2024-09-11T00:37:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.580+0000 7f1d65c00640 1 -- start start 2024-09-11T00:37:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.581+0000 7f1d65c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d60072480 0x7f1d60151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.581+0000 7f1d65c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d600c23a0 0x7f1d6006e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.581+0000 7f1d65c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d601518b0 0x7f1d6006efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.581+0000 7f1d65c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1d60074290 con 0x7f1d60072480 2024-09-11T00:37:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.581+0000 7f1d65c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1d60072860 con 0x7f1d600c23a0 2024-09-11T00:37:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.581+0000 7f1d65c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1d600729e0 con 0x7f1d601518b0 2024-09-11T00:37:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.581+0000 7f1d5ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d60072480 0x7f1d60151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d5ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d60072480 0x7f1d60151eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51998/0 (socket says 172.21.15.31:51998) 2024-09-11T00:37:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d5ea00640 1 -- 172.21.15.31:0/3346306237 learned_addr learned my addr 172.21.15.31:0/3346306237 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d5f400640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d601518b0 0x7f1d6006efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d56000640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d600c23a0 0x7f1d6006e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d5ea00640 1 -- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d601518b0 msgr2=0x7f1d6006efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d5ea00640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d601518b0 0x7f1d6006efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d5ea00640 1 -- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d600c23a0 msgr2=0x7f1d6006e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d5ea00640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d600c23a0 0x7f1d6006e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d5ea00640 1 -- 172.21.15.31:0/3346306237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6006e330 con 0x7f1d60072480 2024-09-11T00:37:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d56000640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d600c23a0 0x7f1d6006e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:37:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d5ea00640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d60072480 0x7f1d60151eb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000c710 tx=0x7f1d5000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d57400640 1 -- 172.21.15.31:0/3346306237 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d50016020 con 0x7f1d60072480 2024-09-11T00:37:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d57400640 1 -- 172.21.15.31:0/3346306237 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d5000ce50 con 0x7f1d60072480 2024-09-11T00:37:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d57400640 1 -- 172.21.15.31:0/3346306237 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5000abb0 con 0x7f1d60072480 2024-09-11T00:37:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.582+0000 7f1d65c00640 1 -- 172.21.15.31:0/3346306237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d601f0200 con 0x7f1d60072480 2024-09-11T00:37:17.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.583+0000 7f1d65c00640 1 -- 172.21.15.31:0/3346306237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d601e0de0 con 0x7f1d60072480 2024-09-11T00:37:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.584+0000 7f1d65c00640 1 -- 172.21.15.31:0/3346306237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d6014d610 con 0x7f1d60072480 2024-09-11T00:37:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.585+0000 7f1d57400640 1 -- 172.21.15.31:0/3346306237 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1d5001b400 con 0x7f1d60072480 2024-09-11T00:37:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.586+0000 7f1d57400640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d3c077fa0 0x7f1d3c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:17.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.586+0000 7f1d57400640 1 -- 172.21.15.31:0/3346306237 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f1d5009a730 con 0x7f1d60072480 2024-09-11T00:37:17.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.586+0000 7f1d56000640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d3c077fa0 0x7f1d3c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:17.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.587+0000 7f1d56000640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d3c077fa0 0x7f1d3c07a460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1d48034000 tx=0x7f1d48004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:17.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.590+0000 7f1d57400640 1 -- 172.21.15.31:0/3346306237 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1d50020020 con 0x7f1d60072480 2024-09-11T00:37:17.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.824+0000 7f1d65c00640 1 -- 172.21.15.31:0/3346306237 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d600c1300 con 0x7f1d60072480 2024-09-11T00:37:17.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.824+0000 7f1d57400640 1 -- 172.21.15.31:0/3346306237 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5705 (secure 0 0 0) 0x7f1d50062b80 con 0x7f1d60072480 2024-09-11T00:37:17.828 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:17.828 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":58,"btime":"2024-09-11T00:37:17:372636+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:16.632243+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:17.830 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 58 2024-09-11T00:37:17.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 -- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d3c077fa0 msgr2=0x7f1d3c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:17.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d3c077fa0 0x7f1d3c07a460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1d48034000 tx=0x7f1d48004730 comp rx=0 tx=0).stop 2024-09-11T00:37:17.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 -- 172.21.15.31:0/3346306237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d60072480 msgr2=0x7f1d60151eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:17.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d60072480 0x7f1d60151eb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1d5000c710 tx=0x7f1d5000cb10 comp rx=0 tx=0).stop 2024-09-11T00:37:17.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 -- 172.21.15.31:0/3346306237 shutdown_connections 2024-09-11T00:37:17.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d3c077fa0 0x7f1d3c07a460 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:17.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d601518b0 0x7f1d6006efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:17.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d600c23a0 0x7f1d6006e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:17.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 --2- 172.21.15.31:0/3346306237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d60072480 0x7f1d60151eb0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:17.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 -- 172.21.15.31:0/3346306237 >> 172.21.15.31:0/3346306237 conn(0x7f1d600a9750 msgr2=0x7f1d601503b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:17.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 -- 172.21.15.31:0/3346306237 shutdown_connections 2024-09-11T00:37:17.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:17.827+0000 7f1d65c00640 1 -- 172.21.15.31:0/3346306237 wait complete. 2024-09-11T00:37:17.844 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-11T00:37:17.845 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:37:17.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:18.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.011+0000 7f3514a00640 1 -- 172.21.15.31:0/2630132046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35101518d0 msgr2=0x7f35100c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:18.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.011+0000 7f3514a00640 1 --2- 172.21.15.31:0/2630132046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35101518d0 0x7f35100c3660 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f35100708a0 tx=0x7f350802ab50 comp rx=0 tx=0).stop 2024-09-11T00:37:18.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.012+0000 7f3514a00640 1 -- 172.21.15.31:0/2630132046 shutdown_connections 2024-09-11T00:37:18.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.012+0000 7f3514a00640 1 --2- 172.21.15.31:0/2630132046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35101518d0 0x7f35100c3660 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.012+0000 7f3514a00640 1 --2- 172.21.15.31:0/2630132046 >> v2:172.21.15.204:3300/0 conn(0x7f35100c2410 0x7f35100c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.012+0000 7f3514a00640 1 -- 172.21.15.31:0/2630132046 >> 172.21.15.31:0/2630132046 conn(0x7f35100a9750 msgr2=0x7f35100a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:18.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.012+0000 7f3514a00640 1 -- 172.21.15.31:0/2630132046 shutdown_connections 2024-09-11T00:37:18.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.012+0000 7f3514a00640 1 -- 172.21.15.31:0/2630132046 wait complete. 2024-09-11T00:37:18.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.013+0000 7f3514a00640 1 Processor -- start 2024-09-11T00:37:18.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.013+0000 7f3514a00640 1 -- start start 2024-09-11T00:37:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.014+0000 7f3514a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3510072480 0x7f3510157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.014+0000 7f3514a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35100c2410 0x7f35101570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.014+0000 7f3514a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35101518d0 0x7f351006f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.014+0000 7f3514a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f35100729e0 con 0x7f35100c2410 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.014+0000 7f3514a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3510072860 con 0x7f3510072480 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.014+0000 7f3514a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f35100c3ba0 con 0x7f35101518d0 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.014+0000 7f3507e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35100c2410 0x7f35101570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.014+0000 7f3507e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35100c2410 0x7f35101570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47548/0 (socket says 172.21.15.31:47548) 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f3507e00640 1 -- 172.21.15.31:0/2534031554 learned_addr learned my addr 172.21.15.31:0/2534031554 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f350de00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3510072480 0x7f3510157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f350e800640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35101518d0 0x7f351006f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f3507e00640 1 -- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35101518d0 msgr2=0x7f351006f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f3507e00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35101518d0 0x7f351006f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f3507e00640 1 -- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3510072480 msgr2=0x7f3510157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f3507e00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3510072480 0x7f3510157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f3507e00640 1 -- 172.21.15.31:0/2534031554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3510157620 con 0x7f35100c2410 2024-09-11T00:37:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f3507e00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35100c2410 0x7f35101570e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f34f800e9f0 tx=0x7f34f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f3506000640 1 -- 172.21.15.31:0/2534031554 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f800cd80 con 0x7f35100c2410 2024-09-11T00:37:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.015+0000 7f3506000640 1 -- 172.21.15.31:0/2534031554 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34f8010430 con 0x7f35100c2410 2024-09-11T00:37:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.016+0000 7f3514a00640 1 -- 172.21.15.31:0/2534031554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35101553b0 con 0x7f35100c2410 2024-09-11T00:37:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.016+0000 7f3506000640 1 -- 172.21.15.31:0/2534031554 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8010840 con 0x7f35100c2410 2024-09-11T00:37:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.016+0000 7f3514a00640 1 -- 172.21.15.31:0/2534031554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f351006fb30 con 0x7f35100c2410 2024-09-11T00:37:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.017+0000 7f3514a00640 1 -- 172.21.15.31:0/2534031554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35100a9540 con 0x7f35100c2410 2024-09-11T00:37:18.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.018+0000 7f3506000640 1 -- 172.21.15.31:0/2534031554 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f34f80109e0 con 0x7f35100c2410 2024-09-11T00:37:18.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.019+0000 7f3506000640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ec077fa0 0x7f34ec07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:18.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.019+0000 7f3506000640 1 -- 172.21.15.31:0/2534031554 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f34f809aee0 con 0x7f35100c2410 2024-09-11T00:37:18.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.019+0000 7f350de00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ec077fa0 0x7f34ec07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:18.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.020+0000 7f350de00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ec077fa0 0x7f34ec07a460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f35000017c0 tx=0x7f3500009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:18.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.023+0000 7f3506000640 1 -- 172.21.15.31:0/2534031554 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f35100a9540 con 0x7f35100c2410 2024-09-11T00:37:18.131 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.260+0000 7f3514a00640 1 -- 172.21.15.31:0/2534031554 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3510074080 con 0x7f35100c2410 2024-09-11T00:37:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.261+0000 7f3506000640 1 -- 172.21.15.31:0/2534031554 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5705 (secure 0 0 0) 0x7f3510157620 con 0x7f35100c2410 2024-09-11T00:37:18.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:18.264 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":58,"btime":"2024-09-11T00:37:17:372636+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:16.632243+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 58 2024-09-11T00:37:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 -- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ec077fa0 msgr2=0x7f34ec07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ec077fa0 0x7f34ec07a460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f35000017c0 tx=0x7f3500009040 comp rx=0 tx=0).stop 2024-09-11T00:37:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 -- 172.21.15.31:0/2534031554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35100c2410 msgr2=0x7f35101570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35100c2410 0x7f35101570e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f34f800e9f0 tx=0x7f34f800eeb0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 -- 172.21.15.31:0/2534031554 shutdown_connections 2024-09-11T00:37:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f34ec077fa0 0x7f34ec07a460 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35101518d0 0x7f351006f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35100c2410 0x7f35101570e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 --2- 172.21.15.31:0/2534031554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3510072480 0x7f3510157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 -- 172.21.15.31:0/2534031554 >> 172.21.15.31:0/2534031554 conn(0x7f35100a9750 msgr2=0x7f35100c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 -- 172.21.15.31:0/2534031554 shutdown_connections 2024-09-11T00:37:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.263+0000 7f3514a00640 1 -- 172.21.15.31:0/2534031554 wait complete. 2024-09-11T00:37:18.282 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:37:17:372636+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 58, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 58, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5673": { "addr": "172.21.15.31:6835/3242133790", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6833", "nonce": 3242133790, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3242133790, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5673, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:37:16.632243+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5673 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 55, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5449": { "addr": "172.21.15.204:6837/1278135080", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 1278135080, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 1278135080, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5449, "incarnation": 52, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:37:12.342188+0000", "qdb_cluster": [ 5449 ], "qdb_leader": 5449, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5449 } } } ], "standbys": [ { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:37:18.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.292+0000 7f76d0e00640 1 -- 172.21.15.31:0/2988376975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76cc0c9fe0 msgr2=0x7f76cc0cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:18.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.292+0000 7f76d0e00640 1 --2- 172.21.15.31:0/2988376975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76cc0c9fe0 0x7f76cc0cc310 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0c2aa0 tx=0x7f76c402a9a0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.292+0000 7f76d0e00640 1 -- 172.21.15.31:0/2988376975 shutdown_connections 2024-09-11T00:37:18.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.292+0000 7f76d0e00640 1 --2- 172.21.15.31:0/2988376975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76cc0c9fe0 0x7f76cc0cc310 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.292+0000 7f76d0e00640 1 --2- 172.21.15.31:0/2988376975 >> v2:172.21.15.204:3300/0 conn(0x7f76cc0c9b00 0x7f76cc156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.292+0000 7f76d0e00640 1 -- 172.21.15.31:0/2988376975 >> 172.21.15.31:0/2988376975 conn(0x7f76cc068f10 msgr2=0x7f76cc0cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.292+0000 7f76d0e00640 1 -- 172.21.15.31:0/2988376975 shutdown_connections 2024-09-11T00:37:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.292+0000 7f76d0e00640 1 -- 172.21.15.31:0/2988376975 wait complete. 2024-09-11T00:37:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.294+0000 7f76d0e00640 1 Processor -- start 2024-09-11T00:37:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.294+0000 7f76d0e00640 1 -- start start 2024-09-11T00:37:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76d0e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f76cc0b1230 0x7f76cc1dfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76d0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76cc0c9b00 0x7f76cc1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76d0e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76cc0c9fe0 0x7f76cc1ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76d0e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc156fa0 con 0x7f76cc0c9b00 2024-09-11T00:37:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76d0e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc156540 con 0x7f76cc0c9fe0 2024-09-11T00:37:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76d0e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f76cc0bf390 con 0x7f76cc0b1230 2024-09-11T00:37:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76c9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76cc0c9b00 0x7f76cc1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76c9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76cc0c9b00 0x7f76cc1584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47566/0 (socket says 172.21.15.31:47566) 2024-09-11T00:37:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76c9400640 1 -- 172.21.15.31:0/2124318124 learned_addr learned my addr 172.21.15.31:0/2124318124 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76c9e00640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f76cc0b1230 0x7f76cc1dfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76ca800640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76cc0c9fe0 0x7f76cc1ef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76c9400640 1 -- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f76cc0b1230 msgr2=0x7f76cc1dfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76c9400640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f76cc0b1230 0x7f76cc1dfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76c9400640 1 -- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76cc0c9fe0 msgr2=0x7f76cc1ef4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76c9400640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76cc0c9fe0 0x7f76cc1ef4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.295+0000 7f76c9400640 1 -- 172.21.15.31:0/2124318124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76cc0c1e30 con 0x7f76cc0c9b00 2024-09-11T00:37:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.296+0000 7f76c9400640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76cc0c9b00 0x7f76cc1584b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f76b800ea20 tx=0x7f76b800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.296+0000 7f76c2000640 1 -- 172.21.15.31:0/2124318124 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b800cd80 con 0x7f76cc0c9b00 2024-09-11T00:37:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.296+0000 7f76c2000640 1 -- 172.21.15.31:0/2124318124 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76b8010430 con 0x7f76cc0c9b00 2024-09-11T00:37:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.296+0000 7f76c2000640 1 -- 172.21.15.31:0/2124318124 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b8010840 con 0x7f76cc0c9b00 2024-09-11T00:37:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.296+0000 7f76d0e00640 1 -- 172.21.15.31:0/2124318124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76cc0c2110 con 0x7f76cc0c9b00 2024-09-11T00:37:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.296+0000 7f76d0e00640 1 -- 172.21.15.31:0/2124318124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76cc1e0370 con 0x7f76cc0c9b00 2024-09-11T00:37:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.297+0000 7f76d0e00640 1 -- 172.21.15.31:0/2124318124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76cc0766d0 con 0x7f76cc0c9b00 2024-09-11T00:37:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.298+0000 7f76c2000640 1 -- 172.21.15.31:0/2124318124 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f76b80109e0 con 0x7f76cc0c9b00 2024-09-11T00:37:18.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.299+0000 7f76c2000640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f76ac077f70 0x7f76ac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:18.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.299+0000 7f76c2000640 1 -- 172.21.15.31:0/2124318124 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f76b809afe0 con 0x7f76cc0c9b00 2024-09-11T00:37:18.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.300+0000 7f76c9e00640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f76ac077f70 0x7f76ac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.300+0000 7f76c9e00640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f76ac077f70 0x7f76ac07a430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f76bc007fd0 tx=0x7f76bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.303+0000 7f76c2000640 1 -- 172.21.15.31:0/2124318124 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f76b8062d00 con 0x7f76cc0c9b00 2024-09-11T00:37:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.538+0000 7f76d0e00640 1 -- 172.21.15.31:0/2124318124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76cc156a60 con 0x7f76cc0c9b00 2024-09-11T00:37:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.539+0000 7f76c2000640 1 -- 172.21.15.31:0/2124318124 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5702 (secure 0 0 0) 0x7f76b8067730 con 0x7f76cc0c9b00 2024-09-11T00:37:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:18.541 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":59,"btime":"2024-09-11T00:37:18:376585+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:17.379164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 59 2024-09-11T00:37:18.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 -- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f76ac077f70 msgr2=0x7f76ac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:18.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f76ac077f70 0x7f76ac07a430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f76bc007fd0 tx=0x7f76bc009040 comp rx=0 tx=0).stop 2024-09-11T00:37:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 -- 172.21.15.31:0/2124318124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76cc0c9b00 msgr2=0x7f76cc1584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76cc0c9b00 0x7f76cc1584b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f76b800ea20 tx=0x7f76b800eee0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 -- 172.21.15.31:0/2124318124 shutdown_connections 2024-09-11T00:37:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f76ac077f70 0x7f76ac07a430 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76cc0c9fe0 0x7f76cc1ef4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f76cc0c9b00 0x7f76cc1584b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 --2- 172.21.15.31:0/2124318124 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f76cc0b1230 0x7f76cc1dfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 -- 172.21.15.31:0/2124318124 >> 172.21.15.31:0/2124318124 conn(0x7f76cc068f10 msgr2=0x7f76cc152a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 -- 172.21.15.31:0/2124318124 shutdown_connections 2024-09-11T00:37:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:18.541+0000 7f76d0e00640 1 -- 172.21.15.31:0/2124318124 wait complete. 2024-09-11T00:37:18.559 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-11T00:37:20.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:20.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.458+0000 7f434ca00640 1 -- 172.21.15.31:0/415576911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43480c4fe0 msgr2=0x7f4348157530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.458+0000 7f434ca00640 1 --2- 172.21.15.31:0/415576911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43480c4fe0 0x7f4348157530 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f43481585c0 tx=0x7f433402a560 comp rx=0 tx=0).stop 2024-09-11T00:37:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.458+0000 7f434ca00640 1 -- 172.21.15.31:0/415576911 shutdown_connections 2024-09-11T00:37:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.458+0000 7f434ca00640 1 --2- 172.21.15.31:0/415576911 >> v2:172.21.15.204:3300/0 conn(0x7f4348157a70 0x7f434815e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.458+0000 7f434ca00640 1 --2- 172.21.15.31:0/415576911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43480c4fe0 0x7f4348157530 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.458+0000 7f434ca00640 1 -- 172.21.15.31:0/415576911 >> 172.21.15.31:0/415576911 conn(0x7f43480ab8a0 msgr2=0x7f43480abcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.459+0000 7f434ca00640 1 -- 172.21.15.31:0/415576911 shutdown_connections 2024-09-11T00:37:20.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.459+0000 7f434ca00640 1 -- 172.21.15.31:0/415576911 wait complete. 2024-09-11T00:37:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.460+0000 7f434ca00640 1 Processor -- start 2024-09-11T00:37:20.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.460+0000 7f434ca00640 1 -- start start 2024-09-11T00:37:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.461+0000 7f434ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43480c44f0 0x7f43480ad670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.461+0000 7f434ca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4348157a70 0x7f43480adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.461+0000 7f434ca00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43480c81f0 0x7f43481df980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.461+0000 7f434ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f43480701b0 con 0x7f43480c44f0 2024-09-11T00:37:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.461+0000 7f434ca00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4348151af0 con 0x7f4348157a70 2024-09-11T00:37:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.461+0000 7f434ca00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f43480a9380 con 0x7f43480c81f0 2024-09-11T00:37:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4345e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43480c44f0 0x7f43480ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4345e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43480c44f0 0x7f43480ad670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47582/0 (socket says 172.21.15.31:47582) 2024-09-11T00:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4345e00640 1 -- 172.21.15.31:0/2733715940 learned_addr learned my addr 172.21.15.31:0/2733715940 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4345400640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4348157a70 0x7f43480adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4346800640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43480c81f0 0x7f43481df980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4345e00640 1 -- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43480c81f0 msgr2=0x7f43481df980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4345e00640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43480c81f0 0x7f43481df980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4345e00640 1 -- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4348157a70 msgr2=0x7f43480adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4345e00640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4348157a70 0x7f43480adeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4345e00640 1 -- 172.21.15.31:0/2733715940 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43480af6c0 con 0x7f43480c44f0 2024-09-11T00:37:20.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.462+0000 7f4345e00640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43480c44f0 0x7f43480ad670 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f433800c500 tx=0x7f433800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.463+0000 7f433e000640 1 -- 172.21.15.31:0/2733715940 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4338017070 con 0x7f43480c44f0 2024-09-11T00:37:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.463+0000 7f433e000640 1 -- 172.21.15.31:0/2733715940 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43380035f0 con 0x7f43480c44f0 2024-09-11T00:37:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.463+0000 7f433e000640 1 -- 172.21.15.31:0/2733715940 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4338003a40 con 0x7f43480c44f0 2024-09-11T00:37:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.463+0000 7f434ca00640 1 -- 172.21.15.31:0/2733715940 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43480c6710 con 0x7f43480c44f0 2024-09-11T00:37:20.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.463+0000 7f434ca00640 1 -- 172.21.15.31:0/2733715940 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43480ac8a0 con 0x7f43480c44f0 2024-09-11T00:37:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.464+0000 7f434ca00640 1 -- 172.21.15.31:0/2733715940 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43480c6950 con 0x7f43480c44f0 2024-09-11T00:37:20.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.466+0000 7f433e000640 1 -- 172.21.15.31:0/2733715940 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4338003be0 con 0x7f43480c44f0 2024-09-11T00:37:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.467+0000 7f433e000640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4328077fa0 0x7f432807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.467+0000 7f433e000640 1 -- 172.21.15.31:0/2733715940 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f4338099a80 con 0x7f43480c44f0 2024-09-11T00:37:20.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.467+0000 7f4345400640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4328077fa0 0x7f432807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:20.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.468+0000 7f4345400640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4328077fa0 0x7f432807a460 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4334033000 tx=0x7f4334004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:20.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.472+0000 7f433e000640 1 -- 172.21.15.31:0/2733715940 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f43480c6950 con 0x7f43480c44f0 2024-09-11T00:37:20.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.741+0000 7f328bc00640 1 -- 172.21.15.31:0/3503555986 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32840b1230 msgr2=0x7f32841596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.741+0000 7f328bc00640 1 --2- 172.21.15.31:0/3503555986 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32840b1230 0x7f32841596b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3278005fd0 tx=0x7f327802a4b0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.741+0000 7f328bc00640 1 -- 172.21.15.31:0/3503555986 shutdown_connections 2024-09-11T00:37:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.741+0000 7f328bc00640 1 --2- 172.21.15.31:0/3503555986 >> v2:172.21.15.204:3300/0 conn(0x7f32840c9b00 0x7f3284156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.741+0000 7f328bc00640 1 --2- 172.21.15.31:0/3503555986 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32840b1230 0x7f32841596b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.741+0000 7f328bc00640 1 -- 172.21.15.31:0/3503555986 >> 172.21.15.31:0/3503555986 conn(0x7f3284068f10 msgr2=0x7f32840cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.741+0000 7f328bc00640 1 -- 172.21.15.31:0/3503555986 shutdown_connections 2024-09-11T00:37:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.741+0000 7f328bc00640 1 -- 172.21.15.31:0/3503555986 wait complete. 2024-09-11T00:37:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.742+0000 7f328bc00640 1 Processor -- start 2024-09-11T00:37:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.742+0000 7f328bc00640 1 -- start start 2024-09-11T00:37:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.743+0000 7f328bc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32840b1230 0x7f32841514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.743+0000 7f328bc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32840c9b00 0x7f32840c3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.743+0000 7f328bc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32840c9fe0 0x7f32841ebff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.743+0000 7f328bc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3284156540 con 0x7f32840b1230 2024-09-11T00:37:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.743+0000 7f328bc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3284156260 con 0x7f32840c9b00 2024-09-11T00:37:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.743+0000 7f328bc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3284156fa0 con 0x7f32840c9fe0 2024-09-11T00:37:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.744+0000 7f3289200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32840b1230 0x7f32841514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.744+0000 7f3289200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32840b1230 0x7f32841514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47596/0 (socket says 172.21.15.31:47596) 2024-09-11T00:37:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.744+0000 7f3289200640 1 -- 172.21.15.31:0/3307516025 learned_addr learned my addr 172.21.15.31:0/3307516025 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:20.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.744+0000 7f3289c00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32840c9fe0 0x7f32841ebff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:20.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.744+0000 7f3283e00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32840c9b00 0x7f32840c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:20.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.744+0000 7f3289200640 1 -- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32840c9fe0 msgr2=0x7f32841ebff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:20.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.744+0000 7f3289200640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32840c9fe0 0x7f32841ebff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.744+0000 7f3289200640 1 -- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32840c9b00 msgr2=0x7f32840c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:20.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.744+0000 7f3289200640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32840c9b00 0x7f32840c3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.744+0000 7f3289200640 1 -- 172.21.15.31:0/3307516025 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32841ef7b0 con 0x7f32840b1230 2024-09-11T00:37:20.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.745+0000 7f3289200640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32840b1230 0x7f32841514d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3278034040 tx=0x7f32780038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:20.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.745+0000 7f3280c00640 1 -- 172.21.15.31:0/3307516025 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327803e030 con 0x7f32840b1230 2024-09-11T00:37:20.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.745+0000 7f3280c00640 1 -- 172.21.15.31:0/3307516025 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3278003d90 con 0x7f32840b1230 2024-09-11T00:37:20.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.745+0000 7f3280c00640 1 -- 172.21.15.31:0/3307516025 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278035040 con 0x7f32840b1230 2024-09-11T00:37:20.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.745+0000 7f328bc00640 1 -- 172.21.15.31:0/3307516025 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32841efa90 con 0x7f32840b1230 2024-09-11T00:37:20.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.745+0000 7f328bc00640 1 -- 172.21.15.31:0/3307516025 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32841f0050 con 0x7f32840b1230 2024-09-11T00:37:20.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.746+0000 7f328bc00640 1 -- 172.21.15.31:0/3307516025 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32840766d0 con 0x7f32840b1230 2024-09-11T00:37:20.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.747+0000 7f3280c00640 1 -- 172.21.15.31:0/3307516025 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3278004a00 con 0x7f32840b1230 2024-09-11T00:37:20.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.748+0000 7f3280c00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3264077f70 0x7f326407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:20.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.748+0000 7f3280c00640 1 -- 172.21.15.31:0/3307516025 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f32780bac00 con 0x7f32840b1230 2024-09-11T00:37:20.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.749+0000 7f3283e00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3264077f70 0x7f326407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:20.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.749+0000 7f3283e00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3264077f70 0x7f326407a430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f328415a5d0 tx=0x7f3270005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:20.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.753+0000 7f3280c00640 1 -- 172.21.15.31:0/3307516025 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3278082a00 con 0x7f32840b1230 2024-09-11T00:37:20.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.792+0000 7f434ca00640 1 -- 172.21.15.31:0/2733715940 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4348157f20 con 0x7f43480c44f0 2024-09-11T00:37:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.793+0000 7f433e000640 1 -- 172.21.15.31:0/2733715940 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5709 (secure 0 0 0) 0x7f4338061900 con 0x7f43480c44f0 2024-09-11T00:37:20.796 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:20.796 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":60,"btime":"2024-09-11T00:37:19:382253+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 60 2024-09-11T00:37:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 -- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4328077fa0 msgr2=0x7f432807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4328077fa0 0x7f432807a460 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4334033000 tx=0x7f4334004730 comp rx=0 tx=0).stop 2024-09-11T00:37:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 -- 172.21.15.31:0/2733715940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43480c44f0 msgr2=0x7f43480ad670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43480c44f0 0x7f43480ad670 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f433800c500 tx=0x7f433800c9c0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 -- 172.21.15.31:0/2733715940 shutdown_connections 2024-09-11T00:37:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4328077fa0 0x7f432807a460 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f43480c81f0 0x7f43481df980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4348157a70 0x7f43480adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 --2- 172.21.15.31:0/2733715940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f43480c44f0 0x7f43480ad670 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 -- 172.21.15.31:0/2733715940 >> 172.21.15.31:0/2733715940 conn(0x7f43480ab8a0 msgr2=0x7f4348157340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 -- 172.21.15.31:0/2733715940 shutdown_connections 2024-09-11T00:37:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:20.796+0000 7f434ca00640 1 -- 172.21.15.31:0/2733715940 wait complete. 2024-09-11T00:37:20.817 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5302 2024-09-11T00:37:20.817 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.d 2024-09-11T00:37:21.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.054+0000 7f328bc00640 1 -- 172.21.15.31:0/3307516025 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32840ccb00 con 0x7f32840b1230 2024-09-11T00:37:21.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.055+0000 7f3280c00640 1 -- 172.21.15.31:0/3307516025 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5709 (secure 0 0 0) 0x7f3278087430 con 0x7f32840b1230 2024-09-11T00:37:21.058 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:21.058 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":60,"btime":"2024-09-11T00:37:19:382253+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:21.058 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 60 2024-09-11T00:37:21.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.057+0000 7f328bc00640 1 -- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3264077f70 msgr2=0x7f326407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:21.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.057+0000 7f328bc00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3264077f70 0x7f326407a430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f328415a5d0 tx=0x7f3270005f90 comp rx=0 tx=0).stop 2024-09-11T00:37:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.057+0000 7f328bc00640 1 -- 172.21.15.31:0/3307516025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32840b1230 msgr2=0x7f32841514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.057+0000 7f328bc00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32840b1230 0x7f32841514d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3278034040 tx=0x7f32780038a0 comp rx=0 tx=0).stop 2024-09-11T00:37:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.057+0000 7f328bc00640 1 -- 172.21.15.31:0/3307516025 shutdown_connections 2024-09-11T00:37:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.057+0000 7f328bc00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3264077f70 0x7f326407a430 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.057+0000 7f328bc00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32840c9fe0 0x7f32841ebff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.057+0000 7f328bc00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32840c9b00 0x7f32840c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.057+0000 7f328bc00640 1 --2- 172.21.15.31:0/3307516025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f32840b1230 0x7f32841514d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.057+0000 7f328bc00640 1 -- 172.21.15.31:0/3307516025 >> 172.21.15.31:0/3307516025 conn(0x7f3284068f10 msgr2=0x7f32840caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.058+0000 7f328bc00640 1 -- 172.21.15.31:0/3307516025 shutdown_connections 2024-09-11T00:37:21.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:21.058+0000 7f328bc00640 1 -- 172.21.15.31:0/3307516025 wait complete. 2024-09-11T00:37:21.079 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:active state 2024-09-11T00:37:21.080 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2024-09-11T00:37:22.377 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-11T00:37:22.374+0000 7f9240400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:37:26.818 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-11T00:37:26.818 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:37:26.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:37:26.822 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:37:26.823 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:26.964 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:37:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.995+0000 7faf18800640 1 -- 172.21.15.31:0/1496014774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf100c5b10 msgr2=0x7faf10156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.995+0000 7faf18800640 1 --2- 172.21.15.31:0/1496014774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf100c5b10 0x7faf10156ad0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7faf1006e7f0 tx=0x7faf0402a560 comp rx=0 tx=0).stop 2024-09-11T00:37:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.995+0000 7faf18800640 1 -- 172.21.15.31:0/1496014774 shutdown_connections 2024-09-11T00:37:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.995+0000 7faf18800640 1 --2- 172.21.15.31:0/1496014774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf100c6340 0x7faf101539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.995+0000 7faf18800640 1 --2- 172.21.15.31:0/1496014774 >> v2:172.21.15.204:3300/0 conn(0x7faf100c5ef0 0x7faf10153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.995+0000 7faf18800640 1 --2- 172.21.15.31:0/1496014774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf100c5b10 0x7faf10156ad0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.995+0000 7faf18800640 1 -- 172.21.15.31:0/1496014774 >> 172.21.15.31:0/1496014774 conn(0x7faf100a9750 msgr2=0x7faf100a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.995+0000 7faf18800640 1 -- 172.21.15.31:0/1496014774 shutdown_connections 2024-09-11T00:37:27.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.996+0000 7faf18800640 1 -- 172.21.15.31:0/1496014774 wait complete. 2024-09-11T00:37:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.998+0000 7faf18800640 1 Processor -- start 2024-09-11T00:37:27.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.998+0000 7faf18800640 1 -- start start 2024-09-11T00:37:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf18800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf100c5b10 0x7faf10158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf18800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf100c5ef0 0x7faf1014c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf18800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf100c6340 0x7faf1014c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf18800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faf101597f0 con 0x7faf100c5ef0 2024-09-11T00:37:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf18800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faf10159670 con 0x7faf100c6340 2024-09-11T00:37:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf18800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faf10159970 con 0x7faf100c5b10 2024-09-11T00:37:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf15400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf100c5ef0 0x7faf1014c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf16800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf100c6340 0x7faf1014c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf15400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf100c5ef0 0x7faf1014c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47614/0 (socket says 172.21.15.31:47614) 2024-09-11T00:37:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf15400640 1 -- 172.21.15.31:0/2206673813 learned_addr learned my addr 172.21.15.31:0/2206673813 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:27.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:26.999+0000 7faf15e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf100c5b10 0x7faf10158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.000+0000 7faf15400640 1 -- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf100c5b10 msgr2=0x7faf10158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:27.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.000+0000 7faf15400640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf100c5b10 0x7faf10158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.000+0000 7faf15400640 1 -- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf100c6340 msgr2=0x7faf1014c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.000+0000 7faf15400640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf100c6340 0x7faf1014c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.000+0000 7faf15400640 1 -- 172.21.15.31:0/2206673813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf10152c90 con 0x7faf100c5ef0 2024-09-11T00:37:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.000+0000 7faf16800640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf100c6340 0x7faf1014c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:37:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.000+0000 7faf15400640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf100c5ef0 0x7faf1014c050 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faf0000e9f0 tx=0x7faf0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.001+0000 7faf0a000640 1 -- 172.21.15.31:0/2206673813 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0000cd80 con 0x7faf100c5ef0 2024-09-11T00:37:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.001+0000 7faf0a000640 1 -- 172.21.15.31:0/2206673813 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf00010430 con 0x7faf100c5ef0 2024-09-11T00:37:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.001+0000 7faf18800640 1 -- 172.21.15.31:0/2206673813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf1014cd60 con 0x7faf100c5ef0 2024-09-11T00:37:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.001+0000 7faf0a000640 1 -- 172.21.15.31:0/2206673813 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00010840 con 0x7faf100c5ef0 2024-09-11T00:37:27.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.001+0000 7faf18800640 1 -- 172.21.15.31:0/2206673813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf101ec370 con 0x7faf100c5ef0 2024-09-11T00:37:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.002+0000 7faf18800640 1 -- 172.21.15.31:0/2206673813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf10156ad0 con 0x7faf100c5ef0 2024-09-11T00:37:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.004+0000 7faf0a000640 1 -- 172.21.15.31:0/2206673813 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7faf000109e0 con 0x7faf100c5ef0 2024-09-11T00:37:27.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.005+0000 7faf0a000640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faef4077f70 0x7faef407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:27.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.005+0000 7faf0a000640 1 -- 172.21.15.31:0/2206673813 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7faf0009af70 con 0x7faf100c5ef0 2024-09-11T00:37:27.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.006+0000 7faf15e00640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faef4077f70 0x7faef407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:27.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.006+0000 7faf15e00640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faef4077f70 0x7faef407a430 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faf04033000 tx=0x7faf04004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:27.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.009+0000 7faf0a000640 1 -- 172.21.15.31:0/2206673813 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7faf00062da0 con 0x7faf100c5ef0 2024-09-11T00:37:27.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.288+0000 7faf18800640 1 -- 172.21.15.31:0/2206673813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf10152f10 con 0x7faf100c5ef0 2024-09-11T00:37:27.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.289+0000 7faf0a000640 1 -- 172.21.15.31:0/2206673813 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5709 (secure 0 0 0) 0x7faf000677d0 con 0x7faf100c5ef0 2024-09-11T00:37:27.292 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:27.292 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":60,"btime":"2024-09-11T00:37:19:382253+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:27.292 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 60 2024-09-11T00:37:27.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 -- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faef4077f70 msgr2=0x7faef407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:27.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faef4077f70 0x7faef407a430 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faf04033000 tx=0x7faf04004730 comp rx=0 tx=0).stop 2024-09-11T00:37:27.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 -- 172.21.15.31:0/2206673813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf100c5ef0 msgr2=0x7faf1014c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:27.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf100c5ef0 0x7faf1014c050 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faf0000e9f0 tx=0x7faf0000eeb0 comp rx=0 tx=0).stop 2024-09-11T00:37:27.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 -- 172.21.15.31:0/2206673813 shutdown_connections 2024-09-11T00:37:27.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faef4077f70 0x7faef407a430 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:27.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf100c6340 0x7faf1014c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:27.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf100c5ef0 0x7faf1014c050 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:27.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 --2- 172.21.15.31:0/2206673813 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf100c5b10 0x7faf10158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:27.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 -- 172.21.15.31:0/2206673813 >> 172.21.15.31:0/2206673813 conn(0x7faf100a9750 msgr2=0x7faf100c4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:27.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 -- 172.21.15.31:0/2206673813 shutdown_connections 2024-09-11T00:37:27.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:27.292+0000 7faf18800640 1 -- 172.21.15.31:0/2206673813 wait complete. 2024-09-11T00:37:27.314 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:37:29.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.491+0000 7fd624a00640 1 -- 172.21.15.31:0/3549943860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6200a6e20 msgr2=0x7fd6201510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.491+0000 7fd624a00640 1 --2- 172.21.15.31:0/3549943860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6200a6e20 0x7fd6201510c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009830 tx=0x7fd60c02a3f0 comp rx=0 tx=0).stop 2024-09-11T00:37:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.492+0000 7fd624a00640 1 -- 172.21.15.31:0/3549943860 shutdown_connections 2024-09-11T00:37:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.492+0000 7fd624a00640 1 --2- 172.21.15.31:0/3549943860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6200a6e20 0x7fd6201510c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.492+0000 7fd624a00640 1 --2- 172.21.15.31:0/3549943860 >> v2:172.21.15.204:3300/0 conn(0x7fd6200ac2c0 0x7fd6200ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.492+0000 7fd624a00640 1 -- 172.21.15.31:0/3549943860 >> 172.21.15.31:0/3549943860 conn(0x7fd62006caf0 msgr2=0x7fd62006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.492+0000 7fd624a00640 1 -- 172.21.15.31:0/3549943860 shutdown_connections 2024-09-11T00:37:29.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.492+0000 7fd624a00640 1 -- 172.21.15.31:0/3549943860 wait complete. 2024-09-11T00:37:29.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.493+0000 7fd624a00640 1 Processor -- start 2024-09-11T00:37:29.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.493+0000 7fd624a00640 1 -- start start 2024-09-11T00:37:29.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.494+0000 7fd624a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6200a6e20 0x7fd620151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:29.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.494+0000 7fd624a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6200ac2c0 0x7fd620145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:29.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.494+0000 7fd624a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6200acbe0 0x7fd620145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.494+0000 7fd624a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd620151780 con 0x7fd6200a6e20 2024-09-11T00:37:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.494+0000 7fd624a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd620151600 con 0x7fd6200acbe0 2024-09-11T00:37:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.494+0000 7fd624a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd620151900 con 0x7fd6200ac2c0 2024-09-11T00:37:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6200a6e20 0x7fd620151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6200acbe0 0x7fd620145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6200ac2c0 0x7fd620145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6200a6e20 0x7fd620151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40968/0 (socket says 172.21.15.31:40968) 2024-09-11T00:37:29.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61e800640 1 -- 172.21.15.31:0/3957305486 learned_addr learned my addr 172.21.15.31:0/3957305486 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61e800640 1 -- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6200ac2c0 msgr2=0x7fd620145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61e800640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6200ac2c0 0x7fd620145750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61e800640 1 -- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6200acbe0 msgr2=0x7fd620145c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61e800640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6200acbe0 0x7fd620145c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:29.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61e800640 1 -- 172.21.15.31:0/3957305486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6201461d0 con 0x7fd6200a6e20 2024-09-11T00:37:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.495+0000 7fd61f200640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6200acbe0 0x7fd620145c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:37:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.496+0000 7fd61e800640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6200a6e20 0x7fd620151ba0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd61000ea60 tx=0x7fd61000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.496+0000 7fd616a00640 1 -- 172.21.15.31:0/3957305486 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61000cde0 con 0x7fd6200a6e20 2024-09-11T00:37:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.496+0000 7fd624a00640 1 -- 172.21.15.31:0/3957305486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6200c3f20 con 0x7fd6200a6e20 2024-09-11T00:37:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.496+0000 7fd616a00640 1 -- 172.21.15.31:0/3957305486 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd610004020 con 0x7fd6200a6e20 2024-09-11T00:37:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.496+0000 7fd616a00640 1 -- 172.21.15.31:0/3957305486 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610004450 con 0x7fd6200a6e20 2024-09-11T00:37:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.496+0000 7fd624a00640 1 -- 172.21.15.31:0/3957305486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6200c44e0 con 0x7fd6200a6e20 2024-09-11T00:37:29.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.497+0000 7fd624a00640 1 -- 172.21.15.31:0/3957305486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6201510c0 con 0x7fd6200a6e20 2024-09-11T00:37:29.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.499+0000 7fd616a00640 1 -- 172.21.15.31:0/3957305486 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd6100035f0 con 0x7fd6200a6e20 2024-09-11T00:37:29.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.500+0000 7fd616a00640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd600077f70 0x7fd60007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:29.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.500+0000 7fd616a00640 1 -- 172.21.15.31:0/3957305486 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7fd61009ac80 con 0x7fd6200a6e20 2024-09-11T00:37:29.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.500+0000 7fd61de00640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd600077f70 0x7fd60007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:29.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.501+0000 7fd61de00640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd600077f70 0x7fd60007a430 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd60c034000 tx=0x7fd60c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:29.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.506+0000 7fd616a00640 1 -- 172.21.15.31:0/3957305486 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd610062a80 con 0x7fd6200a6e20 2024-09-11T00:37:29.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.764+0000 7fd624a00640 1 -- 172.21.15.31:0/3957305486 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6201eedf0 con 0x7fd6200a6e20 2024-09-11T00:37:29.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.764+0000 7fd616a00640 1 -- 172.21.15.31:0/3957305486 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7fd6100674b0 con 0x7fd6200a6e20 2024-09-11T00:37:29.767 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:29.767 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:29.768 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:37:29.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 -- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd600077f70 msgr2=0x7fd60007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:29.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd600077f70 0x7fd60007a430 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd60c034000 tx=0x7fd60c004730 comp rx=0 tx=0).stop 2024-09-11T00:37:29.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 -- 172.21.15.31:0/3957305486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6200a6e20 msgr2=0x7fd620151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:29.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6200a6e20 0x7fd620151ba0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd61000ea60 tx=0x7fd61000ef20 comp rx=0 tx=0).stop 2024-09-11T00:37:29.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 -- 172.21.15.31:0/3957305486 shutdown_connections 2024-09-11T00:37:29.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd600077f70 0x7fd60007a430 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:29.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6200acbe0 0x7fd620145c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:29.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6200ac2c0 0x7fd620145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:29.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 --2- 172.21.15.31:0/3957305486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6200a6e20 0x7fd620151ba0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:29.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 -- 172.21.15.31:0/3957305486 >> 172.21.15.31:0/3957305486 conn(0x7fd62006caf0 msgr2=0x7fd620152ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:29.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 -- 172.21.15.31:0/3957305486 shutdown_connections 2024-09-11T00:37:29.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:29.767+0000 7fd624a00640 1 -- 172.21.15.31:0/3957305486 wait complete. 2024-09-11T00:37:29.786 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-11T00:37:29.787 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2024-09-11T00:37:43.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:37:43.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:37:55.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:55.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.257+0000 7ff46be00640 1 -- 172.21.15.31:0/795481815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff46c150020 msgr2=0x7ff46c0ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.257+0000 7ff46be00640 1 --2- 172.21.15.31:0/795481815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff46c150020 0x7ff46c0ad4e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff46c153d20 tx=0x7ff45c02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.257+0000 7ff46be00640 1 -- 172.21.15.31:0/795481815 shutdown_connections 2024-09-11T00:37:55.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.257+0000 7ff46be00640 1 --2- 172.21.15.31:0/795481815 >> v2:172.21.15.204:3300/0 conn(0x7ff46c06e850 0x7ff46c160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.257+0000 7ff46be00640 1 --2- 172.21.15.31:0/795481815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff46c150020 0x7ff46c0ad4e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.257+0000 7ff46be00640 1 -- 172.21.15.31:0/795481815 >> 172.21.15.31:0/795481815 conn(0x7ff46c068f10 msgr2=0x7ff46c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:55.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.257+0000 7ff46be00640 1 -- 172.21.15.31:0/795481815 shutdown_connections 2024-09-11T00:37:55.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.257+0000 7ff46be00640 1 -- 172.21.15.31:0/795481815 wait complete. 2024-09-11T00:37:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.259+0000 7ff46be00640 1 Processor -- start 2024-09-11T00:37:55.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.260+0000 7ff46be00640 1 -- start start 2024-09-11T00:37:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff46c06e850 0x7ff46c157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff46c0ada20 0x7ff46c0c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff46c150020 0x7ff46c0c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c157550 con 0x7ff46c06e850 2024-09-11T00:37:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c1573d0 con 0x7ff46c0ada20 2024-09-11T00:37:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff46c156960 con 0x7ff46c150020 2024-09-11T00:37:55.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff46c06e850 0x7ff46c157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff46c06e850 0x7ff46c157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49468/0 (socket says 172.21.15.31:49468) 2024-09-11T00:37:55.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46aa00640 1 -- 172.21.15.31:0/3413925382 learned_addr learned my addr 172.21.15.31:0/3413925382 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:55.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46b400640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff46c150020 0x7ff46c0c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.261+0000 7ff46a000640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff46c0ada20 0x7ff46c0c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.262+0000 7ff46aa00640 1 -- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff46c150020 msgr2=0x7ff46c0c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.262+0000 7ff46aa00640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff46c150020 0x7ff46c0c5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.262+0000 7ff46aa00640 1 -- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff46c0ada20 msgr2=0x7ff46c0c4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.262+0000 7ff46aa00640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff46c0ada20 0x7ff46c0c4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.262+0000 7ff46aa00640 1 -- 172.21.15.31:0/3413925382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46c0c5e10 con 0x7ff46c06e850 2024-09-11T00:37:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.262+0000 7ff46b400640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff46c150020 0x7ff46c0c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:37:55.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.262+0000 7ff46aa00640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff46c06e850 0x7ff46c157e50 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff45c034040 tx=0x7ff45c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:55.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.262+0000 7ff462a00640 1 -- 172.21.15.31:0/3413925382 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45c03e030 con 0x7ff46c06e850 2024-09-11T00:37:55.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.262+0000 7ff462a00640 1 -- 172.21.15.31:0/3413925382 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff45c003db0 con 0x7ff46c06e850 2024-09-11T00:37:55.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.262+0000 7ff462a00640 1 -- 172.21.15.31:0/3413925382 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45c035040 con 0x7ff46c06e850 2024-09-11T00:37:55.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.263+0000 7ff46be00640 1 -- 172.21.15.31:0/3413925382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff46c14ce00 con 0x7ff46c06e850 2024-09-11T00:37:55.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.263+0000 7ff46be00640 1 -- 172.21.15.31:0/3413925382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff46c158390 con 0x7ff46c06e850 2024-09-11T00:37:55.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.264+0000 7ff46be00640 1 -- 172.21.15.31:0/3413925382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff46c0c5920 con 0x7ff46c06e850 2024-09-11T00:37:55.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.265+0000 7ff462a00640 1 -- 172.21.15.31:0/3413925382 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff45c004a00 con 0x7ff46c06e850 2024-09-11T00:37:55.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.266+0000 7ff462a00640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff44c077f70 0x7ff44c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.266+0000 7ff462a00640 1 -- 172.21.15.31:0/3413925382 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7ff45c0bac20 con 0x7ff46c06e850 2024-09-11T00:37:55.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.266+0000 7ff46a000640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff44c077f70 0x7ff44c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.267+0000 7ff46a000640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff44c077f70 0x7ff44c07a430 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff458007fd0 tx=0x7ff458009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:55.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.271+0000 7ff462a00640 1 -- 172.21.15.31:0/3413925382 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff45c082a20 con 0x7ff46c06e850 2024-09-11T00:37:55.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.553+0000 7ff46be00640 1 -- 172.21.15.31:0/3413925382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff46c1dff50 con 0x7ff46c06e850 2024-09-11T00:37:55.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.553+0000 7ff462a00640 1 -- 172.21.15.31:0/3413925382 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7ff45c087450 con 0x7ff46c06e850 2024-09-11T00:37:55.555 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:55.555 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:55.556 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:37:55.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.555+0000 7ff46be00640 1 -- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff44c077f70 msgr2=0x7ff44c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.555+0000 7ff46be00640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff44c077f70 0x7ff44c07a430 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff458007fd0 tx=0x7ff458009040 comp rx=0 tx=0).stop 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.555+0000 7ff46be00640 1 -- 172.21.15.31:0/3413925382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff46c06e850 msgr2=0x7ff46c157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.555+0000 7ff46be00640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff46c06e850 0x7ff46c157e50 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff45c034040 tx=0x7ff45c0038a0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.556+0000 7ff46be00640 1 -- 172.21.15.31:0/3413925382 shutdown_connections 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.556+0000 7ff46be00640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff44c077f70 0x7ff44c07a430 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.556+0000 7ff46be00640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff46c150020 0x7ff46c0c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.556+0000 7ff46be00640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff46c0ada20 0x7ff46c0c4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.556+0000 7ff46be00640 1 --2- 172.21.15.31:0/3413925382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff46c06e850 0x7ff46c157e50 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.556+0000 7ff46be00640 1 -- 172.21.15.31:0/3413925382 >> 172.21.15.31:0/3413925382 conn(0x7ff46c068f10 msgr2=0x7ff46c14e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.556+0000 7ff46be00640 1 -- 172.21.15.31:0/3413925382 shutdown_connections 2024-09-11T00:37:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.556+0000 7ff46be00640 1 -- 172.21.15.31:0/3413925382 wait complete. 2024-09-11T00:37:55.574 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-11T00:37:55.574 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-11T00:37:55.574 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:37:55.592 INFO:tasks.ceph.mds.b:Stopped 2024-09-11T00:37:55.592 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:55.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.767+0000 7f69aea00640 1 -- 172.21.15.31:0/4106722086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69a80bf2c0 msgr2=0x7f69a815a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.767+0000 7f69aea00640 1 --2- 172.21.15.31:0/4106722086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69a80bf2c0 0x7f69a815a1a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f69a0005fd0 tx=0x7f69a002a8b0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.767+0000 7f69aea00640 1 -- 172.21.15.31:0/4106722086 shutdown_connections 2024-09-11T00:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.767+0000 7f69aea00640 1 --2- 172.21.15.31:0/4106722086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69a80bf2c0 0x7f69a815a1a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.767+0000 7f69aea00640 1 -- 172.21.15.31:0/4106722086 >> 172.21.15.31:0/4106722086 conn(0x7f69a80cc260 msgr2=0x7f69a80ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.767+0000 7f69aea00640 1 -- 172.21.15.31:0/4106722086 shutdown_connections 2024-09-11T00:37:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.767+0000 7f69aea00640 1 -- 172.21.15.31:0/4106722086 wait complete. 2024-09-11T00:37:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.769+0000 7f69aea00640 1 Processor -- start 2024-09-11T00:37:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.769+0000 7f69aea00640 1 -- start start 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.770+0000 7f69aea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69a80b1160 0x7f69a814f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69aea00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f69a80bf2c0 0x7f69a81e1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69aea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80c9b30 0x7f69a81ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69aea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f69a80b1540 con 0x7f69a80c9b30 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69aea00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f69a806e5c0 con 0x7f69a80b1160 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69aea00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f69a80ae540 con 0x7f69a80bf2c0 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69aca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80c9b30 0x7f69a81ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69a7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f69a80bf2c0 0x7f69a81e1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69a7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69a80b1160 0x7f69a814f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69aca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80c9b30 0x7f69a81ebc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49478/0 (socket says 172.21.15.31:49478) 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69a7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69a80b1160 0x7f69a814f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:60686/0 (socket says 172.21.15.31:60686) 2024-09-11T00:37:55.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.771+0000 7f69aca00640 1 -- 172.21.15.31:0/1911743819 learned_addr learned my addr 172.21.15.31:0/1911743819 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f69a7e00640 1 -- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f69a80bf2c0 msgr2=0x7f69a81e1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f69a7e00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f69a80bf2c0 0x7f69a81e1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f69a7e00640 1 -- 172.21.15.31:0/1911743819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80c9b30 msgr2=0x7f69a81ebc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f69a7e00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80c9b30 0x7f69a81ebc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f69a7e00640 1 -- 172.21.15.31:0/1911743819 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a80acfa0 con 0x7f69a80b1160 2024-09-11T00:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f69aca00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80c9b30 0x7f69a81ebc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f69a7400640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f69a80bf2c0 0x7f69a81e1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:37:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f69a7e00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69a80b1160 0x7f69a814f8c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f69a81598d0 tx=0x7f699800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f699fe00640 1 -- 172.21.15.31:0/1911743819 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998017070 con 0x7f69a80b1160 2024-09-11T00:37:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f699fe00640 1 -- 172.21.15.31:0/1911743819 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69980053c0 con 0x7f69a80b1160 2024-09-11T00:37:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f69aea00640 1 -- 172.21.15.31:0/1911743819 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a81ef390 con 0x7f69a80b1160 2024-09-11T00:37:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f699fe00640 1 -- 172.21.15.31:0/1911743819 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998003870 con 0x7f69a80b1160 2024-09-11T00:37:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.772+0000 7f69aea00640 1 -- 172.21.15.31:0/1911743819 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a80cb980 con 0x7f69a80b1160 2024-09-11T00:37:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.773+0000 7f69aea00640 1 -- 172.21.15.31:0/1911743819 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69a81ef980 con 0x7f69a80b1160 2024-09-11T00:37:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.775+0000 7f699fe00640 1 -- 172.21.15.31:0/1911743819 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6998003a10 con 0x7f69a80b1160 2024-09-11T00:37:55.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.776+0000 7f699fe00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6988077fa0 0x7f698807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.776+0000 7f699fe00640 1 -- 172.21.15.31:0/1911743819 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f6998099770 con 0x7f69a80b1160 2024-09-11T00:37:55.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.776+0000 7f69a7400640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6988077fa0 0x7f698807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.777+0000 7f69a7400640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6988077fa0 0x7f698807a460 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f69a80acdd0 tx=0x7f6994009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:55.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.781+0000 7f699fe00640 1 -- 172.21.15.31:0/1911743819 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6998061570 con 0x7f69a80b1160 2024-09-11T00:37:55.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:55.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.961+0000 7f0186800640 1 -- 172.21.15.31:0/2276045928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01800c5b10 msgr2=0x7f0180156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.961+0000 7f0186800640 1 --2- 172.21.15.31:0/2276045928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01800c5b10 0x7f0180156ad0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0170005fd0 tx=0x7f017002a5d0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.961+0000 7f0186800640 1 -- 172.21.15.31:0/2276045928 shutdown_connections 2024-09-11T00:37:55.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.961+0000 7f0186800640 1 --2- 172.21.15.31:0/2276045928 >> v2:172.21.15.31:3300/0 conn(0x7f01800c6310 0x7f0180159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.961+0000 7f0186800640 1 --2- 172.21.15.31:0/2276045928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01800c5b10 0x7f0180156ad0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.961+0000 7f0186800640 1 -- 172.21.15.31:0/2276045928 >> 172.21.15.31:0/2276045928 conn(0x7f01800a9750 msgr2=0x7f01800a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:55.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.961+0000 7f0186800640 1 -- 172.21.15.31:0/2276045928 shutdown_connections 2024-09-11T00:37:55.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.961+0000 7f0186800640 1 -- 172.21.15.31:0/2276045928 wait complete. 2024-09-11T00:37:55.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.962+0000 7f0186800640 1 Processor -- start 2024-09-11T00:37:55.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.963+0000 7f0186800640 1 -- start start 2024-09-11T00:37:55.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.963+0000 7f0186800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01800c5b10 0x7f0180152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.963+0000 7f0186800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01800c5ef0 0x7f01801576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.963+0000 7f0186800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01800c6310 0x7f01801eb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f0186800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0180153b90 con 0x7f01800c6310 2024-09-11T00:37:55.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f0186800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f01800c3950 con 0x7f01800c5ef0 2024-09-11T00:37:55.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f0186800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0180153d10 con 0x7f01800c5b10 2024-09-11T00:37:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01800c6310 0x7f01801eb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01800c5ef0 0x7f01801576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01800c6310 0x7f01801eb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49496/0 (socket says 172.21.15.31:49496) 2024-09-11T00:37:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01800c5ef0 0x7f01801576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:60698/0 (socket says 172.21.15.31:60698) 2024-09-11T00:37:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017fe00640 1 -- 172.21.15.31:0/3200432326 learned_addr learned my addr 172.21.15.31:0/3200432326 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017f400640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01800c5b10 0x7f0180152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017ea00640 1 -- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01800c5b10 msgr2=0x7f0180152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017ea00640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01800c5b10 0x7f0180152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017ea00640 1 -- 172.21.15.31:0/3200432326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01800c6310 msgr2=0x7f01801eb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:55.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017ea00640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01800c6310 0x7f01801eb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:55.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.964+0000 7f017ea00640 1 -- 172.21.15.31:0/3200432326 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018014f970 con 0x7f01800c5ef0 2024-09-11T00:37:55.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.965+0000 7f017ea00640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01800c5ef0 0x7f01801576e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f016c00c390 tx=0x7f016c00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:55.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.965+0000 7f0177400640 1 -- 172.21.15.31:0/3200432326 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c017070 con 0x7f01800c5ef0 2024-09-11T00:37:55.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.965+0000 7f0177400640 1 -- 172.21.15.31:0/3200432326 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f016c0053c0 con 0x7f01800c5ef0 2024-09-11T00:37:55.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.965+0000 7f0177400640 1 -- 172.21.15.31:0/3200432326 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c003850 con 0x7f01800c5ef0 2024-09-11T00:37:55.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.965+0000 7f0186800640 1 -- 172.21.15.31:0/3200432326 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018014f690 con 0x7f01800c5ef0 2024-09-11T00:37:55.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.965+0000 7f0186800640 1 -- 172.21.15.31:0/3200432326 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01801ef3a0 con 0x7f01800c5ef0 2024-09-11T00:37:55.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.967+0000 7f0186800640 1 -- 172.21.15.31:0/3200432326 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0180156ad0 con 0x7f01800c5ef0 2024-09-11T00:37:55.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.968+0000 7f0177400640 1 -- 172.21.15.31:0/3200432326 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f016c0039f0 con 0x7f01800c5ef0 2024-09-11T00:37:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.969+0000 7f0177400640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0160077fa0 0x7f016007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.969+0000 7f0177400640 1 -- 172.21.15.31:0/3200432326 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f016c099920 con 0x7f01800c5ef0 2024-09-11T00:37:55.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.970+0000 7f017f400640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0160077fa0 0x7f016007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:55.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.970+0000 7f017f400640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0160077fa0 0x7f016007a460 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0170033000 tx=0x7f0170004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:55.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:55.974+0000 7f0177400640 1 -- 172.21.15.31:0/3200432326 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f016c061720 con 0x7f01800c5ef0 2024-09-11T00:37:56.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.048+0000 7f69aea00640 1 -- 172.21.15.31:0/1911743819 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69a806f300 con 0x7f69a80b1160 2024-09-11T00:37:56.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.049+0000 7f699fe00640 1 -- 172.21.15.31:0/1911743819 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f699806a2e0 con 0x7f69a80b1160 2024-09-11T00:37:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:56.051 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:56.051 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 -- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6988077fa0 msgr2=0x7f698807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6988077fa0 0x7f698807a460 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f69a80acdd0 tx=0x7f6994009040 comp rx=0 tx=0).stop 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 -- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69a80b1160 msgr2=0x7f69a814f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69a80b1160 0x7f69a814f8c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f69a81598d0 tx=0x7f699800c820 comp rx=0 tx=0).stop 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 -- 172.21.15.31:0/1911743819 shutdown_connections 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6988077fa0 0x7f698807a460 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69a80c9b30 0x7f69a81ebc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f69a80bf2c0 0x7f69a81e1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 --2- 172.21.15.31:0/1911743819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69a80b1160 0x7f69a814f8c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 -- 172.21.15.31:0/1911743819 >> 172.21.15.31:0/1911743819 conn(0x7f69a80cc260 msgr2=0x7f69a80ca7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 -- 172.21.15.31:0/1911743819 shutdown_connections 2024-09-11T00:37:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.052+0000 7f69aea00640 1 -- 172.21.15.31:0/1911743819 wait complete. 2024-09-11T00:37:56.073 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:37:56.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.226+0000 7f0186800640 1 -- 172.21.15.31:0/3200432326 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01800c1e70 con 0x7f01800c5ef0 2024-09-11T00:37:56.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.227+0000 7f0177400640 1 -- 172.21.15.31:0/3200432326 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f016c066150 con 0x7f01800c5ef0 2024-09-11T00:37:56.229 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:56.229 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:56.229 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.229+0000 7f0186800640 1 -- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0160077fa0 msgr2=0x7f016007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.229+0000 7f0186800640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0160077fa0 0x7f016007a460 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0170033000 tx=0x7f0170004730 comp rx=0 tx=0).stop 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.229+0000 7f0186800640 1 -- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01800c5ef0 msgr2=0x7f01801576e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.229+0000 7f0186800640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01800c5ef0 0x7f01801576e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f016c00c390 tx=0x7f016c00c850 comp rx=0 tx=0).stop 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.230+0000 7f0186800640 1 -- 172.21.15.31:0/3200432326 shutdown_connections 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.230+0000 7f0186800640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0160077fa0 0x7f016007a460 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.230+0000 7f0186800640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01800c6310 0x7f01801eb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.230+0000 7f0186800640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01800c5ef0 0x7f01801576e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.230+0000 7f0186800640 1 --2- 172.21.15.31:0/3200432326 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01800c5b10 0x7f0180152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.230+0000 7f0186800640 1 -- 172.21.15.31:0/3200432326 >> 172.21.15.31:0/3200432326 conn(0x7f01800a9750 msgr2=0x7f018006ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:56.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.230+0000 7f0186800640 1 -- 172.21.15.31:0/3200432326 shutdown_connections 2024-09-11T00:37:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.230+0000 7f0186800640 1 -- 172.21.15.31:0/3200432326 wait complete. 2024-09-11T00:37:56.248 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-11T00:37:56.249 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-11T00:37:56.249 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:37:56.267 INFO:tasks.ceph.mds.a:Stopped 2024-09-11T00:37:56.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:56.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.473+0000 7f4a5b200640 1 -- 172.21.15.31:0/4060132130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a5c156350 msgr2=0x7f4a5c156800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:56.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.473+0000 7f4a5b200640 1 --2- 172.21.15.31:0/4060132130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a5c156350 0x7f4a5c156800 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4a54005fd0 tx=0x7f4a5402aa90 comp rx=0 tx=0).stop 2024-09-11T00:37:56.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.473+0000 7f4a5b200640 1 -- 172.21.15.31:0/4060132130 shutdown_connections 2024-09-11T00:37:56.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.473+0000 7f4a5b200640 1 --2- 172.21.15.31:0/4060132130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a5c156350 0x7f4a5c156800 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.473+0000 7f4a5b200640 1 --2- 172.21.15.31:0/4060132130 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a5c0c5aa0 0x7f4a5c0c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.473+0000 7f4a5b200640 1 -- 172.21.15.31:0/4060132130 >> 172.21.15.31:0/4060132130 conn(0x7f4a5c06caf0 msgr2=0x7f4a5c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:56.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.473+0000 7f4a5b200640 1 -- 172.21.15.31:0/4060132130 shutdown_connections 2024-09-11T00:37:56.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.473+0000 7f4a5b200640 1 -- 172.21.15.31:0/4060132130 wait complete. 2024-09-11T00:37:56.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.475+0000 7f4a5b200640 1 Processor -- start 2024-09-11T00:37:56.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.475+0000 7f4a5b200640 1 -- start start 2024-09-11T00:37:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a5b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a5c0c5aa0 0x7f4a5c0ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a5b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a5c0c6420 0x7f4a5c1e1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a5b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a5c156350 0x7f4a5c1e1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a5b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5c0c2e80 con 0x7f4a5c156350 2024-09-11T00:37:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a5b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5c0c2ba0 con 0x7f4a5c0c5aa0 2024-09-11T00:37:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a5b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4a5c156d40 con 0x7f4a5c0c6420 2024-09-11T00:37:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a59e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a5c0c5aa0 0x7f4a5c0ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:56.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a59e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a5c0c5aa0 0x7f4a5c0ab6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:60720/0 (socket says 172.21.15.31:60720) 2024-09-11T00:37:56.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a59e00640 1 -- 172.21.15.31:0/2524995530 learned_addr learned my addr 172.21.15.31:0/2524995530 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:56.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a5a800640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a5c156350 0x7f4a5c1e1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:56.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.477+0000 7f4a59400640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a5c0c6420 0x7f4a5c1e1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a59400640 1 -- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a5c0c5aa0 msgr2=0x7f4a5c0ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a59400640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a5c0c5aa0 0x7f4a5c0ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a59400640 1 -- 172.21.15.31:0/2524995530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a5c156350 msgr2=0x7f4a5c1e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a59400640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a5c156350 0x7f4a5c1e1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a59400640 1 -- 172.21.15.31:0/2524995530 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5c1ec4b0 con 0x7f4a5c0c6420 2024-09-11T00:37:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a59e00640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a5c0c5aa0 0x7f4a5c0ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:37:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a5a800640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a5c156350 0x7f4a5c1e1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:37:56.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a59400640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a5c0c6420 0x7f4a5c1e1820 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800c390 tx=0x7f4a4800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:56.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a52000640 1 -- 172.21.15.31:0/2524995530 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48017070 con 0x7f4a5c0c6420 2024-09-11T00:37:56.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a52000640 1 -- 172.21.15.31:0/2524995530 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a480053c0 con 0x7f4a5c0c6420 2024-09-11T00:37:56.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a52000640 1 -- 172.21.15.31:0/2524995530 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48003850 con 0x7f4a5c0c6420 2024-09-11T00:37:56.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a5b200640 1 -- 172.21.15.31:0/2524995530 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5c0ac080 con 0x7f4a5c0c6420 2024-09-11T00:37:56.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.478+0000 7f4a5b200640 1 -- 172.21.15.31:0/2524995530 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a5c0abbe0 con 0x7f4a5c0c6420 2024-09-11T00:37:56.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.480+0000 7f4a5b200640 1 -- 172.21.15.31:0/2524995530 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a5c074760 con 0x7f4a5c0c6420 2024-09-11T00:37:56.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.481+0000 7f4a52000640 1 -- 172.21.15.31:0/2524995530 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4a480039f0 con 0x7f4a5c0c6420 2024-09-11T00:37:56.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.482+0000 7f4a52000640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a3c077f70 0x7f4a3c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:56.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.482+0000 7f4a52000640 1 -- 172.21.15.31:0/2524995530 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f4a48099900 con 0x7f4a5c0c6420 2024-09-11T00:37:56.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.482+0000 7f4a59e00640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a3c077f70 0x7f4a3c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:56.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.483+0000 7f4a59e00640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a3c077f70 0x7f4a3c07a430 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c011550 tx=0x7f4a4c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:56.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.487+0000 7f4a52000640 1 -- 172.21.15.31:0/2524995530 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4a5c074760 con 0x7f4a5c0c6420 2024-09-11T00:37:56.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.798+0000 7f4a5b200640 1 -- 172.21.15.31:0/2524995530 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a5c0c5e80 con 0x7f4a5c0c6420 2024-09-11T00:37:56.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.799+0000 7f4a52000640 1 -- 172.21.15.31:0/2524995530 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f4a5c0c7690 con 0x7f4a5c0c6420 2024-09-11T00:37:56.801 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:56.801 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:56.801 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:37:56.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.801+0000 7f4a5b200640 1 -- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a3c077f70 msgr2=0x7f4a3c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:56.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.801+0000 7f4a5b200640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a3c077f70 0x7f4a3c07a430 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c011550 tx=0x7f4a4c005f90 comp rx=0 tx=0).stop 2024-09-11T00:37:56.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.801+0000 7f4a5b200640 1 -- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a5c0c6420 msgr2=0x7f4a5c1e1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:56.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.801+0000 7f4a5b200640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a5c0c6420 0x7f4a5c1e1820 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800c390 tx=0x7f4a4800c850 comp rx=0 tx=0).stop 2024-09-11T00:37:56.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.802+0000 7f4a5b200640 1 -- 172.21.15.31:0/2524995530 shutdown_connections 2024-09-11T00:37:56.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.802+0000 7f4a5b200640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a3c077f70 0x7f4a3c07a430 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.802+0000 7f4a5b200640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a5c156350 0x7f4a5c1e1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.802+0000 7f4a5b200640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4a5c0c6420 0x7f4a5c1e1820 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.802+0000 7f4a5b200640 1 --2- 172.21.15.31:0/2524995530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a5c0c5aa0 0x7f4a5c0ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:56.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.802+0000 7f4a5b200640 1 -- 172.21.15.31:0/2524995530 >> 172.21.15.31:0/2524995530 conn(0x7f4a5c06caf0 msgr2=0x7f4a5c14ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:56.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.802+0000 7f4a5b200640 1 -- 172.21.15.31:0/2524995530 shutdown_connections 2024-09-11T00:37:56.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:56.802+0000 7f4a5b200640 1 -- 172.21.15.31:0/2524995530 wait complete. 2024-09-11T00:37:56.822 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:37:58.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:58.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.249+0000 7f13e7200640 1 -- 172.21.15.31:0/1021514638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e00cc220 msgr2=0x7f13e006e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:58.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.249+0000 7f13e7200640 1 --2- 172.21.15.31:0/1021514638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e00cc220 0x7f13e006e680 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f13dc005fd0 tx=0x7f13dc02aa10 comp rx=0 tx=0).stop 2024-09-11T00:37:58.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.249+0000 7f13e7200640 1 -- 172.21.15.31:0/1021514638 shutdown_connections 2024-09-11T00:37:58.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.249+0000 7f13e7200640 1 --2- 172.21.15.31:0/1021514638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e00cc220 0x7f13e006e680 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.249+0000 7f13e7200640 1 --2- 172.21.15.31:0/1021514638 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e0156960 0x7f13e006e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.249+0000 7f13e7200640 1 -- 172.21.15.31:0/1021514638 >> 172.21.15.31:0/1021514638 conn(0x7f13e0068f10 msgr2=0x7f13e00cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:58.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.249+0000 7f13e7200640 1 -- 172.21.15.31:0/1021514638 shutdown_connections 2024-09-11T00:37:58.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.249+0000 7f13e7200640 1 -- 172.21.15.31:0/1021514638 wait complete. 2024-09-11T00:37:58.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.251+0000 7f13e7200640 1 Processor -- start 2024-09-11T00:37:58.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.251+0000 7f13e7200640 1 -- start start 2024-09-11T00:37:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.252+0000 7f13e7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e00cc220 0x7f13e01ecb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.252+0000 7f13e7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13e0150020 0x7f13e0152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e0156960 0x7f13e00aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f13e00c6ca0 con 0x7f13e00cc220 2024-09-11T00:37:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13e00c6b20 con 0x7f13e0150020 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f13e01573d0 con 0x7f13e0156960 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e00cc220 0x7f13e01ecb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e00cc220 0x7f13e01ecb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40776/0 (socket says 172.21.15.31:40776) 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e5e00640 1 -- 172.21.15.31:0/2947388740 learned_addr learned my addr 172.21.15.31:0/2947388740 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e5400640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13e0150020 0x7f13e0152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e6800640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e0156960 0x7f13e00aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e5e00640 1 -- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e0156960 msgr2=0x7f13e00aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e5e00640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e0156960 0x7f13e00aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e5e00640 1 -- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13e0150020 msgr2=0x7f13e0152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e5e00640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13e0150020 0x7f13e0152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.253+0000 7f13e5e00640 1 -- 172.21.15.31:0/2947388740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e00ae800 con 0x7f13e00cc220 2024-09-11T00:37:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.254+0000 7f13e5e00640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e00cc220 0x7f13e01ecb90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f13d400c500 tx=0x7f13d400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.254+0000 7f13da000640 1 -- 172.21.15.31:0/2947388740 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d40052d0 con 0x7f13e00cc220 2024-09-11T00:37:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.254+0000 7f13e7200640 1 -- 172.21.15.31:0/2947388740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e00aeae0 con 0x7f13e00cc220 2024-09-11T00:37:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.254+0000 7f13da000640 1 -- 172.21.15.31:0/2947388740 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13d4005470 con 0x7f13e00cc220 2024-09-11T00:37:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.254+0000 7f13da000640 1 -- 172.21.15.31:0/2947388740 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d4012870 con 0x7f13e00cc220 2024-09-11T00:37:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.254+0000 7f13e7200640 1 -- 172.21.15.31:0/2947388740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e0152f50 con 0x7f13e00cc220 2024-09-11T00:37:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.255+0000 7f13e7200640 1 -- 172.21.15.31:0/2947388740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e0050fb0 con 0x7f13e00cc220 2024-09-11T00:37:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.257+0000 7f13da000640 1 -- 172.21.15.31:0/2947388740 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f13d40035f0 con 0x7f13e00cc220 2024-09-11T00:37:58.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.258+0000 7f13da000640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077f70 0x7f13c407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:58.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.258+0000 7f13da000640 1 -- 172.21.15.31:0/2947388740 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f13d4099ef0 con 0x7f13e00cc220 2024-09-11T00:37:58.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.258+0000 7f13e5400640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077f70 0x7f13c407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:58.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.259+0000 7f13e5400640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077f70 0x7f13c407a430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f13e01558e0 tx=0x7f13d000b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:58.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.263+0000 7f13da000640 1 -- 172.21.15.31:0/2947388740 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f13e0050fb0 con 0x7f13e00cc220 2024-09-11T00:37:58.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.569+0000 7f13e7200640 1 -- 172.21.15.31:0/2947388740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13e00c6e40 con 0x7f13e00cc220 2024-09-11T00:37:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.570+0000 7f13da000640 1 -- 172.21.15.31:0/2947388740 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f13e00c6e40 con 0x7f13e00cc220 2024-09-11T00:37:58.572 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:58.572 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:37:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 -- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077f70 msgr2=0x7f13c407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077f70 0x7f13c407a430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f13e01558e0 tx=0x7f13d000b2e0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 -- 172.21.15.31:0/2947388740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e00cc220 msgr2=0x7f13e01ecb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e00cc220 0x7f13e01ecb90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f13d400c500 tx=0x7f13d400c9c0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 -- 172.21.15.31:0/2947388740 shutdown_connections 2024-09-11T00:37:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077f70 0x7f13c407a430 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e0156960 0x7f13e00aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13e0150020 0x7f13e0152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 --2- 172.21.15.31:0/2947388740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e00cc220 0x7f13e01ecb90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 -- 172.21.15.31:0/2947388740 >> 172.21.15.31:0/2947388740 conn(0x7f13e0068f10 msgr2=0x7f13e01548a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:58.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 -- 172.21.15.31:0/2947388740 shutdown_connections 2024-09-11T00:37:58.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.573+0000 7f13e7200640 1 -- 172.21.15.31:0/2947388740 wait complete. 2024-09-11T00:37:58.591 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:37:58.823 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:37:58.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.982+0000 7ffb50200640 1 -- 172.21.15.31:0/3823015142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb480c23a0 msgr2=0x7ffb481516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.982+0000 7ffb50200640 1 --2- 172.21.15.31:0/3823015142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb480c23a0 0x7ffb481516e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ffb34005fd0 tx=0x7ffb3402a660 comp rx=0 tx=0).stop 2024-09-11T00:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.982+0000 7ffb50200640 1 -- 172.21.15.31:0/3823015142 shutdown_connections 2024-09-11T00:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.982+0000 7ffb50200640 1 --2- 172.21.15.31:0/3823015142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb480c23a0 0x7ffb481516e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.982+0000 7ffb50200640 1 --2- 172.21.15.31:0/3823015142 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb48072480 0x7ffb481511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.982+0000 7ffb50200640 1 -- 172.21.15.31:0/3823015142 >> 172.21.15.31:0/3823015142 conn(0x7ffb480a9750 msgr2=0x7ffb480a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.982+0000 7ffb50200640 1 -- 172.21.15.31:0/3823015142 shutdown_connections 2024-09-11T00:37:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.982+0000 7ffb50200640 1 -- 172.21.15.31:0/3823015142 wait complete. 2024-09-11T00:37:58.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.984+0000 7ffb50200640 1 Processor -- start 2024-09-11T00:37:58.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.984+0000 7ffb50200640 1 -- start start 2024-09-11T00:37:58.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.985+0000 7ffb50200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb48072480 0x7ffb481decf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.985+0000 7ffb50200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb480c23a0 0x7ffb481dd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.985+0000 7ffb50200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb48153f20 0x7ffb4806f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.985+0000 7ffb50200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffb48074200 con 0x7ffb480c23a0 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.985+0000 7ffb50200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffb48074080 con 0x7ffb48072480 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.985+0000 7ffb50200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffb48072860 con 0x7ffb48153f20 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.985+0000 7ffb4ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb480c23a0 0x7ffb481dd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.985+0000 7ffb4ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb480c23a0 0x7ffb481dd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40790/0 (socket says 172.21.15.31:40790) 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.985+0000 7ffb4ce00640 1 -- 172.21.15.31:0/1703643610 learned_addr learned my addr 172.21.15.31:0/1703643610 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb4e200640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb48153f20 0x7ffb4806f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb4d800640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb48072480 0x7ffb481decf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb4ce00640 1 -- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb48153f20 msgr2=0x7ffb4806f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:58.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb4ce00640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb48153f20 0x7ffb4806f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb4ce00640 1 -- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb48072480 msgr2=0x7ffb481decf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb4ce00640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb48072480 0x7ffb481decf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb4ce00640 1 -- 172.21.15.31:0/1703643610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb480c50e0 con 0x7ffb480c23a0 2024-09-11T00:37:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb4d800640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb48072480 0x7ffb481decf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:37:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb4ce00640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb480c23a0 0x7ffb481dd680 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffb34000c00 tx=0x7ffb34003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb45600640 1 -- 172.21.15.31:0/1703643610 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3403f070 con 0x7ffb480c23a0 2024-09-11T00:37:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb45600640 1 -- 172.21.15.31:0/1703643610 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb340047a0 con 0x7ffb480c23a0 2024-09-11T00:37:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb45600640 1 -- 172.21.15.31:0/1703643610 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb34035030 con 0x7ffb480c23a0 2024-09-11T00:37:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.986+0000 7ffb50200640 1 -- 172.21.15.31:0/1703643610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb480c4e20 con 0x7ffb480c23a0 2024-09-11T00:37:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.987+0000 7ffb50200640 1 -- 172.21.15.31:0/1703643610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb481ec8b0 con 0x7ffb480c23a0 2024-09-11T00:37:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.988+0000 7ffb50200640 1 -- 172.21.15.31:0/1703643610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb481511a0 con 0x7ffb480c23a0 2024-09-11T00:37:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.989+0000 7ffb45600640 1 -- 172.21.15.31:0/1703643610 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ffb34004c10 con 0x7ffb480c23a0 2024-09-11T00:37:58.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.990+0000 7ffb45600640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb28077f70 0x7ffb2807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:37:58.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.990+0000 7ffb45600640 1 -- 172.21.15.31:0/1703643610 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7ffb340bb9e0 con 0x7ffb480c23a0 2024-09-11T00:37:58.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.990+0000 7ffb4d800640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb28077f70 0x7ffb2807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:37:58.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.991+0000 7ffb4d800640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb28077f70 0x7ffb2807a430 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c009b40 tx=0x7ffb3c005e50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:37:58.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:58.995+0000 7ffb45600640 1 -- 172.21.15.31:0/1703643610 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffb34083db0 con 0x7ffb480c23a0 2024-09-11T00:37:59.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.225+0000 7ffb50200640 1 -- 172.21.15.31:0/1703643610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb48151e20 con 0x7ffb480c23a0 2024-09-11T00:37:59.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.226+0000 7ffb45600640 1 -- 172.21.15.31:0/1703643610 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7ffb340836b0 con 0x7ffb480c23a0 2024-09-11T00:37:59.228 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:37:59.228 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:37:59.228 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.228+0000 7ffb50200640 1 -- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb28077f70 msgr2=0x7ffb2807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.228+0000 7ffb50200640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb28077f70 0x7ffb2807a430 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c009b40 tx=0x7ffb3c005e50 comp rx=0 tx=0).stop 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.229+0000 7ffb50200640 1 -- 172.21.15.31:0/1703643610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb480c23a0 msgr2=0x7ffb481dd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.229+0000 7ffb50200640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb480c23a0 0x7ffb481dd680 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffb34000c00 tx=0x7ffb34003910 comp rx=0 tx=0).stop 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.229+0000 7ffb50200640 1 -- 172.21.15.31:0/1703643610 shutdown_connections 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.229+0000 7ffb50200640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb28077f70 0x7ffb2807a430 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.229+0000 7ffb50200640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffb48153f20 0x7ffb4806f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.229+0000 7ffb50200640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffb480c23a0 0x7ffb481dd680 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.229+0000 7ffb50200640 1 --2- 172.21.15.31:0/1703643610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb48072480 0x7ffb481decf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.229+0000 7ffb50200640 1 -- 172.21.15.31:0/1703643610 >> 172.21.15.31:0/1703643610 conn(0x7ffb480a9750 msgr2=0x7ffb480a8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.229+0000 7ffb50200640 1 -- 172.21.15.31:0/1703643610 shutdown_connections 2024-09-11T00:37:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:37:59.229+0000 7ffb50200640 1 -- 172.21.15.31:0/1703643610 wait complete. 2024-09-11T00:37:59.246 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:38:00.592 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:00.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.761+0000 7fbbe8c00640 1 -- 172.21.15.31:0/1257950387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe40ac2c0 msgr2=0x7fbbe40ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:00.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.761+0000 7fbbe8c00640 1 --2- 172.21.15.31:0/1257950387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe40ac2c0 0x7fbbe40ac6a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbbe40c5ea0 tx=0x7fbbd402a420 comp rx=0 tx=0).stop 2024-09-11T00:38:00.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.761+0000 7fbbe8c00640 1 -- 172.21.15.31:0/1257950387 shutdown_connections 2024-09-11T00:38:00.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.761+0000 7fbbe8c00640 1 --2- 172.21.15.31:0/1257950387 >> v2:172.21.15.204:3300/0 conn(0x7fbbe40acbe0 0x7fbbe40c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:00.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.761+0000 7fbbe8c00640 1 --2- 172.21.15.31:0/1257950387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe40ac2c0 0x7fbbe40ac6a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:00.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.761+0000 7fbbe8c00640 1 -- 172.21.15.31:0/1257950387 >> 172.21.15.31:0/1257950387 conn(0x7fbbe406caf0 msgr2=0x7fbbe406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:00.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.761+0000 7fbbe8c00640 1 -- 172.21.15.31:0/1257950387 shutdown_connections 2024-09-11T00:38:00.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.762+0000 7fbbe8c00640 1 -- 172.21.15.31:0/1257950387 wait complete. 2024-09-11T00:38:00.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.763+0000 7fbbe8c00640 1 Processor -- start 2024-09-11T00:38:00.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.763+0000 7fbbe8c00640 1 -- start start 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.764+0000 7fbbe8c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbbe40a6e20 0x7fbbe41ddd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.764+0000 7fbbe8c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbbe40ac2c0 0x7fbbe406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.764+0000 7fbbe8c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe40acbe0 0x7fbbe406ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.764+0000 7fbbe8c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe4151950 con 0x7fbbe40acbe0 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.764+0000 7fbbe8c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe41517d0 con 0x7fbbe40ac2c0 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.764+0000 7fbbe8c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe40c66f0 con 0x7fbbe40a6e20 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.764+0000 7fbbe3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe40acbe0 0x7fbbe406ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.764+0000 7fbbe3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe40acbe0 0x7fbbe406ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40804/0 (socket says 172.21.15.31:40804) 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.764+0000 7fbbe3e00640 1 -- 172.21.15.31:0/4216135528 learned_addr learned my addr 172.21.15.31:0/4216135528 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbe3e00640 1 -- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbbe40a6e20 msgr2=0x7fbbe41ddd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:00.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbe3e00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbbe40a6e20 0x7fbbe41ddd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbdaa00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbbe40ac2c0 0x7fbbe406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbe3e00640 1 -- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbbe40ac2c0 msgr2=0x7fbbe406e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbe3e00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbbe40ac2c0 0x7fbbe406e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbe3e00640 1 -- 172.21.15.31:0/4216135528 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe41dfd00 con 0x7fbbe40acbe0 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbdaa00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbbe40ac2c0 0x7fbbe406e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbe3e00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe40acbe0 0x7fbbe406ee60 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00e960 tx=0x7fbbdc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbdbe00640 1 -- 172.21.15.31:0/4216135528 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc004d20 con 0x7fbbe40acbe0 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbdbe00640 1 -- 172.21.15.31:0/4216135528 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbdc00cb20 con 0x7fbbe40acbe0 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbe8c00640 1 -- 172.21.15.31:0/4216135528 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbe41dffe0 con 0x7fbbe40acbe0 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbdbe00640 1 -- 172.21.15.31:0/4216135528 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc005020 con 0x7fbbe40acbe0 2024-09-11T00:38:00.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.765+0000 7fbbe8c00640 1 -- 172.21.15.31:0/4216135528 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbe40c3dc0 con 0x7fbbe40acbe0 2024-09-11T00:38:00.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.766+0000 7fbbe8c00640 1 -- 172.21.15.31:0/4216135528 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbe40c6b30 con 0x7fbbe40acbe0 2024-09-11T00:38:00.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.768+0000 7fbbdbe00640 1 -- 172.21.15.31:0/4216135528 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbbdc00f040 con 0x7fbbe40acbe0 2024-09-11T00:38:00.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.768+0000 7fbbdbe00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbc0077f70 0x7fbbc007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:00.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.769+0000 7fbbdbe00640 1 -- 172.21.15.31:0/4216135528 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7fbbdc09aa20 con 0x7fbbe40acbe0 2024-09-11T00:38:00.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.769+0000 7fbbe3400640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbc0077f70 0x7fbbc007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:00.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.770+0000 7fbbe3400640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbc0077f70 0x7fbbc007a430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4033000 tx=0x7fbbd40046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:00.773+0000 7fbbdbe00640 1 -- 172.21.15.31:0/4216135528 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbbdc016030 con 0x7fbbe40acbe0 2024-09-11T00:38:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.022+0000 7fbbe8c00640 1 -- 172.21.15.31:0/4216135528 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbe414f410 con 0x7fbbe40acbe0 2024-09-11T00:38:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.022+0000 7fbbdbe00640 1 -- 172.21.15.31:0/4216135528 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7fbbdc062e70 con 0x7fbbe40acbe0 2024-09-11T00:38:01.024 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:01.024 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:01.025 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:01.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.024+0000 7fbbe8c00640 1 -- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbc0077f70 msgr2=0x7fbbc007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.024+0000 7fbbe8c00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbc0077f70 0x7fbbc007a430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbbd4033000 tx=0x7fbbd40046c0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.024+0000 7fbbe8c00640 1 -- 172.21.15.31:0/4216135528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe40acbe0 msgr2=0x7fbbe406ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.024+0000 7fbbe8c00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe40acbe0 0x7fbbe406ee60 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00e960 tx=0x7fbbdc00ee20 comp rx=0 tx=0).stop 2024-09-11T00:38:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.024+0000 7fbbe8c00640 1 -- 172.21.15.31:0/4216135528 shutdown_connections 2024-09-11T00:38:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.024+0000 7fbbe8c00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbbc0077f70 0x7fbbc007a430 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.024+0000 7fbbe8c00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbe40acbe0 0x7fbbe406ee60 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.025+0000 7fbbe8c00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbbe40ac2c0 0x7fbbe406e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.025+0000 7fbbe8c00640 1 --2- 172.21.15.31:0/4216135528 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbbe40a6e20 0x7fbbe41ddd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.025+0000 7fbbe8c00640 1 -- 172.21.15.31:0/4216135528 >> 172.21.15.31:0/4216135528 conn(0x7fbbe406caf0 msgr2=0x7fbbe41510a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:01.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.025+0000 7fbbe8c00640 1 -- 172.21.15.31:0/4216135528 shutdown_connections 2024-09-11T00:38:01.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.025+0000 7fbbe8c00640 1 -- 172.21.15.31:0/4216135528 wait complete. 2024-09-11T00:38:01.042 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:38:01.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:01.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.409+0000 7fb3ea800640 1 -- 172.21.15.31:0/3942209782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c8004d90 msgr2=0x7fb3c8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:01.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.409+0000 7fb3ea800640 1 --2- 172.21.15.31:0/3942209782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3c8004d90 0x7fb3c8005210 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00c610 tx=0x7fb3c8005830 comp rx=0 tx=0).stop 2024-09-11T00:38:01.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.409+0000 7fb3e3e00640 1 -- 172.21.15.31:0/3942209782 reap_dead start 2024-09-11T00:38:01.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.409+0000 7fb3ea800640 1 -- 172.21.15.31:0/3942209782 shutdown_connections 2024-09-11T00:38:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.409+0000 7fb3ea800640 1 -- 172.21.15.31:0/3942209782 >> 172.21.15.31:0/3942209782 conn(0x7fb3e40a9750 msgr2=0x7fb3e40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.409+0000 7fb3ea800640 1 -- 172.21.15.31:0/3942209782 shutdown_connections 2024-09-11T00:38:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.409+0000 7fb3ea800640 1 -- 172.21.15.31:0/3942209782 wait complete. 2024-09-11T00:38:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.410+0000 7fb3ea800640 1 Processor -- start 2024-09-11T00:38:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.411+0000 7fb3ea800640 1 -- start start 2024-09-11T00:38:01.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.411+0000 7fb3ea800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3c8004d90 0x7fb3e414cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:01.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.411+0000 7fb3ea800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3e41df6c0 0x7fb3e41dfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:01.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3ea800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e41e0080 0x7fb3e41428c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:01.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3ea800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e4074200 con 0x7fb3e41e0080 2024-09-11T00:38:01.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3ea800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e4074080 con 0x7fb3c8004d90 2024-09-11T00:38:01.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3ea800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb3e4072860 con 0x7fb3e41df6c0 2024-09-11T00:38:01.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e41e0080 0x7fb3e41428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:01.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e41e0080 0x7fb3e41428c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40820/0 (socket says 172.21.15.31:40820) 2024-09-11T00:38:01.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3e00640 1 -- 172.21.15.31:0/904644322 learned_addr learned my addr 172.21.15.31:0/904644322 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3400640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3c8004d90 0x7fb3e414cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e2a00640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3e41df6c0 0x7fb3e41dfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3e00640 1 -- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3e41df6c0 msgr2=0x7fb3e41dfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3e00640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3e41df6c0 0x7fb3e41dfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3e00640 1 -- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3c8004d90 msgr2=0x7fb3e414cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3e00640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3c8004d90 0x7fb3e414cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3e00640 1 -- 172.21.15.31:0/904644322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e4152a40 con 0x7fb3e41e0080 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3400640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3c8004d90 0x7fb3e414cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.412+0000 7fb3e3e00640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e41e0080 0x7fb3e41428c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00c730 tx=0x7fb3dc001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.413+0000 7fb3db400640 1 -- 172.21.15.31:0/904644322 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc039070 con 0x7fb3e41e0080 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.413+0000 7fb3db400640 1 -- 172.21.15.31:0/904644322 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3dc006590 con 0x7fb3e41e0080 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.413+0000 7fb3db400640 1 -- 172.21.15.31:0/904644322 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc006d20 con 0x7fb3e41e0080 2024-09-11T00:38:01.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.413+0000 7fb3ea800640 1 -- 172.21.15.31:0/904644322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e40c6bb0 con 0x7fb3e41e0080 2024-09-11T00:38:01.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.413+0000 7fb3ea800640 1 -- 172.21.15.31:0/904644322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e41dd590 con 0x7fb3e41e0080 2024-09-11T00:38:01.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.414+0000 7fb3ea800640 1 -- 172.21.15.31:0/904644322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3e41dde80 con 0x7fb3e41e0080 2024-09-11T00:38:01.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.415+0000 7fb3db400640 1 -- 172.21.15.31:0/904644322 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb3dc02b660 con 0x7fb3e41e0080 2024-09-11T00:38:01.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.417+0000 7fb3db400640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3c4077fa0 0x7fb3c407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:01.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.417+0000 7fb3db400640 1 -- 172.21.15.31:0/904644322 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7fb3dc0b9c30 con 0x7fb3e41e0080 2024-09-11T00:38:01.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.417+0000 7fb3e3400640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3c4077fa0 0x7fb3c407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:01.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.418+0000 7fb3e3400640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3c4077fa0 0x7fb3c407a460 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40c76d0 tx=0x7fb3d4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.421+0000 7fb3db400640 1 -- 172.21.15.31:0/904644322 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb3dc045030 con 0x7fb3e41e0080 2024-09-11T00:38:01.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.652+0000 7fb3ea800640 1 -- 172.21.15.31:0/904644322 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3e4151280 con 0x7fb3e41e0080 2024-09-11T00:38:01.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.653+0000 7fb3db400640 1 -- 172.21.15.31:0/904644322 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7fb3dc082100 con 0x7fb3e41e0080 2024-09-11T00:38:01.655 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:01.655 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:01.656 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:01.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 -- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3c4077fa0 msgr2=0x7fb3c407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:01.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3c4077fa0 0x7fb3c407a460 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40c76d0 tx=0x7fb3d4005f90 comp rx=0 tx=0).stop 2024-09-11T00:38:01.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 -- 172.21.15.31:0/904644322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e41e0080 msgr2=0x7fb3e41428c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:01.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e41e0080 0x7fb3e41428c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00c730 tx=0x7fb3dc001aa0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 -- 172.21.15.31:0/904644322 shutdown_connections 2024-09-11T00:38:01.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3c4077fa0 0x7fb3c407a460 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e41e0080 0x7fb3e41428c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb3e41df6c0 0x7fb3e41dfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 --2- 172.21.15.31:0/904644322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3c8004d90 0x7fb3e414cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:01.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 -- 172.21.15.31:0/904644322 >> 172.21.15.31:0/904644322 conn(0x7fb3e40a9750 msgr2=0x7fb3e40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:01.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 -- 172.21.15.31:0/904644322 shutdown_connections 2024-09-11T00:38:01.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:01.655+0000 7fb3ea800640 1 -- 172.21.15.31:0/904644322 wait complete. 2024-09-11T00:38:01.673 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:38:03.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:03.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.217+0000 7eff17e00640 1 -- 172.21.15.31:0/3677303138 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff10072480 msgr2=0x7eff101511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:03.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.217+0000 7eff17e00640 1 --2- 172.21.15.31:0/3677303138 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff10072480 0x7eff101511a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7eff04009860 tx=0x7eff0402a5a0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.217+0000 7eff17e00640 1 -- 172.21.15.31:0/3677303138 shutdown_connections 2024-09-11T00:38:03.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.217+0000 7eff17e00640 1 --2- 172.21.15.31:0/3677303138 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff10072480 0x7eff101511a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.217+0000 7eff17e00640 1 -- 172.21.15.31:0/3677303138 >> 172.21.15.31:0/3677303138 conn(0x7eff100a9750 msgr2=0x7eff100a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:03.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.217+0000 7eff17e00640 1 -- 172.21.15.31:0/3677303138 shutdown_connections 2024-09-11T00:38:03.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.217+0000 7eff17e00640 1 -- 172.21.15.31:0/3677303138 wait complete. 2024-09-11T00:38:03.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.218+0000 7eff17e00640 1 Processor -- start 2024-09-11T00:38:03.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.219+0000 7eff17e00640 1 -- start start 2024-09-11T00:38:03.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.219+0000 7eff17e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff10072480 0x7eff1006f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:03.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.219+0000 7eff17e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff100c23a0 0x7eff101ec770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:03.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.219+0000 7eff17e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff10153f20 0x7eff101eccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:03.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.219+0000 7eff17e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7eff10074200 con 0x7eff10153f20 2024-09-11T00:38:03.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.219+0000 7eff17e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff10074080 con 0x7eff10072480 2024-09-11T00:38:03.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.219+0000 7eff17e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7eff10072860 con 0x7eff100c23a0 2024-09-11T00:38:03.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.220+0000 7eff15e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff10153f20 0x7eff101eccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:03.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.220+0000 7eff15400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff10072480 0x7eff1006f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:03.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.220+0000 7eff15e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff10153f20 0x7eff101eccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40836/0 (socket says 172.21.15.31:40836) 2024-09-11T00:38:03.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.220+0000 7eff15e00640 1 -- 172.21.15.31:0/1778743366 learned_addr learned my addr 172.21.15.31:0/1778743366 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:03.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.220+0000 7eff14a00640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff100c23a0 0x7eff101ec770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:03.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.220+0000 7eff15e00640 1 -- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff100c23a0 msgr2=0x7eff101ec770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:03.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.220+0000 7eff15e00640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff100c23a0 0x7eff101ec770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.220+0000 7eff15e00640 1 -- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff10072480 msgr2=0x7eff1006f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:03.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.220+0000 7eff15e00640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff10072480 0x7eff1006f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.220+0000 7eff15e00640 1 -- 172.21.15.31:0/1778743366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff101ec290 con 0x7eff10153f20 2024-09-11T00:38:03.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.221+0000 7eff15e00640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff10153f20 0x7eff101eccb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7eff0800cae0 tx=0x7eff0800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:03.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.221+0000 7eff0d600640 1 -- 172.21.15.31:0/1778743366 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08016020 con 0x7eff10153f20 2024-09-11T00:38:03.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.221+0000 7eff0d600640 1 -- 172.21.15.31:0/1778743366 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff0800ad50 con 0x7eff10153f20 2024-09-11T00:38:03.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.221+0000 7eff17e00640 1 -- 172.21.15.31:0/1778743366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff10149f20 con 0x7eff10153f20 2024-09-11T00:38:03.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.221+0000 7eff0d600640 1 -- 172.21.15.31:0/1778743366 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08012680 con 0x7eff10153f20 2024-09-11T00:38:03.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.221+0000 7eff17e00640 1 -- 172.21.15.31:0/1778743366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff10149960 con 0x7eff10153f20 2024-09-11T00:38:03.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.222+0000 7eff17e00640 1 -- 172.21.15.31:0/1778743366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff101511a0 con 0x7eff10153f20 2024-09-11T00:38:03.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.223+0000 7eff0d600640 1 -- 172.21.15.31:0/1778743366 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7eff08012820 con 0x7eff10153f20 2024-09-11T00:38:03.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.224+0000 7eff0d600640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efef0077fa0 0x7efef007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:03.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.224+0000 7eff0d600640 1 -- 172.21.15.31:0/1778743366 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7eff0809ac00 con 0x7eff10153f20 2024-09-11T00:38:03.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.225+0000 7eff15400640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efef0077fa0 0x7efef007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:03.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.226+0000 7eff15400640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efef0077fa0 0x7efef007a460 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7eff04034000 tx=0x7eff04004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:03.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.229+0000 7eff0d600640 1 -- 172.21.15.31:0/1778743366 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7eff08020020 con 0x7eff10153f20 2024-09-11T00:38:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.462+0000 7eff17e00640 1 -- 172.21.15.31:0/1778743366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff101de1a0 con 0x7eff10153f20 2024-09-11T00:38:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.462+0000 7eff0d600640 1 -- 172.21.15.31:0/1778743366 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7eff08062980 con 0x7eff10153f20 2024-09-11T00:38:03.464 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:03.465 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 -- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efef0077fa0 msgr2=0x7efef007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efef0077fa0 0x7efef007a460 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7eff04034000 tx=0x7eff04004730 comp rx=0 tx=0).stop 2024-09-11T00:38:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 -- 172.21.15.31:0/1778743366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff10153f20 msgr2=0x7eff101eccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff10153f20 0x7eff101eccb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7eff0800cae0 tx=0x7eff0800a060 comp rx=0 tx=0).stop 2024-09-11T00:38:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 -- 172.21.15.31:0/1778743366 shutdown_connections 2024-09-11T00:38:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efef0077fa0 0x7efef007a460 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff10153f20 0x7eff101eccb0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff100c23a0 0x7eff101ec770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 --2- 172.21.15.31:0/1778743366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff10072480 0x7eff1006f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 -- 172.21.15.31:0/1778743366 >> 172.21.15.31:0/1778743366 conn(0x7eff100a9750 msgr2=0x7eff100c30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 -- 172.21.15.31:0/1778743366 shutdown_connections 2024-09-11T00:38:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.465+0000 7eff17e00640 1 -- 172.21.15.31:0/1778743366 wait complete. 2024-09-11T00:38:03.482 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:38:03.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:03.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.832+0000 7f36e1200640 1 -- 172.21.15.31:0/266718068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36dc0c23a0 msgr2=0x7f36dc0c2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:03.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.832+0000 7f36e1200640 1 --2- 172.21.15.31:0/266718068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36dc0c23a0 0x7f36dc0c2850 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009ca0 tx=0x7f36d402aa30 comp rx=0 tx=0).stop 2024-09-11T00:38:03.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.832+0000 7f36e1200640 1 -- 172.21.15.31:0/266718068 shutdown_connections 2024-09-11T00:38:03.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.832+0000 7f36e1200640 1 --2- 172.21.15.31:0/266718068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36dc0c23a0 0x7f36dc0c2850 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.832+0000 7f36e1200640 1 --2- 172.21.15.31:0/266718068 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36dc06fa70 0x7f36dc0c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.832+0000 7f36e1200640 1 -- 172.21.15.31:0/266718068 >> 172.21.15.31:0/266718068 conn(0x7f36dc0a9750 msgr2=0x7f36dc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:03.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.832+0000 7f36e1200640 1 -- 172.21.15.31:0/266718068 shutdown_connections 2024-09-11T00:38:03.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.832+0000 7f36e1200640 1 -- 172.21.15.31:0/266718068 wait complete. 2024-09-11T00:38:03.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.834+0000 7f36e1200640 1 Processor -- start 2024-09-11T00:38:03.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.834+0000 7f36e1200640 1 -- start start 2024-09-11T00:38:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36e1200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36dc06fa70 0x7f36dc14ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36e1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36dc0c23a0 0x7f36dc14b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36e1200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36dc156300 0x7f36dc1efc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36e1200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc0c77a0 con 0x7f36dc0c23a0 2024-09-11T00:38:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36e1200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc0c7620 con 0x7f36dc156300 2024-09-11T00:38:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36e1200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f36dc156df0 con 0x7f36dc06fa70 2024-09-11T00:38:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36d1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36dc0c23a0 0x7f36dc14b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36d1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36dc0c23a0 0x7f36dc14b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40846/0 (socket says 172.21.15.31:40846) 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36d1400640 1 -- 172.21.15.31:0/419116090 learned_addr learned my addr 172.21.15.31:0/419116090 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36d9e00640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36dc06fa70 0x7f36dc14ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36da800640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36dc156300 0x7f36dc1efc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36d1400640 1 -- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36dc06fa70 msgr2=0x7f36dc14ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36d1400640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36dc06fa70 0x7f36dc14ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36d1400640 1 -- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36dc156300 msgr2=0x7f36dc1efc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36d1400640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36dc156300 0x7f36dc1efc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.835+0000 7f36d1400640 1 -- 172.21.15.31:0/419116090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36dc0a8bb0 con 0x7f36dc0c23a0 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.836+0000 7f36da800640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36dc156300 0x7f36dc1efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.836+0000 7f36d1400640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36dc0c23a0 0x7f36dc14b380 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f36c400d950 tx=0x7f36c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.836+0000 7f36d2a00640 1 -- 172.21.15.31:0/419116090 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c4017070 con 0x7f36dc0c23a0 2024-09-11T00:38:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.836+0000 7f36d2a00640 1 -- 172.21.15.31:0/419116090 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36c4009d20 con 0x7f36dc0c23a0 2024-09-11T00:38:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.836+0000 7f36d2a00640 1 -- 172.21.15.31:0/419116090 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c4004280 con 0x7f36dc0c23a0 2024-09-11T00:38:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.836+0000 7f36e1200640 1 -- 172.21.15.31:0/419116090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36dc0c1880 con 0x7f36dc0c23a0 2024-09-11T00:38:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.836+0000 7f36e1200640 1 -- 172.21.15.31:0/419116090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36dc074ae0 con 0x7f36dc0c23a0 2024-09-11T00:38:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.837+0000 7f36e1200640 1 -- 172.21.15.31:0/419116090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36dc074900 con 0x7f36dc0c23a0 2024-09-11T00:38:03.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.838+0000 7f36d2a00640 1 -- 172.21.15.31:0/419116090 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f36c4023060 con 0x7f36dc0c23a0 2024-09-11T00:38:03.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.839+0000 7f36d2a00640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f36b8077f70 0x7f36b807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:03.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.839+0000 7f36d2a00640 1 -- 172.21.15.31:0/419116090 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f36c4099ea0 con 0x7f36dc0c23a0 2024-09-11T00:38:03.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.840+0000 7f36d9e00640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f36b8077f70 0x7f36b807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:03.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.840+0000 7f36d9e00640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f36b8077f70 0x7f36b807a430 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f36cc001720 tx=0x7f36cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:03.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:03.843+0000 7f36d2a00640 1 -- 172.21.15.31:0/419116090 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f36dc074900 con 0x7f36dc0c23a0 2024-09-11T00:38:04.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.076+0000 7f36e1200640 1 -- 172.21.15.31:0/419116090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36dc0c7240 con 0x7f36dc0c23a0 2024-09-11T00:38:04.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.076+0000 7f36d2a00640 1 -- 172.21.15.31:0/419116090 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f36c4061ca0 con 0x7f36dc0c23a0 2024-09-11T00:38:04.078 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:04.079 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:04.079 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:04.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 -- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f36b8077f70 msgr2=0x7f36b807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:04.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f36b8077f70 0x7f36b807a430 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f36cc001720 tx=0x7f36cc009040 comp rx=0 tx=0).stop 2024-09-11T00:38:04.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 -- 172.21.15.31:0/419116090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36dc0c23a0 msgr2=0x7f36dc14b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:04.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36dc0c23a0 0x7f36dc14b380 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f36c400d950 tx=0x7f36c400de10 comp rx=0 tx=0).stop 2024-09-11T00:38:04.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 -- 172.21.15.31:0/419116090 shutdown_connections 2024-09-11T00:38:04.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f36b8077f70 0x7f36b807a430 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:04.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36dc156300 0x7f36dc1efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:04.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36dc0c23a0 0x7f36dc14b380 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:04.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 --2- 172.21.15.31:0/419116090 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36dc06fa70 0x7f36dc14ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:04.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 -- 172.21.15.31:0/419116090 >> 172.21.15.31:0/419116090 conn(0x7f36dc0a9750 msgr2=0x7f36dc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:04.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 -- 172.21.15.31:0/419116090 shutdown_connections 2024-09-11T00:38:04.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:04.079+0000 7f36e1200640 1 -- 172.21.15.31:0/419116090 wait complete. 2024-09-11T00:38:04.097 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:38:05.484 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:05.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.654+0000 7f1817000640 1 -- 172.21.15.31:0/2230354947 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18100ac2c0 msgr2=0x7f18100ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:05.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.654+0000 7f1817000640 1 --2- 172.21.15.31:0/2230354947 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18100ac2c0 0x7f18100ac6a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1804009830 tx=0x7f180402a3f0 comp rx=0 tx=0).stop 2024-09-11T00:38:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.654+0000 7f1817000640 1 -- 172.21.15.31:0/2230354947 shutdown_connections 2024-09-11T00:38:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.654+0000 7f1817000640 1 --2- 172.21.15.31:0/2230354947 >> v2:172.21.15.204:3300/0 conn(0x7f18100acbe0 0x7f18100c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.654+0000 7f1817000640 1 --2- 172.21.15.31:0/2230354947 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18100ac2c0 0x7f18100ac6a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.654+0000 7f1817000640 1 -- 172.21.15.31:0/2230354947 >> 172.21.15.31:0/2230354947 conn(0x7f181006caf0 msgr2=0x7f181006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.654+0000 7f1817000640 1 -- 172.21.15.31:0/2230354947 shutdown_connections 2024-09-11T00:38:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.654+0000 7f1817000640 1 -- 172.21.15.31:0/2230354947 wait complete. 2024-09-11T00:38:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.656+0000 7f1817000640 1 Processor -- start 2024-09-11T00:38:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.656+0000 7f1817000640 1 -- start start 2024-09-11T00:38:05.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.657+0000 7f1817000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18100a6e20 0x7f1810152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:05.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.657+0000 7f1817000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18100ac2c0 0x7f1810073650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:05.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.657+0000 7f1817000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18100acbe0 0x7f1810072070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:05.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.657+0000 7f1817000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1810151950 con 0x7f18100acbe0 2024-09-11T00:38:05.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.657+0000 7f1817000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18101517d0 con 0x7f18100a6e20 2024-09-11T00:38:05.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.657+0000 7f1817000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f18100c66f0 con 0x7f18100ac2c0 2024-09-11T00:38:05.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.657+0000 7f1816600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18100acbe0 0x7f1810072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:05.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.657+0000 7f1816600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18100acbe0 0x7f1810072070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40856/0 (socket says 172.21.15.31:40856) 2024-09-11T00:38:05.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.657+0000 7f1816600640 1 -- 172.21.15.31:0/3825029239 learned_addr learned my addr 172.21.15.31:0/3825029239 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:05.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1815c00640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18100a6e20 0x7f1810152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:05.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1815200640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18100ac2c0 0x7f1810073650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:05.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1816600640 1 -- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18100ac2c0 msgr2=0x7f1810073650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:05.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1816600640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18100ac2c0 0x7f1810073650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:05.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1816600640 1 -- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18100a6e20 msgr2=0x7f1810152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1816600640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18100a6e20 0x7f1810152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1816600640 1 -- 172.21.15.31:0/3825029239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18100aa070 con 0x7f18100acbe0 2024-09-11T00:38:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1815c00640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18100a6e20 0x7f1810152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:38:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1815200640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18100ac2c0 0x7f1810073650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:38:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1816600640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18100acbe0 0x7f1810072070 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f180800ea80 tx=0x7f180800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f180d600640 1 -- 172.21.15.31:0/3825029239 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1808004b20 con 0x7f18100acbe0 2024-09-11T00:38:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f180d600640 1 -- 172.21.15.31:0/3825029239 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1808004ce0 con 0x7f18100acbe0 2024-09-11T00:38:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f180d600640 1 -- 172.21.15.31:0/3825029239 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1808005280 con 0x7f18100acbe0 2024-09-11T00:38:05.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.658+0000 7f1817000640 1 -- 172.21.15.31:0/3825029239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18101e1ef0 con 0x7f18100acbe0 2024-09-11T00:38:05.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.659+0000 7f1817000640 1 -- 172.21.15.31:0/3825029239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18100aa5c0 con 0x7f18100acbe0 2024-09-11T00:38:05.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.660+0000 7f1817000640 1 -- 172.21.15.31:0/3825029239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18100ac6a0 con 0x7f18100acbe0 2024-09-11T00:38:05.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.661+0000 7f180d600640 1 -- 172.21.15.31:0/3825029239 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1808005420 con 0x7f18100acbe0 2024-09-11T00:38:05.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.662+0000 7f180d600640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17f0077f70 0x7f17f007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:05.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.662+0000 7f180d600640 1 -- 172.21.15.31:0/3825029239 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f180809b110 con 0x7f18100acbe0 2024-09-11T00:38:05.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.662+0000 7f1815c00640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17f0077f70 0x7f17f007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:05.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.663+0000 7f1815c00640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17f0077f70 0x7f17f007a430 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1804034000 tx=0x7f1804004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:05.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.666+0000 7f180d600640 1 -- 172.21.15.31:0/3825029239 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1808063560 con 0x7f18100acbe0 2024-09-11T00:38:05.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.896+0000 7f1817000640 1 -- 172.21.15.31:0/3825029239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f181014cdd0 con 0x7f18100acbe0 2024-09-11T00:38:05.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.896+0000 7f180d600640 1 -- 172.21.15.31:0/3825029239 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f1808062e60 con 0x7f18100acbe0 2024-09-11T00:38:05.898 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:05.899 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:05.899 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:05.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 -- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17f0077f70 msgr2=0x7f17f007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:05.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17f0077f70 0x7f17f007a430 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1804034000 tx=0x7f1804004730 comp rx=0 tx=0).stop 2024-09-11T00:38:05.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 -- 172.21.15.31:0/3825029239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18100acbe0 msgr2=0x7f1810072070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:05.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18100acbe0 0x7f1810072070 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f180800ea80 tx=0x7f180800ef40 comp rx=0 tx=0).stop 2024-09-11T00:38:05.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 -- 172.21.15.31:0/3825029239 shutdown_connections 2024-09-11T00:38:05.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17f0077f70 0x7f17f007a430 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:05.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18100acbe0 0x7f1810072070 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:05.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18100ac2c0 0x7f1810073650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:05.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 --2- 172.21.15.31:0/3825029239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18100a6e20 0x7f1810152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:05.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 -- 172.21.15.31:0/3825029239 >> 172.21.15.31:0/3825029239 conn(0x7f181006caf0 msgr2=0x7f181014ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:05.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 -- 172.21.15.31:0/3825029239 shutdown_connections 2024-09-11T00:38:05.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:05.899+0000 7f1817000640 1 -- 172.21.15.31:0/3825029239 wait complete. 2024-09-11T00:38:05.918 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:38:06.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:06.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.258+0000 7f59bee00640 1 -- 172.21.15.31:0/3777466753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59b8072480 msgr2=0x7f59b8074a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:06.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.258+0000 7f59bee00640 1 --2- 172.21.15.31:0/3777466753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59b8072480 0x7f59b8074a90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f59a8005fd0 tx=0x7f59a802aa70 comp rx=0 tx=0).stop 2024-09-11T00:38:06.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.258+0000 7f59bee00640 1 -- 172.21.15.31:0/3777466753 shutdown_connections 2024-09-11T00:38:06.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.258+0000 7f59bee00640 1 --2- 172.21.15.31:0/3777466753 >> v2:172.21.15.204:3300/0 conn(0x7f59b8154b60 0x7f59b80c2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:06.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.258+0000 7f59bee00640 1 --2- 172.21.15.31:0/3777466753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59b8072480 0x7f59b8074a90 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:06.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.258+0000 7f59bee00640 1 -- 172.21.15.31:0/3777466753 >> 172.21.15.31:0/3777466753 conn(0x7f59b80a9750 msgr2=0x7f59b80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:06.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.258+0000 7f59bee00640 1 -- 172.21.15.31:0/3777466753 shutdown_connections 2024-09-11T00:38:06.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.258+0000 7f59bee00640 1 -- 172.21.15.31:0/3777466753 wait complete. 2024-09-11T00:38:06.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.260+0000 7f59bee00640 1 Processor -- start 2024-09-11T00:38:06.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.260+0000 7f59bee00640 1 -- start start 2024-09-11T00:38:06.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.261+0000 7f59bee00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59b8072480 0x7f59b81ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:06.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.261+0000 7f59bee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59b80c34e0 0x7f59b81ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:06.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.261+0000 7f59bee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59b8154b60 0x7f59b814dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:06.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.261+0000 7f59bee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f59b80c23a0 con 0x7f59b80c34e0 2024-09-11T00:38:06.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.261+0000 7f59bee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f59b8154640 con 0x7f59b8154b60 2024-09-11T00:38:06.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.261+0000 7f59bee00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f59b80a7140 con 0x7f59b8072480 2024-09-11T00:38:06.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.261+0000 7f59b7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59b80c34e0 0x7f59b81ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:06.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.261+0000 7f59b7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59b80c34e0 0x7f59b81ec620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40884/0 (socket says 172.21.15.31:40884) 2024-09-11T00:38:06.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.261+0000 7f59b7400640 1 -- 172.21.15.31:0/2469582773 learned_addr learned my addr 172.21.15.31:0/2469582773 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:06.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59bce00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59b8154b60 0x7f59b814dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:06.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59b7e00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59b8072480 0x7f59b81ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:06.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59b7400640 1 -- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59b8072480 msgr2=0x7f59b81ef2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:06.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59b7400640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59b8072480 0x7f59b81ef2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:06.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59b7400640 1 -- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59b8154b60 msgr2=0x7f59b814dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:06.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59b7400640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59b8154b60 0x7f59b814dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:06.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59b7400640 1 -- 172.21.15.31:0/2469582773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b81ebe60 con 0x7f59b80c34e0 2024-09-11T00:38:06.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59b7e00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59b8072480 0x7f59b81ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:38:06.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59bce00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59b8154b60 0x7f59b814dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:38:06.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59b7400640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59b80c34e0 0x7f59b81ec620 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f59a400c970 tx=0x7f59a400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:06.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59afe00640 1 -- 172.21.15.31:0/2469582773 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a4007bb0 con 0x7f59b80c34e0 2024-09-11T00:38:06.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59afe00640 1 -- 172.21.15.31:0/2469582773 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59a4007e40 con 0x7f59b80c34e0 2024-09-11T00:38:06.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59afe00640 1 -- 172.21.15.31:0/2469582773 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a400f690 con 0x7f59b80c34e0 2024-09-11T00:38:06.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.262+0000 7f59bee00640 1 -- 172.21.15.31:0/2469582773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b814fb40 con 0x7f59b80c34e0 2024-09-11T00:38:06.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.263+0000 7f59bee00640 1 -- 172.21.15.31:0/2469582773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b814fe20 con 0x7f59b80c34e0 2024-09-11T00:38:06.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.264+0000 7f59bee00640 1 -- 172.21.15.31:0/2469582773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59b8072860 con 0x7f59b80c34e0 2024-09-11T00:38:06.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.265+0000 7f59afe00640 1 -- 172.21.15.31:0/2469582773 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f59a400f830 con 0x7f59b80c34e0 2024-09-11T00:38:06.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.265+0000 7f59afe00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5998077fa0 0x7f599807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:06.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.266+0000 7f59afe00640 1 -- 172.21.15.31:0/2469582773 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f59a409ab80 con 0x7f59b80c34e0 2024-09-11T00:38:06.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.266+0000 7f59b7e00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5998077fa0 0x7f599807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:06.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.267+0000 7f59b7e00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5998077fa0 0x7f599807a460 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f59a8035000 tx=0x7f59a8004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:06.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.270+0000 7f59afe00640 1 -- 172.21.15.31:0/2469582773 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f59a4014030 con 0x7f59b80c34e0 2024-09-11T00:38:06.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.509+0000 7f59bee00640 1 -- 172.21.15.31:0/2469582773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59b80a6f20 con 0x7f59b80c34e0 2024-09-11T00:38:06.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.509+0000 7f59afe00640 1 -- 172.21.15.31:0/2469582773 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f59a4063050 con 0x7f59b80c34e0 2024-09-11T00:38:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:06.511 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:06.512 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 -- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5998077fa0 msgr2=0x7f599807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5998077fa0 0x7f599807a460 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f59a8035000 tx=0x7f59a8004ab0 comp rx=0 tx=0).stop 2024-09-11T00:38:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 -- 172.21.15.31:0/2469582773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59b80c34e0 msgr2=0x7f59b81ec620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59b80c34e0 0x7f59b81ec620 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f59a400c970 tx=0x7f59a400ce30 comp rx=0 tx=0).stop 2024-09-11T00:38:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 -- 172.21.15.31:0/2469582773 shutdown_connections 2024-09-11T00:38:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5998077fa0 0x7f599807a460 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59b8154b60 0x7f59b814dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59b80c34e0 0x7f59b81ec620 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 --2- 172.21.15.31:0/2469582773 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59b8072480 0x7f59b81ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:06.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 -- 172.21.15.31:0/2469582773 >> 172.21.15.31:0/2469582773 conn(0x7f59b80a9750 msgr2=0x7f59b80c41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:06.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 -- 172.21.15.31:0/2469582773 shutdown_connections 2024-09-11T00:38:06.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:06.512+0000 7f59bee00640 1 -- 172.21.15.31:0/2469582773 wait complete. 2024-09-11T00:38:06.530 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:38:07.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:08.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.089+0000 7f5b32800640 1 -- 172.21.15.31:0/1687866366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c0c6bb0 msgr2=0x7f5b2c06dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:08.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.089+0000 7f5b32800640 1 --2- 172.21.15.31:0/1687866366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c0c6bb0 0x7f5b2c06dc60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c070690 tx=0x7f5b2002a440 comp rx=0 tx=0).stop 2024-09-11T00:38:08.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.089+0000 7f5b32800640 1 -- 172.21.15.31:0/1687866366 shutdown_connections 2024-09-11T00:38:08.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.089+0000 7f5b32800640 1 --2- 172.21.15.31:0/1687866366 >> v2:172.21.15.204:3300/0 conn(0x7f5b2c1534f0 0x7f5b2c150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.089+0000 7f5b32800640 1 --2- 172.21.15.31:0/1687866366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c0c6bb0 0x7f5b2c06dc60 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.089+0000 7f5b32800640 1 -- 172.21.15.31:0/1687866366 >> 172.21.15.31:0/1687866366 conn(0x7f5b2c06caf0 msgr2=0x7f5b2c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:08.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.089+0000 7f5b32800640 1 -- 172.21.15.31:0/1687866366 shutdown_connections 2024-09-11T00:38:08.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.089+0000 7f5b32800640 1 -- 172.21.15.31:0/1687866366 wait complete. 2024-09-11T00:38:08.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.090+0000 7f5b32800640 1 Processor -- start 2024-09-11T00:38:08.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.091+0000 7f5b32800640 1 -- start start 2024-09-11T00:38:08.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.091+0000 7f5b32800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b2c0c2700 0x7f5b2c14c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.091+0000 7f5b32800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b2c0c6bb0 0x7f5b2c1499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.091+0000 7f5b32800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c1534f0 0x7f5b2c149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.091+0000 7f5b32800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c0c7110 con 0x7f5b2c1534f0 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.091+0000 7f5b32800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c0c6f90 con 0x7f5b2c0c2700 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.091+0000 7f5b32800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5b2c153900 con 0x7f5b2c0c6bb0 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.092+0000 7f5b31e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c1534f0 0x7f5b2c149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.092+0000 7f5b31e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c1534f0 0x7f5b2c149f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45200/0 (socket says 172.21.15.31:45200) 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.092+0000 7f5b31e00640 1 -- 172.21.15.31:0/1080489182 learned_addr learned my addr 172.21.15.31:0/1080489182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.092+0000 7f5b31400640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b2c0c2700 0x7f5b2c14c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.092+0000 7f5b30a00640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b2c0c6bb0 0x7f5b2c1499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:08.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.092+0000 7f5b31e00640 1 -- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b2c0c6bb0 msgr2=0x7f5b2c1499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:08.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.092+0000 7f5b31e00640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b2c0c6bb0 0x7f5b2c1499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.092+0000 7f5b31e00640 1 -- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b2c0c2700 msgr2=0x7f5b2c14c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:08.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.092+0000 7f5b31e00640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b2c0c2700 0x7f5b2c14c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.092+0000 7f5b31e00640 1 -- 172.21.15.31:0/1080489182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2c0a9680 con 0x7f5b2c1534f0 2024-09-11T00:38:08.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.093+0000 7f5b31e00640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c1534f0 0x7f5b2c149f20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400e960 tx=0x7f5b2400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:08.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.093+0000 7f5b29600640 1 -- 172.21.15.31:0/1080489182 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b24004d20 con 0x7f5b2c1534f0 2024-09-11T00:38:08.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.093+0000 7f5b29600640 1 -- 172.21.15.31:0/1080489182 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b2400cb20 con 0x7f5b2c1534f0 2024-09-11T00:38:08.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.093+0000 7f5b29600640 1 -- 172.21.15.31:0/1080489182 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b24005020 con 0x7f5b2c1534f0 2024-09-11T00:38:08.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.093+0000 7f5b32800640 1 -- 172.21.15.31:0/1080489182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2c074db0 con 0x7f5b2c1534f0 2024-09-11T00:38:08.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.093+0000 7f5b32800640 1 -- 172.21.15.31:0/1080489182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b2c074ad0 con 0x7f5b2c1534f0 2024-09-11T00:38:08.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.094+0000 7f5b32800640 1 -- 172.21.15.31:0/1080489182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b2c06dc60 con 0x7f5b2c1534f0 2024-09-11T00:38:08.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.095+0000 7f5b29600640 1 -- 172.21.15.31:0/1080489182 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5b2400f040 con 0x7f5b2c1534f0 2024-09-11T00:38:08.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.096+0000 7f5b29600640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b0c077fa0 0x7f5b0c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:08.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.096+0000 7f5b29600640 1 -- 172.21.15.31:0/1080489182 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f5b2409a990 con 0x7f5b2c1534f0 2024-09-11T00:38:08.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.097+0000 7f5b31400640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b0c077fa0 0x7f5b0c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:08.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.097+0000 7f5b31400640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b0c077fa0 0x7f5b0c07a460 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5b20033000 tx=0x7f5b20004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:08.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.100+0000 7f5b29600640 1 -- 172.21.15.31:0/1080489182 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5b24016030 con 0x7f5b2c1534f0 2024-09-11T00:38:08.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.333+0000 7f5b32800640 1 -- 172.21.15.31:0/1080489182 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b2c0c48b0 con 0x7f5b2c1534f0 2024-09-11T00:38:08.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.334+0000 7f5b29600640 1 -- 172.21.15.31:0/1080489182 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f5b24062de0 con 0x7f5b2c1534f0 2024-09-11T00:38:08.337 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:08.337 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:08.338 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:08.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 -- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b0c077fa0 msgr2=0x7f5b0c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:08.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b0c077fa0 0x7f5b0c07a460 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5b20033000 tx=0x7f5b20004730 comp rx=0 tx=0).stop 2024-09-11T00:38:08.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 -- 172.21.15.31:0/1080489182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c1534f0 msgr2=0x7f5b2c149f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:08.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c1534f0 0x7f5b2c149f20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400e960 tx=0x7f5b2400ee20 comp rx=0 tx=0).stop 2024-09-11T00:38:08.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 -- 172.21.15.31:0/1080489182 shutdown_connections 2024-09-11T00:38:08.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b0c077fa0 0x7f5b0c07a460 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b2c1534f0 0x7f5b2c149f20 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b2c0c6bb0 0x7f5b2c1499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 --2- 172.21.15.31:0/1080489182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b2c0c2700 0x7f5b2c14c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 -- 172.21.15.31:0/1080489182 >> 172.21.15.31:0/1080489182 conn(0x7f5b2c06caf0 msgr2=0x7f5b2c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:08.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 -- 172.21.15.31:0/1080489182 shutdown_connections 2024-09-11T00:38:08.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.337+0000 7f5b32800640 1 -- 172.21.15.31:0/1080489182 wait complete. 2024-09-11T00:38:08.356 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:38:08.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:08.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.707+0000 7f788fc00640 1 -- 172.21.15.31:0/1613370402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7888072480 msgr2=0x7f78881511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.707+0000 7f788fc00640 1 --2- 172.21.15.31:0/1613370402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7888072480 0x7f78881511a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f78880c5500 tx=0x7f787c02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.707+0000 7f788fc00640 1 -- 172.21.15.31:0/1613370402 shutdown_connections 2024-09-11T00:38:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.707+0000 7f788fc00640 1 --2- 172.21.15.31:0/1613370402 >> v2:172.21.15.204:3300/0 conn(0x7f7888153f20 0x7f78881516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.707+0000 7f788fc00640 1 --2- 172.21.15.31:0/1613370402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7888072480 0x7f78881511a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.707+0000 7f788fc00640 1 -- 172.21.15.31:0/1613370402 >> 172.21.15.31:0/1613370402 conn(0x7f78880a9750 msgr2=0x7f78880a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.707+0000 7f788fc00640 1 -- 172.21.15.31:0/1613370402 shutdown_connections 2024-09-11T00:38:08.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.707+0000 7f788fc00640 1 -- 172.21.15.31:0/1613370402 wait complete. 2024-09-11T00:38:08.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.708+0000 7f788fc00640 1 Processor -- start 2024-09-11T00:38:08.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.709+0000 7f788fc00640 1 -- start start 2024-09-11T00:38:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.709+0000 7f788fc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7888072480 0x7f78881ed180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.709+0000 7f788fc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78880c23a0 0x7f78881416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.709+0000 7f788fc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7888153f20 0x7f78880c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.709+0000 7f788fc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7888074200 con 0x7f7888153f20 2024-09-11T00:38:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.709+0000 7f788fc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7888074080 con 0x7f7888072480 2024-09-11T00:38:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.709+0000 7f788fc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7888072860 con 0x7f78880c23a0 2024-09-11T00:38:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.710+0000 7f788dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7888153f20 0x7f78880c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.710+0000 7f788dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7888153f20 0x7f78880c4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45212/0 (socket says 172.21.15.31:45212) 2024-09-11T00:38:08.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.710+0000 7f788dc00640 1 -- 172.21.15.31:0/191255868 learned_addr learned my addr 172.21.15.31:0/191255868 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.710+0000 7f788d200640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7888072480 0x7f78881ed180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.710+0000 7f7887e00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78880c23a0 0x7f78881416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.710+0000 7f788dc00640 1 -- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78880c23a0 msgr2=0x7f78881416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.710+0000 7f788dc00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78880c23a0 0x7f78881416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.710+0000 7f788dc00640 1 -- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7888072480 msgr2=0x7f78881ed180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.710+0000 7f788dc00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7888072480 0x7f78881ed180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.710+0000 7f788dc00640 1 -- 172.21.15.31:0/191255868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788814d130 con 0x7f7888153f20 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.711+0000 7f788dc00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7888153f20 0x7f78880c4100 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f788000e990 tx=0x7f788000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.711+0000 7f7884c00640 1 -- 172.21.15.31:0/191255868 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880004d20 con 0x7f7888153f20 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.711+0000 7f7884c00640 1 -- 172.21.15.31:0/191255868 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f788000cb00 con 0x7f7888153f20 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.711+0000 7f7884c00640 1 -- 172.21.15.31:0/191255868 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880005020 con 0x7f7888153f20 2024-09-11T00:38:08.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.711+0000 7f788fc00640 1 -- 172.21.15.31:0/191255868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788814d410 con 0x7f7888153f20 2024-09-11T00:38:08.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.711+0000 7f788fc00640 1 -- 172.21.15.31:0/191255868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7888149fb0 con 0x7f7888153f20 2024-09-11T00:38:08.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.712+0000 7f788fc00640 1 -- 172.21.15.31:0/191255868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78881511a0 con 0x7f7888153f20 2024-09-11T00:38:08.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.713+0000 7f7884c00640 1 -- 172.21.15.31:0/191255868 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f788000f040 con 0x7f7888153f20 2024-09-11T00:38:08.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.714+0000 7f7884c00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7868077f70 0x7f786807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:08.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.715+0000 7f7884c00640 1 -- 172.21.15.31:0/191255868 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f788009a8a0 con 0x7f7888153f20 2024-09-11T00:38:08.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.715+0000 7f788d200640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7868077f70 0x7f786807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:08.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.716+0000 7f788d200640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7868077f70 0x7f786807a430 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f787c033000 tx=0x7f787c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:08.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.719+0000 7f7884c00640 1 -- 172.21.15.31:0/191255868 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7880016030 con 0x7f7888153f20 2024-09-11T00:38:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.953+0000 7f788fc00640 1 -- 172.21.15.31:0/191255868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f788814d5e0 con 0x7f7888153f20 2024-09-11T00:38:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.954+0000 7f7884c00640 1 -- 172.21.15.31:0/191255868 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f7880062720 con 0x7f7888153f20 2024-09-11T00:38:08.956 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:08.956 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 -- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7868077f70 msgr2=0x7f786807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7868077f70 0x7f786807a430 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f787c033000 tx=0x7f787c004730 comp rx=0 tx=0).stop 2024-09-11T00:38:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 -- 172.21.15.31:0/191255868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7888153f20 msgr2=0x7f78880c4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7888153f20 0x7f78880c4100 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f788000e990 tx=0x7f788000ee50 comp rx=0 tx=0).stop 2024-09-11T00:38:08.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 -- 172.21.15.31:0/191255868 shutdown_connections 2024-09-11T00:38:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7868077f70 0x7f786807a430 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7888153f20 0x7f78880c4100 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78880c23a0 0x7f78881416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 --2- 172.21.15.31:0/191255868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7888072480 0x7f78881ed180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 -- 172.21.15.31:0/191255868 >> 172.21.15.31:0/191255868 conn(0x7f78880a9750 msgr2=0x7f7888154af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 -- 172.21.15.31:0/191255868 shutdown_connections 2024-09-11T00:38:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:08.956+0000 7f788fc00640 1 -- 172.21.15.31:0/191255868 wait complete. 2024-09-11T00:38:08.974 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:38:10.357 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:10.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.527+0000 7f2bf2e00640 1 -- 172.21.15.31:0/662269740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bec0c9a10 msgr2=0x7f2bec159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:10.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.527+0000 7f2bf2e00640 1 --2- 172.21.15.31:0/662269740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bec0c9a10 0x7f2bec159670 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2be4005fd0 tx=0x7f2be402a8b0 comp rx=0 tx=0).stop 2024-09-11T00:38:10.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.528+0000 7f2bf2e00640 1 -- 172.21.15.31:0/662269740 shutdown_connections 2024-09-11T00:38:10.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.528+0000 7f2bf2e00640 1 --2- 172.21.15.31:0/662269740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bec0c9a10 0x7f2bec159670 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:10.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.528+0000 7f2bf2e00640 1 --2- 172.21.15.31:0/662269740 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bec14c8e0 0x7f2bec156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:10.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.528+0000 7f2bf2e00640 1 -- 172.21.15.31:0/662269740 >> 172.21.15.31:0/662269740 conn(0x7f2bec068ee0 msgr2=0x7f2bec0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:10.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.528+0000 7f2bf2e00640 1 -- 172.21.15.31:0/662269740 shutdown_connections 2024-09-11T00:38:10.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.528+0000 7f2bf2e00640 1 -- 172.21.15.31:0/662269740 wait complete. 2024-09-11T00:38:10.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.529+0000 7f2bf2e00640 1 Processor -- start 2024-09-11T00:38:10.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.529+0000 7f2bf2e00640 1 -- start start 2024-09-11T00:38:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.530+0000 7f2bf2e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bec0c9a10 0x7f2bec14ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.530+0000 7f2bf2e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bec14c8e0 0x7f2bec14bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.530+0000 7f2bf2e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bec14ccc0 0x7f2bec0c12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.530+0000 7f2bf2e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec0c6c90 con 0x7f2bec0c9a10 2024-09-11T00:38:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.530+0000 7f2bf2e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec0c6490 con 0x7f2bec14ccc0 2024-09-11T00:38:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.530+0000 7f2bf2e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2bec0adca0 con 0x7f2bec14c8e0 2024-09-11T00:38:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.530+0000 7f2bf1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bec0c9a10 0x7f2bec14ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.530+0000 7f2bf1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bec0c9a10 0x7f2bec14ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45218/0 (socket says 172.21.15.31:45218) 2024-09-11T00:38:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf1a00640 1 -- 172.21.15.31:0/1580421495 learned_addr learned my addr 172.21.15.31:0/1580421495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf2400640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bec14ccc0 0x7f2bec0c12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf1000640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bec14c8e0 0x7f2bec14bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf1a00640 1 -- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bec14c8e0 msgr2=0x7f2bec14bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf1a00640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bec14c8e0 0x7f2bec14bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf1a00640 1 -- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bec14ccc0 msgr2=0x7f2bec0c12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf1a00640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bec14ccc0 0x7f2bec0c12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf1a00640 1 -- 172.21.15.31:0/1580421495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bec14bb00 con 0x7f2bec0c9a10 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf2400640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bec14ccc0 0x7f2bec0c12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf1000640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bec14c8e0 0x7f2bec14bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf1a00640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bec0c9a10 0x7f2bec14ebc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2be000ea90 tx=0x7f2be000ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2be9600640 1 -- 172.21.15.31:0/1580421495 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be000ce40 con 0x7f2bec0c9a10 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2be9600640 1 -- 172.21.15.31:0/1580421495 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2be0004020 con 0x7f2bec0c9a10 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2be9600640 1 -- 172.21.15.31:0/1580421495 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0004470 con 0x7f2bec0c9a10 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.531+0000 7f2bf2e00640 1 -- 172.21.15.31:0/1580421495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bec150030 con 0x7f2bec0c9a10 2024-09-11T00:38:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.532+0000 7f2bf2e00640 1 -- 172.21.15.31:0/1580421495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bec0c17f0 con 0x7f2bec0c9a10 2024-09-11T00:38:10.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.533+0000 7f2bf2e00640 1 -- 172.21.15.31:0/1580421495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bec156b00 con 0x7f2bec0c9a10 2024-09-11T00:38:10.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.534+0000 7f2be9600640 1 -- 172.21.15.31:0/1580421495 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2be00035f0 con 0x7f2bec0c9a10 2024-09-11T00:38:10.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.535+0000 7f2be9600640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2bcc077f70 0x7f2bcc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:10.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.535+0000 7f2be9600640 1 -- 172.21.15.31:0/1580421495 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f2be009ae20 con 0x7f2bec0c9a10 2024-09-11T00:38:10.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.535+0000 7f2bf1000640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2bcc077f70 0x7f2bcc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:10.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.536+0000 7f2bf1000640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2bcc077f70 0x7f2bcc07a430 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8001220 tx=0x7f2bd8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:10.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.539+0000 7f2be9600640 1 -- 172.21.15.31:0/1580421495 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2be0063270 con 0x7f2bec0c9a10 2024-09-11T00:38:10.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.769+0000 7f2bf2e00640 1 -- 172.21.15.31:0/1580421495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2bec154b30 con 0x7f2bec0c9a10 2024-09-11T00:38:10.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.769+0000 7f2be9600640 1 -- 172.21.15.31:0/1580421495 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f2be0062b70 con 0x7f2bec0c9a10 2024-09-11T00:38:10.771 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:10.772 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:10.772 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 -- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2bcc077f70 msgr2=0x7f2bcc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2bcc077f70 0x7f2bcc07a430 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8001220 tx=0x7f2bd8009040 comp rx=0 tx=0).stop 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 -- 172.21.15.31:0/1580421495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bec0c9a10 msgr2=0x7f2bec14ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bec0c9a10 0x7f2bec14ebc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2be000ea90 tx=0x7f2be000ef50 comp rx=0 tx=0).stop 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 -- 172.21.15.31:0/1580421495 shutdown_connections 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2bcc077f70 0x7f2bcc07a430 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bec14ccc0 0x7f2bec0c12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bec14c8e0 0x7f2bec14bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 --2- 172.21.15.31:0/1580421495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bec0c9a10 0x7f2bec14ebc0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 -- 172.21.15.31:0/1580421495 >> 172.21.15.31:0/1580421495 conn(0x7f2bec068ee0 msgr2=0x7f2bec0ac550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 -- 172.21.15.31:0/1580421495 shutdown_connections 2024-09-11T00:38:10.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:10.772+0000 7f2bf2e00640 1 -- 172.21.15.31:0/1580421495 wait complete. 2024-09-11T00:38:10.789 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:38:10.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:11.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.134+0000 7f1712e00640 1 -- 172.21.15.31:0/2148820588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f170c0c5b10 msgr2=0x7f170c156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:11.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.134+0000 7f1712e00640 1 --2- 172.21.15.31:0/2148820588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f170c0c5b10 0x7f170c156ad0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f170c06e7f0 tx=0x7f16fc02a560 comp rx=0 tx=0).stop 2024-09-11T00:38:11.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.134+0000 7f1712e00640 1 -- 172.21.15.31:0/2148820588 shutdown_connections 2024-09-11T00:38:11.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.134+0000 7f1712e00640 1 --2- 172.21.15.31:0/2148820588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f170c0c6340 0x7f170c1539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:11.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.134+0000 7f1712e00640 1 --2- 172.21.15.31:0/2148820588 >> v2:172.21.15.204:3300/0 conn(0x7f170c0c5ef0 0x7f170c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:11.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.134+0000 7f1712e00640 1 --2- 172.21.15.31:0/2148820588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f170c0c5b10 0x7f170c156ad0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:11.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.134+0000 7f1712e00640 1 -- 172.21.15.31:0/2148820588 >> 172.21.15.31:0/2148820588 conn(0x7f170c0a9750 msgr2=0x7f170c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:11.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.134+0000 7f1712e00640 1 -- 172.21.15.31:0/2148820588 shutdown_connections 2024-09-11T00:38:11.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.134+0000 7f1712e00640 1 -- 172.21.15.31:0/2148820588 wait complete. 2024-09-11T00:38:11.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.136+0000 7f1712e00640 1 Processor -- start 2024-09-11T00:38:11.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.136+0000 7f1712e00640 1 -- start start 2024-09-11T00:38:11.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1712e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f170c0c5b10 0x7f170c1f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:11.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1712e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f170c0c5ef0 0x7f170c0c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:11.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1712e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f170c0c6340 0x7f170c151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:11.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1712e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f170c1597f0 con 0x7f170c0c6340 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1712e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f170c159670 con 0x7f170c0c5ef0 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1712e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f170c159970 con 0x7f170c0c5b10 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1710e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f170c0c6340 0x7f170c151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1710e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f170c0c6340 0x7f170c151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45228/0 (socket says 172.21.15.31:45228) 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1710e00640 1 -- 172.21.15.31:0/1361037353 learned_addr learned my addr 172.21.15.31:0/1361037353 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f170be00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f170c0c5b10 0x7f170c1f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1703400640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f170c0c5ef0 0x7f170c0c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1710e00640 1 -- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f170c0c5b10 msgr2=0x7f170c1f03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1710e00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f170c0c5b10 0x7f170c1f03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1710e00640 1 -- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f170c0c5ef0 msgr2=0x7f170c0c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1710e00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f170c0c5ef0 0x7f170c0c4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:11.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.137+0000 7f1710e00640 1 -- 172.21.15.31:0/1361037353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170c1e1eb0 con 0x7f170c0c6340 2024-09-11T00:38:11.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.138+0000 7f1710e00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f170c0c6340 0x7f170c151ce0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f170400e9c0 tx=0x7f170400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:11.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.138+0000 7f1708c00640 1 -- 172.21.15.31:0/1361037353 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704004d20 con 0x7f170c0c6340 2024-09-11T00:38:11.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.138+0000 7f1708c00640 1 -- 172.21.15.31:0/1361037353 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f170400cb20 con 0x7f170c0c6340 2024-09-11T00:38:11.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.138+0000 7f1708c00640 1 -- 172.21.15.31:0/1361037353 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704005020 con 0x7f170c0c6340 2024-09-11T00:38:11.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.138+0000 7f1712e00640 1 -- 172.21.15.31:0/1361037353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170c1ec570 con 0x7f170c0c6340 2024-09-11T00:38:11.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.138+0000 7f1712e00640 1 -- 172.21.15.31:0/1361037353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170c1efed0 con 0x7f170c0c6340 2024-09-11T00:38:11.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.139+0000 7f1712e00640 1 -- 172.21.15.31:0/1361037353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f170c1f00d0 con 0x7f170c0c6340 2024-09-11T00:38:11.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.140+0000 7f1708c00640 1 -- 172.21.15.31:0/1361037353 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f170400f040 con 0x7f170c0c6340 2024-09-11T00:38:11.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.141+0000 7f1708c00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f16e8077f70 0x7f16e807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:11.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.141+0000 7f1708c00640 1 -- 172.21.15.31:0/1361037353 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f170409a9c0 con 0x7f170c0c6340 2024-09-11T00:38:11.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.142+0000 7f170be00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f16e8077f70 0x7f16e807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:11.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.142+0000 7f170be00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f16e8077f70 0x7f16e807a430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f16fc033000 tx=0x7f16fc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:11.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.145+0000 7f1708c00640 1 -- 172.21.15.31:0/1361037353 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f170c1f00d0 con 0x7f170c0c6340 2024-09-11T00:38:11.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.375+0000 7f1712e00640 1 -- 172.21.15.31:0/1361037353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f170c156ad0 con 0x7f170c0c6340 2024-09-11T00:38:11.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.375+0000 7f1708c00640 1 -- 172.21.15.31:0/1361037353 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7f1704016030 con 0x7f170c0c6340 2024-09-11T00:38:11.377 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:11.377 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:11.378 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:11.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.377+0000 7f1712e00640 1 -- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f16e8077f70 msgr2=0x7f16e807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:11.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.377+0000 7f1712e00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f16e8077f70 0x7f16e807a430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f16fc033000 tx=0x7f16fc004730 comp rx=0 tx=0).stop 2024-09-11T00:38:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.377+0000 7f1712e00640 1 -- 172.21.15.31:0/1361037353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f170c0c6340 msgr2=0x7f170c151ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.377+0000 7f1712e00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f170c0c6340 0x7f170c151ce0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f170400e9c0 tx=0x7f170400ee80 comp rx=0 tx=0).stop 2024-09-11T00:38:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.377+0000 7f1712e00640 1 -- 172.21.15.31:0/1361037353 shutdown_connections 2024-09-11T00:38:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.377+0000 7f1712e00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f16e8077f70 0x7f16e807a430 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.377+0000 7f1712e00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f170c0c6340 0x7f170c151ce0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.377+0000 7f1712e00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f170c0c5ef0 0x7f170c0c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.377+0000 7f1712e00640 1 --2- 172.21.15.31:0/1361037353 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f170c0c5b10 0x7f170c1f03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.377+0000 7f1712e00640 1 -- 172.21.15.31:0/1361037353 >> 172.21.15.31:0/1361037353 conn(0x7f170c0a9750 msgr2=0x7f170c0c2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.378+0000 7f1712e00640 1 -- 172.21.15.31:0/1361037353 shutdown_connections 2024-09-11T00:38:11.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:11.378+0000 7f1712e00640 1 -- 172.21.15.31:0/1361037353 wait complete. 2024-09-11T00:38:11.395 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:38:12.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:12.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.962+0000 7fb503000640 1 -- 172.21.15.31:0/193298844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4fc072480 msgr2=0x7fb4fc1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:12.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.962+0000 7fb503000640 1 --2- 172.21.15.31:0/193298844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4fc072480 0x7fb4fc1511a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009830 tx=0x7fb4ec02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:38:12.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.962+0000 7fb503000640 1 -- 172.21.15.31:0/193298844 shutdown_connections 2024-09-11T00:38:12.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.962+0000 7fb503000640 1 --2- 172.21.15.31:0/193298844 >> v2:172.21.15.204:3300/0 conn(0x7fb4fc0c23a0 0x7fb4fc1516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:12.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.962+0000 7fb503000640 1 --2- 172.21.15.31:0/193298844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4fc072480 0x7fb4fc1511a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:12.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.962+0000 7fb503000640 1 -- 172.21.15.31:0/193298844 >> 172.21.15.31:0/193298844 conn(0x7fb4fc0a9750 msgr2=0x7fb4fc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:12.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.962+0000 7fb503000640 1 -- 172.21.15.31:0/193298844 shutdown_connections 2024-09-11T00:38:12.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.962+0000 7fb503000640 1 -- 172.21.15.31:0/193298844 wait complete. 2024-09-11T00:38:12.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.964+0000 7fb503000640 1 Processor -- start 2024-09-11T00:38:12.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.964+0000 7fb503000640 1 -- start start 2024-09-11T00:38:12.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb503000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4fc072480 0x7fb4fc0c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:12.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb503000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4fc0c23a0 0x7fb4fc06f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:12.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb503000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4fc153f20 0x7fb4fc06fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:12.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb503000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc074200 con 0x7fb4fc072480 2024-09-11T00:38:12.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb503000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc074080 con 0x7fb4fc0c23a0 2024-09-11T00:38:12.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb503000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb4fc072860 con 0x7fb4fc153f20 2024-09-11T00:38:12.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb4fbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4fc072480 0x7fb4fc0c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:12.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb4fbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4fc072480 0x7fb4fc0c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45240/0 (socket says 172.21.15.31:45240) 2024-09-11T00:38:12.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb4fbe00640 1 -- 172.21.15.31:0/401304084 learned_addr learned my addr 172.21.15.31:0/401304084 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:12.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb501000640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4fc153f20 0x7fb4fc06fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:12.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.965+0000 7fb4f3e00640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4fc0c23a0 0x7fb4fc06f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:12.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb4fbe00640 1 -- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4fc153f20 msgr2=0x7fb4fc06fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:12.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb4fbe00640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4fc153f20 0x7fb4fc06fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:12.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb4fbe00640 1 -- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4fc0c23a0 msgr2=0x7fb4fc06f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:12.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb4fbe00640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4fc0c23a0 0x7fb4fc06f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:12.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb4fbe00640 1 -- 172.21.15.31:0/401304084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4fc0c6c20 con 0x7fb4fc072480 2024-09-11T00:38:12.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb4fbe00640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4fc072480 0x7fb4fc0c1040 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009950 tx=0x7fb4ec032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:12.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb4f8c00640 1 -- 172.21.15.31:0/401304084 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec040070 con 0x7fb4fc072480 2024-09-11T00:38:12.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb4f8c00640 1 -- 172.21.15.31:0/401304084 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4ec005310 con 0x7fb4fc072480 2024-09-11T00:38:12.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb4f8c00640 1 -- 172.21.15.31:0/401304084 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec005c90 con 0x7fb4fc072480 2024-09-11T00:38:12.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb503000640 1 -- 172.21.15.31:0/401304084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4fc0c6e90 con 0x7fb4fc072480 2024-09-11T00:38:12.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.966+0000 7fb503000640 1 -- 172.21.15.31:0/401304084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4fc0c7450 con 0x7fb4fc072480 2024-09-11T00:38:12.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.967+0000 7fb503000640 1 -- 172.21.15.31:0/401304084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4fc1511a0 con 0x7fb4fc072480 2024-09-11T00:38:12.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.968+0000 7fb4f8c00640 1 -- 172.21.15.31:0/401304084 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb4ec0049b0 con 0x7fb4fc072480 2024-09-11T00:38:12.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.969+0000 7fb4f8c00640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb4d8077fa0 0x7fb4d807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:12.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.969+0000 7fb4f8c00640 1 -- 172.21.15.31:0/401304084 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7fb4ec0bb7a0 con 0x7fb4fc072480 2024-09-11T00:38:12.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.970+0000 7fb4f3e00640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb4d8077fa0 0x7fb4d807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:12.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.970+0000 7fb4f3e00640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb4d8077fa0 0x7fb4d807a460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc1ef070 tx=0x7fb4e4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:12.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:12.973+0000 7fb4f8c00640 1 -- 172.21.15.31:0/401304084 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb4ec0835a0 con 0x7fb4fc072480 2024-09-11T00:38:13.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:38:13.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:38:13.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.206+0000 7fb503000640 1 -- 172.21.15.31:0/401304084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4fc0c5290 con 0x7fb4fc072480 2024-09-11T00:38:13.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.207+0000 7fb4f8c00640 1 -- 172.21.15.31:0/401304084 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7fb4ec087fd0 con 0x7fb4fc072480 2024-09-11T00:38:13.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:13.209 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:13.210 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.209+0000 7fb503000640 1 -- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb4d8077fa0 msgr2=0x7fb4d807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.209+0000 7fb503000640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb4d8077fa0 0x7fb4d807a460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc1ef070 tx=0x7fb4e4005f90 comp rx=0 tx=0).stop 2024-09-11T00:38:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.209+0000 7fb503000640 1 -- 172.21.15.31:0/401304084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4fc072480 msgr2=0x7fb4fc0c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.209+0000 7fb503000640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4fc072480 0x7fb4fc0c1040 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009950 tx=0x7fb4ec032c10 comp rx=0 tx=0).stop 2024-09-11T00:38:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.210+0000 7fb503000640 1 -- 172.21.15.31:0/401304084 shutdown_connections 2024-09-11T00:38:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.210+0000 7fb503000640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb4d8077fa0 0x7fb4d807a460 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.210+0000 7fb503000640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4fc153f20 0x7fb4fc06fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.210+0000 7fb503000640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4fc0c23a0 0x7fb4fc06f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.210+0000 7fb503000640 1 --2- 172.21.15.31:0/401304084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4fc072480 0x7fb4fc0c1040 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.210+0000 7fb503000640 1 -- 172.21.15.31:0/401304084 >> 172.21.15.31:0/401304084 conn(0x7fb4fc0a9750 msgr2=0x7fb4fc152e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.210+0000 7fb503000640 1 -- 172.21.15.31:0/401304084 shutdown_connections 2024-09-11T00:38:13.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.210+0000 7fb503000640 1 -- 172.21.15.31:0/401304084 wait complete. 2024-09-11T00:38:13.228 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:38:13.396 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:13.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.558+0000 7fc547e00640 1 -- 172.21.15.31:0/3523072157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc52c004e60 msgr2=0x7fc52c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.558+0000 7fc547e00640 1 --2- 172.21.15.31:0/3523072157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc52c004e60 0x7fc52c00b9c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc540001dc0 tx=0x7fc52c005830 comp rx=0 tx=0).stop 2024-09-11T00:38:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.558+0000 7fc547400640 1 -- 172.21.15.31:0/3523072157 reap_dead start 2024-09-11T00:38:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.558+0000 7fc547e00640 1 -- 172.21.15.31:0/3523072157 shutdown_connections 2024-09-11T00:38:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.558+0000 7fc547e00640 1 -- 172.21.15.31:0/3523072157 >> 172.21.15.31:0/3523072157 conn(0x7fc54806caf0 msgr2=0x7fc54806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.558+0000 7fc547e00640 1 -- 172.21.15.31:0/3523072157 shutdown_connections 2024-09-11T00:38:13.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.558+0000 7fc547e00640 1 -- 172.21.15.31:0/3523072157 wait complete. 2024-09-11T00:38:13.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.559+0000 7fc547e00640 1 Processor -- start 2024-09-11T00:38:13.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.560+0000 7fc547e00640 1 -- start start 2024-09-11T00:38:13.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.560+0000 7fc547e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc52c004e60 0x7fc5481ed1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:13.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.560+0000 7fc547e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc5481ed700 0x7fc548148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.560+0000 7fc547e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5480742d0 0x7fc548149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.560+0000 7fc547e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc5481510c0 con 0x7fc5480742d0 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.560+0000 7fc547e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc5480c5d20 con 0x7fc52c004e60 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.560+0000 7fc547e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc548151240 con 0x7fc5481ed700 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc547400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5480742d0 0x7fc548149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc547400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5480742d0 0x7fc548149160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45250/0 (socket says 172.21.15.31:45250) 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc547400640 1 -- 172.21.15.31:0/36718610 learned_addr learned my addr 172.21.15.31:0/36718610 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc546a00640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc52c004e60 0x7fc5481ed1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc53e000640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc5481ed700 0x7fc548148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc547400640 1 -- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc5481ed700 msgr2=0x7fc548148c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:13.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc547400640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc5481ed700 0x7fc548148c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc547400640 1 -- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc52c004e60 msgr2=0x7fc5481ed1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc547400640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc52c004e60 0x7fc5481ed1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc547400640 1 -- 172.21.15.31:0/36718610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5481ede90 con 0x7fc5480742d0 2024-09-11T00:38:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc546a00640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc52c004e60 0x7fc5481ed1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:38:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.561+0000 7fc547400640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5480742d0 0x7fc548149160 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc540001b30 tx=0x7fc540006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.562+0000 7fc53f400640 1 -- 172.21.15.31:0/36718610 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5400062b0 con 0x7fc5480742d0 2024-09-11T00:38:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.562+0000 7fc53f400640 1 -- 172.21.15.31:0/36718610 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc540006450 con 0x7fc5480742d0 2024-09-11T00:38:13.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.562+0000 7fc53f400640 1 -- 172.21.15.31:0/36718610 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc540006820 con 0x7fc5480742d0 2024-09-11T00:38:13.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.562+0000 7fc547e00640 1 -- 172.21.15.31:0/36718610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5481edbb0 con 0x7fc5480742d0 2024-09-11T00:38:13.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.562+0000 7fc547e00640 1 -- 172.21.15.31:0/36718610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5481559d0 con 0x7fc5480742d0 2024-09-11T00:38:13.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.563+0000 7fc547e00640 1 -- 172.21.15.31:0/36718610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5480ac6a0 con 0x7fc5480742d0 2024-09-11T00:38:13.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.564+0000 7fc53f400640 1 -- 172.21.15.31:0/36718610 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc54003e020 con 0x7fc5480742d0 2024-09-11T00:38:13.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.565+0000 7fc53f400640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc524077fa0 0x7fc52407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:13.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.565+0000 7fc53f400640 1 -- 172.21.15.31:0/36718610 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7fc5400b9d20 con 0x7fc5480742d0 2024-09-11T00:38:13.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.566+0000 7fc546a00640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc524077fa0 0x7fc52407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.566+0000 7fc546a00640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc524077fa0 0x7fc52407a460 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc538006a70 tx=0x7fc538006720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:13.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.569+0000 7fc53f400640 1 -- 172.21.15.31:0/36718610 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc540047030 con 0x7fc5480742d0 2024-09-11T00:38:13.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.799+0000 7fc547e00640 1 -- 172.21.15.31:0/36718610 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc54814f7e0 con 0x7fc5480742d0 2024-09-11T00:38:13.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.800+0000 7fc53f400640 1 -- 172.21.15.31:0/36718610 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6537 (secure 0 0 0) 0x7fc540082170 con 0x7fc5480742d0 2024-09-11T00:38:13.802 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:13.802 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":61,"btime":"2024-09-11T00:37:27:551055+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4307,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":5201,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:37:19.382248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5673":{"gid":5673,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":4,"addr":"172.21.15.31:6835/3242133790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3242133790},{"type":"v1","addr":"172.21.15.31:6835","nonce":3242133790}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5673,"qdb_cluster":[5673]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:37:12.342188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5449":{"gid":5449,"name":"b","rank":0,"incarnation":52,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6837/1278135080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1278135080},{"type":"v1","addr":"172.21.15.204:6837","nonce":1278135080}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5449,"qdb_cluster":[5449]},"id":2}]} 2024-09-11T00:38:13.802 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 61 2024-09-11T00:38:13.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 -- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc524077fa0 msgr2=0x7fc52407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:13.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc524077fa0 0x7fc52407a460 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc538006a70 tx=0x7fc538006720 comp rx=0 tx=0).stop 2024-09-11T00:38:13.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 -- 172.21.15.31:0/36718610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5480742d0 msgr2=0x7fc548149160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:13.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5480742d0 0x7fc548149160 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc540001b30 tx=0x7fc540006ca0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 -- 172.21.15.31:0/36718610 shutdown_connections 2024-09-11T00:38:13.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc524077fa0 0x7fc52407a460 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5480742d0 0x7fc548149160 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc5481ed700 0x7fc548148c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 --2- 172.21.15.31:0/36718610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc52c004e60 0x7fc5481ed1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:13.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 -- 172.21.15.31:0/36718610 >> 172.21.15.31:0/36718610 conn(0x7fc54806caf0 msgr2=0x7fc54814fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:13.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 -- 172.21.15.31:0/36718610 shutdown_connections 2024-09-11T00:38:13.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:13.802+0000 7fc547e00640 1 -- 172.21.15.31:0/36718610 wait complete. 2024-09-11T00:38:13.819 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:38:15.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:15.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.398+0000 7ffbc1000640 1 -- 172.21.15.31:0/604418395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba0004800 msgr2=0x7ffba0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.398+0000 7ffbc1000640 1 --2- 172.21.15.31:0/604418395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba0004800 0x7ffba0004c80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ffba4001690 tx=0x7ffba0005830 comp rx=0 tx=0).stop 2024-09-11T00:38:15.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.398+0000 7ffbc1000640 1 -- 172.21.15.31:0/604418395 shutdown_connections 2024-09-11T00:38:15.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.398+0000 7ffbc1000640 1 --2- 172.21.15.31:0/604418395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba0004800 0x7ffba0004c80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.398+0000 7ffbc1000640 1 -- 172.21.15.31:0/604418395 >> 172.21.15.31:0/604418395 conn(0x7ffbbc0a9750 msgr2=0x7ffbbc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:15.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.398+0000 7ffbc1000640 1 -- 172.21.15.31:0/604418395 shutdown_connections 2024-09-11T00:38:15.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.398+0000 7ffbc1000640 1 -- 172.21.15.31:0/604418395 wait complete. 2024-09-11T00:38:15.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.399+0000 7ffbc1000640 1 Processor -- start 2024-09-11T00:38:15.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.400+0000 7ffbc1000640 1 -- start start 2024-09-11T00:38:15.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.400+0000 7ffbc1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba0004800 0x7ffbbc149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.400+0000 7ffbc1000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbbc0c4b90 0x7ffbbc14a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.400+0000 7ffbc1000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffbbc1ef490 0x7ffbbc1ef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.400+0000 7ffbc1000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc074200 con 0x7ffba0004800 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.400+0000 7ffbc1000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc074080 con 0x7ffbbc0c4b90 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.400+0000 7ffbc1000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffbbc072860 con 0x7ffbbc1ef490 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba0004800 0x7ffbbc149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba0004800 0x7ffbbc149c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45272/0 (socket says 172.21.15.31:45272) 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb9e00640 1 -- 172.21.15.31:0/4040299553 learned_addr learned my addr 172.21.15.31:0/4040299553 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb1400640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbbc0c4b90 0x7ffbbc14a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbba800640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffbbc1ef490 0x7ffbbc1ef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb9e00640 1 -- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffbbc1ef490 msgr2=0x7ffbbc1ef8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb9e00640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffbbc1ef490 0x7ffbbc1ef8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb9e00640 1 -- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbbc0c4b90 msgr2=0x7ffbbc14a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb9e00640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbbc0c4b90 0x7ffbbc14a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb9e00640 1 -- 172.21.15.31:0/4040299553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbbc14b160 con 0x7ffba0004800 2024-09-11T00:38:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb1400640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbbc0c4b90 0x7ffbbc14a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:38:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbba800640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffbbc1ef490 0x7ffbbc1ef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:38:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.401+0000 7ffbb9e00640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba0004800 0x7ffbbc149c00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0062c0 tx=0x7ffbac006780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.402+0000 7ffbb2a00640 1 -- 172.21.15.31:0/4040299553 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac00fb40 con 0x7ffba0004800 2024-09-11T00:38:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.402+0000 7ffbb2a00640 1 -- 172.21.15.31:0/4040299553 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffbac016020 con 0x7ffba0004800 2024-09-11T00:38:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.402+0000 7ffbb2a00640 1 -- 172.21.15.31:0/4040299553 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac011430 con 0x7ffba0004800 2024-09-11T00:38:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.402+0000 7ffbc1000640 1 -- 172.21.15.31:0/4040299553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbbc0c70a0 con 0x7ffba0004800 2024-09-11T00:38:15.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.402+0000 7ffbc1000640 1 -- 172.21.15.31:0/4040299553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbbc1506f0 con 0x7ffba0004800 2024-09-11T00:38:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.403+0000 7ffbc1000640 1 -- 172.21.15.31:0/4040299553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbbc154000 con 0x7ffba0004800 2024-09-11T00:38:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.404+0000 7ffbb2a00640 1 -- 172.21.15.31:0/4040299553 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ffbac0115d0 con 0x7ffba0004800 2024-09-11T00:38:15.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.405+0000 7ffbb2a00640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb98077fa0 0x7ffb9807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.405+0000 7ffbb2a00640 1 -- 172.21.15.31:0/4040299553 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7ffbac09b2a0 con 0x7ffba0004800 2024-09-11T00:38:15.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.405+0000 7ffbb1400640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb98077fa0 0x7ffb9807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.406+0000 7ffbb1400640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb98077fa0 0x7ffb9807a460 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ffba4032000 tx=0x7ffba4001190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:15.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.409+0000 7ffbb2a00640 1 -- 172.21.15.31:0/4040299553 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffbac020020 con 0x7ffba0004800 2024-09-11T00:38:15.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.644+0000 7ffbc1000640 1 -- 172.21.15.31:0/4040299553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffbbc151e60 con 0x7ffba0004800 2024-09-11T00:38:15.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.644+0000 7ffbb2a00640 1 -- 172.21.15.31:0/4040299553 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+4880 (secure 0 0 0) 0x7ffbac063590 con 0x7ffba0004800 2024-09-11T00:38:15.646 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:15.646 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":63,"btime":"2024-09-11T00:38:15:345156+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:14.600788+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":115,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:14.625586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:38:15.646 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 63 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 -- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb98077fa0 msgr2=0x7ffb9807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb98077fa0 0x7ffb9807a460 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ffba4032000 tx=0x7ffba4001190 comp rx=0 tx=0).stop 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 -- 172.21.15.31:0/4040299553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba0004800 msgr2=0x7ffbbc149c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba0004800 0x7ffbbc149c00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ffbac0062c0 tx=0x7ffbac006780 comp rx=0 tx=0).stop 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 -- 172.21.15.31:0/4040299553 shutdown_connections 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffb98077fa0 0x7ffb9807a460 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffbbc1ef490 0x7ffbbc1ef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbbc0c4b90 0x7ffbbc14a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 --2- 172.21.15.31:0/4040299553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffba0004800 0x7ffbbc149c00 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 -- 172.21.15.31:0/4040299553 >> 172.21.15.31:0/4040299553 conn(0x7ffbbc0a9750 msgr2=0x7ffbbc152570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 -- 172.21.15.31:0/4040299553 shutdown_connections 2024-09-11T00:38:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.647+0000 7ffbc1000640 1 -- 172.21.15.31:0/4040299553 wait complete. 2024-09-11T00:38:15.664 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-11T00:38:15.665 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:38:15.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:15.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.825+0000 7f8f3e000640 1 -- 172.21.15.31:0/1101262899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f38153f20 msgr2=0x7f8f381516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.825+0000 7f8f3e000640 1 --2- 172.21.15.31:0/1101262899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f38153f20 0x7f8f381516e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8f380c4400 tx=0x7f8f3002ab40 comp rx=0 tx=0).stop 2024-09-11T00:38:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.825+0000 7f8f3e000640 1 -- 172.21.15.31:0/1101262899 shutdown_connections 2024-09-11T00:38:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.825+0000 7f8f3e000640 1 --2- 172.21.15.31:0/1101262899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f38153f20 0x7f8f381516e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.825+0000 7f8f3e000640 1 --2- 172.21.15.31:0/1101262899 >> v2:172.21.15.204:3300/0 conn(0x7f8f38072480 0x7f8f381511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.825+0000 7f8f3e000640 1 -- 172.21.15.31:0/1101262899 >> 172.21.15.31:0/1101262899 conn(0x7f8f380a9750 msgr2=0x7f8f380a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.825+0000 7f8f3e000640 1 -- 172.21.15.31:0/1101262899 shutdown_connections 2024-09-11T00:38:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.825+0000 7f8f3e000640 1 -- 172.21.15.31:0/1101262899 wait complete. 2024-09-11T00:38:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.826+0000 7f8f3e000640 1 Processor -- start 2024-09-11T00:38:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.827+0000 7f8f3e000640 1 -- start start 2024-09-11T00:38:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.827+0000 7f8f3e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8f38072480 0x7f8f381ecb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.827+0000 7f8f3e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f380c23a0 0x7f8f3806e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.827+0000 7f8f3e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f38153f20 0x7f8f380c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.827+0000 7f8f3e000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8f38074200 con 0x7f8f38153f20 2024-09-11T00:38:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.827+0000 7f8f3e000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8f38074080 con 0x7f8f38072480 2024-09-11T00:38:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.827+0000 7f8f3e000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8f38072860 con 0x7f8f380c23a0 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f37e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f38153f20 0x7f8f380c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f37e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f38153f20 0x7f8f380c46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45286/0 (socket says 172.21.15.31:45286) 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f37e00640 1 -- 172.21.15.31:0/759168483 learned_addr learned my addr 172.21.15.31:0/759168483 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f37400640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8f38072480 0x7f8f381ecb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f2ea00640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f380c23a0 0x7f8f3806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f37e00640 1 -- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f380c23a0 msgr2=0x7f8f3806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f37e00640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f380c23a0 0x7f8f3806e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f37e00640 1 -- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8f38072480 msgr2=0x7f8f381ecb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f37e00640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8f38072480 0x7f8f381ecb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f37e00640 1 -- 172.21.15.31:0/759168483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f38143df0 con 0x7f8f38153f20 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.828+0000 7f8f2ea00640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f380c23a0 0x7f8f3806e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:38:15.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.829+0000 7f8f37e00640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f38153f20 0x7f8f380c46b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8f3002b130 tx=0x7f8f3002c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:15.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.829+0000 7f8f2fe00640 1 -- 172.21.15.31:0/759168483 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f30004250 con 0x7f8f38153f20 2024-09-11T00:38:15.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.829+0000 7f8f2fe00640 1 -- 172.21.15.31:0/759168483 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f300043f0 con 0x7f8f38153f20 2024-09-11T00:38:15.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.829+0000 7f8f2fe00640 1 -- 172.21.15.31:0/759168483 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f30004800 con 0x7f8f38153f20 2024-09-11T00:38:15.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.829+0000 7f8f3e000640 1 -- 172.21.15.31:0/759168483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f380a90f0 con 0x7f8f38153f20 2024-09-11T00:38:15.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.829+0000 7f8f3e000640 1 -- 172.21.15.31:0/759168483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f381ec080 con 0x7f8f38153f20 2024-09-11T00:38:15.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.830+0000 7f8f3e000640 1 -- 172.21.15.31:0/759168483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f381511a0 con 0x7f8f38153f20 2024-09-11T00:38:15.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.832+0000 7f8f2fe00640 1 -- 172.21.15.31:0/759168483 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8f30049430 con 0x7f8f38153f20 2024-09-11T00:38:15.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.833+0000 7f8f2fe00640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8f14077f70 0x7f8f1407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.833+0000 7f8f2fe00640 1 -- 172.21.15.31:0/759168483 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f8f300bb610 con 0x7f8f38153f20 2024-09-11T00:38:15.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.833+0000 7f8f37400640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8f14077f70 0x7f8f1407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.834+0000 7f8f37400640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8f14077f70 0x7f8f1407a430 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8f38151150 tx=0x7f8f28005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:15.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.835+0000 7f8f2fe00640 1 -- 172.21.15.31:0/759168483 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8f30032030 con 0x7f8f38153f20 2024-09-11T00:38:15.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.983+0000 7f3217200640 1 -- 172.21.15.31:0/2691500012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3210157ac0 msgr2=0x7f321006f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.983+0000 7f3217200640 1 --2- 172.21.15.31:0/2691500012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3210157ac0 0x7f321006f820 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3210072e20 tx=0x7f31fc02a4a0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.983+0000 7f3217200640 1 -- 172.21.15.31:0/2691500012 shutdown_connections 2024-09-11T00:38:15.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.983+0000 7f3217200640 1 --2- 172.21.15.31:0/2691500012 >> v2:172.21.15.204:3300/0 conn(0x7f321006fd60 0x7f321015e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.983+0000 7f3217200640 1 --2- 172.21.15.31:0/2691500012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3210157ac0 0x7f321006f820 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.983+0000 7f3217200640 1 -- 172.21.15.31:0/2691500012 >> 172.21.15.31:0/2691500012 conn(0x7f32100c8320 msgr2=0x7f32100c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:15.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.983+0000 7f3217200640 1 -- 172.21.15.31:0/2691500012 shutdown_connections 2024-09-11T00:38:15.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.983+0000 7f3217200640 1 -- 172.21.15.31:0/2691500012 wait complete. 2024-09-11T00:38:15.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.985+0000 7f3217200640 1 Processor -- start 2024-09-11T00:38:15.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.985+0000 7f3217200640 1 -- start start 2024-09-11T00:38:15.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.985+0000 7f3217200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f321006fd60 0x7f3210156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.985+0000 7f3217200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32101576e0 0x7f3210156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.985+0000 7f3217200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3210157ac0 0x7f32100c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.985+0000 7f3217200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3210074da0 con 0x7f321006fd60 2024-09-11T00:38:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f3217200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3210074c20 con 0x7f3210157ac0 2024-09-11T00:38:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f3217200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f321006dfc0 con 0x7f32101576e0 2024-09-11T00:38:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f320fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f321006fd60 0x7f3210156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f320fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f321006fd60 0x7f3210156c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45310/0 (socket says 172.21.15.31:45310) 2024-09-11T00:38:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f320fe00640 1 -- 172.21.15.31:0/1484892360 learned_addr learned my addr 172.21.15.31:0/1484892360 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f3215200640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3210157ac0 0x7f32100c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f320f400640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32101576e0 0x7f3210156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f320fe00640 1 -- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32101576e0 msgr2=0x7f3210156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f320fe00640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32101576e0 0x7f3210156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f320fe00640 1 -- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3210157ac0 msgr2=0x7f32100c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f320fe00640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3210157ac0 0x7f32100c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f320fe00640 1 -- 172.21.15.31:0/1484892360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3210144de0 con 0x7f321006fd60 2024-09-11T00:38:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.986+0000 7f320f400640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32101576e0 0x7f3210156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:38:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.987+0000 7f320fe00640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f321006fd60 0x7f3210156c20 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f320000c490 tx=0x7f320000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.987+0000 7f3207e00640 1 -- 172.21.15.31:0/1484892360 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3200016020 con 0x7f321006fd60 2024-09-11T00:38:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.987+0000 7f3207e00640 1 -- 172.21.15.31:0/1484892360 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3200012940 con 0x7f321006fd60 2024-09-11T00:38:15.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.987+0000 7f3207e00640 1 -- 172.21.15.31:0/1484892360 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3200012d90 con 0x7f321006fd60 2024-09-11T00:38:15.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.987+0000 7f3217200640 1 -- 172.21.15.31:0/1484892360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32101ea6f0 con 0x7f321006fd60 2024-09-11T00:38:15.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.987+0000 7f3217200640 1 -- 172.21.15.31:0/1484892360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3210157160 con 0x7f321006fd60 2024-09-11T00:38:15.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.988+0000 7f3217200640 1 -- 172.21.15.31:0/1484892360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3210068d30 con 0x7f321006fd60 2024-09-11T00:38:15.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.989+0000 7f3207e00640 1 -- 172.21.15.31:0/1484892360 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f320001b400 con 0x7f321006fd60 2024-09-11T00:38:15.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.991+0000 7f3207e00640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31f0077f70 0x7f31f007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:15.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.991+0000 7f3207e00640 1 -- 172.21.15.31:0/1484892360 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f320009a680 con 0x7f321006fd60 2024-09-11T00:38:15.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.991+0000 7f320f400640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31f0077f70 0x7f31f007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:15.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.992+0000 7f320f400640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31f0077f70 0x7f31f007a430 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f31fc034000 tx=0x7f31fc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:15.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:15.995+0000 7f3207e00640 1 -- 172.21.15.31:0/1484892360 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3200020020 con 0x7f321006fd60 2024-09-11T00:38:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.072+0000 7f8f3e000640 1 -- 172.21.15.31:0/759168483 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f381445b0 con 0x7f8f38153f20 2024-09-11T00:38:16.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.073+0000 7f8f2fe00640 1 -- 172.21.15.31:0/759168483 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+4880 (secure 0 0 0) 0x7f8f30083a10 con 0x7f8f38153f20 2024-09-11T00:38:16.075 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:16.075 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":63,"btime":"2024-09-11T00:38:15:345156+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:14.600788+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":115,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:14.625586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:38:16.075 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 63 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.075+0000 7f8f3e000640 1 -- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8f14077f70 msgr2=0x7f8f1407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.075+0000 7f8f3e000640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8f14077f70 0x7f8f1407a430 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8f38151150 tx=0x7f8f28005f90 comp rx=0 tx=0).stop 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.075+0000 7f8f3e000640 1 -- 172.21.15.31:0/759168483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f38153f20 msgr2=0x7f8f380c46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.075+0000 7f8f3e000640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f38153f20 0x7f8f380c46b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8f3002b130 tx=0x7f8f3002c8b0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.076+0000 7f8f3e000640 1 -- 172.21.15.31:0/759168483 shutdown_connections 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.076+0000 7f8f3e000640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8f14077f70 0x7f8f1407a430 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.076+0000 7f8f3e000640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f38153f20 0x7f8f380c46b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.076+0000 7f8f3e000640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8f380c23a0 0x7f8f3806e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.076+0000 7f8f3e000640 1 --2- 172.21.15.31:0/759168483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8f38072480 0x7f8f381ecb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.076+0000 7f8f3e000640 1 -- 172.21.15.31:0/759168483 >> 172.21.15.31:0/759168483 conn(0x7f8f380a9750 msgr2=0x7f8f3814cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.076+0000 7f8f3e000640 1 -- 172.21.15.31:0/759168483 shutdown_connections 2024-09-11T00:38:16.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.076+0000 7f8f3e000640 1 -- 172.21.15.31:0/759168483 wait complete. 2024-09-11T00:38:16.097 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:38:15:345156+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4307": { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 62, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 115 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:38:14.600788+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4307 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5201": { "addr": "172.21.15.204:6835/3506150555", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 3506150555, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 3506150555, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5201, "incarnation": 62, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:38:14.625586+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5201 } } } ], "standbys": [] } 2024-09-11T00:38:16.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.230+0000 7f3217200640 1 -- 172.21.15.31:0/1484892360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32100c26b0 con 0x7f321006fd60 2024-09-11T00:38:16.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.231+0000 7f3207e00640 1 -- 172.21.15.31:0/1484892360 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+4880 (secure 0 0 0) 0x7f3200062aa0 con 0x7f321006fd60 2024-09-11T00:38:16.233 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:16.233 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":63,"btime":"2024-09-11T00:38:15:345156+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:14.600788+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":115,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:14.625586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:38:16.233 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 63 2024-09-11T00:38:16.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 -- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31f0077f70 msgr2=0x7f31f007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:16.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31f0077f70 0x7f31f007a430 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f31fc034000 tx=0x7f31fc004730 comp rx=0 tx=0).stop 2024-09-11T00:38:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 -- 172.21.15.31:0/1484892360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f321006fd60 msgr2=0x7f3210156c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f321006fd60 0x7f3210156c20 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f320000c490 tx=0x7f320000c950 comp rx=0 tx=0).stop 2024-09-11T00:38:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 -- 172.21.15.31:0/1484892360 shutdown_connections 2024-09-11T00:38:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f31f0077f70 0x7f31f007a430 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3210157ac0 0x7f32100c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f32101576e0 0x7f3210156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 --2- 172.21.15.31:0/1484892360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f321006fd60 0x7f3210156c20 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 -- 172.21.15.31:0/1484892360 >> 172.21.15.31:0/1484892360 conn(0x7f32100c8320 msgr2=0x7f32100c2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.233+0000 7f3217200640 1 -- 172.21.15.31:0/1484892360 shutdown_connections 2024-09-11T00:38:16.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.234+0000 7f3217200640 1 -- 172.21.15.31:0/1484892360 wait complete. 2024-09-11T00:38:16.252 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-11T00:38:16.252 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:38:16.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:16.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.413+0000 7efd1e000640 1 -- 172.21.15.31:0/2002153186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd180c61b0 msgr2=0x7efd180c6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.413+0000 7efd1e000640 1 --2- 172.21.15.31:0/2002153186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd180c61b0 0x7efd180c6600 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efd10005fd0 tx=0x7efd1002a820 comp rx=0 tx=0).stop 2024-09-11T00:38:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.413+0000 7efd1e000640 1 -- 172.21.15.31:0/2002153186 shutdown_connections 2024-09-11T00:38:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.413+0000 7efd1e000640 1 --2- 172.21.15.31:0/2002153186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd180c61b0 0x7efd180c6600 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.413+0000 7efd1e000640 1 -- 172.21.15.31:0/2002153186 >> 172.21.15.31:0/2002153186 conn(0x7efd1806caf0 msgr2=0x7efd1806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.413+0000 7efd1e000640 1 -- 172.21.15.31:0/2002153186 shutdown_connections 2024-09-11T00:38:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.414+0000 7efd1e000640 1 -- 172.21.15.31:0/2002153186 wait complete. 2024-09-11T00:38:16.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.415+0000 7efd1e000640 1 Processor -- start 2024-09-11T00:38:16.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.415+0000 7efd1e000640 1 -- start start 2024-09-11T00:38:16.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.416+0000 7efd1e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd180a6e20 0x7efd181575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.416+0000 7efd1e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd180ac2c0 0x7efd18074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.416+0000 7efd1e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd180c61b0 0x7efd181524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.416+0000 7efd1e000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efd181517d0 con 0x7efd180a6e20 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.416+0000 7efd1e000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd180c5d20 con 0x7efd180ac2c0 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.416+0000 7efd1e000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efd18151950 con 0x7efd180c61b0 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.416+0000 7efd1cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd180a6e20 0x7efd181575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.416+0000 7efd1cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd180a6e20 0x7efd181575e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45326/0 (socket says 172.21.15.31:45326) 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.416+0000 7efd1cc00640 1 -- 172.21.15.31:0/1412422248 learned_addr learned my addr 172.21.15.31:0/1412422248 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd17e00640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd180ac2c0 0x7efd18074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd1d600640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd180c61b0 0x7efd181524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:16.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd1cc00640 1 -- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd180c61b0 msgr2=0x7efd181524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:16.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd1cc00640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd180c61b0 0x7efd181524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd1cc00640 1 -- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd180ac2c0 msgr2=0x7efd18074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:16.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd1cc00640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd180ac2c0 0x7efd18074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd1cc00640 1 -- 172.21.15.31:0/1412422248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd18074f60 con 0x7efd180a6e20 2024-09-11T00:38:16.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd17e00640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd180ac2c0 0x7efd18074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:38:16.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd1d600640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd180c61b0 0x7efd181524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:38:16.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd1cc00640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd180a6e20 0x7efd181575e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00c500 tx=0x7efd0c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd14c00640 1 -- 172.21.15.31:0/1412422248 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c016020 con 0x7efd180a6e20 2024-09-11T00:38:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd14c00640 1 -- 172.21.15.31:0/1412422248 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd0c012940 con 0x7efd180a6e20 2024-09-11T00:38:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd1e000640 1 -- 172.21.15.31:0/1412422248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd18156910 con 0x7efd180a6e20 2024-09-11T00:38:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd14c00640 1 -- 172.21.15.31:0/1412422248 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c012d90 con 0x7efd180a6e20 2024-09-11T00:38:16.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.417+0000 7efd1e000640 1 -- 172.21.15.31:0/1412422248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd18152a00 con 0x7efd180a6e20 2024-09-11T00:38:16.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.418+0000 7efd1e000640 1 -- 172.21.15.31:0/1412422248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd181510c0 con 0x7efd180a6e20 2024-09-11T00:38:16.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.420+0000 7efd14c00640 1 -- 172.21.15.31:0/1412422248 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7efd0c01b400 con 0x7efd180a6e20 2024-09-11T00:38:16.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.421+0000 7efd14c00640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcf8077f70 0x7efcf807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:16.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.421+0000 7efd14c00640 1 -- 172.21.15.31:0/1412422248 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7efd0c09a8e0 con 0x7efd180a6e20 2024-09-11T00:38:16.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.421+0000 7efd17e00640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcf8077f70 0x7efcf807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:16.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.422+0000 7efd17e00640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcf8077f70 0x7efcf807a430 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efd04000c00 tx=0x7efd04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:16.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.425+0000 7efd14c00640 1 -- 172.21.15.31:0/1412422248 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efd0c020020 con 0x7efd180a6e20 2024-09-11T00:38:16.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.656+0000 7efd1e000640 1 -- 172.21.15.31:0/1412422248 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd1814c180 con 0x7efd180a6e20 2024-09-11T00:38:16.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.657+0000 7efd14c00640 1 -- 172.21.15.31:0/1412422248 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+4874 (secure 0 0 0) 0x7efd0c062be0 con 0x7efd180a6e20 2024-09-11T00:38:16.659 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:16.659 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":64,"btime":"2024-09-11T00:38:16:350539+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:15.352583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":116,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:15.353285+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:38:16.659 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 64 2024-09-11T00:38:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 -- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcf8077f70 msgr2=0x7efcf807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcf8077f70 0x7efcf807a430 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efd04000c00 tx=0x7efd04009040 comp rx=0 tx=0).stop 2024-09-11T00:38:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 -- 172.21.15.31:0/1412422248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd180a6e20 msgr2=0x7efd181575e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd180a6e20 0x7efd181575e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7efd0c00c500 tx=0x7efd0c00c9c0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 -- 172.21.15.31:0/1412422248 shutdown_connections 2024-09-11T00:38:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efcf8077f70 0x7efcf807a430 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efd180c61b0 0x7efd181524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd180ac2c0 0x7efd18074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 --2- 172.21.15.31:0/1412422248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd180a6e20 0x7efd181575e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:16.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 -- 172.21.15.31:0/1412422248 >> 172.21.15.31:0/1412422248 conn(0x7efd1806caf0 msgr2=0x7efd1814eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:16.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 -- 172.21.15.31:0/1412422248 shutdown_connections 2024-09-11T00:38:16.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:16.660+0000 7efd1e000640 1 -- 172.21.15.31:0/1412422248 wait complete. 2024-09-11T00:38:16.679 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:38:16:350539+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 64, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 64, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4307": { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 62, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 116 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:38:15.352583+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4307 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 64, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5201": { "addr": "172.21.15.204:6835/3506150555", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 3506150555, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 3506150555, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5201, "incarnation": 62, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:38:15.353285+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5201 } } } ], "standbys": [] } 2024-09-11T00:38:17.566 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:38:17.565+0000 7f2061a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:38:17.568 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:38:17.566+0000 7f2064c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:38:18.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:18.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.274+0000 7f6bfd800640 1 -- 172.21.15.31:0/59292198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf81539c0 msgr2=0x7f6bf8159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:18.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.274+0000 7f6bfd800640 1 --2- 172.21.15.31:0/59292198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf81539c0 0x7f6bf8159670 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0005fd0 tx=0x7f6bf002aa90 comp rx=0 tx=0).stop 2024-09-11T00:38:18.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.274+0000 7f6bfd800640 1 -- 172.21.15.31:0/59292198 shutdown_connections 2024-09-11T00:38:18.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.274+0000 7f6bfd800640 1 --2- 172.21.15.31:0/59292198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf81539c0 0x7f6bf8159670 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.274+0000 7f6bfd800640 1 --2- 172.21.15.31:0/59292198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bf8156ad0 0x7f6bf8153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.274+0000 7f6bfd800640 1 -- 172.21.15.31:0/59292198 >> 172.21.15.31:0/59292198 conn(0x7f6bf80a9750 msgr2=0x7f6bf80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:18.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.274+0000 7f6bfd800640 1 -- 172.21.15.31:0/59292198 shutdown_connections 2024-09-11T00:38:18.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.274+0000 7f6bfd800640 1 -- 172.21.15.31:0/59292198 wait complete. 2024-09-11T00:38:18.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.276+0000 7f6bfd800640 1 Processor -- start 2024-09-11T00:38:18.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.276+0000 7f6bfd800640 1 -- start start 2024-09-11T00:38:18.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.277+0000 7f6bfd800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bf80c5b10 0x7f6bf81529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:18.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.277+0000 7f6bfd800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf81539c0 0x7f6bf80c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.277+0000 7f6bfd800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf8156ad0 0x7f6bf8150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.277+0000 7f6bfd800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf8156ef0 con 0x7f6bf81539c0 2024-09-11T00:38:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.277+0000 7f6bfd800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf80c7320 con 0x7f6bf8156ad0 2024-09-11T00:38:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.277+0000 7f6bfd800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf80c3950 con 0x7f6bf80c5b10 2024-09-11T00:38:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf81539c0 0x7f6bf80c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf81539c0 0x7f6bf80c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41020/0 (socket says 172.21.15.31:41020) 2024-09-11T00:38:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf6000640 1 -- 172.21.15.31:0/3854662710 learned_addr learned my addr 172.21.15.31:0/3854662710 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf6a00640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bf80c5b10 0x7f6bf81529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:18.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf7400640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf8156ad0 0x7f6bf8150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:18.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf6000640 1 -- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bf80c5b10 msgr2=0x7f6bf81529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:18.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf6000640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bf80c5b10 0x7f6bf81529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf6000640 1 -- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf8156ad0 msgr2=0x7f6bf8150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:18.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf6000640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf8156ad0 0x7f6bf8150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf6000640 1 -- 172.21.15.31:0/3854662710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf81efcf0 con 0x7f6bf81539c0 2024-09-11T00:38:18.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.278+0000 7f6bf6a00640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bf80c5b10 0x7f6bf81529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:38:18.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.279+0000 7f6bf6000640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf81539c0 0x7f6bf80c1040 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6be400e790 tx=0x7f6be400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:18.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.279+0000 7f6beea00640 1 -- 172.21.15.31:0/3854662710 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be400eea0 con 0x7f6bf81539c0 2024-09-11T00:38:18.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.279+0000 7f6beea00640 1 -- 172.21.15.31:0/3854662710 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6be400ccc0 con 0x7f6bf81539c0 2024-09-11T00:38:18.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.279+0000 7f6beea00640 1 -- 172.21.15.31:0/3854662710 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4004280 con 0x7f6bf81539c0 2024-09-11T00:38:18.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.279+0000 7f6bfd800640 1 -- 172.21.15.31:0/3854662710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf814e750 con 0x7f6bf81539c0 2024-09-11T00:38:18.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.279+0000 7f6bfd800640 1 -- 172.21.15.31:0/3854662710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf80c1580 con 0x7f6bf81539c0 2024-09-11T00:38:18.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.280+0000 7f6bfd800640 1 -- 172.21.15.31:0/3854662710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf8159670 con 0x7f6bf81539c0 2024-09-11T00:38:18.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.281+0000 7f6beea00640 1 -- 172.21.15.31:0/3854662710 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6be40035f0 con 0x7f6bf81539c0 2024-09-11T00:38:18.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.283+0000 7f6beea00640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6bd8077f70 0x7f6bd807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:18.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.283+0000 7f6beea00640 1 -- 172.21.15.31:0/3854662710 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f6be409acb0 con 0x7f6bf81539c0 2024-09-11T00:38:18.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.283+0000 7f6bf6a00640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6bd8077f70 0x7f6bd807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:18.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.284+0000 7f6bf6a00640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6bd8077f70 0x7f6bd807a430 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6be8001bb0 tx=0x7f6be8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:18.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.287+0000 7f6beea00640 1 -- 172.21.15.31:0/3854662710 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6be40629e0 con 0x7f6bf81539c0 2024-09-11T00:38:18.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.588+0000 7f6bfd800640 1 -- 172.21.15.31:0/3854662710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bf80c1e70 con 0x7f6bf81539c0 2024-09-11T00:38:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.589+0000 7f6beea00640 1 -- 172.21.15.31:0/3854662710 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+4888 (secure 0 0 0) 0x7f6be4067410 con 0x7f6bf81539c0 2024-09-11T00:38:18.591 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:18.591 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"btime":"2024-09-11T00:38:17:360594+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:38:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2024-09-11T00:38:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.591+0000 7f6bfd800640 1 -- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6bd8077f70 msgr2=0x7f6bd807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.591+0000 7f6bfd800640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6bd8077f70 0x7f6bd807a430 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6be8001bb0 tx=0x7f6be8009040 comp rx=0 tx=0).stop 2024-09-11T00:38:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.591+0000 7f6bfd800640 1 -- 172.21.15.31:0/3854662710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf81539c0 msgr2=0x7f6bf80c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.591+0000 7f6bfd800640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf81539c0 0x7f6bf80c1040 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6be400e790 tx=0x7f6be400ec50 comp rx=0 tx=0).stop 2024-09-11T00:38:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.591+0000 7f6bfd800640 1 -- 172.21.15.31:0/3854662710 shutdown_connections 2024-09-11T00:38:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.591+0000 7f6bfd800640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6bd8077f70 0x7f6bd807a430 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.591+0000 7f6bfd800640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf8156ad0 0x7f6bf8150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.591+0000 7f6bfd800640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6bf81539c0 0x7f6bf80c1040 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.591+0000 7f6bfd800640 1 --2- 172.21.15.31:0/3854662710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6bf80c5b10 0x7f6bf81529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.591+0000 7f6bfd800640 1 -- 172.21.15.31:0/3854662710 >> 172.21.15.31:0/3854662710 conn(0x7f6bf80a9750 msgr2=0x7f6bf814fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.592+0000 7f6bfd800640 1 -- 172.21.15.31:0/3854662710 shutdown_connections 2024-09-11T00:38:18.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.592+0000 7f6bfd800640 1 -- 172.21.15.31:0/3854662710 wait complete. 2024-09-11T00:38:18.610 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=5449 2024-09-11T00:38:18.610 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.b 2024-09-11T00:38:18.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:18.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.865+0000 7f7243200640 1 -- 172.21.15.31:0/657098732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f723c150020 msgr2=0x7f723c0ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:18.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.865+0000 7f7243200640 1 --2- 172.21.15.31:0/657098732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f723c150020 0x7f723c0ad4e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f7230009830 tx=0x7f723002a510 comp rx=0 tx=0).stop 2024-09-11T00:38:18.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.865+0000 7f7243200640 1 -- 172.21.15.31:0/657098732 shutdown_connections 2024-09-11T00:38:18.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.865+0000 7f7243200640 1 --2- 172.21.15.31:0/657098732 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f723c0ada20 0x7f723c06e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.865+0000 7f7243200640 1 --2- 172.21.15.31:0/657098732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f723c150020 0x7f723c0ad4e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.865+0000 7f7243200640 1 -- 172.21.15.31:0/657098732 >> 172.21.15.31:0/657098732 conn(0x7f723c068f10 msgr2=0x7f723c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:18.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.865+0000 7f7243200640 1 -- 172.21.15.31:0/657098732 shutdown_connections 2024-09-11T00:38:18.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.865+0000 7f7243200640 1 -- 172.21.15.31:0/657098732 wait complete. 2024-09-11T00:38:18.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.867+0000 7f7243200640 1 Processor -- start 2024-09-11T00:38:18.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.867+0000 7f7243200640 1 -- start start 2024-09-11T00:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.868+0000 7f7243200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f723c0ada20 0x7f723c1ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.868+0000 7f7243200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f723c150020 0x7f723c0c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.868+0000 7f7243200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f723c156960 0x7f723c14e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.868+0000 7f7243200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f723c157550 con 0x7f723c150020 2024-09-11T00:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.868+0000 7f7243200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f723c1573d0 con 0x7f723c156960 2024-09-11T00:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.868+0000 7f7243200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f723c0cc220 con 0x7f723c0ada20 2024-09-11T00:38:18.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.868+0000 7f7241400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f723c150020 0x7f723c0c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.868+0000 7f7241e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f723c0ada20 0x7f723c1ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.868+0000 7f7241e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f723c0ada20 0x7f723c1ef5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:43776/0 (socket says 172.21.15.31:43776) 2024-09-11T00:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.869+0000 7f7241e00640 1 -- 172.21.15.31:0/3330220080 learned_addr learned my addr 172.21.15.31:0/3330220080 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.869+0000 7f7241400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f723c150020 0x7f723c0c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41046/0 (socket says 172.21.15.31:41046) 2024-09-11T00:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.869+0000 7f7242800640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f723c156960 0x7f723c14e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.869+0000 7f7241400640 1 -- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f723c0ada20 msgr2=0x7f723c1ef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.869+0000 7f7241400640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f723c0ada20 0x7f723c1ef5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.869+0000 7f7241400640 1 -- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f723c156960 msgr2=0x7f723c14e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.869+0000 7f7241400640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f723c156960 0x7f723c14e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:18.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.869+0000 7f7241400640 1 -- 172.21.15.31:0/3330220080 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f723c0c4100 con 0x7f723c150020 2024-09-11T00:38:18.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.869+0000 7f7241400640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f723c150020 0x7f723c0c9860 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f722c00ea20 tx=0x7f722c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:18.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.869+0000 7f7236000640 1 -- 172.21.15.31:0/3330220080 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c00ccc0 con 0x7f723c150020 2024-09-11T00:38:18.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.870+0000 7f7236000640 1 -- 172.21.15.31:0/3330220080 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f722c010430 con 0x7f723c150020 2024-09-11T00:38:18.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.870+0000 7f7236000640 1 -- 172.21.15.31:0/3330220080 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c010840 con 0x7f723c150020 2024-09-11T00:38:18.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.870+0000 7f7243200640 1 -- 172.21.15.31:0/3330220080 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f723c1efe70 con 0x7f723c150020 2024-09-11T00:38:18.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.870+0000 7f7243200640 1 -- 172.21.15.31:0/3330220080 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f723c0c2b40 con 0x7f723c150020 2024-09-11T00:38:18.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.871+0000 7f7243200640 1 -- 172.21.15.31:0/3330220080 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f723c0cc8e0 con 0x7f723c150020 2024-09-11T00:38:18.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.872+0000 7f7236000640 1 -- 172.21.15.31:0/3330220080 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f722c0109e0 con 0x7f723c150020 2024-09-11T00:38:18.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.873+0000 7f7236000640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7220077f70 0x7f722007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:18.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.873+0000 7f7236000640 1 -- 172.21.15.31:0/3330220080 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f722c09af20 con 0x7f723c150020 2024-09-11T00:38:18.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.874+0000 7f7241e00640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7220077f70 0x7f722007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:18.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.874+0000 7f7241e00640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7220077f70 0x7f722007a430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f72300095f0 tx=0x7f7230004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:18.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:18.878+0000 7f7236000640 1 -- 172.21.15.31:0/3330220080 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f722c062c00 con 0x7f723c150020 2024-09-11T00:38:19.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.139+0000 7f7243200640 1 -- 172.21.15.31:0/3330220080 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f723c0b17b0 con 0x7f723c150020 2024-09-11T00:38:19.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.140+0000 7f7236000640 1 -- 172.21.15.31:0/3330220080 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+4888 (secure 0 0 0) 0x7f722c067630 con 0x7f723c150020 2024-09-11T00:38:19.142 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:19.142 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"btime":"2024-09-11T00:38:17:360594+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:38:19.142 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2024-09-11T00:38:19.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 -- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7220077f70 msgr2=0x7f722007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:19.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7220077f70 0x7f722007a430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f72300095f0 tx=0x7f7230004730 comp rx=0 tx=0).stop 2024-09-11T00:38:19.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 -- 172.21.15.31:0/3330220080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f723c150020 msgr2=0x7f723c0c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:19.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f723c150020 0x7f723c0c9860 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f722c00ea20 tx=0x7f722c00eee0 comp rx=0 tx=0).stop 2024-09-11T00:38:19.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 -- 172.21.15.31:0/3330220080 shutdown_connections 2024-09-11T00:38:19.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7220077f70 0x7f722007a430 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:19.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f723c156960 0x7f723c14e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:19.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f723c150020 0x7f723c0c9860 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:19.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 --2- 172.21.15.31:0/3330220080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f723c0ada20 0x7f723c1ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:19.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 -- 172.21.15.31:0/3330220080 >> 172.21.15.31:0/3330220080 conn(0x7f723c068f10 msgr2=0x7f723c1576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:19.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 -- 172.21.15.31:0/3330220080 shutdown_connections 2024-09-11T00:38:19.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:19.142+0000 7f7243200640 1 -- 172.21.15.31:0/3330220080 wait complete. 2024-09-11T00:38:19.161 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=5673 2024-09-11T00:38:19.161 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before reviving mds.a 2024-09-11T00:38:19.891 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-11T00:38:19.889+0000 7f52d2200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:38:24.611 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-11T00:38:24.612 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-11T00:38:24.612 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-11T00:38:24.616 INFO:tasks.ceph.mds.b:Started 2024-09-11T00:38:24.617 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:24.750 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:38:24.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.802+0000 7f64ada00640 1 -- 172.21.15.31:0/4095300427 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f64a806fa70 msgr2=0x7f64a81566e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:24.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.802+0000 7f64ada00640 1 --2- 172.21.15.31:0/4095300427 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f64a806fa70 0x7f64a81566e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6494009860 tx=0x7f649402a4b0 comp rx=0 tx=0).stop 2024-09-11T00:38:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.802+0000 7f64ada00640 1 -- 172.21.15.31:0/4095300427 shutdown_connections 2024-09-11T00:38:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.802+0000 7f64ada00640 1 --2- 172.21.15.31:0/4095300427 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f64a806fa70 0x7f64a81566e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.802+0000 7f64ada00640 1 --2- 172.21.15.31:0/4095300427 >> v2:172.21.15.204:3300/0 conn(0x7f64a8156300 0x7f64a80c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.802+0000 7f64ada00640 1 -- 172.21.15.31:0/4095300427 >> 172.21.15.31:0/4095300427 conn(0x7f64a80a9750 msgr2=0x7f64a80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.802+0000 7f64ada00640 1 -- 172.21.15.31:0/4095300427 shutdown_connections 2024-09-11T00:38:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.802+0000 7f64ada00640 1 -- 172.21.15.31:0/4095300427 wait complete. 2024-09-11T00:38:24.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.804+0000 7f64ada00640 1 Processor -- start 2024-09-11T00:38:24.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.804+0000 7f64ada00640 1 -- start start 2024-09-11T00:38:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.805+0000 7f64ada00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a806fa70 0x7f64a80aa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.805+0000 7f64ada00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8156300 0x7f64a81e9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.805+0000 7f64ada00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f64a8156c20 0x7f64a80c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.805+0000 7f64ada00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f64a80c77a0 con 0x7f64a806fa70 2024-09-11T00:38:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.805+0000 7f64ada00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f64a80c7620 con 0x7f64a8156300 2024-09-11T00:38:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.805+0000 7f64ada00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f64a814d040 con 0x7f64a8156c20 2024-09-11T00:38:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a806fa70 0x7f64a80aa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8156300 0x7f64a81e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f64a8156c20 0x7f64a80c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8156300 0x7f64a81e9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:45994/0 (socket says 172.21.15.31:45994) 2024-09-11T00:38:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a806fa70 0x7f64a80aa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41060/0 (socket says 172.21.15.31:41060) 2024-09-11T00:38:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6000640 1 -- 172.21.15.31:0/3305207122 learned_addr learned my addr 172.21.15.31:0/3305207122 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:24.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6a00640 1 -- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f64a8156c20 msgr2=0x7f64a80c0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:24.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6a00640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f64a8156c20 0x7f64a80c0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:24.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6a00640 1 -- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8156300 msgr2=0x7f64a81e9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6a00640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8156300 0x7f64a81e9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6a00640 1 -- 172.21.15.31:0/3305207122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a814ef90 con 0x7f64a806fa70 2024-09-11T00:38:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.806+0000 7f64a6000640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8156300 0x7f64a81e9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:38:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.807+0000 7f64a6a00640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a806fa70 0x7f64a80aa9f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f649800e9f0 tx=0x7f649800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.807+0000 7f649ea00640 1 -- 172.21.15.31:0/3305207122 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649800cd80 con 0x7f64a806fa70 2024-09-11T00:38:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.807+0000 7f649ea00640 1 -- 172.21.15.31:0/3305207122 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6498004020 con 0x7f64a806fa70 2024-09-11T00:38:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.807+0000 7f649ea00640 1 -- 172.21.15.31:0/3305207122 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6498004450 con 0x7f64a806fa70 2024-09-11T00:38:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.807+0000 7f64ada00640 1 -- 172.21.15.31:0/3305207122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a80aaf30 con 0x7f64a806fa70 2024-09-11T00:38:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.807+0000 7f64ada00640 1 -- 172.21.15.31:0/3305207122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a814ffd0 con 0x7f64a806fa70 2024-09-11T00:38:24.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.808+0000 7f64ada00640 1 -- 172.21.15.31:0/3305207122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64a81516c0 con 0x7f64a806fa70 2024-09-11T00:38:24.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.809+0000 7f649ea00640 1 -- 172.21.15.31:0/3305207122 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f64980035f0 con 0x7f64a806fa70 2024-09-11T00:38:24.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.810+0000 7f649ea00640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6488077fa0 0x7f648807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:24.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.810+0000 7f649ea00640 1 -- 172.21.15.31:0/3305207122 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f649809ad00 con 0x7f64a806fa70 2024-09-11T00:38:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.811+0000 7f64a6000640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6488077fa0 0x7f648807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.812+0000 7f64a6000640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6488077fa0 0x7f648807a460 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f64a80c4cf0 tx=0x7f6494004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:24.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:24.815+0000 7f649ea00640 1 -- 172.21.15.31:0/3305207122 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f64a81516c0 con 0x7f64a806fa70 2024-09-11T00:38:25.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.088+0000 7f64ada00640 1 -- 172.21.15.31:0/3305207122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64a80aa7f0 con 0x7f64a806fa70 2024-09-11T00:38:25.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.089+0000 7f649ea00640 1 -- 172.21.15.31:0/3305207122 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+4888 (secure 0 0 0) 0x7f64a80aa7f0 con 0x7f64a806fa70 2024-09-11T00:38:25.092 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:25.092 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":65,"btime":"2024-09-11T00:38:17:360594+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:38:25.092 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 65 2024-09-11T00:38:25.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 -- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6488077fa0 msgr2=0x7f648807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:25.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6488077fa0 0x7f648807a460 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f64a80c4cf0 tx=0x7f6494004730 comp rx=0 tx=0).stop 2024-09-11T00:38:25.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 -- 172.21.15.31:0/3305207122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a806fa70 msgr2=0x7f64a80aa9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:25.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a806fa70 0x7f64a80aa9f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f649800e9f0 tx=0x7f649800eeb0 comp rx=0 tx=0).stop 2024-09-11T00:38:25.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 -- 172.21.15.31:0/3305207122 shutdown_connections 2024-09-11T00:38:25.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6488077fa0 0x7f648807a460 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:25.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f64a8156c20 0x7f64a80c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:25.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8156300 0x7f64a81e9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:25.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 --2- 172.21.15.31:0/3305207122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f64a806fa70 0x7f64a80aa9f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:25.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 -- 172.21.15.31:0/3305207122 >> 172.21.15.31:0/3305207122 conn(0x7f64a80a9750 msgr2=0x7f64a80c2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:25.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 -- 172.21.15.31:0/3305207122 shutdown_connections 2024-09-11T00:38:25.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:25.092+0000 7f64ada00640 1 -- 172.21.15.31:0/3305207122 wait complete. 2024-09-11T00:38:25.114 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-11T00:38:27.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:27.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.297+0000 7f23fa400640 1 -- 172.21.15.31:0/2191486649 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23f40bf2c0 msgr2=0x7f23f415a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:27.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.297+0000 7f23fa400640 1 --2- 172.21.15.31:0/2191486649 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23f40bf2c0 0x7f23f415a1a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f23ec009be0 tx=0x7f23ec02a890 comp rx=0 tx=0).stop 2024-09-11T00:38:27.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.298+0000 7f23fa400640 1 -- 172.21.15.31:0/2191486649 shutdown_connections 2024-09-11T00:38:27.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.298+0000 7f23fa400640 1 --2- 172.21.15.31:0/2191486649 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23f40bf2c0 0x7f23f415a1a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:27.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.298+0000 7f23fa400640 1 -- 172.21.15.31:0/2191486649 >> 172.21.15.31:0/2191486649 conn(0x7f23f40cc260 msgr2=0x7f23f40ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:27.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.298+0000 7f23fa400640 1 -- 172.21.15.31:0/2191486649 shutdown_connections 2024-09-11T00:38:27.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.298+0000 7f23fa400640 1 -- 172.21.15.31:0/2191486649 wait complete. 2024-09-11T00:38:27.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.299+0000 7f23fa400640 1 Processor -- start 2024-09-11T00:38:27.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.300+0000 7f23fa400640 1 -- start start 2024-09-11T00:38:27.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.300+0000 7f23fa400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23f40b1160 0x7f23f41ed090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:27.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23fa400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23f40bf2c0 0x7f23f40ac810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:27.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23fa400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23f40c9b30 0x7f23f40c5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:27.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23fa400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f23f40b1540 con 0x7f23f40bf2c0 2024-09-11T00:38:27.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23fa400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f23f406e5c0 con 0x7f23f40c9b30 2024-09-11T00:38:27.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23fa400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f23f40ae540 con 0x7f23f40b1160 2024-09-11T00:38:27.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23f40b1160 0x7f23f41ed090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:27.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23f40b1160 0x7f23f41ed090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:43802/0 (socket says 172.21.15.31:43802) 2024-09-11T00:38:27.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f3400640 1 -- 172.21.15.31:0/2909069052 learned_addr learned my addr 172.21.15.31:0/2909069052 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:27.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f3e00640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23f40c9b30 0x7f23f40c5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:27.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f2a00640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23f40bf2c0 0x7f23f40ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:27.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f3400640 1 -- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23f40c9b30 msgr2=0x7f23f40c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f3400640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23f40c9b30 0x7f23f40c5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f3400640 1 -- 172.21.15.31:0/2909069052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23f40bf2c0 msgr2=0x7f23f40ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f3400640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23f40bf2c0 0x7f23f40ac810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f3400640 1 -- 172.21.15.31:0/2909069052 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f40c3540 con 0x7f23f40b1160 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.301+0000 7f23f2a00640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23f40bf2c0 0x7f23f40ac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.302+0000 7f23f3400640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23f40b1160 0x7f23f41ed090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f23e400c4c0 tx=0x7f23e400c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.302+0000 7f23eb400640 1 -- 172.21.15.31:0/2909069052 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e40052d0 con 0x7f23f40b1160 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.302+0000 7f23eb400640 1 -- 172.21.15.31:0/2909069052 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f23e4005490 con 0x7f23f40b1160 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.302+0000 7f23eb400640 1 -- 172.21.15.31:0/2909069052 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4012890 con 0x7f23f40b1160 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.302+0000 7f23fa400640 1 -- 172.21.15.31:0/2909069052 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f40c3820 con 0x7f23f40b1160 2024-09-11T00:38:27.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.302+0000 7f23fa400640 1 -- 172.21.15.31:0/2909069052 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f40acd50 con 0x7f23f40b1160 2024-09-11T00:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.303+0000 7f23fa400640 1 -- 172.21.15.31:0/2909069052 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f40af000 con 0x7f23f40b1160 2024-09-11T00:38:27.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.305+0000 7f23eb400640 1 -- 172.21.15.31:0/2909069052 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f23e4012a30 con 0x7f23f40b1160 2024-09-11T00:38:27.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.306+0000 7f23eb400640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f23d4077f70 0x7f23d407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:27.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.306+0000 7f23eb400640 1 -- 172.21.15.31:0/2909069052 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f23e4099f10 con 0x7f23f40b1160 2024-09-11T00:38:27.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.306+0000 7f23f2a00640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f23d4077f70 0x7f23d407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:27.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.307+0000 7f23f2a00640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f23d4077f70 0x7f23d407a430 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f23f40af570 tx=0x7f23e0007aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:27.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.310+0000 7f23eb400640 1 -- 172.21.15.31:0/2909069052 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f23f40af000 con 0x7f23f40b1160 2024-09-11T00:38:27.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.581+0000 7f23fa400640 1 -- 172.21.15.31:0/2909069052 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23f4158fb0 con 0x7f23f40b1160 2024-09-11T00:38:27.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.582+0000 7f23eb400640 1 -- 172.21.15.31:0/2909069052 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5715 (secure 0 0 0) 0x7f23e4061d00 con 0x7f23f40b1160 2024-09-11T00:38:27.585 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:27.585 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":66,"btime":"2024-09-11T00:38:25:380536+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:38:27.585 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 66 2024-09-11T00:38:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 -- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f23d4077f70 msgr2=0x7f23d407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f23d4077f70 0x7f23d407a430 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f23f40af570 tx=0x7f23e0007aa0 comp rx=0 tx=0).stop 2024-09-11T00:38:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 -- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23f40b1160 msgr2=0x7f23f41ed090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23f40b1160 0x7f23f41ed090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f23e400c4c0 tx=0x7f23e400c980 comp rx=0 tx=0).stop 2024-09-11T00:38:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 -- 172.21.15.31:0/2909069052 shutdown_connections 2024-09-11T00:38:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f23d4077f70 0x7f23d407a430 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f23f40c9b30 0x7f23f40c5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23f40bf2c0 0x7f23f40ac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 --2- 172.21.15.31:0/2909069052 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f23f40b1160 0x7f23f41ed090 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:27.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 -- 172.21.15.31:0/2909069052 >> 172.21.15.31:0/2909069052 conn(0x7f23f40cc260 msgr2=0x7f23f40c9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:27.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 -- 172.21.15.31:0/2909069052 shutdown_connections 2024-09-11T00:38:27.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:27.585+0000 7f23fa400640 1 -- 172.21.15.31:0/2909069052 wait complete. 2024-09-11T00:38:27.605 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-11T00:38:27.605 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2024-09-11T00:38:28.162 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-11T00:38:28.162 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-11T00:38:28.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-11T00:38:28.167 INFO:tasks.ceph.mds.a:Started 2024-09-11T00:38:28.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:38:28.313 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-11T00:38:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.356+0000 7f8b6b400640 1 -- 172.21.15.31:0/1593562361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b640c4e10 msgr2=0x7f8b64157530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.356+0000 7f8b6b400640 1 --2- 172.21.15.31:0/1593562361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b640c4e10 0x7f8b64157530 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8b64074310 tx=0x7f8b5002a4d0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.357+0000 7f8b6b400640 1 -- 172.21.15.31:0/1593562361 shutdown_connections 2024-09-11T00:38:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.357+0000 7f8b6b400640 1 --2- 172.21.15.31:0/1593562361 >> v2:172.21.15.204:3300/0 conn(0x7f8b64157a70 0x7f8b64071e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.357+0000 7f8b6b400640 1 --2- 172.21.15.31:0/1593562361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b640c4e10 0x7f8b64157530 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.357+0000 7f8b6b400640 1 --2- 172.21.15.31:0/1593562361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b640c44f0 0x7f8b640c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.357+0000 7f8b6b400640 1 -- 172.21.15.31:0/1593562361 >> 172.21.15.31:0/1593562361 conn(0x7f8b640ab8a0 msgr2=0x7f8b640abcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.357+0000 7f8b6b400640 1 -- 172.21.15.31:0/1593562361 shutdown_connections 2024-09-11T00:38:28.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.357+0000 7f8b6b400640 1 -- 172.21.15.31:0/1593562361 wait complete. 2024-09-11T00:38:28.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.359+0000 7f8b6b400640 1 Processor -- start 2024-09-11T00:38:28.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.359+0000 7f8b6b400640 1 -- start start 2024-09-11T00:38:28.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b6b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b640c44f0 0x7f8b641efb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:28.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b6b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b640c4e10 0x7f8b640aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:28.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b6b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b64157a70 0x7f8b640af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:28.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b6b400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64151af0 con 0x7f8b640c44f0 2024-09-11T00:38:28.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b6b400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8b640c54e0 con 0x7f8b64157a70 2024-09-11T00:38:28.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b6b400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8b640701b0 con 0x7f8b640c4e10 2024-09-11T00:38:28.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b68a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b640c44f0 0x7f8b641efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:28.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b68a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b640c44f0 0x7f8b641efb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46172/0 (socket says 172.21.15.31:46172) 2024-09-11T00:38:28.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b68a00640 1 -- 172.21.15.31:0/855586842 learned_addr learned my addr 172.21.15.31:0/855586842 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:38:28.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b69400640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b64157a70 0x7f8b640af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:28.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.360+0000 7f8b63e00640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b640c4e10 0x7f8b640aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:28.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.361+0000 7f8b68a00640 1 -- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b640c4e10 msgr2=0x7f8b640aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:28.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.361+0000 7f8b68a00640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b640c4e10 0x7f8b640aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.361+0000 7f8b68a00640 1 -- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b64157a70 msgr2=0x7f8b640af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:28.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.361+0000 7f8b68a00640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b64157a70 0x7f8b640af4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.361+0000 7f8b68a00640 1 -- 172.21.15.31:0/855586842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b641f0360 con 0x7f8b640c44f0 2024-09-11T00:38:28.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.361+0000 7f8b68a00640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b640c44f0 0x7f8b641efb20 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800e9f0 tx=0x7f8b5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:28.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.361+0000 7f8b60c00640 1 -- 172.21.15.31:0/855586842 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5800ccc0 con 0x7f8b640c44f0 2024-09-11T00:38:28.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.362+0000 7f8b60c00640 1 -- 172.21.15.31:0/855586842 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b58010430 con 0x7f8b640c44f0 2024-09-11T00:38:28.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.362+0000 7f8b60c00640 1 -- 172.21.15.31:0/855586842 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58010840 con 0x7f8b640c44f0 2024-09-11T00:38:28.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.362+0000 7f8b6b400640 1 -- 172.21.15.31:0/855586842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b641f0600 con 0x7f8b640c44f0 2024-09-11T00:38:28.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.362+0000 7f8b6b400640 1 -- 172.21.15.31:0/855586842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b641572c0 con 0x7f8b640c44f0 2024-09-11T00:38:28.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.363+0000 7f8b6b400640 1 -- 172.21.15.31:0/855586842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b640a9380 con 0x7f8b640c44f0 2024-09-11T00:38:28.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.364+0000 7f8b60c00640 1 -- 172.21.15.31:0/855586842 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8b580109e0 con 0x7f8b640c44f0 2024-09-11T00:38:28.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.365+0000 7f8b60c00640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b44077f70 0x7f8b4407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:38:28.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.366+0000 7f8b60c00640 1 -- 172.21.15.31:0/855586842 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f8b5809add0 con 0x7f8b640c44f0 2024-09-11T00:38:28.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.366+0000 7f8b63e00640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b44077f70 0x7f8b4407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:38:28.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.367+0000 7f8b63e00640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b44077f70 0x7f8b4407a430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8b50033000 tx=0x7f8b50004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:38:28.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.370+0000 7f8b60c00640 1 -- 172.21.15.31:0/855586842 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8b58062ba0 con 0x7f8b640c44f0 2024-09-11T00:38:28.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.676+0000 7f8b6b400640 1 -- 172.21.15.31:0/855586842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b640c41d0 con 0x7f8b640c44f0 2024-09-11T00:38:28.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.677+0000 7f8b60c00640 1 -- 172.21.15.31:0/855586842 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7f8b580675d0 con 0x7f8b640c44f0 2024-09-11T00:38:28.870 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:38:28.870 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 -- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b44077f70 msgr2=0x7f8b4407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b44077f70 0x7f8b4407a430 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8b50033000 tx=0x7f8b50004730 comp rx=0 tx=0).stop 2024-09-11T00:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 -- 172.21.15.31:0/855586842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b640c44f0 msgr2=0x7f8b641efb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b640c44f0 0x7f8b641efb20 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8b5800e9f0 tx=0x7f8b5800edf0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 -- 172.21.15.31:0/855586842 shutdown_connections 2024-09-11T00:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8b44077f70 0x7f8b4407a430 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8b64157a70 0x7f8b640af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8b640c4e10 0x7f8b640aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 --2- 172.21.15.31:0/855586842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b640c44f0 0x7f8b641efb20 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 -- 172.21.15.31:0/855586842 >> 172.21.15.31:0/855586842 conn(0x7f8b640ab8a0 msgr2=0x7f8b64152920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 -- 172.21.15.31:0/855586842 shutdown_connections 2024-09-11T00:38:28.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:38:28.679+0000 7f8b6b400640 1 -- 172.21.15.31:0/855586842 wait complete. 2024-09-11T00:38:28.873 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-11T00:38:28.873 INFO:tasks.mds_thrash.fs.[a]:waiting for 111 secs before thrashing 2024-09-11T00:38:43.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:38:43.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:39:13.393 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:39:13.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:39:44.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:39:44.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:40:04.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:04.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.793+0000 7f8daca00640 1 -- 172.21.15.31:0/533517264 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da8070380 msgr2=0x7f8da8157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:04.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.793+0000 7f8daca00640 1 --2- 172.21.15.31:0/533517264 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da8070380 0x7f8da8157dc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009860 tx=0x7f8d9002a5a0 comp rx=0 tx=0).stop 2024-09-11T00:40:04.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.793+0000 7f8daca00640 1 -- 172.21.15.31:0/533517264 shutdown_connections 2024-09-11T00:40:04.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.793+0000 7f8daca00640 1 --2- 172.21.15.31:0/533517264 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da8070380 0x7f8da8157dc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:04.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.793+0000 7f8daca00640 1 -- 172.21.15.31:0/533517264 >> 172.21.15.31:0/533517264 conn(0x7f8da80c52f0 msgr2=0x7f8da80c56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:04.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.793+0000 7f8daca00640 1 -- 172.21.15.31:0/533517264 shutdown_connections 2024-09-11T00:40:04.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.794+0000 7f8daca00640 1 -- 172.21.15.31:0/533517264 wait complete. 2024-09-11T00:40:04.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.796+0000 7f8daca00640 1 Processor -- start 2024-09-11T00:40:04.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.796+0000 7f8daca00640 1 -- start start 2024-09-11T00:40:04.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8daca00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da806d6f0 0x7f8da80ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:04.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8daca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da8070380 0x7f8da814eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:04.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8daca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da81515e0 0x7f8da81ed400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:04.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8daca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8da8158300 con 0x7f8da81515e0 2024-09-11T00:40:04.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8daca00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8da8157930 con 0x7f8da8070380 2024-09-11T00:40:04.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8daca00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8da806d200 con 0x7f8da806d6f0 2024-09-11T00:40:04.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8da6800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da806d6f0 0x7f8da80ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:04.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8da7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da81515e0 0x7f8da81ed400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:04.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8d9fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da8070380 0x7f8da814eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:04.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8da6800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da806d6f0 0x7f8da80ac940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:42992/0 (socket says 172.21.15.31:42992) 2024-09-11T00:40:04.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.798+0000 7f8da6800640 1 -- 172.21.15.31:0/3018776347 learned_addr learned my addr 172.21.15.31:0/3018776347 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:04.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.799+0000 7f8d9fe00640 1 -- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da806d6f0 msgr2=0x7f8da80ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:04.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.799+0000 7f8d9fe00640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da806d6f0 0x7f8da80ac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:04.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.799+0000 7f8d9fe00640 1 -- 172.21.15.31:0/3018776347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da81515e0 msgr2=0x7f8da81ed400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:04.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.799+0000 7f8d9fe00640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da81515e0 0x7f8da81ed400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:04.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.799+0000 7f8d9fe00640 1 -- 172.21.15.31:0/3018776347 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da81ee090 con 0x7f8da8070380 2024-09-11T00:40:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.799+0000 7f8d9fe00640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da8070380 0x7f8da814eb30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8d90031000 tx=0x7f8d90032cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.800+0000 7f8d9ea00640 1 -- 172.21.15.31:0/3018776347 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d90040070 con 0x7f8da8070380 2024-09-11T00:40:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.800+0000 7f8d9ea00640 1 -- 172.21.15.31:0/3018776347 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d90005330 con 0x7f8da8070380 2024-09-11T00:40:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.800+0000 7f8d9ea00640 1 -- 172.21.15.31:0/3018776347 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d90003800 con 0x7f8da8070380 2024-09-11T00:40:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.800+0000 7f8daca00640 1 -- 172.21.15.31:0/3018776347 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da80a8030 con 0x7f8da8070380 2024-09-11T00:40:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.800+0000 7f8daca00640 1 -- 172.21.15.31:0/3018776347 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da81ea9e0 con 0x7f8da8070380 2024-09-11T00:40:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.801+0000 7f8daca00640 1 -- 172.21.15.31:0/3018776347 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da80c2e90 con 0x7f8da8070380 2024-09-11T00:40:04.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.802+0000 7f8d9ea00640 1 -- 172.21.15.31:0/3018776347 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8d900039a0 con 0x7f8da8070380 2024-09-11T00:40:04.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.804+0000 7f8d9ea00640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8d84077fa0 0x7f8d8407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:04.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.804+0000 7f8d9ea00640 1 -- 172.21.15.31:0/3018776347 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f8d900bba10 con 0x7f8da8070380 2024-09-11T00:40:04.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.804+0000 7f8da6800640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8d84077fa0 0x7f8d8407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:04.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.805+0000 7f8da6800640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8d84077fa0 0x7f8d8407a460 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8d98001870 tx=0x7f8d98009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:04.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:04.808+0000 7f8d9ea00640 1 -- 172.21.15.31:0/3018776347 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8d900837c0 con 0x7f8da8070380 2024-09-11T00:40:05.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.112+0000 7f8daca00640 1 -- 172.21.15.31:0/3018776347 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8da8070e30 con 0x7f8da8070380 2024-09-11T00:40:05.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.113+0000 7f8d9ea00640 1 -- 172.21.15.31:0/3018776347 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7f8d900881f0 con 0x7f8da8070380 2024-09-11T00:40:05.115 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:05.115 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:05.115 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:05.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 -- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8d84077fa0 msgr2=0x7f8d8407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:05.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8d84077fa0 0x7f8d8407a460 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8d98001870 tx=0x7f8d98009040 comp rx=0 tx=0).stop 2024-09-11T00:40:05.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 -- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da8070380 msgr2=0x7f8da814eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:05.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da8070380 0x7f8da814eb30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8d90031000 tx=0x7f8d90032cd0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 -- 172.21.15.31:0/3018776347 shutdown_connections 2024-09-11T00:40:05.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8d84077fa0 0x7f8d8407a460 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8da81515e0 0x7f8da81ed400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8da8070380 0x7f8da814eb30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 --2- 172.21.15.31:0/3018776347 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8da806d6f0 0x7f8da80ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 -- 172.21.15.31:0/3018776347 >> 172.21.15.31:0/3018776347 conn(0x7f8da80c52f0 msgr2=0x7f8da80c63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:05.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 -- 172.21.15.31:0/3018776347 shutdown_connections 2024-09-11T00:40:05.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.115+0000 7f8daca00640 1 -- 172.21.15.31:0/3018776347 wait complete. 2024-09-11T00:40:05.134 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-11T00:40:05.135 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:40:05.135 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:40:05.152 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:40:05.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.322+0000 7f5561400640 1 -- 172.21.15.31:0/3999158363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f555c0c9eb0 msgr2=0x7f555c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.322+0000 7f5561400640 1 --2- 172.21.15.31:0/3999158363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f555c0c9eb0 0x7f555c06dd20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f555c06f360 tx=0x7f554c02a510 comp rx=0 tx=0).stop 2024-09-11T00:40:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.323+0000 7f5561400640 1 -- 172.21.15.31:0/3999158363 shutdown_connections 2024-09-11T00:40:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.323+0000 7f5561400640 1 --2- 172.21.15.31:0/3999158363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f555c0c9eb0 0x7f555c06dd20 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.323+0000 7f5561400640 1 -- 172.21.15.31:0/3999158363 >> 172.21.15.31:0/3999158363 conn(0x7f555c0cbec0 msgr2=0x7f555c0cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.323+0000 7f5561400640 1 -- 172.21.15.31:0/3999158363 shutdown_connections 2024-09-11T00:40:05.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.323+0000 7f5561400640 1 -- 172.21.15.31:0/3999158363 wait complete. 2024-09-11T00:40:05.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.325+0000 7f5561400640 1 Processor -- start 2024-09-11T00:40:05.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.325+0000 7f5561400640 1 -- start start 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.326+0000 7f5561400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f555c0bf1a0 0x7f555c1ed190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.326+0000 7f5561400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f555c0c6690 0x7f555c0ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.326+0000 7f5561400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f555c1ef4c0 0x7f555c149840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.326+0000 7f5561400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f555c0c6a70 con 0x7f555c0bf1a0 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.326+0000 7f5561400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f555c155f90 con 0x7f555c0c6690 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.326+0000 7f5561400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f555c0c6bf0 con 0x7f555c1ef4c0 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.326+0000 7f555be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f555c0bf1a0 0x7f555c1ed190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.326+0000 7f555be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f555c0bf1a0 0x7f555c1ed190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34294/0 (socket says 172.21.15.31:34294) 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.326+0000 7f555be00640 1 -- 172.21.15.31:0/2874909529 learned_addr learned my addr 172.21.15.31:0/2874909529 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.326+0000 7f555b400640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f555c0c6690 0x7f555c0ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:05.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.327+0000 7f5560a00640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f555c1ef4c0 0x7f555c149840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:05.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.327+0000 7f555be00640 1 -- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f555c1ef4c0 msgr2=0x7f555c149840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:05.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.327+0000 7f555be00640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f555c1ef4c0 0x7f555c149840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.327+0000 7f555be00640 1 -- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f555c0c6690 msgr2=0x7f555c0ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:05.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.327+0000 7f555be00640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f555c0c6690 0x7f555c0ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.327+0000 7f555be00640 1 -- 172.21.15.31:0/2874909529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555c14a730 con 0x7f555c0bf1a0 2024-09-11T00:40:05.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.327+0000 7f555be00640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f555c0bf1a0 0x7f555c1ed190 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f554400c4b0 tx=0x7f554400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:05.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.327+0000 7f5553e00640 1 -- 172.21.15.31:0/2874909529 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544016020 con 0x7f555c0bf1a0 2024-09-11T00:40:05.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.327+0000 7f5553e00640 1 -- 172.21.15.31:0/2874909529 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5544012920 con 0x7f555c0bf1a0 2024-09-11T00:40:05.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.327+0000 7f5561400640 1 -- 172.21.15.31:0/2874909529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f555c0abeb0 con 0x7f555c0bf1a0 2024-09-11T00:40:05.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.328+0000 7f5553e00640 1 -- 172.21.15.31:0/2874909529 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544012d70 con 0x7f555c0bf1a0 2024-09-11T00:40:05.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.328+0000 7f5561400640 1 -- 172.21.15.31:0/2874909529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f555c0af5a0 con 0x7f555c0bf1a0 2024-09-11T00:40:05.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.329+0000 7f5561400640 1 -- 172.21.15.31:0/2874909529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f555c0b1410 con 0x7f555c0bf1a0 2024-09-11T00:40:05.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.330+0000 7f5553e00640 1 -- 172.21.15.31:0/2874909529 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f554401b400 con 0x7f555c0bf1a0 2024-09-11T00:40:05.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.331+0000 7f5553e00640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5538077f70 0x7f553807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:05.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.331+0000 7f5553e00640 1 -- 172.21.15.31:0/2874909529 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f554409a720 con 0x7f555c0bf1a0 2024-09-11T00:40:05.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.331+0000 7f555b400640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5538077f70 0x7f553807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:05.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.332+0000 7f555b400640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5538077f70 0x7f553807a430 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f554c034000 tx=0x7f554c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:05.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.339+0000 7f5553e00640 1 -- 172.21.15.31:0/2874909529 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5544020020 con 0x7f555c0bf1a0 2024-09-11T00:40:05.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.632+0000 7f5561400640 1 -- 172.21.15.31:0/2874909529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f555c06dec0 con 0x7f555c0bf1a0 2024-09-11T00:40:05.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.632+0000 7f5553e00640 1 -- 172.21.15.31:0/2874909529 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7f5544062450 con 0x7f555c0bf1a0 2024-09-11T00:40:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:05.635 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:05.635 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:05.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 -- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5538077f70 msgr2=0x7f553807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:05.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5538077f70 0x7f553807a430 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f554c034000 tx=0x7f554c004730 comp rx=0 tx=0).stop 2024-09-11T00:40:05.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 -- 172.21.15.31:0/2874909529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f555c0bf1a0 msgr2=0x7f555c1ed190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:05.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f555c0bf1a0 0x7f555c1ed190 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f554400c4b0 tx=0x7f554400c970 comp rx=0 tx=0).stop 2024-09-11T00:40:05.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 -- 172.21.15.31:0/2874909529 shutdown_connections 2024-09-11T00:40:05.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5538077f70 0x7f553807a430 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f555c1ef4c0 0x7f555c149840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f555c0c6690 0x7f555c0ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 --2- 172.21.15.31:0/2874909529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f555c0bf1a0 0x7f555c1ed190 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:05.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 -- 172.21.15.31:0/2874909529 >> 172.21.15.31:0/2874909529 conn(0x7f555c0cbec0 msgr2=0x7f555c0b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:05.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 -- 172.21.15.31:0/2874909529 shutdown_connections 2024-09-11T00:40:05.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:05.635+0000 7f5561400640 1 -- 172.21.15.31:0/2874909529 wait complete. 2024-09-11T00:40:05.655 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:40:07.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:07.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.847+0000 7f142a400640 1 -- 172.21.15.31:0/2080617180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f142406fd60 msgr2=0x7f142415e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:07.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.847+0000 7f142a400640 1 --2- 172.21.15.31:0/2080617180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f142406fd60 0x7f142415e7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f141c005fd0 tx=0x7f141c02a820 comp rx=0 tx=0).stop 2024-09-11T00:40:07.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.847+0000 7f142a400640 1 -- 172.21.15.31:0/2080617180 shutdown_connections 2024-09-11T00:40:07.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.847+0000 7f142a400640 1 --2- 172.21.15.31:0/2080617180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f142406fd60 0x7f142415e7a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:07.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.847+0000 7f142a400640 1 --2- 172.21.15.31:0/2080617180 >> v2:172.21.15.204:3300/0 conn(0x7f1424157ac0 0x7f142406f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:07.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.847+0000 7f142a400640 1 -- 172.21.15.31:0/2080617180 >> 172.21.15.31:0/2080617180 conn(0x7f14240c8320 msgr2=0x7f14240c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:07.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.847+0000 7f142a400640 1 -- 172.21.15.31:0/2080617180 shutdown_connections 2024-09-11T00:40:07.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.847+0000 7f142a400640 1 -- 172.21.15.31:0/2080617180 wait complete. 2024-09-11T00:40:07.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.849+0000 7f142a400640 1 Processor -- start 2024-09-11T00:40:07.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.850+0000 7f142a400640 1 -- start start 2024-09-11T00:40:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f142a400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f142406fd60 0x7f1424156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f142a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14241576e0 0x7f1424150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f142a400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1424157ac0 0x7f14241507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f142a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1424074da0 con 0x7f14241576e0 2024-09-11T00:40:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f142a400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1424074c20 con 0x7f142406fd60 2024-09-11T00:40:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f142a400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f142406dfc0 con 0x7f1424157ac0 2024-09-11T00:40:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f1422a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14241576e0 0x7f1424150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f1422a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14241576e0 0x7f1424150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34298/0 (socket says 172.21.15.31:34298) 2024-09-11T00:40:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f1422a00640 1 -- 172.21.15.31:0/2463616436 learned_addr learned my addr 172.21.15.31:0/2463616436 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:07.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f1423e00640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1424157ac0 0x7f14241507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:07.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f1423400640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f142406fd60 0x7f1424156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f1422a00640 1 -- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1424157ac0 msgr2=0x7f14241507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.851+0000 7f1422a00640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1424157ac0 0x7f14241507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.852+0000 7f1422a00640 1 -- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f142406fd60 msgr2=0x7f1424156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.852+0000 7f1422a00640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f142406fd60 0x7f1424156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.852+0000 7f1422a00640 1 -- 172.21.15.31:0/2463616436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1424144b40 con 0x7f14241576e0 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.852+0000 7f1422a00640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14241576e0 0x7f1424150260 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1410009fd0 tx=0x7f141000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.852+0000 7f141b400640 1 -- 172.21.15.31:0/2463616436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141000cc60 con 0x7f14241576e0 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.852+0000 7f141b400640 1 -- 172.21.15.31:0/2463616436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f141000ce00 con 0x7f14241576e0 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.852+0000 7f141b400640 1 -- 172.21.15.31:0/2463616436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410005280 con 0x7f14241576e0 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.852+0000 7f142a400640 1 -- 172.21.15.31:0/2463616436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14240abb50 con 0x7f14241576e0 2024-09-11T00:40:07.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.852+0000 7f142a400640 1 -- 172.21.15.31:0/2463616436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14240ac110 con 0x7f14241576e0 2024-09-11T00:40:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.854+0000 7f142a400640 1 -- 172.21.15.31:0/2463616436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1424068d30 con 0x7f14241576e0 2024-09-11T00:40:07.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.855+0000 7f141b400640 1 -- 172.21.15.31:0/2463616436 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1410005420 con 0x7f14241576e0 2024-09-11T00:40:07.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.856+0000 7f141b400640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1404077f70 0x7f140407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:07.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.856+0000 7f141b400640 1 -- 172.21.15.31:0/2463616436 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f141009aca0 con 0x7f14241576e0 2024-09-11T00:40:07.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.857+0000 7f1423400640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1404077f70 0x7f140407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:07.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.858+0000 7f1423400640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1404077f70 0x7f140407a430 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1414007fd0 tx=0x7f1414001410 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:07.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:07.862+0000 7f141b400640 1 -- 172.21.15.31:0/2463616436 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f14100629d0 con 0x7f14241576e0 2024-09-11T00:40:08.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.116+0000 7f142a400640 1 -- 172.21.15.31:0/2463616436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1424070a70 con 0x7f14241576e0 2024-09-11T00:40:08.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.117+0000 7f141b400640 1 -- 172.21.15.31:0/2463616436 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7f1410067400 con 0x7f14241576e0 2024-09-11T00:40:08.119 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:08.119 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:08.119 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:08.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 -- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1404077f70 msgr2=0x7f140407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:08.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1404077f70 0x7f140407a430 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1414007fd0 tx=0x7f1414001410 comp rx=0 tx=0).stop 2024-09-11T00:40:08.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 -- 172.21.15.31:0/2463616436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14241576e0 msgr2=0x7f1424150260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:08.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14241576e0 0x7f1424150260 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1410009fd0 tx=0x7f141000ec20 comp rx=0 tx=0).stop 2024-09-11T00:40:08.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 -- 172.21.15.31:0/2463616436 shutdown_connections 2024-09-11T00:40:08.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1404077f70 0x7f140407a430 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:08.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1424157ac0 0x7f14241507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:08.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14241576e0 0x7f1424150260 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:08.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 --2- 172.21.15.31:0/2463616436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f142406fd60 0x7f1424156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:08.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 -- 172.21.15.31:0/2463616436 >> 172.21.15.31:0/2463616436 conn(0x7f14240c8320 msgr2=0x7f14240c2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:08.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.119+0000 7f142a400640 1 -- 172.21.15.31:0/2463616436 shutdown_connections 2024-09-11T00:40:08.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:08.120+0000 7f142a400640 1 -- 172.21.15.31:0/2463616436 wait complete. 2024-09-11T00:40:08.139 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:40:10.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:10.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.314+0000 7f0a7be00640 1 -- 172.21.15.31:0/377461909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c0acbe0 msgr2=0x7f0a7c0c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:10.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.314+0000 7f0a7be00640 1 --2- 172.21.15.31:0/377461909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c0acbe0 0x7f0a7c0c61b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0c5ea0 tx=0x7f0a7402a980 comp rx=0 tx=0).stop 2024-09-11T00:40:10.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.314+0000 7f0a7be00640 1 -- 172.21.15.31:0/377461909 shutdown_connections 2024-09-11T00:40:10.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.314+0000 7f0a7be00640 1 --2- 172.21.15.31:0/377461909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c0acbe0 0x7f0a7c0c61b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:10.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.314+0000 7f0a7be00640 1 --2- 172.21.15.31:0/377461909 >> v2:172.21.15.204:3300/0 conn(0x7f0a7c0ac2c0 0x7f0a7c0ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.314+0000 7f0a7be00640 1 -- 172.21.15.31:0/377461909 >> 172.21.15.31:0/377461909 conn(0x7f0a7c06caf0 msgr2=0x7f0a7c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.314+0000 7f0a7be00640 1 -- 172.21.15.31:0/377461909 shutdown_connections 2024-09-11T00:40:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.314+0000 7f0a7be00640 1 -- 172.21.15.31:0/377461909 wait complete. 2024-09-11T00:40:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.316+0000 7f0a7be00640 1 Processor -- start 2024-09-11T00:40:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.316+0000 7f0a7be00640 1 -- start start 2024-09-11T00:40:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c0a6e20 0x7f0a7c073910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a7c0ac2c0 0x7f0a7c14ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7c0acbe0 0x7f0a7c149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c151950 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c1517d0 con 0x7f0a7c0ac2c0 2024-09-11T00:40:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0a7c0c66f0 con 0x7f0a7c0acbe0 2024-09-11T00:40:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c0a6e20 0x7f0a7c073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c0a6e20 0x7f0a7c073910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56220/0 (socket says 172.21.15.31:56220) 2024-09-11T00:40:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7aa00640 1 -- 172.21.15.31:0/3410785373 learned_addr learned my addr 172.21.15.31:0/3410785373 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7a000640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a7c0ac2c0 0x7f0a7c14ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7b400640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7c0acbe0 0x7f0a7c149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7aa00640 1 -- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7c0acbe0 msgr2=0x7f0a7c149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7aa00640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7c0acbe0 0x7f0a7c149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7aa00640 1 -- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a7c0ac2c0 msgr2=0x7f0a7c14ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7aa00640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a7c0ac2c0 0x7f0a7c14ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.317+0000 7f0a7aa00640 1 -- 172.21.15.31:0/3410785373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7c074610 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.318+0000 7f0a7b400640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7c0acbe0 0x7f0a7c149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:40:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.318+0000 7f0a7aa00640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c0a6e20 0x7f0a7c073910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00e9f0 tx=0x7f0a6c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.318+0000 7f0a72a00640 1 -- 172.21.15.31:0/3410785373 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c00cd80 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.318+0000 7f0a72a00640 1 -- 172.21.15.31:0/3410785373 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a6c010430 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.318+0000 7f0a7be00640 1 -- 172.21.15.31:0/3410785373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a7c0748f0 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.318+0000 7f0a72a00640 1 -- 172.21.15.31:0/3410785373 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c010840 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.318+0000 7f0a7be00640 1 -- 172.21.15.31:0/3410785373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a7c152c90 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.320+0000 7f0a7be00640 1 -- 172.21.15.31:0/3410785373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a7c0ac6a0 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.321+0000 7f0a72a00640 1 -- 172.21.15.31:0/3410785373 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0a6c0109e0 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.322+0000 7f0a72a00640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a5c077f70 0x7f0a5c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.322+0000 7f0a72a00640 1 -- 172.21.15.31:0/3410785373 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f0a6c09aff0 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.322+0000 7f0a7a000640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a5c077f70 0x7f0a5c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.323+0000 7f0a7a000640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a5c077f70 0x7f0a5c07a430 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0a6d40 tx=0x7f0a68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:10.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.327+0000 7f0a72a00640 1 -- 172.21.15.31:0/3410785373 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0a6c062de0 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.582+0000 7f0a7be00640 1 -- 172.21.15.31:0/3410785373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a7c0c74f0 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.582+0000 7f0a72a00640 1 -- 172.21.15.31:0/3410785373 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7f0a6c067810 con 0x7f0a7c0a6e20 2024-09-11T00:40:10.585 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:10.585 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:10.585 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:10.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 -- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a5c077f70 msgr2=0x7f0a5c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:10.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a5c077f70 0x7f0a5c07a430 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0a6d40 tx=0x7f0a68009040 comp rx=0 tx=0).stop 2024-09-11T00:40:10.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 -- 172.21.15.31:0/3410785373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c0a6e20 msgr2=0x7f0a7c073910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:10.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c0a6e20 0x7f0a7c073910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00e9f0 tx=0x7f0a6c00eeb0 comp rx=0 tx=0).stop 2024-09-11T00:40:10.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 -- 172.21.15.31:0/3410785373 shutdown_connections 2024-09-11T00:40:10.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a5c077f70 0x7f0a5c07a430 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:10.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7c0acbe0 0x7f0a7c149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:10.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a7c0ac2c0 0x7f0a7c14ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:10.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 --2- 172.21.15.31:0/3410785373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a7c0a6e20 0x7f0a7c073910 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:10.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 -- 172.21.15.31:0/3410785373 >> 172.21.15.31:0/3410785373 conn(0x7f0a7c06caf0 msgr2=0x7f0a7c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:10.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 -- 172.21.15.31:0/3410785373 shutdown_connections 2024-09-11T00:40:10.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:10.585+0000 7f0a7be00640 1 -- 172.21.15.31:0/3410785373 wait complete. 2024-09-11T00:40:10.604 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:40:12.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:12.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.791+0000 7fa850a00640 1 -- 172.21.15.31:0/1715860622 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa84c157ac0 msgr2=0x7fa84c06f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:12.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.791+0000 7fa850a00640 1 --2- 172.21.15.31:0/1715860622 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa84c157ac0 0x7fa84c06f820 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa838009860 tx=0x7fa83802a510 comp rx=0 tx=0).stop 2024-09-11T00:40:12.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.791+0000 7fa850a00640 1 -- 172.21.15.31:0/1715860622 shutdown_connections 2024-09-11T00:40:12.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.791+0000 7fa850a00640 1 --2- 172.21.15.31:0/1715860622 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa84c157ac0 0x7fa84c06f820 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:12.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.791+0000 7fa850a00640 1 -- 172.21.15.31:0/1715860622 >> 172.21.15.31:0/1715860622 conn(0x7fa84c0c8320 msgr2=0x7fa84c0c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:12.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.791+0000 7fa850a00640 1 -- 172.21.15.31:0/1715860622 shutdown_connections 2024-09-11T00:40:12.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.792+0000 7fa850a00640 1 -- 172.21.15.31:0/1715860622 wait complete. 2024-09-11T00:40:12.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.793+0000 7fa850a00640 1 Processor -- start 2024-09-11T00:40:12.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.793+0000 7fa850a00640 1 -- start start 2024-09-11T00:40:12.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.794+0000 7fa850a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa84c06fd60 0x7fa84c156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:12.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.794+0000 7fa850a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa84c1576e0 0x7fa84c155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:12.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.794+0000 7fa850a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa84c157ac0 0x7fa84c1e17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.794+0000 7fa850a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c074da0 con 0x7fa84c157ac0 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.794+0000 7fa850a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c074c20 con 0x7fa84c06fd60 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.794+0000 7fa850a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa84c06dfc0 con 0x7fa84c1576e0 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.795+0000 7fa84a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa84c157ac0 0x7fa84c1e17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.795+0000 7fa84a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa84c157ac0 0x7fa84c1e17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56222/0 (socket says 172.21.15.31:56222) 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.795+0000 7fa849400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa84c1576e0 0x7fa84c155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.795+0000 7fa84a800640 1 -- 172.21.15.31:0/1753251991 learned_addr learned my addr 172.21.15.31:0/1753251991 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.795+0000 7fa849e00640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa84c06fd60 0x7fa84c156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.795+0000 7fa84a800640 1 -- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa84c1576e0 msgr2=0x7fa84c155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.795+0000 7fa84a800640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa84c1576e0 0x7fa84c155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.795+0000 7fa84a800640 1 -- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa84c06fd60 msgr2=0x7fa84c156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:12.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.795+0000 7fa84a800640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa84c06fd60 0x7fa84c156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:12.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.795+0000 7fa84a800640 1 -- 172.21.15.31:0/1753251991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84c150ba0 con 0x7fa84c157ac0 2024-09-11T00:40:12.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.796+0000 7fa84a800640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa84c157ac0 0x7fa84c1e17c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa84400cbf0 tx=0x7fa84400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:12.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.796+0000 7fa842000640 1 -- 172.21.15.31:0/1753251991 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844016020 con 0x7fa84c157ac0 2024-09-11T00:40:12.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.796+0000 7fa850a00640 1 -- 172.21.15.31:0/1753251991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa84c0c7fc0 con 0x7fa84c157ac0 2024-09-11T00:40:12.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.796+0000 7fa842000640 1 -- 172.21.15.31:0/1753251991 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa84400ad50 con 0x7fa84c157ac0 2024-09-11T00:40:12.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.796+0000 7fa842000640 1 -- 172.21.15.31:0/1753251991 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844012680 con 0x7fa84c157ac0 2024-09-11T00:40:12.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.796+0000 7fa850a00640 1 -- 172.21.15.31:0/1753251991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa84c1ecee0 con 0x7fa84c157ac0 2024-09-11T00:40:12.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.797+0000 7fa850a00640 1 -- 172.21.15.31:0/1753251991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa84c068e10 con 0x7fa84c157ac0 2024-09-11T00:40:12.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.799+0000 7fa842000640 1 -- 172.21.15.31:0/1753251991 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa844012820 con 0x7fa84c157ac0 2024-09-11T00:40:12.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.800+0000 7fa842000640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa82c077f70 0x7fa82c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:12.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.800+0000 7fa842000640 1 -- 172.21.15.31:0/1753251991 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7fa84409ab50 con 0x7fa84c157ac0 2024-09-11T00:40:12.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.800+0000 7fa849e00640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa82c077f70 0x7fa82c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:12.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.801+0000 7fa849e00640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa82c077f70 0x7fa82c07a430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa83c001810 tx=0x7fa83c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:12.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:12.806+0000 7fa842000640 1 -- 172.21.15.31:0/1753251991 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa844020030 con 0x7fa84c157ac0 2024-09-11T00:40:13.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.064+0000 7fa850a00640 1 -- 172.21.15.31:0/1753251991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa84c14b520 con 0x7fa84c157ac0 2024-09-11T00:40:13.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.064+0000 7fa842000640 1 -- 172.21.15.31:0/1753251991 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7fa844062800 con 0x7fa84c157ac0 2024-09-11T00:40:13.067 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:13.067 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:13.067 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:13.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 -- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa82c077f70 msgr2=0x7fa82c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:13.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa82c077f70 0x7fa82c07a430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa83c001810 tx=0x7fa83c009040 comp rx=0 tx=0).stop 2024-09-11T00:40:13.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 -- 172.21.15.31:0/1753251991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa84c157ac0 msgr2=0x7fa84c1e17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:13.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa84c157ac0 0x7fa84c1e17c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa84400cbf0 tx=0x7fa84400a430 comp rx=0 tx=0).stop 2024-09-11T00:40:13.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 -- 172.21.15.31:0/1753251991 shutdown_connections 2024-09-11T00:40:13.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa82c077f70 0x7fa82c07a430 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:13.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa84c157ac0 0x7fa84c1e17c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:13.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa84c1576e0 0x7fa84c155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:13.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 --2- 172.21.15.31:0/1753251991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa84c06fd60 0x7fa84c156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:13.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 -- 172.21.15.31:0/1753251991 >> 172.21.15.31:0/1753251991 conn(0x7fa84c0c8320 msgr2=0x7fa84c1511a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:13.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 -- 172.21.15.31:0/1753251991 shutdown_connections 2024-09-11T00:40:13.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:13.067+0000 7fa850a00640 1 -- 172.21.15.31:0/1753251991 wait complete. 2024-09-11T00:40:13.085 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:40:15.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:15.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:40:15.236 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:40:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.261+0000 7f287aa00640 1 -- 172.21.15.31:0/3120023541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28740c6340 msgr2=0x7f2874159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.261+0000 7f287aa00640 1 --2- 172.21.15.31:0/3120023541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28740c6340 0x7f2874159670 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f2874070470 tx=0x7f286c02aac0 comp rx=0 tx=0).stop 2024-09-11T00:40:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.261+0000 7f287aa00640 1 -- 172.21.15.31:0/3120023541 shutdown_connections 2024-09-11T00:40:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.261+0000 7f287aa00640 1 --2- 172.21.15.31:0/3120023541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28740c6340 0x7f2874159670 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.261+0000 7f287aa00640 1 --2- 172.21.15.31:0/3120023541 >> v2:172.21.15.204:3300/0 conn(0x7f28740c5b10 0x7f2874156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:15.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.261+0000 7f287aa00640 1 -- 172.21.15.31:0/3120023541 >> 172.21.15.31:0/3120023541 conn(0x7f28740a9750 msgr2=0x7f28740a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:15.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.261+0000 7f287aa00640 1 -- 172.21.15.31:0/3120023541 shutdown_connections 2024-09-11T00:40:15.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.261+0000 7f287aa00640 1 -- 172.21.15.31:0/3120023541 wait complete. 2024-09-11T00:40:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.263+0000 7f287aa00640 1 Processor -- start 2024-09-11T00:40:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.264+0000 7f287aa00640 1 -- start start 2024-09-11T00:40:15.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f287aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28740c5b10 0x7f28741ef660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:15.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f287aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28740c5ef0 0x7f287414a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:15.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f287aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28740c6340 0x7f287414a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:15.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f287aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2874153b90 con 0x7f28740c6340 2024-09-11T00:40:15.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f287aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28740c3950 con 0x7f28740c5ef0 2024-09-11T00:40:15.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f287aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2874153d10 con 0x7f28740c5b10 2024-09-11T00:40:15.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f2878a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28740c6340 0x7f287414a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:15.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f2873e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28740c5b10 0x7f28741ef660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:15.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f2873400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28740c5ef0 0x7f287414a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f2873e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28740c5b10 0x7f28741ef660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50850/0 (socket says 172.21.15.31:50850) 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.265+0000 7f2873e00640 1 -- 172.21.15.31:0/3525598901 learned_addr learned my addr 172.21.15.31:0/3525598901 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f2878a00640 1 -- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28740c5b10 msgr2=0x7f28741ef660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f2878a00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28740c5b10 0x7f28741ef660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f2878a00640 1 -- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28740c5ef0 msgr2=0x7f287414a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f2878a00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28740c5ef0 0x7f287414a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f2878a00640 1 -- 172.21.15.31:0/3525598901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28741f02f0 con 0x7f28740c6340 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f2878a00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28740c6340 0x7f287414a930 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f286c032000 tx=0x7f286c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f286be00640 1 -- 172.21.15.31:0/3525598901 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c044070 con 0x7f28740c6340 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f287aa00640 1 -- 172.21.15.31:0/3525598901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f287414b1c0 con 0x7f28740c6340 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f286be00640 1 -- 172.21.15.31:0/3525598901 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f286c004240 con 0x7f28740c6340 2024-09-11T00:40:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f286be00640 1 -- 172.21.15.31:0/3525598901 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286c004630 con 0x7f28740c6340 2024-09-11T00:40:15.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.266+0000 7f287aa00640 1 -- 172.21.15.31:0/3525598901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28740a6d10 con 0x7f28740c6340 2024-09-11T00:40:15.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.268+0000 7f287aa00640 1 -- 172.21.15.31:0/3525598901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2874156ad0 con 0x7f28740c6340 2024-09-11T00:40:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.269+0000 7f286be00640 1 -- 172.21.15.31:0/3525598901 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f286c03e030 con 0x7f28740c6340 2024-09-11T00:40:15.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.271+0000 7f286be00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f28540781a0 0x7f285407a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:15.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.271+0000 7f286be00640 1 -- 172.21.15.31:0/3525598901 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f286c0043f0 con 0x7f28740c6340 2024-09-11T00:40:15.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.271+0000 7f2873e00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f28540781a0 0x7f285407a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:15.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.272+0000 7f2873e00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f28540781a0 0x7f285407a660 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2864009c90 tx=0x7f2864005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:15.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.275+0000 7f286be00640 1 -- 172.21.15.31:0/3525598901 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f286c03c070 con 0x7f28740c6340 2024-09-11T00:40:15.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.612+0000 7f287aa00640 1 -- 172.21.15.31:0/3525598901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2874152ea0 con 0x7f28740c6340 2024-09-11T00:40:15.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.612+0000 7f286be00640 1 -- 172.21.15.31:0/3525598901 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7f286c032030 con 0x7f28740c6340 2024-09-11T00:40:15.615 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:15.615 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:15.615 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 -- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f28540781a0 msgr2=0x7f285407a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:15.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f28540781a0 0x7f285407a660 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2864009c90 tx=0x7f2864005f90 comp rx=0 tx=0).stop 2024-09-11T00:40:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 -- 172.21.15.31:0/3525598901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28740c6340 msgr2=0x7f287414a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28740c6340 0x7f287414a930 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f286c032000 tx=0x7f286c003c10 comp rx=0 tx=0).stop 2024-09-11T00:40:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 -- 172.21.15.31:0/3525598901 shutdown_connections 2024-09-11T00:40:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f28540781a0 0x7f285407a660 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28740c6340 0x7f287414a930 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28740c5ef0 0x7f287414a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 --2- 172.21.15.31:0/3525598901 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28740c5b10 0x7f28741ef660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 -- 172.21.15.31:0/3525598901 >> 172.21.15.31:0/3525598901 conn(0x7f28740a9750 msgr2=0x7f2874150300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 -- 172.21.15.31:0/3525598901 shutdown_connections 2024-09-11T00:40:15.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:15.615+0000 7f287aa00640 1 -- 172.21.15.31:0/3525598901 wait complete. 2024-09-11T00:40:15.638 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:40:17.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:17.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.831+0000 7f22bf400640 1 -- 172.21.15.31:0/3092212582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8156300 msgr2=0x7f22b80c5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:17.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.831+0000 7f22bf400640 1 --2- 172.21.15.31:0/3092212582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8156300 0x7f22b80c5b40 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009900 tx=0x7f22ac02a580 comp rx=0 tx=0).stop 2024-09-11T00:40:17.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.831+0000 7f22bf400640 1 -- 172.21.15.31:0/3092212582 shutdown_connections 2024-09-11T00:40:17.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.831+0000 7f22bf400640 1 --2- 172.21.15.31:0/3092212582 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22b806fa70 0x7f22b81566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:17.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.831+0000 7f22bf400640 1 --2- 172.21.15.31:0/3092212582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8156300 0x7f22b80c5b40 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:17.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.831+0000 7f22bf400640 1 -- 172.21.15.31:0/3092212582 >> 172.21.15.31:0/3092212582 conn(0x7f22b80a9750 msgr2=0x7f22b80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:17.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.831+0000 7f22bf400640 1 -- 172.21.15.31:0/3092212582 shutdown_connections 2024-09-11T00:40:17.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.831+0000 7f22bf400640 1 -- 172.21.15.31:0/3092212582 wait complete. 2024-09-11T00:40:17.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.833+0000 7f22bf400640 1 Processor -- start 2024-09-11T00:40:17.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.834+0000 7f22bf400640 1 -- start start 2024-09-11T00:40:17.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22bf400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22b806fa70 0x7f22b8074930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:17.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22bf400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22b8156300 0x7f22b814aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:17.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22bf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8156c20 0x7f22b814b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:17.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22bf400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f22b80c77a0 con 0x7f22b8156c20 2024-09-11T00:40:17.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22bf400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f22b80c7620 con 0x7f22b8156300 2024-09-11T00:40:17.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22bf400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f22b814d040 con 0x7f22b806fa70 2024-09-11T00:40:17.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22bd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8156c20 0x7f22b814b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:17.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22bd400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8156c20 0x7f22b814b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56272/0 (socket says 172.21.15.31:56272) 2024-09-11T00:40:17.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22bd400640 1 -- 172.21.15.31:0/109155826 learned_addr learned my addr 172.21.15.31:0/109155826 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:17.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22bca00640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22b806fa70 0x7f22b8074930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:17.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.835+0000 7f22b7e00640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22b8156300 0x7f22b814aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:17.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22bd400640 1 -- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22b806fa70 msgr2=0x7f22b8074930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:17.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22bd400640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22b806fa70 0x7f22b8074930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:17.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22bd400640 1 -- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22b8156300 msgr2=0x7f22b814aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:17.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22bd400640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22b8156300 0x7f22b814aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:17.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22bd400640 1 -- 172.21.15.31:0/109155826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b80c1770 con 0x7f22b8156c20 2024-09-11T00:40:17.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22bd400640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8156c20 0x7f22b814b3f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f22b000c730 tx=0x7f22b000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22b4c00640 1 -- 172.21.15.31:0/109155826 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0003b70 con 0x7f22b8156c20 2024-09-11T00:40:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22b4c00640 1 -- 172.21.15.31:0/109155826 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22b0003d10 con 0x7f22b8156c20 2024-09-11T00:40:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22bf400640 1 -- 172.21.15.31:0/109155826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b81ecf10 con 0x7f22b8156c20 2024-09-11T00:40:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22b4c00640 1 -- 172.21.15.31:0/109155826 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b00052d0 con 0x7f22b8156c20 2024-09-11T00:40:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.836+0000 7f22bf400640 1 -- 172.21.15.31:0/109155826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b80c1250 con 0x7f22b8156c20 2024-09-11T00:40:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.838+0000 7f22bf400640 1 -- 172.21.15.31:0/109155826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b81ed150 con 0x7f22b8156c20 2024-09-11T00:40:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.839+0000 7f22b4c00640 1 -- 172.21.15.31:0/109155826 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f22b0005470 con 0x7f22b8156c20 2024-09-11T00:40:17.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.840+0000 7f22b4c00640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2298077f70 0x7f229807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:17.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.840+0000 7f22b4c00640 1 -- 172.21.15.31:0/109155826 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f22b009a130 con 0x7f22b8156c20 2024-09-11T00:40:17.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.840+0000 7f22bca00640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2298077f70 0x7f229807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:17.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.841+0000 7f22bca00640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2298077f70 0x7f229807a430 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f22ac033000 tx=0x7f22ac004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:17.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:17.844+0000 7f22b4c00640 1 -- 172.21.15.31:0/109155826 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f22b81ed150 con 0x7f22b8156c20 2024-09-11T00:40:18.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.126+0000 7f22bf400640 1 -- 172.21.15.31:0/109155826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22b80c2be0 con 0x7f22b8156c20 2024-09-11T00:40:18.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.126+0000 7f22b4c00640 1 -- 172.21.15.31:0/109155826 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7f22b0061dd0 con 0x7f22b8156c20 2024-09-11T00:40:18.128 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:18.128 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:18.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 -- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2298077f70 msgr2=0x7f229807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:18.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2298077f70 0x7f229807a430 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f22ac033000 tx=0x7f22ac004730 comp rx=0 tx=0).stop 2024-09-11T00:40:18.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 -- 172.21.15.31:0/109155826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8156c20 msgr2=0x7f22b814b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8156c20 0x7f22b814b3f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f22b000c730 tx=0x7f22b000cb30 comp rx=0 tx=0).stop 2024-09-11T00:40:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 -- 172.21.15.31:0/109155826 shutdown_connections 2024-09-11T00:40:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2298077f70 0x7f229807a430 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8156c20 0x7f22b814b3f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22b8156300 0x7f22b814aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 --2- 172.21.15.31:0/109155826 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22b806fa70 0x7f22b8074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 -- 172.21.15.31:0/109155826 >> 172.21.15.31:0/109155826 conn(0x7f22b80a9750 msgr2=0x7f22b80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 -- 172.21.15.31:0/109155826 shutdown_connections 2024-09-11T00:40:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:18.129+0000 7f22bf400640 1 -- 172.21.15.31:0/109155826 wait complete. 2024-09-11T00:40:18.147 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:40:19.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:20.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.045+0000 7eff87000640 1 -- 172.21.15.31:0/290375337 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff800a6e20 msgr2=0x7eff801510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.045+0000 7eff87000640 1 --2- 172.21.15.31:0/290375337 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff800a6e20 0x7eff801510c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009830 tx=0x7eff6c02a3f0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.045+0000 7eff87000640 1 -- 172.21.15.31:0/290375337 shutdown_connections 2024-09-11T00:40:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.045+0000 7eff87000640 1 --2- 172.21.15.31:0/290375337 >> v2:172.21.15.204:3300/0 conn(0x7eff80151600 0x7eff800c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.045+0000 7eff87000640 1 --2- 172.21.15.31:0/290375337 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff800a6e20 0x7eff801510c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.045+0000 7eff87000640 1 -- 172.21.15.31:0/290375337 >> 172.21.15.31:0/290375337 conn(0x7eff8006caf0 msgr2=0x7eff8006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.045+0000 7eff87000640 1 -- 172.21.15.31:0/290375337 shutdown_connections 2024-09-11T00:40:20.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.045+0000 7eff87000640 1 -- 172.21.15.31:0/290375337 wait complete. 2024-09-11T00:40:20.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.047+0000 7eff87000640 1 Processor -- start 2024-09-11T00:40:20.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.047+0000 7eff87000640 1 -- start start 2024-09-11T00:40:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.048+0000 7eff87000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff800a6e20 0x7eff80157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.048+0000 7eff87000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff800ac2c0 0x7eff80157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.048+0000 7eff87000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80151600 0x7eff801ed9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.048+0000 7eff87000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7eff800acf30 con 0x7eff80151600 2024-09-11T00:40:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.048+0000 7eff87000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff800acdb0 con 0x7eff800ac2c0 2024-09-11T00:40:20.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.048+0000 7eff87000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7eff800c66f0 con 0x7eff800a6e20 2024-09-11T00:40:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.048+0000 7eff86600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80151600 0x7eff801ed9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.049+0000 7eff86600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80151600 0x7eff801ed9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44560/0 (socket says 172.21.15.31:44560) 2024-09-11T00:40:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.049+0000 7eff86600640 1 -- 172.21.15.31:0/581814432 learned_addr learned my addr 172.21.15.31:0/581814432 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.049+0000 7eff85200640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff800ac2c0 0x7eff80157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.049+0000 7eff85c00640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff800a6e20 0x7eff80157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.049+0000 7eff86600640 1 -- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff800a6e20 msgr2=0x7eff80157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.049+0000 7eff86600640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff800a6e20 0x7eff80157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.049+0000 7eff86600640 1 -- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff800ac2c0 msgr2=0x7eff80157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.049+0000 7eff86600640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff800ac2c0 0x7eff80157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.049+0000 7eff86600640 1 -- 172.21.15.31:0/581814432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff80156090 con 0x7eff80151600 2024-09-11T00:40:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.049+0000 7eff86600640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80151600 0x7eff801ed9f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7eff7800e740 tx=0x7eff7800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.050+0000 7eff7d600640 1 -- 172.21.15.31:0/581814432 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7800ccc0 con 0x7eff80151600 2024-09-11T00:40:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.050+0000 7eff7d600640 1 -- 172.21.15.31:0/581814432 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff7800ce80 con 0x7eff80151600 2024-09-11T00:40:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.050+0000 7eff7d600640 1 -- 172.21.15.31:0/581814432 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff78005280 con 0x7eff80151600 2024-09-11T00:40:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.050+0000 7eff87000640 1 -- 172.21.15.31:0/581814432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff80158200 con 0x7eff80151600 2024-09-11T00:40:20.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.050+0000 7eff87000640 1 -- 172.21.15.31:0/581814432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff80072330 con 0x7eff80151600 2024-09-11T00:40:20.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.051+0000 7eff87000640 1 -- 172.21.15.31:0/581814432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff8014cf60 con 0x7eff80151600 2024-09-11T00:40:20.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.053+0000 7eff7d600640 1 -- 172.21.15.31:0/581814432 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7eff78005420 con 0x7eff80151600 2024-09-11T00:40:20.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.054+0000 7eff7d600640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff60077f70 0x7eff6007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.054+0000 7eff7d600640 1 -- 172.21.15.31:0/581814432 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7eff7809ad30 con 0x7eff80151600 2024-09-11T00:40:20.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.054+0000 7eff85c00640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff60077f70 0x7eff6007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.055+0000 7eff85c00640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff60077f70 0x7eff6007a430 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7eff74001840 tx=0x7eff740017d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:20.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.060+0000 7eff7d600640 1 -- 172.21.15.31:0/581814432 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7eff8014cf60 con 0x7eff80151600 2024-09-11T00:40:20.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.316+0000 7f5bb2000640 1 -- 172.21.15.31:0/3562302258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bac0c9a10 msgr2=0x7f5bac159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.316+0000 7f5bb2000640 1 --2- 172.21.15.31:0/3562302258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bac0c9a10 0x7f5bac159670 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0766d0 tx=0x7f5ba402aa10 comp rx=0 tx=0).stop 2024-09-11T00:40:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.316+0000 7f5bb2000640 1 -- 172.21.15.31:0/3562302258 shutdown_connections 2024-09-11T00:40:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.316+0000 7f5bb2000640 1 --2- 172.21.15.31:0/3562302258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bac0c9a10 0x7f5bac159670 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.316+0000 7f5bb2000640 1 --2- 172.21.15.31:0/3562302258 >> v2:172.21.15.204:3300/0 conn(0x7f5bac14ccc0 0x7f5bac0ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.316+0000 7f5bb2000640 1 --2- 172.21.15.31:0/3562302258 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bac14c8e0 0x7f5bac156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.316+0000 7f5bb2000640 1 -- 172.21.15.31:0/3562302258 >> 172.21.15.31:0/3562302258 conn(0x7f5bac068ee0 msgr2=0x7f5bac0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.316+0000 7f5bb2000640 1 -- 172.21.15.31:0/3562302258 shutdown_connections 2024-09-11T00:40:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.316+0000 7f5bb2000640 1 -- 172.21.15.31:0/3562302258 wait complete. 2024-09-11T00:40:20.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.319+0000 7f5bb2000640 1 Processor -- start 2024-09-11T00:40:20.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.319+0000 7f5bb2000640 1 -- start start 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.320+0000 7f5bb2000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bac0c9a10 0x7f5bac1e1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.320+0000 7f5bb2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bac14c8e0 0x7f5bac1ebfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.321+0000 7f5bb2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bac14ccc0 0x7f5bac1ec520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.321+0000 7f5bb2000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac0c6490 con 0x7f5bac14ccc0 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.321+0000 7f5bb2000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac0cc620 con 0x7f5bac14c8e0 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.321+0000 7f5bb2000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5bac0c6c90 con 0x7f5bac0c9a10 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.321+0000 7f5bb0c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bac0c9a10 0x7f5bac1e1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.321+0000 7f5bb0c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bac0c9a10 0x7f5bac1e1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:49090/0 (socket says 172.21.15.31:49090) 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.321+0000 7f5bb1600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bac14ccc0 0x7f5bac1ec520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.321+0000 7f5bb0c00640 1 -- 172.21.15.31:0/4211973251 learned_addr learned my addr 172.21.15.31:0/4211973251 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.321+0000 7f5babe00640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bac14c8e0 0x7f5bac1ebfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.322+0000 7f5bb0c00640 1 -- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bac14c8e0 msgr2=0x7f5bac1ebfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.322+0000 7f5bb0c00640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bac14c8e0 0x7f5bac1ebfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.322+0000 7f5bb0c00640 1 -- 172.21.15.31:0/4211973251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bac14ccc0 msgr2=0x7f5bac1ec520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.322+0000 7f5bb0c00640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bac14ccc0 0x7f5bac1ec520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.322+0000 7f5bb0c00640 1 -- 172.21.15.31:0/4211973251 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bac1587d0 con 0x7f5bac0c9a10 2024-09-11T00:40:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.322+0000 7f5bb1600640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bac14ccc0 0x7f5bac1ec520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.322+0000 7f5bb0c00640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bac0c9a10 0x7f5bac1e1b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000d770 tx=0x7f5ba000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.322+0000 7f5ba8c00640 1 -- 172.21.15.31:0/4211973251 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0017070 con 0x7f5bac0c9a10 2024-09-11T00:40:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.323+0000 7f5ba8c00640 1 -- 172.21.15.31:0/4211973251 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ba0009d20 con 0x7f5bac0c9a10 2024-09-11T00:40:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.323+0000 7f5bb2000640 1 -- 172.21.15.31:0/4211973251 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bac1480a0 con 0x7f5bac0c9a10 2024-09-11T00:40:20.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.323+0000 7f5ba8c00640 1 -- 172.21.15.31:0/4211973251 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba00106c0 con 0x7f5bac0c9a10 2024-09-11T00:40:20.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.323+0000 7f5bb2000640 1 -- 172.21.15.31:0/4211973251 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bac1e2060 con 0x7f5bac0c9a10 2024-09-11T00:40:20.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.324+0000 7f5bb2000640 1 -- 172.21.15.31:0/4211973251 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bac0adc40 con 0x7f5bac0c9a10 2024-09-11T00:40:20.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.325+0000 7f5ba8c00640 1 -- 172.21.15.31:0/4211973251 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5ba0010860 con 0x7f5bac0c9a10 2024-09-11T00:40:20.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.327+0000 7f5ba8c00640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b8c077f70 0x7f5b8c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.327+0000 7f5ba8c00640 1 -- 172.21.15.31:0/4211973251 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f5ba009a8b0 con 0x7f5bac0c9a10 2024-09-11T00:40:20.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.327+0000 7f5babe00640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b8c077f70 0x7f5b8c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.328+0000 7f5babe00640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b8c077f70 0x7f5b8c07a430 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5bac14ad60 tx=0x7f5b9800b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:20.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.333+0000 7f5ba8c00640 1 -- 172.21.15.31:0/4211973251 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5bac1587d0 con 0x7f5bac0c9a10 2024-09-11T00:40:20.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.363+0000 7eff87000640 1 -- 172.21.15.31:0/581814432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff801510c0 con 0x7eff80151600 2024-09-11T00:40:20.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.364+0000 7eff7d600640 1 -- 172.21.15.31:0/581814432 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7eff78062a60 con 0x7eff80151600 2024-09-11T00:40:20.367 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:20.367 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:20.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.366+0000 7eff87000640 1 -- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff60077f70 msgr2=0x7eff6007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.366+0000 7eff87000640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff60077f70 0x7eff6007a430 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7eff74001840 tx=0x7eff740017d0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.366+0000 7eff87000640 1 -- 172.21.15.31:0/581814432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80151600 msgr2=0x7eff801ed9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.366+0000 7eff87000640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80151600 0x7eff801ed9f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7eff7800e740 tx=0x7eff7800ec00 comp rx=0 tx=0).stop 2024-09-11T00:40:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.366+0000 7eff87000640 1 -- 172.21.15.31:0/581814432 shutdown_connections 2024-09-11T00:40:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.366+0000 7eff87000640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff60077f70 0x7eff6007a430 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.366+0000 7eff87000640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff80151600 0x7eff801ed9f0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.366+0000 7eff87000640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff800ac2c0 0x7eff80157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.366+0000 7eff87000640 1 --2- 172.21.15.31:0/581814432 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff800a6e20 0x7eff80157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.366+0000 7eff87000640 1 -- 172.21.15.31:0/581814432 >> 172.21.15.31:0/581814432 conn(0x7eff8006caf0 msgr2=0x7eff8014c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.367+0000 7eff87000640 1 -- 172.21.15.31:0/581814432 shutdown_connections 2024-09-11T00:40:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.367+0000 7eff87000640 1 -- 172.21.15.31:0/581814432 wait complete. 2024-09-11T00:40:20.387 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-11T00:40:20.388 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-11T00:40:20.388 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:40:20.406 INFO:tasks.ceph.mds.c:Stopped 2024-09-11T00:40:20.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:20.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.606+0000 7f5bb2000640 1 -- 172.21.15.31:0/4211973251 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bac150400 con 0x7f5bac0c9a10 2024-09-11T00:40:20.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.607+0000 7f5ba8c00640 1 -- 172.21.15.31:0/4211973251 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7f5ba00625e0 con 0x7f5bac0c9a10 2024-09-11T00:40:20.609 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:20.609 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:20.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.609+0000 7f5bb2000640 1 -- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b8c077f70 msgr2=0x7f5b8c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.609+0000 7f5bb2000640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b8c077f70 0x7f5b8c07a430 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5bac14ad60 tx=0x7f5b9800b2e0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.609+0000 7f5bb2000640 1 -- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bac0c9a10 msgr2=0x7f5bac1e1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.609+0000 7f5bb2000640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bac0c9a10 0x7f5bac1e1b20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000d770 tx=0x7f5ba000dc30 comp rx=0 tx=0).stop 2024-09-11T00:40:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.609+0000 7f5bb2000640 1 -- 172.21.15.31:0/4211973251 shutdown_connections 2024-09-11T00:40:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.609+0000 7f5bb2000640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b8c077f70 0x7f5b8c07a430 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.609+0000 7f5bb2000640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bac14ccc0 0x7f5bac1ec520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.609+0000 7f5bb2000640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bac14c8e0 0x7f5bac1ebfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.609+0000 7f5bb2000640 1 --2- 172.21.15.31:0/4211973251 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5bac0c9a10 0x7f5bac1e1b20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.609+0000 7f5bb2000640 1 -- 172.21.15.31:0/4211973251 >> 172.21.15.31:0/4211973251 conn(0x7f5bac068ee0 msgr2=0x7f5bac155ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.610+0000 7f5bb2000640 1 -- 172.21.15.31:0/4211973251 shutdown_connections 2024-09-11T00:40:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.610+0000 7f5bb2000640 1 -- 172.21.15.31:0/4211973251 wait complete. 2024-09-11T00:40:20.628 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:40:20.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.638+0000 7fcf56800640 1 -- 172.21.15.31:0/2386978567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf38004d40 msgr2=0x7fcf380051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.638+0000 7fcf56800640 1 --2- 172.21.15.31:0/2386978567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf38004d40 0x7fcf380051c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009ed0 tx=0x7fcf38005830 comp rx=0 tx=0).stop 2024-09-11T00:40:20.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.638+0000 7fcf55e00640 1 -- 172.21.15.31:0/2386978567 reap_dead start 2024-09-11T00:40:20.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.638+0000 7fcf56800640 1 -- 172.21.15.31:0/2386978567 shutdown_connections 2024-09-11T00:40:20.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.638+0000 7fcf56800640 1 -- 172.21.15.31:0/2386978567 >> 172.21.15.31:0/2386978567 conn(0x7fcf50068f10 msgr2=0x7fcf500cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:20.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.638+0000 7fcf56800640 1 -- 172.21.15.31:0/2386978567 shutdown_connections 2024-09-11T00:40:20.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.638+0000 7fcf56800640 1 -- 172.21.15.31:0/2386978567 wait complete. 2024-09-11T00:40:20.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.640+0000 7fcf56800640 1 Processor -- start 2024-09-11T00:40:20.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.640+0000 7fcf56800640 1 -- start start 2024-09-11T00:40:20.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.641+0000 7fcf56800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf38004d40 0x7fcf501ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.641+0000 7fcf56800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf50150400 0x7fcf500c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.641+0000 7fcf56800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5014e710 0x7fcf501e2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.641+0000 7fcf56800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcf500c6ca0 con 0x7fcf5014e710 2024-09-11T00:40:20.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.641+0000 7fcf56800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcf500c6b20 con 0x7fcf38004d40 2024-09-11T00:40:20.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.641+0000 7fcf56800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcf501573d0 con 0x7fcf50150400 2024-09-11T00:40:20.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.642+0000 7fcf55e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5014e710 0x7fcf501e2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.642+0000 7fcf55400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf38004d40 0x7fcf501ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.642+0000 7fcf55e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5014e710 0x7fcf501e2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44614/0 (socket says 172.21.15.31:44614) 2024-09-11T00:40:20.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.642+0000 7fcf55e00640 1 -- 172.21.15.31:0/3510320768 learned_addr learned my addr 172.21.15.31:0/3510320768 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:20.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.642+0000 7fcf54a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf50150400 0x7fcf500c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.642+0000 7fcf55e00640 1 -- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf50150400 msgr2=0x7fcf500c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.642+0000 7fcf55e00640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf50150400 0x7fcf500c3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.642+0000 7fcf55e00640 1 -- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf38004d40 msgr2=0x7fcf501ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.642+0000 7fcf55e00640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf38004d40 0x7fcf501ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.642+0000 7fcf55e00640 1 -- 172.21.15.31:0/3510320768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf500c2f40 con 0x7fcf5014e710 2024-09-11T00:40:20.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.643+0000 7fcf55e00640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5014e710 0x7fcf501e2d00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fcf48036040 tx=0x7fcf48034c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:20.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.643+0000 7fcf4d600640 1 -- 172.21.15.31:0/3510320768 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf48043070 con 0x7fcf5014e710 2024-09-11T00:40:20.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.643+0000 7fcf4d600640 1 -- 172.21.15.31:0/3510320768 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf48034cf0 con 0x7fcf5014e710 2024-09-11T00:40:20.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.643+0000 7fcf4d600640 1 -- 172.21.15.31:0/3510320768 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf48038040 con 0x7fcf5014e710 2024-09-11T00:40:20.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.643+0000 7fcf56800640 1 -- 172.21.15.31:0/3510320768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf500c3220 con 0x7fcf5014e710 2024-09-11T00:40:20.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.643+0000 7fcf56800640 1 -- 172.21.15.31:0/3510320768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5014eff0 con 0x7fcf5014e710 2024-09-11T00:40:20.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.644+0000 7fcf56800640 1 -- 172.21.15.31:0/3510320768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf500c5ae0 con 0x7fcf5014e710 2024-09-11T00:40:20.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.646+0000 7fcf4d600640 1 -- 172.21.15.31:0/3510320768 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fcf4803b050 con 0x7fcf5014e710 2024-09-11T00:40:20.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.647+0000 7fcf4d600640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf30077f70 0x7fcf3007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:20.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.647+0000 7fcf4d600640 1 -- 172.21.15.31:0/3510320768 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7fcf480ba120 con 0x7fcf5014e710 2024-09-11T00:40:20.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.647+0000 7fcf55400640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf30077f70 0x7fcf3007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:20.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.648+0000 7fcf55400640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf30077f70 0x7fcf3007a430 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcf500ace60 tx=0x7fcf44006720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:20.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.651+0000 7fcf4d600640 1 -- 172.21.15.31:0/3510320768 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcf500c5ae0 con 0x7fcf5014e710 2024-09-11T00:40:20.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.961+0000 7fcf56800640 1 -- 172.21.15.31:0/3510320768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf5006ef00 con 0x7fcf5014e710 2024-09-11T00:40:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.962+0000 7fcf4d600640 1 -- 172.21.15.31:0/3510320768 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7fcf5006ef00 con 0x7fcf5014e710 2024-09-11T00:40:20.964 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:20.964 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 -- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf30077f70 msgr2=0x7fcf3007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf30077f70 0x7fcf3007a430 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fcf500ace60 tx=0x7fcf44006720 comp rx=0 tx=0).stop 2024-09-11T00:40:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 -- 172.21.15.31:0/3510320768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5014e710 msgr2=0x7fcf501e2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5014e710 0x7fcf501e2d00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fcf48036040 tx=0x7fcf48034c00 comp rx=0 tx=0).stop 2024-09-11T00:40:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 -- 172.21.15.31:0/3510320768 shutdown_connections 2024-09-11T00:40:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf30077f70 0x7fcf3007a430 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf5014e710 0x7fcf501e2d00 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf50150400 0x7fcf500c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 --2- 172.21.15.31:0/3510320768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf38004d40 0x7fcf501ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:20.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 -- 172.21.15.31:0/3510320768 >> 172.21.15.31:0/3510320768 conn(0x7fcf50068f10 msgr2=0x7fcf50159100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:20.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 -- 172.21.15.31:0/3510320768 shutdown_connections 2024-09-11T00:40:20.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:20.965+0000 7fcf56800640 1 -- 172.21.15.31:0/3510320768 wait complete. 2024-09-11T00:40:20.986 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:40:22.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:22.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.803+0000 7f1281800640 1 -- 172.21.15.31:0/2082640251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f127c153f20 msgr2=0x7f127c1516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:22.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.803+0000 7f1281800640 1 --2- 172.21.15.31:0/2082640251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f127c153f20 0x7f127c1516e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f127c0c4400 tx=0x7f127402ab50 comp rx=0 tx=0).stop 2024-09-11T00:40:22.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.804+0000 7f1281800640 1 -- 172.21.15.31:0/2082640251 shutdown_connections 2024-09-11T00:40:22.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.804+0000 7f1281800640 1 --2- 172.21.15.31:0/2082640251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f127c153f20 0x7f127c1516e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:22.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.804+0000 7f1281800640 1 --2- 172.21.15.31:0/2082640251 >> v2:172.21.15.204:3300/0 conn(0x7f127c072480 0x7f127c1511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:22.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.804+0000 7f1281800640 1 -- 172.21.15.31:0/2082640251 >> 172.21.15.31:0/2082640251 conn(0x7f127c0a9750 msgr2=0x7f127c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:22.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.804+0000 7f1281800640 1 -- 172.21.15.31:0/2082640251 shutdown_connections 2024-09-11T00:40:22.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.804+0000 7f1281800640 1 -- 172.21.15.31:0/2082640251 wait complete. 2024-09-11T00:40:22.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.805+0000 7f1281800640 1 Processor -- start 2024-09-11T00:40:22.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.805+0000 7f1281800640 1 -- start start 2024-09-11T00:40:22.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.806+0000 7f1281800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f127c072480 0x7f127c1ecb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:22.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.806+0000 7f1281800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f127c0c23a0 0x7f127c06e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:22.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.806+0000 7f1281800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f127c153f20 0x7f127c0c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:22.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.806+0000 7f1281800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f127c074200 con 0x7f127c0c23a0 2024-09-11T00:40:22.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.806+0000 7f1281800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f127c074080 con 0x7f127c153f20 2024-09-11T00:40:22.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.806+0000 7f1281800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f127c072860 con 0x7f127c072480 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f127c0c23a0 0x7f127c06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f127c0c23a0 0x7f127c06e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44632/0 (socket says 172.21.15.31:44632) 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127a000640 1 -- 172.21.15.31:0/3114613621 learned_addr learned my addr 172.21.15.31:0/3114613621 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127aa00640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f127c072480 0x7f127c1ecb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127b400640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f127c153f20 0x7f127c0c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127a000640 1 -- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f127c072480 msgr2=0x7f127c1ecb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127a000640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f127c072480 0x7f127c1ecb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127a000640 1 -- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f127c153f20 msgr2=0x7f127c0c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127a000640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f127c153f20 0x7f127c0c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127a000640 1 -- 172.21.15.31:0/3114613621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127c143df0 con 0x7f127c0c23a0 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.807+0000 7f127a000640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f127c0c23a0 0x7f127c06e200 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f126800c450 tx=0x7f126800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:22.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.808+0000 7f1272a00640 1 -- 172.21.15.31:0/3114613621 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12680052d0 con 0x7f127c0c23a0 2024-09-11T00:40:22.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.808+0000 7f1272a00640 1 -- 172.21.15.31:0/3114613621 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1268005470 con 0x7f127c0c23a0 2024-09-11T00:40:22.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.808+0000 7f1272a00640 1 -- 172.21.15.31:0/3114613621 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1268012870 con 0x7f127c0c23a0 2024-09-11T00:40:22.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.808+0000 7f1281800640 1 -- 172.21.15.31:0/3114613621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f127c0a90f0 con 0x7f127c0c23a0 2024-09-11T00:40:22.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.808+0000 7f1281800640 1 -- 172.21.15.31:0/3114613621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f127c1ec080 con 0x7f127c0c23a0 2024-09-11T00:40:22.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.809+0000 7f1281800640 1 -- 172.21.15.31:0/3114613621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f127c1511a0 con 0x7f127c0c23a0 2024-09-11T00:40:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.810+0000 7f1272a00640 1 -- 172.21.15.31:0/3114613621 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f12680035f0 con 0x7f127c0c23a0 2024-09-11T00:40:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.811+0000 7f1272a00640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f125c077f70 0x7f125c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.811+0000 7f1272a00640 1 -- 172.21.15.31:0/3114613621 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7f1268099f10 con 0x7f127c0c23a0 2024-09-11T00:40:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.811+0000 7f127aa00640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f125c077f70 0x7f125c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:22.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.812+0000 7f127aa00640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f125c077f70 0x7f125c07a430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f127c151150 tx=0x7f126c00b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:22.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:22.815+0000 7f1272a00640 1 -- 172.21.15.31:0/3114613621 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1268061c40 con 0x7f127c0c23a0 2024-09-11T00:40:22.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:23.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.051+0000 7f1281800640 1 -- 172.21.15.31:0/3114613621 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f127c1445b0 con 0x7f127c0c23a0 2024-09-11T00:40:23.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.052+0000 7f1272a00640 1 -- 172.21.15.31:0/3114613621 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7f1268066670 con 0x7f127c0c23a0 2024-09-11T00:40:23.054 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:23.054 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:23.054 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:23.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 -- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f125c077f70 msgr2=0x7f125c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:23.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f125c077f70 0x7f125c07a430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f127c151150 tx=0x7f126c00b390 comp rx=0 tx=0).stop 2024-09-11T00:40:23.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 -- 172.21.15.31:0/3114613621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f127c0c23a0 msgr2=0x7f127c06e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:23.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f127c0c23a0 0x7f127c06e200 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f126800c450 tx=0x7f126800c910 comp rx=0 tx=0).stop 2024-09-11T00:40:23.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 -- 172.21.15.31:0/3114613621 shutdown_connections 2024-09-11T00:40:23.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f125c077f70 0x7f125c07a430 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f127c153f20 0x7f127c0c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f127c0c23a0 0x7f127c06e200 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 --2- 172.21.15.31:0/3114613621 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f127c072480 0x7f127c1ecb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 -- 172.21.15.31:0/3114613621 >> 172.21.15.31:0/3114613621 conn(0x7f127c0a9750 msgr2=0x7f127c14cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:23.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 -- 172.21.15.31:0/3114613621 shutdown_connections 2024-09-11T00:40:23.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.054+0000 7f1281800640 1 -- 172.21.15.31:0/3114613621 wait complete. 2024-09-11T00:40:23.072 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:40:23.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.151+0000 7fd775a00640 1 -- 172.21.15.31:0/3136240481 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd7700c34e0 msgr2=0x7fd7701511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:23.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.151+0000 7fd775a00640 1 --2- 172.21.15.31:0/3136240481 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd7700c34e0 0x7fd7701511c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd768009860 tx=0x7fd76802a4b0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.152+0000 7fd775a00640 1 -- 172.21.15.31:0/3136240481 shutdown_connections 2024-09-11T00:40:23.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.152+0000 7fd775a00640 1 --2- 172.21.15.31:0/3136240481 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd7700c34e0 0x7fd7701511c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.152+0000 7fd775a00640 1 --2- 172.21.15.31:0/3136240481 >> v2:172.21.15.204:3300/0 conn(0x7fd770154b60 0x7fd7700c2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.152+0000 7fd775a00640 1 -- 172.21.15.31:0/3136240481 >> 172.21.15.31:0/3136240481 conn(0x7fd7700a9750 msgr2=0x7fd7700a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:23.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.152+0000 7fd775a00640 1 -- 172.21.15.31:0/3136240481 shutdown_connections 2024-09-11T00:40:23.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.152+0000 7fd775a00640 1 -- 172.21.15.31:0/3136240481 wait complete. 2024-09-11T00:40:23.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.153+0000 7fd775a00640 1 Processor -- start 2024-09-11T00:40:23.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.153+0000 7fd775a00640 1 -- start start 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.154+0000 7fd775a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd770072480 0x7fd7701531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.154+0000 7fd775a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd7700c34e0 0x7fd770148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.154+0000 7fd775a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd770154b60 0x7fd770149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.154+0000 7fd775a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd7700a7140 con 0x7fd770072480 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.154+0000 7fd775a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd7700c23a0 con 0x7fd7700c34e0 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.154+0000 7fd775a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd770072860 con 0x7fd770154b60 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd76ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd770072480 0x7fd7701531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd76ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd770072480 0x7fd7701531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44634/0 (socket says 172.21.15.31:44634) 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd76ea00640 1 -- 172.21.15.31:0/3421970219 learned_addr learned my addr 172.21.15.31:0/3421970219 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd766000640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd7700c34e0 0x7fd770148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:23.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd76f400640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd770154b60 0x7fd770149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:23.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd76ea00640 1 -- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd770154b60 msgr2=0x7fd770149110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:23.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd76ea00640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd770154b60 0x7fd770149110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd76ea00640 1 -- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd7700c34e0 msgr2=0x7fd770148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:23.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd76ea00640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd7700c34e0 0x7fd770148bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd76ea00640 1 -- 172.21.15.31:0/3421970219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77014f620 con 0x7fd770072480 2024-09-11T00:40:23.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd766000640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd7700c34e0 0x7fd770148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:40:23.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.155+0000 7fd76ea00640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd770072480 0x7fd7701531c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd76000cba0 tx=0x7fd76000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:23.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.156+0000 7fd767400640 1 -- 172.21.15.31:0/3421970219 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd760016020 con 0x7fd770072480 2024-09-11T00:40:23.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.156+0000 7fd767400640 1 -- 172.21.15.31:0/3421970219 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd76000ae30 con 0x7fd770072480 2024-09-11T00:40:23.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.156+0000 7fd775a00640 1 -- 172.21.15.31:0/3421970219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd770157d00 con 0x7fd770072480 2024-09-11T00:40:23.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.156+0000 7fd767400640 1 -- 172.21.15.31:0/3421970219 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd760012660 con 0x7fd770072480 2024-09-11T00:40:23.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.156+0000 7fd775a00640 1 -- 172.21.15.31:0/3421970219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7701dd740 con 0x7fd770072480 2024-09-11T00:40:23.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.157+0000 7fd775a00640 1 -- 172.21.15.31:0/3421970219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd77014d050 con 0x7fd770072480 2024-09-11T00:40:23.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.158+0000 7fd767400640 1 -- 172.21.15.31:0/3421970219 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd76001b400 con 0x7fd770072480 2024-09-11T00:40:23.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.159+0000 7fd767400640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd74c077f70 0x7fd74c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:23.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.159+0000 7fd767400640 1 -- 172.21.15.31:0/3421970219 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7612+0+0 (secure 0 0 0) 0x7fd76009afe0 con 0x7fd770072480 2024-09-11T00:40:23.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.160+0000 7fd766000640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd74c077f70 0x7fd74c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:23.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.160+0000 7fd766000640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd74c077f70 0x7fd74c07a430 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd7700a9de0 tx=0x7fd758005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:23.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.164+0000 7fd767400640 1 -- 172.21.15.31:0/3421970219 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd77014d050 con 0x7fd770072480 2024-09-11T00:40:23.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.394+0000 7fd775a00640 1 -- 172.21.15.31:0/3421970219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7700c2fa0 con 0x7fd770072480 2024-09-11T00:40:23.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.394+0000 7fd767400640 1 -- 172.21.15.31:0/3421970219 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6540 (secure 0 0 0) 0x7fd760020020 con 0x7fd770072480 2024-09-11T00:40:23.396 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:23.397 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":67,"btime":"2024-09-11T00:38:28:438472+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:38:17.360585+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5201},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5201":{"gid":5201,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.204:6835/3506150555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3506150555},{"type":"v1","addr":"172.21.15.204:6835","nonce":3506150555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5201,"qdb_cluster":[5201]},"id":2}]} 2024-09-11T00:40:23.397 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 67 2024-09-11T00:40:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.396+0000 7fd775a00640 1 -- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd74c077f70 msgr2=0x7fd74c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd74c077f70 0x7fd74c07a430 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd7700a9de0 tx=0x7fd758005f90 comp rx=0 tx=0).stop 2024-09-11T00:40:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 -- 172.21.15.31:0/3421970219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd770072480 msgr2=0x7fd7701531c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd770072480 0x7fd7701531c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd76000cba0 tx=0x7fd76000a430 comp rx=0 tx=0).stop 2024-09-11T00:40:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 -- 172.21.15.31:0/3421970219 shutdown_connections 2024-09-11T00:40:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd74c077f70 0x7fd74c07a430 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd770154b60 0x7fd770149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd7700c34e0 0x7fd770148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 --2- 172.21.15.31:0/3421970219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd770072480 0x7fd7701531c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 -- 172.21.15.31:0/3421970219 >> 172.21.15.31:0/3421970219 conn(0x7fd7700a9750 msgr2=0x7fd770150f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 -- 172.21.15.31:0/3421970219 shutdown_connections 2024-09-11T00:40:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:23.397+0000 7fd775a00640 1 -- 172.21.15.31:0/3421970219 wait complete. 2024-09-11T00:40:23.416 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:40:25.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:25.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.242+0000 7fe9d1800640 1 -- 172.21.15.31:0/950290374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9cc150d60 msgr2=0x7fe9cc1511d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.242+0000 7fe9d1800640 1 --2- 172.21.15.31:0/950290374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9cc150d60 0x7fe9cc1511d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4005fd0 tx=0x7fe9c402aa10 comp rx=0 tx=0).stop 2024-09-11T00:40:25.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.242+0000 7fe9d1800640 1 -- 172.21.15.31:0/950290374 shutdown_connections 2024-09-11T00:40:25.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.242+0000 7fe9d1800640 1 --2- 172.21.15.31:0/950290374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9cc150d60 0x7fe9cc1511d0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.242+0000 7fe9d1800640 1 --2- 172.21.15.31:0/950290374 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9cc074a30 0x7fe9cc154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.242+0000 7fe9d1800640 1 -- 172.21.15.31:0/950290374 >> 172.21.15.31:0/950290374 conn(0x7fe9cc0c4d20 msgr2=0x7fe9cc0c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:25.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.243+0000 7fe9d1800640 1 -- 172.21.15.31:0/950290374 shutdown_connections 2024-09-11T00:40:25.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.243+0000 7fe9d1800640 1 -- 172.21.15.31:0/950290374 wait complete. 2024-09-11T00:40:25.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.244+0000 7fe9d1800640 1 Processor -- start 2024-09-11T00:40:25.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.244+0000 7fe9d1800640 1 -- start start 2024-09-11T00:40:25.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.245+0000 7fe9d1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9cc06dd70 0x7fe9cc1ed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.245+0000 7fe9d1800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9cc074a30 0x7fe9cc14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.245+0000 7fe9d1800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9cc150d60 0x7fe9cc1e1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.245+0000 7fe9d1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc154410 con 0x7fe9cc06dd70 2024-09-11T00:40:25.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.245+0000 7fe9d1800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc153f10 con 0x7fe9cc150d60 2024-09-11T00:40:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.245+0000 7fe9d1800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe9cc070c10 con 0x7fe9cc074a30 2024-09-11T00:40:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.245+0000 7fe9cbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9cc06dd70 0x7fe9cc1ed4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.245+0000 7fe9cbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9cc06dd70 0x7fe9cc1ed4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44654/0 (socket says 172.21.15.31:44654) 2024-09-11T00:40:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.245+0000 7fe9cbe00640 1 -- 172.21.15.31:0/1913469774 learned_addr learned my addr 172.21.15.31:0/1913469774 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.246+0000 7fe9c3400640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9cc074a30 0x7fe9cc14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.246+0000 7fe9d0e00640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9cc150d60 0x7fe9cc1e1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.246+0000 7fe9cbe00640 1 -- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9cc074a30 msgr2=0x7fe9cc14ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.246+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9cc074a30 0x7fe9cc14ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.246+0000 7fe9cbe00640 1 -- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9cc150d60 msgr2=0x7fe9cc1e1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.246+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9cc150d60 0x7fe9cc1e1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.246+0000 7fe9cbe00640 1 -- 172.21.15.31:0/1913469774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc1e07c0 con 0x7fe9cc06dd70 2024-09-11T00:40:25.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.246+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9cc06dd70 0x7fe9cc1ed4a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c450 tx=0x7fe9bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:25.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.246+0000 7fe9c8c00640 1 -- 172.21.15.31:0/1913469774 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc016020 con 0x7fe9cc06dd70 2024-09-11T00:40:25.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.247+0000 7fe9c8c00640 1 -- 172.21.15.31:0/1913469774 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9bc005400 con 0x7fe9cc06dd70 2024-09-11T00:40:25.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.247+0000 7fe9c8c00640 1 -- 172.21.15.31:0/1913469774 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc007280 con 0x7fe9cc06dd70 2024-09-11T00:40:25.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.247+0000 7fe9d1800640 1 -- 172.21.15.31:0/1913469774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9cc1ecc30 con 0x7fe9cc06dd70 2024-09-11T00:40:25.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.247+0000 7fe9d1800640 1 -- 172.21.15.31:0/1913469774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9cc150520 con 0x7fe9cc06dd70 2024-09-11T00:40:25.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.248+0000 7fe9d1800640 1 -- 172.21.15.31:0/1913469774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9cc06f470 con 0x7fe9cc06dd70 2024-09-11T00:40:25.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.249+0000 7fe9c8c00640 1 -- 172.21.15.31:0/1913469774 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe9bc007420 con 0x7fe9cc06dd70 2024-09-11T00:40:25.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.250+0000 7fe9c8c00640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a8077f70 0x7fe9a807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.250+0000 7fe9c8c00640 1 -- 172.21.15.31:0/1913469774 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7fe9bc099e10 con 0x7fe9cc06dd70 2024-09-11T00:40:25.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.250+0000 7fe9c3400640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a8077f70 0x7fe9a807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.251+0000 7fe9c3400640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a8077f70 0x7fe9a807a430 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc1ef500 tx=0x7fe9b400b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:25.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.254+0000 7fe9c8c00640 1 -- 172.21.15.31:0/1913469774 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe9bc09d030 con 0x7fe9cc06dd70 2024-09-11T00:40:25.417 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:25.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.484+0000 7fe9d1800640 1 -- 172.21.15.31:0/1913469774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9cc154c80 con 0x7fe9cc06dd70 2024-09-11T00:40:25.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.485+0000 7fe9c8c00640 1 -- 172.21.15.31:0/1913469774 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5708 (secure 0 0 0) 0x7fe9bc020020 con 0x7fe9cc06dd70 2024-09-11T00:40:25.487 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:25.487 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":69,"btime":"2024-09-11T00:40:25:369970+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:24.647410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:40:25.488 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 69 2024-09-11T00:40:25.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 -- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a8077f70 msgr2=0x7fe9a807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a8077f70 0x7fe9a807a430 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc1ef500 tx=0x7fe9b400b390 comp rx=0 tx=0).stop 2024-09-11T00:40:25.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 -- 172.21.15.31:0/1913469774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9cc06dd70 msgr2=0x7fe9cc1ed4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9cc06dd70 0x7fe9cc1ed4a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c450 tx=0x7fe9bc00c910 comp rx=0 tx=0).stop 2024-09-11T00:40:25.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 -- 172.21.15.31:0/1913469774 shutdown_connections 2024-09-11T00:40:25.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a8077f70 0x7fe9a807a430 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9cc150d60 0x7fe9cc1e1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9cc074a30 0x7fe9cc14ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 --2- 172.21.15.31:0/1913469774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9cc06dd70 0x7fe9cc1ed4a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 -- 172.21.15.31:0/1913469774 >> 172.21.15.31:0/1913469774 conn(0x7fe9cc0c4d20 msgr2=0x7fe9cc0bd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:25.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 -- 172.21.15.31:0/1913469774 shutdown_connections 2024-09-11T00:40:25.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.488+0000 7fe9d1800640 1 -- 172.21.15.31:0/1913469774 wait complete. 2024-09-11T00:40:25.506 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-11T00:40:25.507 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:40:25.507 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:25.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.579+0000 7f293c000640 1 -- 172.21.15.31:0/1040827331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2934072480 msgr2=0x7f29341511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.579+0000 7f293c000640 1 --2- 172.21.15.31:0/1040827331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2934072480 0x7f29341511a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f29340c5500 tx=0x7f292802a6c0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.579+0000 7f293c000640 1 -- 172.21.15.31:0/1040827331 shutdown_connections 2024-09-11T00:40:25.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.579+0000 7f293c000640 1 --2- 172.21.15.31:0/1040827331 >> v2:172.21.15.204:3300/0 conn(0x7f2934153f20 0x7f29341516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.579+0000 7f293c000640 1 --2- 172.21.15.31:0/1040827331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2934072480 0x7f29341511a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.579+0000 7f293c000640 1 -- 172.21.15.31:0/1040827331 >> 172.21.15.31:0/1040827331 conn(0x7f29340a9750 msgr2=0x7f29340a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:25.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.579+0000 7f293c000640 1 -- 172.21.15.31:0/1040827331 shutdown_connections 2024-09-11T00:40:25.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.579+0000 7f293c000640 1 -- 172.21.15.31:0/1040827331 wait complete. 2024-09-11T00:40:25.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.581+0000 7f293c000640 1 Processor -- start 2024-09-11T00:40:25.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.581+0000 7f293c000640 1 -- start start 2024-09-11T00:40:25.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293c000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2934072480 0x7f29340c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293c000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29340c23a0 0x7f29341ec510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293c000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2934153f20 0x7f29341eca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293c000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2934074200 con 0x7f29340c23a0 2024-09-11T00:40:25.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293c000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2934074080 con 0x7f2934153f20 2024-09-11T00:40:25.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293c000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2934072860 con 0x7f2934072480 2024-09-11T00:40:25.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f2938c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29340c23a0 0x7f29341ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2934153f20 0x7f29341eca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2934153f20 0x7f29341eca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:57672/0 (socket says 172.21.15.31:57672) 2024-09-11T00:40:25.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f2939600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2934072480 0x7f29340c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293a000640 1 -- 172.21.15.31:0/2558316752 learned_addr learned my addr 172.21.15.31:0/2558316752 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:25.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f2938c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29340c23a0 0x7f29341ec510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44670/0 (socket says 172.21.15.31:44670) 2024-09-11T00:40:25.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293a000640 1 -- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2934072480 msgr2=0x7f29340c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293a000640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2934072480 0x7f29340c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293a000640 1 -- 172.21.15.31:0/2558316752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29340c23a0 msgr2=0x7f29341ec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293a000640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29340c23a0 0x7f29341ec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f293a000640 1 -- 172.21.15.31:0/2558316752 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29340abbc0 con 0x7f2934153f20 2024-09-11T00:40:25.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.582+0000 7f2939600640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2934072480 0x7f29340c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:25.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.583+0000 7f293a000640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2934153f20 0x7f29341eca50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f292c00e960 tx=0x7f292c00ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:25.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.583+0000 7f2931600640 1 -- 172.21.15.31:0/2558316752 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c004d20 con 0x7f2934153f20 2024-09-11T00:40:25.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.583+0000 7f2931600640 1 -- 172.21.15.31:0/2558316752 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f292c00cb20 con 0x7f2934153f20 2024-09-11T00:40:25.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.583+0000 7f2931600640 1 -- 172.21.15.31:0/2558316752 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292c005020 con 0x7f2934153f20 2024-09-11T00:40:25.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.583+0000 7f293c000640 1 -- 172.21.15.31:0/2558316752 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29341ecf90 con 0x7f2934153f20 2024-09-11T00:40:25.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.583+0000 7f293c000640 1 -- 172.21.15.31:0/2558316752 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29341f0310 con 0x7f2934153f20 2024-09-11T00:40:25.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.584+0000 7f293c000640 1 -- 172.21.15.31:0/2558316752 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29341511a0 con 0x7f2934153f20 2024-09-11T00:40:25.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.585+0000 7f2931600640 1 -- 172.21.15.31:0/2558316752 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f292c00f040 con 0x7f2934153f20 2024-09-11T00:40:25.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.586+0000 7f2931600640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2914077f70 0x7f291407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.586+0000 7f2931600640 1 -- 172.21.15.31:0/2558316752 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7f292c09aae0 con 0x7f2934153f20 2024-09-11T00:40:25.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.587+0000 7f2939600640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2914077f70 0x7f291407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.588+0000 7f2939600640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2914077f70 0x7f291407a430 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2928033000 tx=0x7f2928004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:25.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.591+0000 7f2931600640 1 -- 172.21.15.31:0/2558316752 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f29341ecf90 con 0x7f2934153f20 2024-09-11T00:40:25.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.667+0000 7fbb7b200640 1 -- 172.21.15.31:0/3596052713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb741518b0 msgr2=0x7fbb740c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.667+0000 7fbb7b200640 1 --2- 172.21.15.31:0/3596052713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb741518b0 0x7fbb740c3600 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009900 tx=0x7fbb6c02a690 comp rx=0 tx=0).stop 2024-09-11T00:40:25.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.667+0000 7fbb7b200640 1 -- 172.21.15.31:0/3596052713 shutdown_connections 2024-09-11T00:40:25.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.667+0000 7fbb7b200640 1 --2- 172.21.15.31:0/3596052713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb741518b0 0x7fbb740c3600 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.667+0000 7fbb7b200640 1 --2- 172.21.15.31:0/3596052713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb740c23a0 0x7fbb740c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.667+0000 7fbb7b200640 1 -- 172.21.15.31:0/3596052713 >> 172.21.15.31:0/3596052713 conn(0x7fbb740a9750 msgr2=0x7fbb740a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:25.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.667+0000 7fbb7b200640 1 -- 172.21.15.31:0/3596052713 shutdown_connections 2024-09-11T00:40:25.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.667+0000 7fbb7b200640 1 -- 172.21.15.31:0/3596052713 wait complete. 2024-09-11T00:40:25.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.669+0000 7fbb7b200640 1 Processor -- start 2024-09-11T00:40:25.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.669+0000 7fbb7b200640 1 -- start start 2024-09-11T00:40:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb7b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb74072480 0x7fbb741500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb7b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb740c23a0 0x7fbb741efda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb7b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb741518b0 0x7fbb741531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb7b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbb74074290 con 0x7fbb740c23a0 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb7b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbb74072860 con 0x7fbb74072480 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb7b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbb740729e0 con 0x7fbb741518b0 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb73400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb740c23a0 0x7fbb741efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb73400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb740c23a0 0x7fbb741efda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44680/0 (socket says 172.21.15.31:44680) 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb73400640 1 -- 172.21.15.31:0/3507069754 learned_addr learned my addr 172.21.15.31:0/3507069754 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb79200640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb741518b0 0x7fbb741531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.670+0000 7fbb73e00640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb74072480 0x7fbb741500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.671+0000 7fbb73400640 1 -- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb741518b0 msgr2=0x7fbb741531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.671+0000 7fbb73400640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb741518b0 0x7fbb741531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.671+0000 7fbb73400640 1 -- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb74072480 msgr2=0x7fbb741500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.671+0000 7fbb73400640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb74072480 0x7fbb741500f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.671+0000 7fbb73400640 1 -- 172.21.15.31:0/3507069754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7406e200 con 0x7fbb740c23a0 2024-09-11T00:40:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.671+0000 7fbb73400640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb740c23a0 0x7fbb741efda0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000d770 tx=0x7fbb6000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.671+0000 7fbb6be00640 1 -- 172.21.15.31:0/3507069754 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60017070 con 0x7fbb740c23a0 2024-09-11T00:40:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.671+0000 7fbb6be00640 1 -- 172.21.15.31:0/3507069754 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb60009d20 con 0x7fbb740c23a0 2024-09-11T00:40:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.671+0000 7fbb6be00640 1 -- 172.21.15.31:0/3507069754 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60004280 con 0x7fbb740c23a0 2024-09-11T00:40:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.671+0000 7fbb7b200640 1 -- 172.21.15.31:0/3507069754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb741dff80 con 0x7fbb740c23a0 2024-09-11T00:40:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.672+0000 7fbb7b200640 1 -- 172.21.15.31:0/3507069754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb741f02e0 con 0x7fbb740c23a0 2024-09-11T00:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.673+0000 7fbb7b200640 1 -- 172.21.15.31:0/3507069754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb74153f20 con 0x7fbb740c23a0 2024-09-11T00:40:25.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.675+0000 7fbb6be00640 1 -- 172.21.15.31:0/3507069754 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbb600035f0 con 0x7fbb740c23a0 2024-09-11T00:40:25.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.676+0000 7fbb6be00640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb54077fa0 0x7fbb5407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:25.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.676+0000 7fbb6be00640 1 -- 172.21.15.31:0/3507069754 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7fbb6009a570 con 0x7fbb740c23a0 2024-09-11T00:40:25.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.676+0000 7fbb73e00640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb54077fa0 0x7fbb5407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:25.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.677+0000 7fbb73e00640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb54077fa0 0x7fbb5407a460 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbb640012e0 tx=0x7fbb64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:25.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.680+0000 7fbb6be00640 1 -- 172.21.15.31:0/3507069754 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbb60062240 con 0x7fbb740c23a0 2024-09-11T00:40:25.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.826+0000 7f293c000640 1 -- 172.21.15.31:0/2558316752 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f293414bf30 con 0x7f2934153f20 2024-09-11T00:40:25.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.827+0000 7f2931600640 1 -- 172.21.15.31:0/2558316752 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5708 (secure 0 0 0) 0x7f292c016030 con 0x7f2934153f20 2024-09-11T00:40:25.829 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:25.829 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":69,"btime":"2024-09-11T00:40:25:369970+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:24.647410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:40:25.829 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 69 2024-09-11T00:40:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.829+0000 7f293c000640 1 -- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2914077f70 msgr2=0x7f291407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2914077f70 0x7f291407a430 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2928033000 tx=0x7f2928004730 comp rx=0 tx=0).stop 2024-09-11T00:40:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 -- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2934153f20 msgr2=0x7f29341eca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2934153f20 0x7f29341eca50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f292c00e960 tx=0x7f292c00ee20 comp rx=0 tx=0).stop 2024-09-11T00:40:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 -- 172.21.15.31:0/2558316752 shutdown_connections 2024-09-11T00:40:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2914077f70 0x7f291407a430 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2934153f20 0x7f29341eca50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29340c23a0 0x7f29341ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 --2- 172.21.15.31:0/2558316752 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2934072480 0x7f29340c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 -- 172.21.15.31:0/2558316752 >> 172.21.15.31:0/2558316752 conn(0x7f29340a9750 msgr2=0x7f29340a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 -- 172.21.15.31:0/2558316752 shutdown_connections 2024-09-11T00:40:25.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.830+0000 7f293c000640 1 -- 172.21.15.31:0/2558316752 wait complete. 2024-09-11T00:40:25.848 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:40:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.911+0000 7fbb7b200640 1 -- 172.21.15.31:0/3507069754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb741e0150 con 0x7fbb740c23a0 2024-09-11T00:40:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.912+0000 7fbb6be00640 1 -- 172.21.15.31:0/3507069754 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5708 (secure 0 0 0) 0x7fbb60066c70 con 0x7fbb740c23a0 2024-09-11T00:40:25.914 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:25.914 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":69,"btime":"2024-09-11T00:40:25:369970+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:24.647410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:40:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 69 2024-09-11T00:40:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 -- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb54077fa0 msgr2=0x7fbb5407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb54077fa0 0x7fbb5407a460 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbb640012e0 tx=0x7fbb64009040 comp rx=0 tx=0).stop 2024-09-11T00:40:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 -- 172.21.15.31:0/3507069754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb740c23a0 msgr2=0x7fbb741efda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb740c23a0 0x7fbb741efda0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000d770 tx=0x7fbb6000dc30 comp rx=0 tx=0).stop 2024-09-11T00:40:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 -- 172.21.15.31:0/3507069754 shutdown_connections 2024-09-11T00:40:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb54077fa0 0x7fbb5407a460 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb741518b0 0x7fbb741531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb740c23a0 0x7fbb741efda0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 --2- 172.21.15.31:0/3507069754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb74072480 0x7fbb741500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 -- 172.21.15.31:0/3507069754 >> 172.21.15.31:0/3507069754 conn(0x7fbb740a9750 msgr2=0x7fbb740c30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 -- 172.21.15.31:0/3507069754 shutdown_connections 2024-09-11T00:40:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:25.914+0000 7fbb7b200640 1 -- 172.21.15.31:0/3507069754 wait complete. 2024-09-11T00:40:25.932 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:40:25:369970+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 69, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 65, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4307": { "addr": "172.21.15.31:6834/3881548149", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3881548149, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 3881548149, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4307, "incarnation": 62, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 117 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:38:17.360584+0000", "qdb_cluster": [ 4307 ], "qdb_leader": 4307, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4307 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 69, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5901": { "addr": "172.21.15.31:6835/3925687919", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6833", "nonce": 3925687919, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3925687919, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5901, "incarnation": 68, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:40:24.647410+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5901 } } } ], "standbys": [ { "addr": "172.21.15.204:6837/3513641573", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 3513641573, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 3513641573, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5773, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:40:27.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:27.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:28.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.023+0000 7f2568e00640 1 -- 172.21.15.31:0/2185422804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25640c5b10 msgr2=0x7f2564156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:28.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.023+0000 7f2568e00640 1 --2- 172.21.15.31:0/2185422804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25640c5b10 0x7f2564156ad0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2554005fd0 tx=0x7f255402a5c0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.024+0000 7f2568e00640 1 -- 172.21.15.31:0/2185422804 shutdown_connections 2024-09-11T00:40:28.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.024+0000 7f2568e00640 1 --2- 172.21.15.31:0/2185422804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25640c6340 0x7f2564159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.024+0000 7f2568e00640 1 --2- 172.21.15.31:0/2185422804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25640c5b10 0x7f2564156ad0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.024+0000 7f2568e00640 1 -- 172.21.15.31:0/2185422804 >> 172.21.15.31:0/2185422804 conn(0x7f25640a9750 msgr2=0x7f25640a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:28.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.024+0000 7f2568e00640 1 -- 172.21.15.31:0/2185422804 shutdown_connections 2024-09-11T00:40:28.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.024+0000 7f2568e00640 1 -- 172.21.15.31:0/2185422804 wait complete. 2024-09-11T00:40:28.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.025+0000 7f2568e00640 1 Processor -- start 2024-09-11T00:40:28.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.025+0000 7f2568e00640 1 -- start start 2024-09-11T00:40:28.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.026+0000 7f2568e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25640c5b10 0x7f256414b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:28.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.026+0000 7f2568e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25640c5ef0 0x7f256414beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:28.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.026+0000 7f2568e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25640c6340 0x7f25640ac6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:28.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.026+0000 7f2568e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2564153b90 con 0x7f25640c6340 2024-09-11T00:40:28.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.026+0000 7f2568e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f25640c3950 con 0x7f25640c5ef0 2024-09-11T00:40:28.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.026+0000 7f2568e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2564153d10 con 0x7f25640c5b10 2024-09-11T00:40:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2562800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25640c6340 0x7f25640ac6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2561e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25640c5b10 0x7f256414b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2559400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25640c5ef0 0x7f256414beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2562800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25640c6340 0x7f25640ac6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55658/0 (socket says 172.21.15.31:55658) 2024-09-11T00:40:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2561e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25640c5b10 0x7f256414b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:42112/0 (socket says 172.21.15.31:42112) 2024-09-11T00:40:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2562800640 1 -- 172.21.15.31:0/3087184499 learned_addr learned my addr 172.21.15.31:0/3087184499 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2562800640 1 -- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25640c5b10 msgr2=0x7f256414b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2562800640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25640c5b10 0x7f256414b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2562800640 1 -- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25640c5ef0 msgr2=0x7f256414beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:28.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2562800640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25640c5ef0 0x7f256414beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2562800640 1 -- 172.21.15.31:0/3087184499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25640abfa0 con 0x7f25640c6340 2024-09-11T00:40:28.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f2562800640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25640c6340 0x7f25640ac6c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f255c00ea40 tx=0x7f255c00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:28.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.027+0000 7f255aa00640 1 -- 172.21.15.31:0/3087184499 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c00ce40 con 0x7f25640c6340 2024-09-11T00:40:28.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.028+0000 7f255aa00640 1 -- 172.21.15.31:0/3087184499 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f255c004c10 con 0x7f25640c6340 2024-09-11T00:40:28.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.028+0000 7f255aa00640 1 -- 172.21.15.31:0/3087184499 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c005020 con 0x7f25640c6340 2024-09-11T00:40:28.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.028+0000 7f2568e00640 1 -- 172.21.15.31:0/3087184499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25641ecd60 con 0x7f25640c6340 2024-09-11T00:40:28.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.028+0000 7f2568e00640 1 -- 172.21.15.31:0/3087184499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25640aab70 con 0x7f25640c6340 2024-09-11T00:40:28.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.029+0000 7f2568e00640 1 -- 172.21.15.31:0/3087184499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2564156ad0 con 0x7f25640c6340 2024-09-11T00:40:28.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.030+0000 7f255aa00640 1 -- 172.21.15.31:0/3087184499 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f255c020030 con 0x7f25640c6340 2024-09-11T00:40:28.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.031+0000 7f255aa00640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2540077f70 0x7f254007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:28.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.031+0000 7f255aa00640 1 -- 172.21.15.31:0/3087184499 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7f255c09af70 con 0x7f25640c6340 2024-09-11T00:40:28.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.031+0000 7f2561e00640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2540077f70 0x7f254007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:28.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.032+0000 7f2561e00640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2540077f70 0x7f254007a430 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2554033000 tx=0x7f2554004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:28.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.035+0000 7f255aa00640 1 -- 172.21.15.31:0/3087184499 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f255c062c40 con 0x7f25640c6340 2024-09-11T00:40:28.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.097+0000 7f6c38a00640 1 -- 172.21.15.31:0/246121427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c34156300 msgr2=0x7f6c340c5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:28.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.097+0000 7f6c38a00640 1 --2- 172.21.15.31:0/246121427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c34156300 0x7f6c340c5b40 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6c24005fd0 tx=0x7f6c2402a5f0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.097+0000 7f6c38a00640 1 -- 172.21.15.31:0/246121427 shutdown_connections 2024-09-11T00:40:28.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.097+0000 7f6c38a00640 1 --2- 172.21.15.31:0/246121427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c34156300 0x7f6c340c5b40 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.097+0000 7f6c38a00640 1 -- 172.21.15.31:0/246121427 >> 172.21.15.31:0/246121427 conn(0x7f6c340a9750 msgr2=0x7f6c340a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.097+0000 7f6c38a00640 1 -- 172.21.15.31:0/246121427 shutdown_connections 2024-09-11T00:40:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.097+0000 7f6c38a00640 1 -- 172.21.15.31:0/246121427 wait complete. 2024-09-11T00:40:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.099+0000 7f6c38a00640 1 Processor -- start 2024-09-11T00:40:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.099+0000 7f6c38a00640 1 -- start start 2024-09-11T00:40:28.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c38a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c3406fa70 0x7f6c3414bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:28.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c38a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c340c23a0 0x7f6c34073d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:28.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c38a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c34156300 0x7f6c341eff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c38a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c34156df0 con 0x7f6c340c23a0 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c38a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6c340c7660 con 0x7f6c3406fa70 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c38a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6c34156f70 con 0x7f6c34156300 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c31400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c340c23a0 0x7f6c34073d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c31400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c340c23a0 0x7f6c34073d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55688/0 (socket says 172.21.15.31:55688) 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c31e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c3406fa70 0x7f6c3414bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c31400640 1 -- 172.21.15.31:0/3465618606 learned_addr learned my addr 172.21.15.31:0/3465618606 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c32800640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c34156300 0x7f6c341eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c31400640 1 -- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c34156300 msgr2=0x7f6c341eff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c31400640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c34156300 0x7f6c341eff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c31400640 1 -- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c3406fa70 msgr2=0x7f6c3414bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c31400640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c3406fa70 0x7f6c3414bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.100+0000 7f6c31400640 1 -- 172.21.15.31:0/3465618606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c341dd9e0 con 0x7f6c340c23a0 2024-09-11T00:40:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.101+0000 7f6c31400640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c340c23a0 0x7f6c34073d30 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000c410 tx=0x7f6c2000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.101+0000 7f6c2a000640 1 -- 172.21.15.31:0/3465618606 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20017070 con 0x7f6c340c23a0 2024-09-11T00:40:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.101+0000 7f6c2a000640 1 -- 172.21.15.31:0/3465618606 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c200035f0 con 0x7f6c340c23a0 2024-09-11T00:40:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.101+0000 7f6c38a00640 1 -- 172.21.15.31:0/3465618606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c340748e0 con 0x7f6c340c23a0 2024-09-11T00:40:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.101+0000 7f6c2a000640 1 -- 172.21.15.31:0/3465618606 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20003a80 con 0x7f6c340c23a0 2024-09-11T00:40:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.101+0000 7f6c38a00640 1 -- 172.21.15.31:0/3465618606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c341dd670 con 0x7f6c340c23a0 2024-09-11T00:40:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.102+0000 7f6c38a00640 1 -- 172.21.15.31:0/3465618606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c340c5b40 con 0x7f6c340c23a0 2024-09-11T00:40:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.103+0000 7f6c2a000640 1 -- 172.21.15.31:0/3465618606 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6c20003c20 con 0x7f6c340c23a0 2024-09-11T00:40:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.104+0000 7f6c2a000640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c14077fa0 0x7f6c1407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.104+0000 7f6c2a000640 1 -- 172.21.15.31:0/3465618606 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7f6c20099a70 con 0x7f6c340c23a0 2024-09-11T00:40:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.105+0000 7f6c31e00640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c14077fa0 0x7f6c1407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:28.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.105+0000 7f6c31e00640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c14077fa0 0x7f6c1407a460 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6c24033000 tx=0x7f6c24004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:28.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.108+0000 7f6c2a000640 1 -- 172.21.15.31:0/3465618606 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6c20007e20 con 0x7f6c340c23a0 2024-09-11T00:40:28.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.278+0000 7f2568e00640 1 -- 172.21.15.31:0/3087184499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25640a70e0 con 0x7f25640c6340 2024-09-11T00:40:28.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.278+0000 7f255aa00640 1 -- 172.21.15.31:0/3087184499 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5712 (secure 0 0 0) 0x7f255c067670 con 0x7f25640c6340 2024-09-11T00:40:28.281 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:28.281 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"btime":"2024-09-11T00:40:27:381676+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:28.281 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2024-09-11T00:40:28.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 -- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2540077f70 msgr2=0x7f254007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:28.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2540077f70 0x7f254007a430 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2554033000 tx=0x7f2554004730 comp rx=0 tx=0).stop 2024-09-11T00:40:28.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 -- 172.21.15.31:0/3087184499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25640c6340 msgr2=0x7f25640ac6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:28.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25640c6340 0x7f25640ac6c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f255c00ea40 tx=0x7f255c00ef00 comp rx=0 tx=0).stop 2024-09-11T00:40:28.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 -- 172.21.15.31:0/3087184499 shutdown_connections 2024-09-11T00:40:28.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2540077f70 0x7f254007a430 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25640c6340 0x7f25640ac6c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25640c5ef0 0x7f256414beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 --2- 172.21.15.31:0/3087184499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25640c5b10 0x7f256414b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 -- 172.21.15.31:0/3087184499 >> 172.21.15.31:0/3087184499 conn(0x7f25640a9750 msgr2=0x7f25640c4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:28.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 -- 172.21.15.31:0/3087184499 shutdown_connections 2024-09-11T00:40:28.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.281+0000 7f2568e00640 1 -- 172.21.15.31:0/3087184499 wait complete. 2024-09-11T00:40:28.298 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:40:28.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.349+0000 7f6c38a00640 1 -- 172.21.15.31:0/3465618606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c3414c630 con 0x7f6c340c23a0 2024-09-11T00:40:28.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.350+0000 7f6c2a000640 1 -- 172.21.15.31:0/3465618606 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5712 (secure 0 0 0) 0x7f6c20061740 con 0x7f6c340c23a0 2024-09-11T00:40:28.352 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:28.352 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"btime":"2024-09-11T00:40:27:381676+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:28.352 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2024-09-11T00:40:28.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.352+0000 7f6c38a00640 1 -- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c14077fa0 msgr2=0x7f6c1407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:28.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.352+0000 7f6c38a00640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c14077fa0 0x7f6c1407a460 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6c24033000 tx=0x7f6c24004730 comp rx=0 tx=0).stop 2024-09-11T00:40:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.352+0000 7f6c38a00640 1 -- 172.21.15.31:0/3465618606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c340c23a0 msgr2=0x7f6c34073d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.352+0000 7f6c38a00640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c340c23a0 0x7f6c34073d30 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000c410 tx=0x7f6c2000c8d0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.352+0000 7f6c38a00640 1 -- 172.21.15.31:0/3465618606 shutdown_connections 2024-09-11T00:40:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.352+0000 7f6c38a00640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c14077fa0 0x7f6c1407a460 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.352+0000 7f6c38a00640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c34156300 0x7f6c341eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.352+0000 7f6c38a00640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c340c23a0 0x7f6c34073d30 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.352+0000 7f6c38a00640 1 --2- 172.21.15.31:0/3465618606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c3406fa70 0x7f6c3414bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.352+0000 7f6c38a00640 1 -- 172.21.15.31:0/3465618606 >> 172.21.15.31:0/3465618606 conn(0x7f6c340a9750 msgr2=0x7f6c340c4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.353+0000 7f6c38a00640 1 -- 172.21.15.31:0/3465618606 shutdown_connections 2024-09-11T00:40:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:28.353+0000 7f6c38a00640 1 -- 172.21.15.31:0/3465618606 wait complete. 2024-09-11T00:40:28.370 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5201 2024-09-11T00:40:28.371 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.d 2024-09-11T00:40:28.455 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-11T00:40:28.453+0000 7fb6dd800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:40:29.371 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-11T00:40:29.371 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:40:29.372 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:40:29.376 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:40:29.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:29.513 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:40:29.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.541+0000 7fec4a800640 1 -- 172.21.15.31:0/3587463601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec4414ccc0 msgr2=0x7fec44156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:29.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.541+0000 7fec4a800640 1 --2- 172.21.15.31:0/3587463601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec4414ccc0 0x7fec44156b00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fec30009830 tx=0x7fec3002a5c0 comp rx=0 tx=0).stop 2024-09-11T00:40:29.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.541+0000 7fec4a800640 1 -- 172.21.15.31:0/3587463601 shutdown_connections 2024-09-11T00:40:29.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.541+0000 7fec4a800640 1 --2- 172.21.15.31:0/3587463601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec4414ccc0 0x7fec44156b00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:29.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.541+0000 7fec4a800640 1 -- 172.21.15.31:0/3587463601 >> 172.21.15.31:0/3587463601 conn(0x7fec44068ee0 msgr2=0x7fec440cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:29.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.541+0000 7fec4a800640 1 -- 172.21.15.31:0/3587463601 shutdown_connections 2024-09-11T00:40:29.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.541+0000 7fec4a800640 1 -- 172.21.15.31:0/3587463601 wait complete. 2024-09-11T00:40:29.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.543+0000 7fec4a800640 1 Processor -- start 2024-09-11T00:40:29.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.543+0000 7fec4a800640 1 -- start start 2024-09-11T00:40:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec4a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec440c9a10 0x7fec441ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec4a800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec4414c8e0 0x7fec440c19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec4a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec4414ccc0 0x7fec440c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec4a800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec440c6800 con 0x7fec4414ccc0 2024-09-11T00:40:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec4a800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fec440cb880 con 0x7fec4414c8e0 2024-09-11T00:40:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec4a800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fec440c6c90 con 0x7fec440c9a10 2024-09-11T00:40:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec49400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec440c9a10 0x7fec441ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec48a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec4414c8e0 0x7fec440c19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:29.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec49400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec440c9a10 0x7fec441ebf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:42130/0 (socket says 172.21.15.31:42130) 2024-09-11T00:40:29.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec49400640 1 -- 172.21.15.31:0/3404509950 learned_addr learned my addr 172.21.15.31:0/3404509950 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:29.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec49e00640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec4414ccc0 0x7fec440c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec49e00640 1 -- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec440c9a10 msgr2=0x7fec441ebf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec49e00640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec440c9a10 0x7fec441ebf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec49e00640 1 -- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec4414c8e0 msgr2=0x7fec440c19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.544+0000 7fec49e00640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec4414c8e0 0x7fec440c19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.545+0000 7fec49e00640 1 -- 172.21.15.31:0/3404509950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec44153eb0 con 0x7fec4414ccc0 2024-09-11T00:40:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.545+0000 7fec49400640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec440c9a10 0x7fec441ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.545+0000 7fec48a00640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec4414c8e0 0x7fec440c19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.545+0000 7fec49e00640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec4414ccc0 0x7fec440c1f00 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00cb40 tx=0x7fec3c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:29.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.545+0000 7fec41600640 1 -- 172.21.15.31:0/3404509950 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c016020 con 0x7fec4414ccc0 2024-09-11T00:40:29.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.545+0000 7fec41600640 1 -- 172.21.15.31:0/3404509950 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec3c00ad30 con 0x7fec4414ccc0 2024-09-11T00:40:29.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.545+0000 7fec41600640 1 -- 172.21.15.31:0/3404509950 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c012680 con 0x7fec4414ccc0 2024-09-11T00:40:29.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.545+0000 7fec4a800640 1 -- 172.21.15.31:0/3404509950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec440c2440 con 0x7fec4414ccc0 2024-09-11T00:40:29.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.545+0000 7fec4a800640 1 -- 172.21.15.31:0/3404509950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec441ec970 con 0x7fec4414ccc0 2024-09-11T00:40:29.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.546+0000 7fec4a800640 1 -- 172.21.15.31:0/3404509950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec44156b00 con 0x7fec4414ccc0 2024-09-11T00:40:29.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.547+0000 7fec41600640 1 -- 172.21.15.31:0/3404509950 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fec3c012820 con 0x7fec4414ccc0 2024-09-11T00:40:29.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.548+0000 7fec41600640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec24077f70 0x7fec2407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:29.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.548+0000 7fec41600640 1 -- 172.21.15.31:0/3404509950 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7fec3c09ad00 con 0x7fec4414ccc0 2024-09-11T00:40:29.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.549+0000 7fec49400640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec24077f70 0x7fec2407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:29.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.549+0000 7fec49400640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec24077f70 0x7fec2407a430 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fec38000c00 tx=0x7fec38009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:29.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.553+0000 7fec41600640 1 -- 172.21.15.31:0/3404509950 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fec3c020020 con 0x7fec4414ccc0 2024-09-11T00:40:29.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.796+0000 7fec4a800640 1 -- 172.21.15.31:0/3404509950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec440ca910 con 0x7fec4414ccc0 2024-09-11T00:40:29.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.797+0000 7fec41600640 1 -- 172.21.15.31:0/3404509950 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5712 (secure 0 0 0) 0x7fec3c0629d0 con 0x7fec4414ccc0 2024-09-11T00:40:29.801 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:29.801 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"btime":"2024-09-11T00:40:27:381676+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:29.801 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 71 2024-09-11T00:40:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 -- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec24077f70 msgr2=0x7fec2407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec24077f70 0x7fec2407a430 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fec38000c00 tx=0x7fec38009040 comp rx=0 tx=0).stop 2024-09-11T00:40:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 -- 172.21.15.31:0/3404509950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec4414ccc0 msgr2=0x7fec440c1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec4414ccc0 0x7fec440c1f00 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00cb40 tx=0x7fec3c00a060 comp rx=0 tx=0).stop 2024-09-11T00:40:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 -- 172.21.15.31:0/3404509950 shutdown_connections 2024-09-11T00:40:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fec24077f70 0x7fec2407a430 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec4414ccc0 0x7fec440c1f00 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec4414c8e0 0x7fec440c19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 --2- 172.21.15.31:0/3404509950 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fec440c9a10 0x7fec441ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 -- 172.21.15.31:0/3404509950 >> 172.21.15.31:0/3404509950 conn(0x7fec44068ee0 msgr2=0x7fec440ab570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 -- 172.21.15.31:0/3404509950 shutdown_connections 2024-09-11T00:40:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:29.800+0000 7fec4a800640 1 -- 172.21.15.31:0/3404509950 wait complete. 2024-09-11T00:40:29.820 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:40:30.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.460+0000 7f7e0dc00640 1 -- 172.21.15.31:0/1654432680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e080708b0 msgr2=0x7f7e08068a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.460+0000 7f7e0dc00640 1 --2- 172.21.15.31:0/1654432680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e080708b0 0x7f7e08068a00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f7df4005fd0 tx=0x7f7df402a5d0 comp rx=0 tx=0).stop 2024-09-11T00:40:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.460+0000 7f7e0dc00640 1 -- 172.21.15.31:0/1654432680 shutdown_connections 2024-09-11T00:40:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.460+0000 7f7e0dc00640 1 --2- 172.21.15.31:0/1654432680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e080708b0 0x7f7e08068a00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.460+0000 7f7e0dc00640 1 --2- 172.21.15.31:0/1654432680 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e0814ca20 0x7f7e08155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.460+0000 7f7e0dc00640 1 -- 172.21.15.31:0/1654432680 >> 172.21.15.31:0/1654432680 conn(0x7f7e080c8320 msgr2=0x7f7e080c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.460+0000 7f7e0dc00640 1 -- 172.21.15.31:0/1654432680 shutdown_connections 2024-09-11T00:40:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.460+0000 7f7e0dc00640 1 -- 172.21.15.31:0/1654432680 wait complete. 2024-09-11T00:40:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.461+0000 7f7e0dc00640 1 Processor -- start 2024-09-11T00:40:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.462+0000 7f7e0dc00640 1 -- start start 2024-09-11T00:40:30.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.462+0000 7f7e0dc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e080708b0 0x7f7e081e1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:30.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.462+0000 7f7e0dc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e080c4810 0x7f7e081e09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:30.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e0dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e0814ca20 0x7f7e0814a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:30.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e0dc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7e080c2dc0 con 0x7f7e0814ca20 2024-09-11T00:40:30.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e0dc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0815a3d0 con 0x7f7e080708b0 2024-09-11T00:40:30.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e0dc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7e0814ce00 con 0x7f7e080c4810 2024-09-11T00:40:30.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e080708b0 0x7f7e081e1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:30.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e080c4810 0x7f7e081e09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:30.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e07400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e0814ca20 0x7f7e0814a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:30.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e080708b0 0x7f7e081e1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:51370/0 (socket says 172.21.15.31:51370) 2024-09-11T00:40:30.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e080c4810 0x7f7e081e09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:42156/0 (socket says 172.21.15.31:42156) 2024-09-11T00:40:30.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06a00640 1 -- 172.21.15.31:0/3576288449 learned_addr learned my addr 172.21.15.31:0/3576288449 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:30.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06a00640 1 -- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e080c4810 msgr2=0x7f7e081e09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:30.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06a00640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e080c4810 0x7f7e081e09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:30.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06a00640 1 -- 172.21.15.31:0/3576288449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e0814ca20 msgr2=0x7f7e0814a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:30.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06a00640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e0814ca20 0x7f7e0814a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:30.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06a00640 1 -- 172.21.15.31:0/3576288449 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0814b970 con 0x7f7e080708b0 2024-09-11T00:40:30.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e06000640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e080c4810 0x7f7e081e09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:30.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.463+0000 7f7e07400640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e0814ca20 0x7f7e0814a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:30.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.464+0000 7f7e06a00640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e080708b0 0x7f7e081e1920 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7df800e790 tx=0x7f7df800ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:30.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.464+0000 7f7dfea00640 1 -- 172.21.15.31:0/3576288449 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df800eea0 con 0x7f7e080708b0 2024-09-11T00:40:30.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.464+0000 7f7dfea00640 1 -- 172.21.15.31:0/3576288449 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7df800cdb0 con 0x7f7e080708b0 2024-09-11T00:40:30.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.464+0000 7f7dfea00640 1 -- 172.21.15.31:0/3576288449 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8010690 con 0x7f7e080708b0 2024-09-11T00:40:30.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.464+0000 7f7e0dc00640 1 -- 172.21.15.31:0/3576288449 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e0814b690 con 0x7f7e080708b0 2024-09-11T00:40:30.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.464+0000 7f7e0dc00640 1 -- 172.21.15.31:0/3576288449 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e08155ec0 con 0x7f7e080708b0 2024-09-11T00:40:30.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.465+0000 7f7e0dc00640 1 -- 172.21.15.31:0/3576288449 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e0806caf0 con 0x7f7e080708b0 2024-09-11T00:40:30.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.467+0000 7f7dfea00640 1 -- 172.21.15.31:0/3576288449 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7df803eb10 con 0x7f7e080708b0 2024-09-11T00:40:30.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.467+0000 7f7dfea00640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7de8077f70 0x7f7de807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:30.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.468+0000 7f7dfea00640 1 -- 172.21.15.31:0/3576288449 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7f7df809b070 con 0x7f7e080708b0 2024-09-11T00:40:30.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.468+0000 7f7e06000640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7de8077f70 0x7f7de807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:30.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.468+0000 7f7e06000640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7de8077f70 0x7f7de807a430 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7df4033000 tx=0x7f7df4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:30.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.471+0000 7f7dfea00640 1 -- 172.21.15.31:0/3576288449 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7df8062dc0 con 0x7f7e080708b0 2024-09-11T00:40:30.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.710+0000 7f7e0dc00640 1 -- 172.21.15.31:0/3576288449 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e0806dc50 con 0x7f7e080708b0 2024-09-11T00:40:30.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.711+0000 7f7dfea00640 1 -- 172.21.15.31:0/3576288449 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6534 (secure 0 0 0) 0x7f7df80677f0 con 0x7f7e080708b0 2024-09-11T00:40:30.714 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:30.714 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"btime":"2024-09-11T00:40:30:390816+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5911,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:30.714 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2024-09-11T00:40:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 -- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7de8077f70 msgr2=0x7f7de807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7de8077f70 0x7f7de807a430 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7df4033000 tx=0x7f7df4004730 comp rx=0 tx=0).stop 2024-09-11T00:40:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 -- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e080708b0 msgr2=0x7f7e081e1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e080708b0 0x7f7e081e1920 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7df800e790 tx=0x7f7df800ec50 comp rx=0 tx=0).stop 2024-09-11T00:40:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 -- 172.21.15.31:0/3576288449 shutdown_connections 2024-09-11T00:40:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7de8077f70 0x7f7de807a430 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7e0814ca20 0x7f7e0814a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7e080c4810 0x7f7e081e09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 --2- 172.21.15.31:0/3576288449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e080708b0 0x7f7e081e1920 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 -- 172.21.15.31:0/3576288449 >> 172.21.15.31:0/3576288449 conn(0x7f7e080c8320 msgr2=0x7f7e080ab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 -- 172.21.15.31:0/3576288449 shutdown_connections 2024-09-11T00:40:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:30.713+0000 7f7e0dc00640 1 -- 172.21.15.31:0/3576288449 wait complete. 2024-09-11T00:40:30.732 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:40:31.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:31.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.990+0000 7f66dde00640 1 -- 172.21.15.31:0/3124903719 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d8150630 msgr2=0x7f66d806e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:31.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.990+0000 7f66dde00640 1 --2- 172.21.15.31:0/3124903719 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d8150630 0x7f66d806e200 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009860 tx=0x7f66c802a480 comp rx=0 tx=0).stop 2024-09-11T00:40:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.990+0000 7f66dde00640 1 -- 172.21.15.31:0/3124903719 shutdown_connections 2024-09-11T00:40:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.990+0000 7f66dde00640 1 --2- 172.21.15.31:0/3124903719 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d8150630 0x7f66d806e200 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.990+0000 7f66dde00640 1 -- 172.21.15.31:0/3124903719 >> 172.21.15.31:0/3124903719 conn(0x7f66d80a9750 msgr2=0x7f66d80a9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.990+0000 7f66dde00640 1 -- 172.21.15.31:0/3124903719 shutdown_connections 2024-09-11T00:40:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.990+0000 7f66dde00640 1 -- 172.21.15.31:0/3124903719 wait complete. 2024-09-11T00:40:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.992+0000 7f66dde00640 1 Processor -- start 2024-09-11T00:40:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.992+0000 7f66dde00640 1 -- start start 2024-09-11T00:40:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66dde00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d806e740 0x7f66d8155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66dde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d80c3a10 0x7f66d80acb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66dde00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d8150630 0x7f66d80a83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66dde00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66d80c5df0 con 0x7f66d8150630 2024-09-11T00:40:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66dde00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f66d80c7240 con 0x7f66d80c3a10 2024-09-11T00:40:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66dde00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f66d815e680 con 0x7f66d806e740 2024-09-11T00:40:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d806e740 0x7f66d8155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d8150630 0x7f66d80a83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d6000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d80c3a10 0x7f66d80acb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:31.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d806e740 0x7f66d8155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:42178/0 (socket says 172.21.15.31:42178) 2024-09-11T00:40:31.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d6a00640 1 -- 172.21.15.31:0/1814576586 learned_addr learned my addr 172.21.15.31:0/1814576586 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d6a00640 1 -- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d80c3a10 msgr2=0x7f66d80acb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d6a00640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d80c3a10 0x7f66d80acb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d6a00640 1 -- 172.21.15.31:0/1814576586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d8150630 msgr2=0x7f66d80a83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d6a00640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d8150630 0x7f66d80a83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d6a00640 1 -- 172.21.15.31:0/1814576586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d80aae10 con 0x7f66d806e740 2024-09-11T00:40:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.993+0000 7f66d7400640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d8150630 0x7f66d80a83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.994+0000 7f66d6a00640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d806e740 0x7f66d8155830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009980 tx=0x7f66c8033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.994+0000 7f66cf400640 1 -- 172.21.15.31:0/1814576586 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c8005c90 con 0x7f66d806e740 2024-09-11T00:40:31.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.994+0000 7f66cf400640 1 -- 172.21.15.31:0/1814576586 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66c8031e50 con 0x7f66d806e740 2024-09-11T00:40:31.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.994+0000 7f66dde00640 1 -- 172.21.15.31:0/1814576586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66d80ab0f0 con 0x7f66d806e740 2024-09-11T00:40:31.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.994+0000 7f66cf400640 1 -- 172.21.15.31:0/1814576586 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c8005260 con 0x7f66d806e740 2024-09-11T00:40:31.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.994+0000 7f66dde00640 1 -- 172.21.15.31:0/1814576586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f66d8155310 con 0x7f66d806e740 2024-09-11T00:40:31.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.995+0000 7f66dde00640 1 -- 172.21.15.31:0/1814576586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66d8155130 con 0x7f66d806e740 2024-09-11T00:40:31.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.997+0000 7f66cf400640 1 -- 172.21.15.31:0/1814576586 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f66c8005400 con 0x7f66d806e740 2024-09-11T00:40:32.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.998+0000 7f66cf400640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b8077f70 0x7f66b807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:32.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.998+0000 7f66cf400640 1 -- 172.21.15.31:0/1814576586 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7f66c80bb8d0 con 0x7f66d806e740 2024-09-11T00:40:32.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.998+0000 7f66d6000640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b8077f70 0x7f66b807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:32.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:31.999+0000 7f66d6000640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b8077f70 0x7f66b807a430 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f66c4007fd0 tx=0x7f66c4007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:32.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.002+0000 7f66cf400640 1 -- 172.21.15.31:0/1814576586 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f66d8155130 con 0x7f66d806e740 2024-09-11T00:40:32.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.236+0000 7f66dde00640 1 -- 172.21.15.31:0/1814576586 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66d806e200 con 0x7f66d806e740 2024-09-11T00:40:32.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.237+0000 7f66cf400640 1 -- 172.21.15.31:0/1814576586 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6534 (secure 0 0 0) 0x7f66c803f030 con 0x7f66d806e740 2024-09-11T00:40:32.239 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:32.240 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"btime":"2024-09-11T00:40:30:390816+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5911,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:32.240 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2024-09-11T00:40:32.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.239+0000 7f66dde00640 1 -- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b8077f70 msgr2=0x7f66b807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.239+0000 7f66dde00640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b8077f70 0x7f66b807a430 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f66c4007fd0 tx=0x7f66c4007eb0 comp rx=0 tx=0).stop 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.239+0000 7f66dde00640 1 -- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d806e740 msgr2=0x7f66d8155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.239+0000 7f66dde00640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d806e740 0x7f66d8155830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009980 tx=0x7f66c8033c20 comp rx=0 tx=0).stop 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.239+0000 7f66dde00640 1 -- 172.21.15.31:0/1814576586 shutdown_connections 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.239+0000 7f66dde00640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b8077f70 0x7f66b807a430 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.239+0000 7f66dde00640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66d8150630 0x7f66d80a83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.239+0000 7f66dde00640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d80c3a10 0x7f66d80acb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.239+0000 7f66dde00640 1 --2- 172.21.15.31:0/1814576586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d806e740 0x7f66d8155830 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.239+0000 7f66dde00640 1 -- 172.21.15.31:0/1814576586 >> 172.21.15.31:0/1814576586 conn(0x7f66d80a9750 msgr2=0x7f66d80c2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.240+0000 7f66dde00640 1 -- 172.21.15.31:0/1814576586 shutdown_connections 2024-09-11T00:40:32.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.240+0000 7f66dde00640 1 -- 172.21.15.31:0/1814576586 wait complete. 2024-09-11T00:40:32.258 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-11T00:40:32.259 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2024-09-11T00:40:32.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:32.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.893+0000 7f4e58c00640 1 -- 172.21.15.31:0/1495456071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e54072480 msgr2=0x7f4e541511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:32.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.893+0000 7f4e58c00640 1 --2- 172.21.15.31:0/1495456071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e54072480 0x7f4e541511a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4e54075260 tx=0x7f4e4402a5c0 comp rx=0 tx=0).stop 2024-09-11T00:40:32.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.893+0000 7f4e58c00640 1 -- 172.21.15.31:0/1495456071 shutdown_connections 2024-09-11T00:40:32.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.893+0000 7f4e58c00640 1 --2- 172.21.15.31:0/1495456071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e54072480 0x7f4e541511a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:32.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.893+0000 7f4e58c00640 1 -- 172.21.15.31:0/1495456071 >> 172.21.15.31:0/1495456071 conn(0x7f4e540a9750 msgr2=0x7f4e540a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:32.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.893+0000 7f4e58c00640 1 -- 172.21.15.31:0/1495456071 shutdown_connections 2024-09-11T00:40:32.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.893+0000 7f4e58c00640 1 -- 172.21.15.31:0/1495456071 wait complete. 2024-09-11T00:40:32.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.894+0000 7f4e58c00640 1 Processor -- start 2024-09-11T00:40:32.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.895+0000 7f4e58c00640 1 -- start start 2024-09-11T00:40:32.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.895+0000 7f4e58c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e54072480 0x7f4e54152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:32.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.895+0000 7f4e58c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e540c23a0 0x7f4e54148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:32.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.895+0000 7f4e58c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e54153f20 0x7f4e541490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:32.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.895+0000 7f4e58c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4e54074200 con 0x7f4e540c23a0 2024-09-11T00:40:32.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.895+0000 7f4e58c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4e54074080 con 0x7f4e54153f20 2024-09-11T00:40:32.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.895+0000 7f4e58c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4e54072860 con 0x7f4e54072480 2024-09-11T00:40:32.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e49400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e540c23a0 0x7f4e54148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:32.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e49400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e540c23a0 0x7f4e54148b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55742/0 (socket says 172.21.15.31:55742) 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e49400640 1 -- 172.21.15.31:0/144388710 learned_addr learned my addr 172.21.15.31:0/144388710 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e51e00640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e54072480 0x7f4e54152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e52800640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e54153f20 0x7f4e541490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e49400640 1 -- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e54072480 msgr2=0x7f4e54152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e49400640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e54072480 0x7f4e54152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e49400640 1 -- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e54153f20 msgr2=0x7f4e541490a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e49400640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e54153f20 0x7f4e541490a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e49400640 1 -- 172.21.15.31:0/144388710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e540c6730 con 0x7f4e540c23a0 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.896+0000 7f4e49400640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e540c23a0 0x7f4e54148b60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00c400 tx=0x7f4e3c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.897+0000 7f4e4aa00640 1 -- 172.21.15.31:0/144388710 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c011790 con 0x7f4e540c23a0 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.897+0000 7f4e4aa00640 1 -- 172.21.15.31:0/144388710 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e3c017070 con 0x7f4e540c23a0 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.897+0000 7f4e58c00640 1 -- 172.21.15.31:0/144388710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e540c6a10 con 0x7f4e540c23a0 2024-09-11T00:40:32.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.897+0000 7f4e4aa00640 1 -- 172.21.15.31:0/144388710 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c011d00 con 0x7f4e540c23a0 2024-09-11T00:40:32.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.897+0000 7f4e58c00640 1 -- 172.21.15.31:0/144388710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e54152160 con 0x7f4e540c23a0 2024-09-11T00:40:32.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.898+0000 7f4e58c00640 1 -- 172.21.15.31:0/144388710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e541511a0 con 0x7f4e540c23a0 2024-09-11T00:40:32.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.899+0000 7f4e4aa00640 1 -- 172.21.15.31:0/144388710 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4e3c01c460 con 0x7f4e540c23a0 2024-09-11T00:40:32.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.900+0000 7f4e4aa00640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4e30077fa0 0x7f4e3007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:32.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.900+0000 7f4e4aa00640 1 -- 172.21.15.31:0/144388710 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7f4e3c09a230 con 0x7f4e540c23a0 2024-09-11T00:40:32.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.900+0000 7f4e51e00640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4e30077fa0 0x7f4e3007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:32.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.901+0000 7f4e51e00640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4e30077fa0 0x7f4e3007a460 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4e44034000 tx=0x7f4e44004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:32.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:32.904+0000 7f4e4aa00640 1 -- 172.21.15.31:0/144388710 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4e3c061e70 con 0x7f4e540c23a0 2024-09-11T00:40:33.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.137+0000 7f4e58c00640 1 -- 172.21.15.31:0/144388710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e5414bd00 con 0x7f4e540c23a0 2024-09-11T00:40:33.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.138+0000 7f4e4aa00640 1 -- 172.21.15.31:0/144388710 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6534 (secure 0 0 0) 0x7f4e3c0668a0 con 0x7f4e540c23a0 2024-09-11T00:40:33.141 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:33.141 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"btime":"2024-09-11T00:40:30:390816+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5911,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:33.142 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2024-09-11T00:40:33.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.140+0000 7f4e58c00640 1 -- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4e30077fa0 msgr2=0x7f4e3007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.140+0000 7f4e58c00640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4e30077fa0 0x7f4e3007a460 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4e44034000 tx=0x7f4e44004730 comp rx=0 tx=0).stop 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.140+0000 7f4e58c00640 1 -- 172.21.15.31:0/144388710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e540c23a0 msgr2=0x7f4e54148b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.140+0000 7f4e58c00640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e540c23a0 0x7f4e54148b60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00c400 tx=0x7f4e3c00c8c0 comp rx=0 tx=0).stop 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.140+0000 7f4e58c00640 1 -- 172.21.15.31:0/144388710 shutdown_connections 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.140+0000 7f4e58c00640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4e30077fa0 0x7f4e3007a460 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.140+0000 7f4e58c00640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4e54153f20 0x7f4e541490a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.140+0000 7f4e58c00640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e540c23a0 0x7f4e54148b60 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.140+0000 7f4e58c00640 1 --2- 172.21.15.31:0/144388710 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4e54072480 0x7f4e54152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.140+0000 7f4e58c00640 1 -- 172.21.15.31:0/144388710 >> 172.21.15.31:0/144388710 conn(0x7f4e540a9750 msgr2=0x7f4e540c36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.141+0000 7f4e58c00640 1 -- 172.21.15.31:0/144388710 shutdown_connections 2024-09-11T00:40:33.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:33.141+0000 7f4e58c00640 1 -- 172.21.15.31:0/144388710 wait complete. 2024-09-11T00:40:33.159 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:40:35.161 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.323+0000 7f17cb200640 1 -- 172.21.15.31:0/3757407046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c40ac2c0 msgr2=0x7f17c40ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.323+0000 7f17cb200640 1 --2- 172.21.15.31:0/3757407046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c40ac2c0 0x7f17c40ac6a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009830 tx=0x7f17b802a4a0 comp rx=0 tx=0).stop 2024-09-11T00:40:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.324+0000 7f17cb200640 1 -- 172.21.15.31:0/3757407046 shutdown_connections 2024-09-11T00:40:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.324+0000 7f17cb200640 1 --2- 172.21.15.31:0/3757407046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c40ac2c0 0x7f17c40ac6a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.324+0000 7f17cb200640 1 -- 172.21.15.31:0/3757407046 >> 172.21.15.31:0/3757407046 conn(0x7f17c406caf0 msgr2=0x7f17c406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.324+0000 7f17cb200640 1 -- 172.21.15.31:0/3757407046 shutdown_connections 2024-09-11T00:40:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.324+0000 7f17cb200640 1 -- 172.21.15.31:0/3757407046 wait complete. 2024-09-11T00:40:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.325+0000 7f17cb200640 1 Processor -- start 2024-09-11T00:40:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.325+0000 7f17cb200640 1 -- start start 2024-09-11T00:40:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.326+0000 7f17cb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c40a6e20 0x7f17c40a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.326+0000 7f17cb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17c40ac2c0 0x7f17c41e0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.326+0000 7f17cb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17c40acbe0 0x7f17c4072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.326+0000 7f17cb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f17c40c5d20 con 0x7f17c40ac2c0 2024-09-11T00:40:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.326+0000 7f17cb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17c41519e0 con 0x7f17c40a6e20 2024-09-11T00:40:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.326+0000 7f17cb200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f17c40c68c0 con 0x7f17c40acbe0 2024-09-11T00:40:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.326+0000 7f17c9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17c40ac2c0 0x7f17c41e0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.327+0000 7f17c9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17c40ac2c0 0x7f17c41e0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55748/0 (socket says 172.21.15.31:55748) 2024-09-11T00:40:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.327+0000 7f17c9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c40a6e20 0x7f17c40a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.327+0000 7f17c9400640 1 -- 172.21.15.31:0/2264875452 learned_addr learned my addr 172.21.15.31:0/2264875452 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.327+0000 7f17ca800640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17c40acbe0 0x7f17c4072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.327+0000 7f17c9400640 1 -- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17c40acbe0 msgr2=0x7f17c4072a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.327+0000 7f17c9400640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17c40acbe0 0x7f17c4072a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.327+0000 7f17c9400640 1 -- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c40a6e20 msgr2=0x7f17c40a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.327+0000 7f17c9400640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c40a6e20 0x7f17c40a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.327+0000 7f17c9400640 1 -- 172.21.15.31:0/2264875452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c414c9a0 con 0x7f17c40ac2c0 2024-09-11T00:40:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.327+0000 7f17c9400640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17c40ac2c0 0x7f17c41e0c40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f17b400c460 tx=0x7f17b400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:35.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.328+0000 7f17be000640 1 -- 172.21.15.31:0/2264875452 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4016020 con 0x7f17c40ac2c0 2024-09-11T00:40:35.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.328+0000 7f17be000640 1 -- 172.21.15.31:0/2264875452 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17b4012920 con 0x7f17c40ac2c0 2024-09-11T00:40:35.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.328+0000 7f17be000640 1 -- 172.21.15.31:0/2264875452 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4012d70 con 0x7f17c40ac2c0 2024-09-11T00:40:35.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.328+0000 7f17cb200640 1 -- 172.21.15.31:0/2264875452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c4073bf0 con 0x7f17c40ac2c0 2024-09-11T00:40:35.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.328+0000 7f17cb200640 1 -- 172.21.15.31:0/2264875452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c40a9e10 con 0x7f17c40ac2c0 2024-09-11T00:40:35.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.329+0000 7f17cb200640 1 -- 172.21.15.31:0/2264875452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17c40c6c90 con 0x7f17c40ac2c0 2024-09-11T00:40:35.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.330+0000 7f17be000640 1 -- 172.21.15.31:0/2264875452 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f17b401a3b0 con 0x7f17c40ac2c0 2024-09-11T00:40:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.331+0000 7f17be000640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17a8077fa0 0x7f17a807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.331+0000 7f17be000640 1 -- 172.21.15.31:0/2264875452 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7f17b409a340 con 0x7f17c40ac2c0 2024-09-11T00:40:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.331+0000 7f17c9e00640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17a8077fa0 0x7f17a807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.332+0000 7f17c9e00640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17a8077fa0 0x7f17a807a460 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f17b8034000 tx=0x7f17b8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:35.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.335+0000 7f17be000640 1 -- 172.21.15.31:0/2264875452 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f17b4020030 con 0x7f17c40ac2c0 2024-09-11T00:40:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.570+0000 7f17cb200640 1 -- 172.21.15.31:0/2264875452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17c414efd0 con 0x7f17c40ac2c0 2024-09-11T00:40:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.570+0000 7f17be000640 1 -- 172.21.15.31:0/2264875452 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6534 (secure 0 0 0) 0x7f17b4062010 con 0x7f17c40ac2c0 2024-09-11T00:40:35.573 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"btime":"2024-09-11T00:40:30:390816+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5911,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2024-09-11T00:40:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 -- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17a8077fa0 msgr2=0x7f17a807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17a8077fa0 0x7f17a807a460 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f17b8034000 tx=0x7f17b8004730 comp rx=0 tx=0).stop 2024-09-11T00:40:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 -- 172.21.15.31:0/2264875452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17c40ac2c0 msgr2=0x7f17c41e0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17c40ac2c0 0x7f17c41e0c40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f17b400c460 tx=0x7f17b400c920 comp rx=0 tx=0).stop 2024-09-11T00:40:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 -- 172.21.15.31:0/2264875452 shutdown_connections 2024-09-11T00:40:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17a8077fa0 0x7f17a807a460 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17c40acbe0 0x7f17c4072a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17c40ac2c0 0x7f17c41e0c40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 --2- 172.21.15.31:0/2264875452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17c40a6e20 0x7f17c40a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 -- 172.21.15.31:0/2264875452 >> 172.21.15.31:0/2264875452 conn(0x7f17c406caf0 msgr2=0x7f17c40c6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 -- 172.21.15.31:0/2264875452 shutdown_connections 2024-09-11T00:40:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:35.573+0000 7f17cb200640 1 -- 172.21.15.31:0/2264875452 wait complete. 2024-09-11T00:40:35.592 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:40:37.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.762+0000 7f17d1400640 1 -- 172.21.15.31:0/530660322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17cc0a6e20 msgr2=0x7f17cc1510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.762+0000 7f17d1400640 1 --2- 172.21.15.31:0/530660322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17cc0a6e20 0x7f17cc1510c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009830 tx=0x7f17b802a3f0 comp rx=0 tx=0).stop 2024-09-11T00:40:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.762+0000 7f17d1400640 1 -- 172.21.15.31:0/530660322 shutdown_connections 2024-09-11T00:40:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.762+0000 7f17d1400640 1 --2- 172.21.15.31:0/530660322 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17cc151600 0x7f17cc0c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.762+0000 7f17d1400640 1 --2- 172.21.15.31:0/530660322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17cc0a6e20 0x7f17cc1510c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.762+0000 7f17d1400640 1 -- 172.21.15.31:0/530660322 >> 172.21.15.31:0/530660322 conn(0x7f17cc06caf0 msgr2=0x7f17cc06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:37.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.762+0000 7f17d1400640 1 -- 172.21.15.31:0/530660322 shutdown_connections 2024-09-11T00:40:37.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.762+0000 7f17d1400640 1 -- 172.21.15.31:0/530660322 wait complete. 2024-09-11T00:40:37.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.763+0000 7f17d1400640 1 Processor -- start 2024-09-11T00:40:37.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.764+0000 7f17d1400640 1 -- start start 2024-09-11T00:40:37.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17d1400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17cc0a6e20 0x7f17cc1dd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:37.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17d1400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17cc0ac2c0 0x7f17cc146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:37.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17d1400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17cc151600 0x7f17cc1dfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:37.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17d1400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc0acf30 con 0x7f17cc0ac2c0 2024-09-11T00:40:37.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17d1400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc0acdb0 con 0x7f17cc151600 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17d1400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f17cc0c66f0 con 0x7f17cc0a6e20 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17cc0ac2c0 0x7f17cc146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cbe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17cc0a6e20 0x7f17cc1dd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cbe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17cc0a6e20 0x7f17cc1dd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:42222/0 (socket says 172.21.15.31:42222) 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17cc0ac2c0 0x7f17cc146ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55760/0 (socket says 172.21.15.31:55760) 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cbe00640 1 -- 172.21.15.31:0/2152756943 learned_addr learned my addr 172.21.15.31:0/2152756943 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17d0a00640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17cc151600 0x7f17cc1dfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cb400640 1 -- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17cc0a6e20 msgr2=0x7f17cc1dd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cb400640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17cc0a6e20 0x7f17cc1dd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cb400640 1 -- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17cc151600 msgr2=0x7f17cc1dfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cb400640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17cc151600 0x7f17cc1dfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cb400640 1 -- 172.21.15.31:0/2152756943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17cc072400 con 0x7f17cc0ac2c0 2024-09-11T00:40:37.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.765+0000 7f17cbe00640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17cc0a6e20 0x7f17cc1dd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:37.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.766+0000 7f17cb400640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17cc0ac2c0 0x7f17cc146ef0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f17b8031000 tx=0x7f17b8032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:37.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.766+0000 7f17c3e00640 1 -- 172.21.15.31:0/2152756943 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8040070 con 0x7f17cc0ac2c0 2024-09-11T00:40:37.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.766+0000 7f17c3e00640 1 -- 172.21.15.31:0/2152756943 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17b8003900 con 0x7f17cc0ac2c0 2024-09-11T00:40:37.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.766+0000 7f17c3e00640 1 -- 172.21.15.31:0/2152756943 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8003d50 con 0x7f17cc0ac2c0 2024-09-11T00:40:37.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.766+0000 7f17d1400640 1 -- 172.21.15.31:0/2152756943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17cc1573d0 con 0x7f17cc0ac2c0 2024-09-11T00:40:37.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.766+0000 7f17d1400640 1 -- 172.21.15.31:0/2152756943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17cc155eb0 con 0x7f17cc0ac2c0 2024-09-11T00:40:37.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.767+0000 7f17d1400640 1 -- 172.21.15.31:0/2152756943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17cc0c6b50 con 0x7f17cc0ac2c0 2024-09-11T00:40:37.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.769+0000 7f17c3e00640 1 -- 172.21.15.31:0/2152756943 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f17b804d030 con 0x7f17cc0ac2c0 2024-09-11T00:40:37.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.770+0000 7f17c3e00640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17ac077f70 0x7f17ac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:37.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.770+0000 7f17c3e00640 1 -- 172.21.15.31:0/2152756943 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7698+0+0 (secure 0 0 0) 0x7f17b80bb760 con 0x7f17cc0ac2c0 2024-09-11T00:40:37.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.770+0000 7f17cbe00640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17ac077f70 0x7f17ac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:37.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.771+0000 7f17cbe00640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17ac077f70 0x7f17ac07a430 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f17bc007b10 tx=0x7f17bc007a60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:37.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:37.774+0000 7f17c3e00640 1 -- 172.21.15.31:0/2152756943 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f17b8083430 con 0x7f17cc0ac2c0 2024-09-11T00:40:38.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.010+0000 7f17d1400640 1 -- 172.21.15.31:0/2152756943 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17cc147430 con 0x7f17cc0ac2c0 2024-09-11T00:40:38.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.011+0000 7f17c3e00640 1 -- 172.21.15.31:0/2152756943 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6534 (secure 0 0 0) 0x7f17b8087e60 con 0x7f17cc0ac2c0 2024-09-11T00:40:38.014 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:38.014 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":72,"btime":"2024-09-11T00:40:30:390816+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5911,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:38:17.360584+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4307},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4307":{"gid":4307,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":117,"addr":"172.21.15.31:6834/3881548149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3881548149},{"type":"v1","addr":"172.21.15.31:6834","nonce":3881548149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4307,"qdb_cluster":[4307]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:38.014 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 72 2024-09-11T00:40:38.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 -- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17ac077f70 msgr2=0x7f17ac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:38.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17ac077f70 0x7f17ac07a430 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f17bc007b10 tx=0x7f17bc007a60 comp rx=0 tx=0).stop 2024-09-11T00:40:38.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 -- 172.21.15.31:0/2152756943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17cc0ac2c0 msgr2=0x7f17cc146ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:38.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17cc0ac2c0 0x7f17cc146ef0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f17b8031000 tx=0x7f17b8032b90 comp rx=0 tx=0).stop 2024-09-11T00:40:38.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 -- 172.21.15.31:0/2152756943 shutdown_connections 2024-09-11T00:40:38.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17ac077f70 0x7f17ac07a430 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:38.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17cc151600 0x7f17cc1dfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:38.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17cc0ac2c0 0x7f17cc146ef0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:38.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 --2- 172.21.15.31:0/2152756943 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17cc0a6e20 0x7f17cc1dd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:38.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 -- 172.21.15.31:0/2152756943 >> 172.21.15.31:0/2152756943 conn(0x7f17cc06caf0 msgr2=0x7f17cc14fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:38.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.013+0000 7f17d1400640 1 -- 172.21.15.31:0/2152756943 shutdown_connections 2024-09-11T00:40:38.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:38.014+0000 7f17d1400640 1 -- 172.21.15.31:0/2152756943 wait complete. 2024-09-11T00:40:38.032 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:40:40.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:40.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.192+0000 7fe135e00640 1 -- 172.21.15.31:0/4089602278 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe130074080 msgr2=0x7fe130151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:40.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.192+0000 7fe135e00640 1 --2- 172.21.15.31:0/4089602278 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe130074080 0x7fe130151700 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe128005fd0 tx=0x7fe12802a930 comp rx=0 tx=0).stop 2024-09-11T00:40:40.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.192+0000 7fe135e00640 1 -- 172.21.15.31:0/4089602278 shutdown_connections 2024-09-11T00:40:40.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.192+0000 7fe135e00640 1 --2- 172.21.15.31:0/4089602278 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe130074080 0x7fe130151700 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.192+0000 7fe135e00640 1 --2- 172.21.15.31:0/4089602278 >> v2:172.21.15.204:3300/0 conn(0x7fe130072480 0x7fe1301511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.192+0000 7fe135e00640 1 -- 172.21.15.31:0/4089602278 >> 172.21.15.31:0/4089602278 conn(0x7fe1300a9750 msgr2=0x7fe1300a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:40.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.192+0000 7fe135e00640 1 -- 172.21.15.31:0/4089602278 shutdown_connections 2024-09-11T00:40:40.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.192+0000 7fe135e00640 1 -- 172.21.15.31:0/4089602278 wait complete. 2024-09-11T00:40:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.194+0000 7fe135e00640 1 Processor -- start 2024-09-11T00:40:40.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.194+0000 7fe135e00640 1 -- start start 2024-09-11T00:40:40.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe135e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe130072480 0x7fe13014f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:40.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe135e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe130074080 0x7fe1301e0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:40.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe135e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1300c2410 0x7fe130152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:40.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe135e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1300729e0 con 0x7fe130074080 2024-09-11T00:40:40.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe135e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe130072860 con 0x7fe130072480 2024-09-11T00:40:40.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe135e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe130153f20 con 0x7fe1300c2410 2024-09-11T00:40:40.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe126000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe130074080 0x7fe1301e0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:40.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe12ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe130072480 0x7fe13014f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:40.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe12ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe130072480 0x7fe13014f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:42096/0 (socket says 172.21.15.31:42096) 2024-09-11T00:40:40.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe12ea00640 1 -- 172.21.15.31:0/1349681865 learned_addr learned my addr 172.21.15.31:0/1349681865 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:40.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.195+0000 7fe12f400640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1300c2410 0x7fe130152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:40.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe12ea00640 1 -- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1300c2410 msgr2=0x7fe130152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:40.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe12ea00640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1300c2410 0x7fe130152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe12ea00640 1 -- 172.21.15.31:0/1349681865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe130074080 msgr2=0x7fe1301e0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:40.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe12ea00640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe130074080 0x7fe1301e0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe12ea00640 1 -- 172.21.15.31:0/1349681865 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13014ae40 con 0x7fe130072480 2024-09-11T00:40:40.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe126000640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe130074080 0x7fe1301e0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:40.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe12ea00640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe130072480 0x7fe13014f500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe12000e9f0 tx=0x7fe12000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:40.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe127e00640 1 -- 172.21.15.31:0/1349681865 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12000cd80 con 0x7fe130072480 2024-09-11T00:40:40.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe127e00640 1 -- 172.21.15.31:0/1349681865 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe120010430 con 0x7fe130072480 2024-09-11T00:40:40.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe127e00640 1 -- 172.21.15.31:0/1349681865 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120010840 con 0x7fe130072480 2024-09-11T00:40:40.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.196+0000 7fe135e00640 1 -- 172.21.15.31:0/1349681865 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe130152130 con 0x7fe130072480 2024-09-11T00:40:40.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.197+0000 7fe135e00640 1 -- 172.21.15.31:0/1349681865 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe130152970 con 0x7fe130072480 2024-09-11T00:40:40.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.198+0000 7fe135e00640 1 -- 172.21.15.31:0/1349681865 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1301511c0 con 0x7fe130072480 2024-09-11T00:40:40.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.199+0000 7fe127e00640 1 -- 172.21.15.31:0/1349681865 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe12003eb10 con 0x7fe130072480 2024-09-11T00:40:40.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.200+0000 7fe127e00640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe10c077fa0 0x7fe10c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:40.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.200+0000 7fe127e00640 1 -- 172.21.15.31:0/1349681865 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7fe12009b3a0 con 0x7fe130072480 2024-09-11T00:40:40.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.200+0000 7fe126000640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe10c077fa0 0x7fe10c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:40.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.201+0000 7fe126000640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe10c077fa0 0x7fe10c07a460 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe1300c7900 tx=0x7fe118007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:40.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.204+0000 7fe127e00640 1 -- 172.21.15.31:0/1349681865 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe130152130 con 0x7fe130072480 2024-09-11T00:40:40.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.437+0000 7fe135e00640 1 -- 172.21.15.31:0/1349681865 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1301e14a0 con 0x7fe130072480 2024-09-11T00:40:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.438+0000 7fe127e00640 1 -- 172.21.15.31:0/1349681865 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5700 (secure 0 0 0) 0x7fe120063040 con 0x7fe130072480 2024-09-11T00:40:40.441 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:40.441 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":73,"btime":"2024-09-11T00:40:39:371415+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:39.371408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:40.441 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 73 2024-09-11T00:40:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 -- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe10c077fa0 msgr2=0x7fe10c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe10c077fa0 0x7fe10c07a460 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe1300c7900 tx=0x7fe118007d40 comp rx=0 tx=0).stop 2024-09-11T00:40:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 -- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe130072480 msgr2=0x7fe13014f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe130072480 0x7fe13014f500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe12000e9f0 tx=0x7fe12000eeb0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 -- 172.21.15.31:0/1349681865 shutdown_connections 2024-09-11T00:40:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe10c077fa0 0x7fe10c07a460 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1300c2410 0x7fe130152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe130074080 0x7fe1301e0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 --2- 172.21.15.31:0/1349681865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe130072480 0x7fe13014f500 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 -- 172.21.15.31:0/1349681865 >> 172.21.15.31:0/1349681865 conn(0x7fe1300a9750 msgr2=0x7fe13014cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 -- 172.21.15.31:0/1349681865 shutdown_connections 2024-09-11T00:40:40.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.441+0000 7fe135e00640 1 -- 172.21.15.31:0/1349681865 wait complete. 2024-09-11T00:40:40.460 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-11T00:40:40.460 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:40:40.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.621+0000 7fe1a4a00640 1 -- 172.21.15.31:0/904239764 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1a0150020 msgr2=0x7fe1a00ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.621+0000 7fe1a4a00640 1 --2- 172.21.15.31:0/904239764 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1a0150020 0x7fe1a00ad4e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe190009890 tx=0x7fe19002a5d0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.621+0000 7fe1a4a00640 1 -- 172.21.15.31:0/904239764 shutdown_connections 2024-09-11T00:40:40.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.621+0000 7fe1a4a00640 1 --2- 172.21.15.31:0/904239764 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1a0150020 0x7fe1a00ad4e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.621+0000 7fe1a4a00640 1 -- 172.21.15.31:0/904239764 >> 172.21.15.31:0/904239764 conn(0x7fe1a0068f10 msgr2=0x7fe1a00cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.621+0000 7fe1a4a00640 1 -- 172.21.15.31:0/904239764 shutdown_connections 2024-09-11T00:40:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.621+0000 7fe1a4a00640 1 -- 172.21.15.31:0/904239764 wait complete. 2024-09-11T00:40:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.622+0000 7fe1a4a00640 1 Processor -- start 2024-09-11T00:40:40.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.623+0000 7fe1a4a00640 1 -- start start 2024-09-11T00:40:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.623+0000 7fe1a4a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1a006e850 0x7fe1a00ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.623+0000 7fe1a4a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1a00ada20 0x7fe1a00bf500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.623+0000 7fe1a4a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1a0150020 0x7fe1a014a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.623+0000 7fe1a4a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a0157550 con 0x7fe1a00ada20 2024-09-11T00:40:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.623+0000 7fe1a4a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a01573d0 con 0x7fe1a006e850 2024-09-11T00:40:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.623+0000 7fe1a4a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe1a0156960 con 0x7fe1a0150020 2024-09-11T00:40:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1a00ada20 0x7fe1a00bf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1a00ada20 0x7fe1a00bf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46580/0 (socket says 172.21.15.31:46580) 2024-09-11T00:40:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19de00640 1 -- 172.21.15.31:0/1996179592 learned_addr learned my addr 172.21.15.31:0/1996179592 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:40.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19e800640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1a006e850 0x7fe1a00ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19f200640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1a0150020 0x7fe1a014a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19de00640 1 -- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1a0150020 msgr2=0x7fe1a014a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19de00640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1a0150020 0x7fe1a014a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19de00640 1 -- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1a006e850 msgr2=0x7fe1a00ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19de00640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1a006e850 0x7fe1a00ac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19de00640 1 -- 172.21.15.31:0/1996179592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a00ad780 con 0x7fe1a00ada20 2024-09-11T00:40:40.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.624+0000 7fe19de00640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1a00ada20 0x7fe1a00bf500 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00c4d0 tx=0x7fe18c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:40.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.625+0000 7fe196a00640 1 -- 172.21.15.31:0/1996179592 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c016020 con 0x7fe1a00ada20 2024-09-11T00:40:40.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.625+0000 7fe196a00640 1 -- 172.21.15.31:0/1996179592 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe18c012920 con 0x7fe1a00ada20 2024-09-11T00:40:40.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.625+0000 7fe196a00640 1 -- 172.21.15.31:0/1996179592 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c012d10 con 0x7fe1a00ada20 2024-09-11T00:40:40.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.625+0000 7fe1a4a00640 1 -- 172.21.15.31:0/1996179592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1a014b430 con 0x7fe1a00ada20 2024-09-11T00:40:40.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.625+0000 7fe1a4a00640 1 -- 172.21.15.31:0/1996179592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1a01f02f0 con 0x7fe1a00ada20 2024-09-11T00:40:40.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.626+0000 7fe1a4a00640 1 -- 172.21.15.31:0/1996179592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1a01efe30 con 0x7fe1a00ada20 2024-09-11T00:40:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.627+0000 7fe196a00640 1 -- 172.21.15.31:0/1996179592 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe18c01b400 con 0x7fe1a00ada20 2024-09-11T00:40:40.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.628+0000 7fe196a00640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe180077f70 0x7fe18007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:40.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.628+0000 7fe196a00640 1 -- 172.21.15.31:0/1996179592 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7fe18c09a680 con 0x7fe1a00ada20 2024-09-11T00:40:40.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.628+0000 7fe19e800640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe180077f70 0x7fe18007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:40.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.629+0000 7fe19e800640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe180077f70 0x7fe18007a430 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe190034000 tx=0x7fe190004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:40.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.632+0000 7fe196a00640 1 -- 172.21.15.31:0/1996179592 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe1a01efe30 con 0x7fe1a00ada20 2024-09-11T00:40:40.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.869+0000 7fe1a4a00640 1 -- 172.21.15.31:0/1996179592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1a00cc8e0 con 0x7fe1a00ada20 2024-09-11T00:40:40.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.870+0000 7fe196a00640 1 -- 172.21.15.31:0/1996179592 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5703 (secure 0 0 0) 0x7fe18c020020 con 0x7fe1a00ada20 2024-09-11T00:40:40.873 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:40.873 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":74,"btime":"2024-09-11T00:40:40:610179+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:39.767290+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:40.874 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 74 2024-09-11T00:40:40.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 -- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe180077f70 msgr2=0x7fe18007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:40.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe180077f70 0x7fe18007a430 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe190034000 tx=0x7fe190004730 comp rx=0 tx=0).stop 2024-09-11T00:40:40.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 -- 172.21.15.31:0/1996179592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1a00ada20 msgr2=0x7fe1a00bf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:40.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1a00ada20 0x7fe1a00bf500 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00c4d0 tx=0x7fe18c00c990 comp rx=0 tx=0).stop 2024-09-11T00:40:40.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 -- 172.21.15.31:0/1996179592 shutdown_connections 2024-09-11T00:40:40.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe180077f70 0x7fe18007a430 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe1a0150020 0x7fe1a014a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1a00ada20 0x7fe1a00bf500 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 --2- 172.21.15.31:0/1996179592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1a006e850 0x7fe1a00ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:40.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 -- 172.21.15.31:0/1996179592 >> 172.21.15.31:0/1996179592 conn(0x7fe1a0068f10 msgr2=0x7fe1a0159100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:40.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 -- 172.21.15.31:0/1996179592 shutdown_connections 2024-09-11T00:40:40.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:40.872+0000 7fe1a4a00640 1 -- 172.21.15.31:0/1996179592 wait complete. 2024-09-11T00:40:40.892 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:40:40:610179+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 74, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 74, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5911": { "addr": "172.21.15.204:6835/98007221", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 98007221, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 98007221, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5911, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:40:39.767290+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5911 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 71, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5901": { "addr": "172.21.15.31:6835/3925687919", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6833", "nonce": 3925687919, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3925687919, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5901, "incarnation": 68, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:40:27.381668+0000", "qdb_cluster": [ 5901 ], "qdb_leader": 5901, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5901 } } } ], "standbys": [ { "addr": "172.21.15.204:6837/3513641573", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 3513641573, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 3513641573, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5773, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:40:42.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.061+0000 7ff94b200640 1 -- 172.21.15.31:0/2798694845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff944150020 msgr2=0x7ff9440ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.061+0000 7ff94b200640 1 --2- 172.21.15.31:0/2798694845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff944150020 0x7ff9440ad4e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff9440b1b70 tx=0x7ff93802a4d0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.061+0000 7ff94b200640 1 -- 172.21.15.31:0/2798694845 shutdown_connections 2024-09-11T00:40:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.061+0000 7ff94b200640 1 --2- 172.21.15.31:0/2798694845 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff944156960 0x7ff94406e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.061+0000 7ff94b200640 1 --2- 172.21.15.31:0/2798694845 >> v2:172.21.15.204:3300/0 conn(0x7ff9440ada20 0x7ff94406e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.061+0000 7ff94b200640 1 --2- 172.21.15.31:0/2798694845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff944150020 0x7ff9440ad4e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.061+0000 7ff94b200640 1 -- 172.21.15.31:0/2798694845 >> 172.21.15.31:0/2798694845 conn(0x7ff944068f10 msgr2=0x7ff9440cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:43.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.061+0000 7ff94b200640 1 -- 172.21.15.31:0/2798694845 shutdown_connections 2024-09-11T00:40:43.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.061+0000 7ff94b200640 1 -- 172.21.15.31:0/2798694845 wait complete. 2024-09-11T00:40:43.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.063+0000 7ff94b200640 1 Processor -- start 2024-09-11T00:40:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.063+0000 7ff94b200640 1 -- start start 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.064+0000 7ff94b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9440ada20 0x7ff9440c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.064+0000 7ff94b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff944150020 0x7ff9441ec410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.064+0000 7ff94b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff944156960 0x7ff9441ec950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.064+0000 7ff94b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff944157550 con 0x7ff944150020 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.064+0000 7ff94b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441573d0 con 0x7ff9440ada20 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.064+0000 7ff94b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff9440cc220 con 0x7ff944156960 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.064+0000 7ff949400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff944150020 0x7ff9441ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.064+0000 7ff949400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff944150020 0x7ff9441ec410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46598/0 (socket says 172.21.15.31:46598) 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.064+0000 7ff949e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9440ada20 0x7ff9440c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.064+0000 7ff949400640 1 -- 172.21.15.31:0/31223793 learned_addr learned my addr 172.21.15.31:0/31223793 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.065+0000 7ff94a800640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff944156960 0x7ff9441ec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.065+0000 7ff949400640 1 -- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff944156960 msgr2=0x7ff9441ec950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.065+0000 7ff949400640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff944156960 0x7ff9441ec950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.065+0000 7ff949400640 1 -- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9440ada20 msgr2=0x7ff9440c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.065+0000 7ff949400640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9440ada20 0x7ff9440c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.065+0000 7ff949400640 1 -- 172.21.15.31:0/31223793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff944153650 con 0x7ff944150020 2024-09-11T00:40:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.065+0000 7ff949400640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff944150020 0x7ff9441ec410 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff93400e710 tx=0x7ff93400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.065+0000 7ff93e000640 1 -- 172.21.15.31:0/31223793 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93400cab0 con 0x7ff944150020 2024-09-11T00:40:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.066+0000 7ff93e000640 1 -- 172.21.15.31:0/31223793 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff93400cd40 con 0x7ff944150020 2024-09-11T00:40:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.066+0000 7ff93e000640 1 -- 172.21.15.31:0/31223793 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934010690 con 0x7ff944150020 2024-09-11T00:40:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.066+0000 7ff94b200640 1 -- 172.21.15.31:0/31223793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff944151040 con 0x7ff944150020 2024-09-11T00:40:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.066+0000 7ff94b200640 1 -- 172.21.15.31:0/31223793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9440c58e0 con 0x7ff944150020 2024-09-11T00:40:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.067+0000 7ff94b200640 1 -- 172.21.15.31:0/31223793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9440ad4e0 con 0x7ff944150020 2024-09-11T00:40:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.068+0000 7ff93e000640 1 -- 172.21.15.31:0/31223793 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff934010a40 con 0x7ff944150020 2024-09-11T00:40:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.069+0000 7ff93e000640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff928077f70 0x7ff92807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.069+0000 7ff93e000640 1 -- 172.21.15.31:0/31223793 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7ff93409b090 con 0x7ff944150020 2024-09-11T00:40:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.069+0000 7ff949e00640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff928077f70 0x7ff92807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.070+0000 7ff949e00640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff928077f70 0x7ff92807a430 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff938033000 tx=0x7ff938004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.074+0000 7ff93e000640 1 -- 172.21.15.31:0/31223793 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff934062d10 con 0x7ff944150020 2024-09-11T00:40:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.348+0000 7ff94b200640 1 -- 172.21.15.31:0/31223793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9440cc8e0 con 0x7ff944150020 2024-09-11T00:40:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.349+0000 7ff93e000640 1 -- 172.21.15.31:0/31223793 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5707 (secure 0 0 0) 0x7ff934067740 con 0x7ff944150020 2024-09-11T00:40:43.352 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:43.352 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":76,"btime":"2024-09-11T00:40:42:621940+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 76 2024-09-11T00:40:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 -- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff928077f70 msgr2=0x7ff92807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff928077f70 0x7ff92807a430 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff938033000 tx=0x7ff938004730 comp rx=0 tx=0).stop 2024-09-11T00:40:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 -- 172.21.15.31:0/31223793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff944150020 msgr2=0x7ff9441ec410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff944150020 0x7ff9441ec410 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff93400e710 tx=0x7ff93400ebd0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 -- 172.21.15.31:0/31223793 shutdown_connections 2024-09-11T00:40:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff928077f70 0x7ff92807a430 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff944156960 0x7ff9441ec950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff944150020 0x7ff9441ec410 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 --2- 172.21.15.31:0/31223793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff9440ada20 0x7ff9440c4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 -- 172.21.15.31:0/31223793 >> 172.21.15.31:0/31223793 conn(0x7ff944068f10 msgr2=0x7ff9441549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 -- 172.21.15.31:0/31223793 shutdown_connections 2024-09-11T00:40:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:43.351+0000 7ff94b200640 1 -- 172.21.15.31:0/31223793 wait complete. 2024-09-11T00:40:43.372 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=4307 2024-09-11T00:40:43.373 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.c 2024-09-11T00:40:45.332 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:40:45.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:40:45.405 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:40:45.403+0000 7f9d6b000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:40:50.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:50.373 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-11T00:40:50.374 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-11T00:40:50.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-11T00:40:50.378 INFO:tasks.ceph.mds.c:Started 2024-09-11T00:40:50.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:50.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.442+0000 7fdcdc600640 1 -- 172.21.15.31:0/2685598663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd41518b0 msgr2=0x7fdcd40c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.442+0000 7fdcdc600640 1 --2- 172.21.15.31:0/2685598663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd41518b0 0x7fdcd40c3600 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009ca0 tx=0x7fdccc02ab50 comp rx=0 tx=0).stop 2024-09-11T00:40:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.442+0000 7fdcdc600640 1 -- 172.21.15.31:0/2685598663 shutdown_connections 2024-09-11T00:40:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.443+0000 7fdcdc600640 1 --2- 172.21.15.31:0/2685598663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd41518b0 0x7fdcd40c3600 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.443+0000 7fdcdc600640 1 --2- 172.21.15.31:0/2685598663 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcd40c23a0 0x7fdcd40c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.443+0000 7fdcdc600640 1 -- 172.21.15.31:0/2685598663 >> 172.21.15.31:0/2685598663 conn(0x7fdcd40a9750 msgr2=0x7fdcd40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.443+0000 7fdcdc600640 1 -- 172.21.15.31:0/2685598663 shutdown_connections 2024-09-11T00:40:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.443+0000 7fdcdc600640 1 -- 172.21.15.31:0/2685598663 wait complete. 2024-09-11T00:40:50.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.445+0000 7fdcdc600640 1 Processor -- start 2024-09-11T00:40:50.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.446+0000 7fdcdc600640 1 -- start start 2024-09-11T00:40:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcdc600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcd4072480 0x7fdcd40ac920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcdc600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcd40c23a0 0x7fdcd41ef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:50.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcdc600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd41518b0 0x7fdcd41efbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:50.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcdc600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4072860 con 0x7fdcd41518b0 2024-09-11T00:40:50.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcdc600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4074290 con 0x7fdcd40c23a0 2024-09-11T00:40:50.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcdc600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdcd40729e0 con 0x7fdcd4072480 2024-09-11T00:40:50.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcda600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd41518b0 0x7fdcd41efbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:50.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcda600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd41518b0 0x7fdcd41efbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51188/0 (socket says 172.21.15.31:51188) 2024-09-11T00:40:50.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcda600640 1 -- 172.21.15.31:0/3288774180 learned_addr learned my addr 172.21.15.31:0/3288774180 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:50.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcd9c00640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcd4072480 0x7fdcd40ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:50.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.447+0000 7fdcd9200640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcd40c23a0 0x7fdcd41ef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:50.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.448+0000 7fdcd9c00640 1 -- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcd40c23a0 msgr2=0x7fdcd41ef6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:50.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.448+0000 7fdcd9c00640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcd40c23a0 0x7fdcd41ef6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.448+0000 7fdcd9c00640 1 -- 172.21.15.31:0/3288774180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd41518b0 msgr2=0x7fdcd41efbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:50.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.448+0000 7fdcd9c00640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd41518b0 0x7fdcd41efbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.448+0000 7fdcd9c00640 1 -- 172.21.15.31:0/3288774180 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd41f0130 con 0x7fdcd4072480 2024-09-11T00:40:50.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.449+0000 7fdcd9c00640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcd4072480 0x7fdcd40ac920 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c390 tx=0x7fdcc800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.449+0000 7fdcd1600640 1 -- 172.21.15.31:0/3288774180 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8017070 con 0x7fdcd4072480 2024-09-11T00:40:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.449+0000 7fdcd1600640 1 -- 172.21.15.31:0/3288774180 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdcc80035f0 con 0x7fdcd4072480 2024-09-11T00:40:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.449+0000 7fdcd1600640 1 -- 172.21.15.31:0/3288774180 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8003a40 con 0x7fdcd4072480 2024-09-11T00:40:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.449+0000 7fdcdc600640 1 -- 172.21.15.31:0/3288774180 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd4153730 con 0x7fdcd4072480 2024-09-11T00:40:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.449+0000 7fdcdc600640 1 -- 172.21.15.31:0/3288774180 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd41f03a0 con 0x7fdcd4072480 2024-09-11T00:40:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.451+0000 7fdcdc600640 1 -- 172.21.15.31:0/3288774180 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd4153560 con 0x7fdcd4072480 2024-09-11T00:40:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.452+0000 7fdcd1600640 1 -- 172.21.15.31:0/3288774180 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdcc8003be0 con 0x7fdcd4072480 2024-09-11T00:40:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.453+0000 7fdcd1600640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcb4077f70 0x7fdcb407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.453+0000 7fdcd1600640 1 -- 172.21.15.31:0/3288774180 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7fdcc8099b10 con 0x7fdcd4072480 2024-09-11T00:40:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.454+0000 7fdcd9200640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcb4077f70 0x7fdcb407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:50.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.455+0000 7fdcd9200640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcb4077f70 0x7fdcb407a430 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00095f0 tx=0x7fdcc0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:50.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.458+0000 7fdcd1600640 1 -- 172.21.15.31:0/3288774180 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdcc8061700 con 0x7fdcd4072480 2024-09-11T00:40:50.530 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-11T00:40:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.570+0000 7f527d000640 1 -- 172.21.15.31:0/3844229911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52780c9eb0 msgr2=0x7f527806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.570+0000 7f527d000640 1 --2- 172.21.15.31:0/3844229911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52780c9eb0 0x7f527806dd20 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5264005fd0 tx=0x7f526402a540 comp rx=0 tx=0).stop 2024-09-11T00:40:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.570+0000 7f527d000640 1 -- 172.21.15.31:0/3844229911 shutdown_connections 2024-09-11T00:40:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.570+0000 7f527d000640 1 --2- 172.21.15.31:0/3844229911 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52780bf1a0 0x7f52780b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.570+0000 7f527d000640 1 --2- 172.21.15.31:0/3844229911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52780c9eb0 0x7f527806dd20 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.570+0000 7f527d000640 1 -- 172.21.15.31:0/3844229911 >> 172.21.15.31:0/3844229911 conn(0x7f52780cbec0 msgr2=0x7f52780cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.570+0000 7f527d000640 1 -- 172.21.15.31:0/3844229911 shutdown_connections 2024-09-11T00:40:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.570+0000 7f527d000640 1 -- 172.21.15.31:0/3844229911 wait complete. 2024-09-11T00:40:50.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.572+0000 7f527d000640 1 Processor -- start 2024-09-11T00:40:50.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.572+0000 7f527d000640 1 -- start start 2024-09-11T00:40:50.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.573+0000 7f527d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52780bf1a0 0x7f52781e17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:50.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.573+0000 7f527d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52780c6690 0x7f52781f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:50.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.573+0000 7f527d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52781e1d00 0x7f52781f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:50.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.573+0000 7f527d000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f52780c6a70 con 0x7f52780c6690 2024-09-11T00:40:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.573+0000 7f527d000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5278155f90 con 0x7f52780bf1a0 2024-09-11T00:40:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.573+0000 7f527d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f52780c6bf0 con 0x7f52781e1d00 2024-09-11T00:40:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5276a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52780c6690 0x7f52781f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5277400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52780bf1a0 0x7f52781e17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5277400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52780bf1a0 0x7f52781e17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:60612/0 (socket says 172.21.15.31:60612) 2024-09-11T00:40:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5277400640 1 -- 172.21.15.31:0/2108245396 learned_addr learned my addr 172.21.15.31:0/2108245396 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5277e00640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52781e1d00 0x7f52781f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5276a00640 1 -- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52781e1d00 msgr2=0x7f52781f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5276a00640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52781e1d00 0x7f52781f3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5276a00640 1 -- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52780bf1a0 msgr2=0x7f52781e17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5276a00640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52780bf1a0 0x7f52781e17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5276a00640 1 -- 172.21.15.31:0/2108245396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52780c1e10 con 0x7f52780c6690 2024-09-11T00:40:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5277400640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52780bf1a0 0x7f52781e17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.574+0000 7f5276a00640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52780c6690 0x7f52781f3990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5264031000 tx=0x7f52640038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.575+0000 7f526f400640 1 -- 172.21.15.31:0/2108245396 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526403e030 con 0x7f52780c6690 2024-09-11T00:40:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.575+0000 7f526f400640 1 -- 172.21.15.31:0/2108245396 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5264003ca0 con 0x7f52780c6690 2024-09-11T00:40:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.575+0000 7f526f400640 1 -- 172.21.15.31:0/2108245396 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5264035040 con 0x7f52780c6690 2024-09-11T00:40:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.575+0000 7f527d000640 1 -- 172.21.15.31:0/2108245396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52780c06d0 con 0x7f52780c6690 2024-09-11T00:40:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.575+0000 7f527d000640 1 -- 172.21.15.31:0/2108245396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52780c20b0 con 0x7f52780c6690 2024-09-11T00:40:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.577+0000 7f527d000640 1 -- 172.21.15.31:0/2108245396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52780c71f0 con 0x7f52780c6690 2024-09-11T00:40:50.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.579+0000 7f526f400640 1 -- 172.21.15.31:0/2108245396 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5264004a00 con 0x7f52780c6690 2024-09-11T00:40:50.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.580+0000 7f526f400640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5258077fa0 0x7f525807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:50.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.580+0000 7f526f400640 1 -- 172.21.15.31:0/2108245396 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7f52640bb160 con 0x7f52780c6690 2024-09-11T00:40:50.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.581+0000 7f5277400640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5258077fa0 0x7f525807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:50.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.582+0000 7f5277400640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5258077fa0 0x7f525807a460 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f52780acb10 tx=0x7f5268009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:50.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.584+0000 7f526f400640 1 -- 172.21.15.31:0/2108245396 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f52780c71f0 con 0x7f52780c6690 2024-09-11T00:40:50.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.747+0000 7fdcdc600640 1 -- 172.21.15.31:0/3288774180 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcd4072d90 con 0x7fdcd4072480 2024-09-11T00:40:50.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.748+0000 7fdcd1600640 1 -- 172.21.15.31:0/3288774180 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6532 (secure 0 0 0) 0x7fdcd4072d90 con 0x7fdcd4072480 2024-09-11T00:40:50.750 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:50.750 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"btime":"2024-09-11T00:40:50:638638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6087,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:50.751 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2024-09-11T00:40:50.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 -- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcb4077f70 msgr2=0x7fdcb407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:50.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcb4077f70 0x7fdcb407a430 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00095f0 tx=0x7fdcc0005f90 comp rx=0 tx=0).stop 2024-09-11T00:40:50.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 -- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcd4072480 msgr2=0x7fdcd40ac920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:50.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcd4072480 0x7fdcd40ac920 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c390 tx=0x7fdcc800c850 comp rx=0 tx=0).stop 2024-09-11T00:40:50.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 -- 172.21.15.31:0/3288774180 shutdown_connections 2024-09-11T00:40:50.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcb4077f70 0x7fdcb407a430 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcd41518b0 0x7fdcd41efbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcd40c23a0 0x7fdcd41ef6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 --2- 172.21.15.31:0/3288774180 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcd4072480 0x7fdcd40ac920 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 -- 172.21.15.31:0/3288774180 >> 172.21.15.31:0/3288774180 conn(0x7fdcd40a9750 msgr2=0x7fdcd414ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:50.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 -- 172.21.15.31:0/3288774180 shutdown_connections 2024-09-11T00:40:50.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.751+0000 7fdcdc600640 1 -- 172.21.15.31:0/3288774180 wait complete. 2024-09-11T00:40:50.771 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-11T00:40:50.772 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-11T00:40:50.772 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:40:50.787 INFO:tasks.ceph.mds.a:Stopped 2024-09-11T00:40:50.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:50.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.908+0000 7f527d000640 1 -- 172.21.15.31:0/2108245396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52780c9b60 con 0x7f52780c6690 2024-09-11T00:40:50.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.908+0000 7f526f400640 1 -- 172.21.15.31:0/2108245396 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6532 (secure 0 0 0) 0x7f5264082e60 con 0x7f52780c6690 2024-09-11T00:40:50.910 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:50.910 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"btime":"2024-09-11T00:40:50:638638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6087,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:50.911 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2024-09-11T00:40:50.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 -- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5258077fa0 msgr2=0x7f525807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:50.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5258077fa0 0x7f525807a460 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f52780acb10 tx=0x7f5268009040 comp rx=0 tx=0).stop 2024-09-11T00:40:50.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 -- 172.21.15.31:0/2108245396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52780c6690 msgr2=0x7f52781f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52780c6690 0x7f52781f3990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5264031000 tx=0x7f52640038a0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 -- 172.21.15.31:0/2108245396 shutdown_connections 2024-09-11T00:40:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5258077fa0 0x7f525807a460 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52781e1d00 0x7f52781f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52780c6690 0x7f52781f3990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 --2- 172.21.15.31:0/2108245396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52780bf1a0 0x7f52781e17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 -- 172.21.15.31:0/2108245396 >> 172.21.15.31:0/2108245396 conn(0x7f52780cbec0 msgr2=0x7f5278153e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 -- 172.21.15.31:0/2108245396 shutdown_connections 2024-09-11T00:40:50.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:50.911+0000 7f527d000640 1 -- 172.21.15.31:0/2108245396 wait complete. 2024-09-11T00:40:50.930 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-11T00:40:50.931 INFO:tasks.mds_thrash.fs.[a]:waiting for 46 secs before thrashing 2024-09-11T00:40:51.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.005+0000 7fbbad400640 1 -- 172.21.15.31:0/4023069884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba80bf240 msgr2=0x7fbba80b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:51.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.005+0000 7fbbad400640 1 --2- 172.21.15.31:0/4023069884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba80bf240 0x7fbba80b0be0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbba8156290 tx=0x7fbb9402a4a0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.005+0000 7fbbad400640 1 -- 172.21.15.31:0/4023069884 shutdown_connections 2024-09-11T00:40:51.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.005+0000 7fbbad400640 1 --2- 172.21.15.31:0/4023069884 >> v2:172.21.15.204:3300/0 conn(0x7fbba8156e40 0x7fbba80b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.005+0000 7fbbad400640 1 --2- 172.21.15.31:0/4023069884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba80bf240 0x7fbba80b0be0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.005+0000 7fbbad400640 1 --2- 172.21.15.31:0/4023069884 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba80c6730 0x7fbba81596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.005+0000 7fbbad400640 1 -- 172.21.15.31:0/4023069884 >> 172.21.15.31:0/4023069884 conn(0x7fbba8068ac0 msgr2=0x7fbba8068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:51.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.005+0000 7fbbad400640 1 -- 172.21.15.31:0/4023069884 shutdown_connections 2024-09-11T00:40:51.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.005+0000 7fbbad400640 1 -- 172.21.15.31:0/4023069884 wait complete. 2024-09-11T00:40:51.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.007+0000 7fbbad400640 1 Processor -- start 2024-09-11T00:40:51.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.007+0000 7fbbad400640 1 -- start start 2024-09-11T00:40:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbbad400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba80bf240 0x7fbba80abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbbad400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba80c6730 0x7fbba8150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:51.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbbad400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba8156e40 0x7fbba81ebfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbbad400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbba80c6b10 con 0x7fbba80bf240 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbbad400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbba8156640 con 0x7fbba80c6730 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbbad400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbba80c6c90 con 0x7fbba8156e40 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbba7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba80bf240 0x7fbba80abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbba7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba80bf240 0x7fbba80abd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51262/0 (socket says 172.21.15.31:51262) 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbba7e00640 1 -- 172.21.15.31:0/3438757859 learned_addr learned my addr 172.21.15.31:0/3438757859 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbba7400640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba80c6730 0x7fbba8150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbbaca00640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba8156e40 0x7fbba81ebfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbba7e00640 1 -- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba8156e40 msgr2=0x7fbba81ebfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbba7e00640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba8156e40 0x7fbba81ebfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbba7e00640 1 -- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba80c6730 msgr2=0x7fbba8150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:51.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbba7e00640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba80c6730 0x7fbba8150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbba7e00640 1 -- 172.21.15.31:0/3438757859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba80c4230 con 0x7fbba80bf240 2024-09-11T00:40:51.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.008+0000 7fbbaca00640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba8156e40 0x7fbba81ebfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:40:51.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.009+0000 7fbba7e00640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba80bf240 0x7fbba80abd40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800e9f0 tx=0x7fbb9800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:51.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.009+0000 7fbb9fe00640 1 -- 172.21.15.31:0/3438757859 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9800ccc0 con 0x7fbba80bf240 2024-09-11T00:40:51.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.009+0000 7fbb9fe00640 1 -- 172.21.15.31:0/3438757859 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb9800ce60 con 0x7fbba80bf240 2024-09-11T00:40:51.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.009+0000 7fbb9fe00640 1 -- 172.21.15.31:0/3438757859 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb98005260 con 0x7fbba80bf240 2024-09-11T00:40:51.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.009+0000 7fbbad400640 1 -- 172.21.15.31:0/3438757859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba8151130 con 0x7fbba80bf240 2024-09-11T00:40:51.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.009+0000 7fbbad400640 1 -- 172.21.15.31:0/3438757859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba80c5270 con 0x7fbba80bf240 2024-09-11T00:40:51.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.010+0000 7fbbad400640 1 -- 172.21.15.31:0/3438757859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba8155090 con 0x7fbba80bf240 2024-09-11T00:40:51.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.012+0000 7fbb9fe00640 1 -- 172.21.15.31:0/3438757859 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbb98005400 con 0x7fbba80bf240 2024-09-11T00:40:51.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.013+0000 7fbb9fe00640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb88077f70 0x7fbb8807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:51.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.013+0000 7fbb9fe00640 1 -- 172.21.15.31:0/3438757859 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7fbb9809afa0 con 0x7fbba80bf240 2024-09-11T00:40:51.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.014+0000 7fbba7400640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb88077f70 0x7fbb8807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:51.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.014+0000 7fbba7400640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb88077f70 0x7fbb8807a430 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbb94034000 tx=0x7fbb94004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:51.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.018+0000 7fbb9fe00640 1 -- 172.21.15.31:0/3438757859 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbba8155090 con 0x7fbba80bf240 2024-09-11T00:40:51.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.308+0000 7fbbad400640 1 -- 172.21.15.31:0/3438757859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbba806ea60 con 0x7fbba80bf240 2024-09-11T00:40:51.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.309+0000 7fbb9fe00640 1 -- 172.21.15.31:0/3438757859 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6532 (secure 0 0 0) 0x7fbb98063270 con 0x7fbba80bf240 2024-09-11T00:40:51.311 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:51.311 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"btime":"2024-09-11T00:40:50:638638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6087,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:51.311 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2024-09-11T00:40:51.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 -- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb88077f70 msgr2=0x7fbb8807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:51.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb88077f70 0x7fbb8807a430 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbb94034000 tx=0x7fbb94004730 comp rx=0 tx=0).stop 2024-09-11T00:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 -- 172.21.15.31:0/3438757859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba80bf240 msgr2=0x7fbba80abd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba80bf240 0x7fbba80abd40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800e9f0 tx=0x7fbb9800eeb0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 -- 172.21.15.31:0/3438757859 shutdown_connections 2024-09-11T00:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb88077f70 0x7fbb8807a430 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbba8156e40 0x7fbba81ebfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbba80c6730 0x7fbba8150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 --2- 172.21.15.31:0/3438757859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbba80bf240 0x7fbba80abd40 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 -- 172.21.15.31:0/3438757859 >> 172.21.15.31:0/3438757859 conn(0x7fbba8068ac0 msgr2=0x7fbba80ae740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 -- 172.21.15.31:0/3438757859 shutdown_connections 2024-09-11T00:40:51.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:51.311+0000 7fbbad400640 1 -- 172.21.15.31:0/3438757859 wait complete. 2024-09-11T00:40:51.330 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:40:53.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:53.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.508+0000 7f763b200640 1 -- 172.21.15.31:0/910626784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7634150020 msgr2=0x7f76340ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:53.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.508+0000 7f763b200640 1 --2- 172.21.15.31:0/910626784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7634150020 0x7f76340ad4e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7628005fd0 tx=0x7f762802a670 comp rx=0 tx=0).stop 2024-09-11T00:40:53.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.508+0000 7f763b200640 1 -- 172.21.15.31:0/910626784 shutdown_connections 2024-09-11T00:40:53.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.508+0000 7f763b200640 1 --2- 172.21.15.31:0/910626784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7634150020 0x7f76340ad4e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:53.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.508+0000 7f763b200640 1 -- 172.21.15.31:0/910626784 >> 172.21.15.31:0/910626784 conn(0x7f7634068f10 msgr2=0x7f76340cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:53.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.508+0000 7f763b200640 1 -- 172.21.15.31:0/910626784 shutdown_connections 2024-09-11T00:40:53.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.509+0000 7f763b200640 1 -- 172.21.15.31:0/910626784 wait complete. 2024-09-11T00:40:53.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.510+0000 7f763b200640 1 Processor -- start 2024-09-11T00:40:53.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.511+0000 7f763b200640 1 -- start start 2024-09-11T00:40:53.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.512+0000 7f763b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f763406e850 0x7f76341e10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:53.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.512+0000 7f7639e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f763406e850 0x7f76341e10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:53.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.512+0000 7f7639e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f763406e850 0x7f76341e10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51280/0 (socket says 172.21.15.31:51280) 2024-09-11T00:40:53.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.512+0000 7f763b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76340ada20 0x7f76341529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:53.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.512+0000 7f7639e00640 1 -- 172.21.15.31:0/4177440709 learned_addr learned my addr 172.21.15.31:0/4177440709 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:53.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.512+0000 7f763b200640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7634150020 0x7f7634152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:53.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.512+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7634157550 con 0x7f763406e850 2024-09-11T00:40:53.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.512+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f76341573d0 con 0x7f76340ada20 2024-09-11T00:40:53.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.512+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7634156960 con 0x7f7634150020 2024-09-11T00:40:53.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.513+0000 7f7639e00640 1 -- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7634150020 msgr2=0x7f7634152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:53.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.513+0000 7f7639e00640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7634150020 0x7f7634152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:53.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.513+0000 7f7639400640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76340ada20 0x7f76341529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:53.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.513+0000 7f7639e00640 1 -- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76340ada20 msgr2=0x7f76341529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:53.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.513+0000 7f7639e00640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76340ada20 0x7f76341529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:53.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.513+0000 7f7639e00640 1 -- 172.21.15.31:0/4177440709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76341525c0 con 0x7f763406e850 2024-09-11T00:40:53.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.513+0000 7f7639400640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76340ada20 0x7f76341529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:40:53.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.513+0000 7f7639e00640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f763406e850 0x7f76341e10d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7628000c00 tx=0x7f7628003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:53.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.513+0000 7f762e000640 1 -- 172.21.15.31:0/4177440709 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762803f070 con 0x7f763406e850 2024-09-11T00:40:53.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.514+0000 7f762e000640 1 -- 172.21.15.31:0/4177440709 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7628004890 con 0x7f763406e850 2024-09-11T00:40:53.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.514+0000 7f762e000640 1 -- 172.21.15.31:0/4177440709 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7628035030 con 0x7f763406e850 2024-09-11T00:40:53.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.514+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7634150eb0 con 0x7f763406e850 2024-09-11T00:40:53.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.514+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76341e1930 con 0x7f763406e850 2024-09-11T00:40:53.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.515+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76340cc220 con 0x7f763406e850 2024-09-11T00:40:53.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.516+0000 7f762e000640 1 -- 172.21.15.31:0/4177440709 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7628004ce0 con 0x7f763406e850 2024-09-11T00:40:53.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.517+0000 7f762e000640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7618077fa0 0x7f761807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:53.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.518+0000 7f762e000640 1 -- 172.21.15.31:0/4177440709 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7f76280bb930 con 0x7f763406e850 2024-09-11T00:40:53.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.518+0000 7f7639400640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7618077fa0 0x7f761807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:53.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.519+0000 7f7639400640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7618077fa0 0x7f761807a460 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f76340be960 tx=0x7f7624009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:53.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.522+0000 7f762e000640 1 -- 172.21.15.31:0/4177440709 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7628083af0 con 0x7f763406e850 2024-09-11T00:40:53.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.780+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7634150bf0 con 0x7f763406e850 2024-09-11T00:40:53.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.781+0000 7f762e000640 1 -- 172.21.15.31:0/4177440709 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6532 (secure 0 0 0) 0x7f76280833f0 con 0x7f763406e850 2024-09-11T00:40:53.783 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:53.783 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"btime":"2024-09-11T00:40:50:638638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6087,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:53.783 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2024-09-11T00:40:53.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.783+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7618077fa0 msgr2=0x7f761807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.783+0000 7f763b200640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7618077fa0 0x7f761807a460 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f76340be960 tx=0x7f7624009040 comp rx=0 tx=0).stop 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.783+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f763406e850 msgr2=0x7f76341e10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.783+0000 7f763b200640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f763406e850 0x7f76341e10d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7628000c00 tx=0x7f7628003970 comp rx=0 tx=0).stop 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.784+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 shutdown_connections 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.784+0000 7f763b200640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7618077fa0 0x7f761807a460 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.784+0000 7f763b200640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7634150020 0x7f7634152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.784+0000 7f763b200640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f76340ada20 0x7f76341529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.784+0000 7f763b200640 1 --2- 172.21.15.31:0/4177440709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f763406e850 0x7f76341e10d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.784+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 >> 172.21.15.31:0/4177440709 conn(0x7f7634068f10 msgr2=0x7f7634159c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.784+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 shutdown_connections 2024-09-11T00:40:53.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:53.784+0000 7f763b200640 1 -- 172.21.15.31:0/4177440709 wait complete. 2024-09-11T00:40:53.802 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:40:55.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.982+0000 7f5deb200640 1 -- 172.21.15.31:0/2758048379 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5de40ada20 msgr2=0x7f5de406e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.982+0000 7f5deb200640 1 --2- 172.21.15.31:0/2758048379 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5de40ada20 0x7f5de406e140 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009830 tx=0x7f5dd402a530 comp rx=0 tx=0).stop 2024-09-11T00:40:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.982+0000 7f5deb200640 1 -- 172.21.15.31:0/2758048379 shutdown_connections 2024-09-11T00:40:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.982+0000 7f5deb200640 1 --2- 172.21.15.31:0/2758048379 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5de40ada20 0x7f5de406e140 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.982+0000 7f5deb200640 1 --2- 172.21.15.31:0/2758048379 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5de4150020 0x7f5de40ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.982+0000 7f5deb200640 1 -- 172.21.15.31:0/2758048379 >> 172.21.15.31:0/2758048379 conn(0x7f5de4068f10 msgr2=0x7f5de40cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.982+0000 7f5deb200640 1 -- 172.21.15.31:0/2758048379 shutdown_connections 2024-09-11T00:40:55.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.982+0000 7f5deb200640 1 -- 172.21.15.31:0/2758048379 wait complete. 2024-09-11T00:40:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.984+0000 7f5deb200640 1 Processor -- start 2024-09-11T00:40:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.984+0000 7f5deb200640 1 -- start start 2024-09-11T00:40:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5deb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5de40ada20 0x7f5de40c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5deb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5de4150020 0x7f5de40c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5deb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de4156960 0x7f5de414e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5deb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5de4157550 con 0x7f5de4156960 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5deb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5de41573d0 con 0x7f5de40ada20 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5deb200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5de40cc220 con 0x7f5de4150020 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5dea800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de4156960 0x7f5de414e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5dea800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de4156960 0x7f5de414e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51292/0 (socket says 172.21.15.31:51292) 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5dea800640 1 -- 172.21.15.31:0/2861174188 learned_addr learned my addr 172.21.15.31:0/2861174188 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5de9400640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5de4150020 0x7f5de40c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.985+0000 7f5de9e00640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5de40ada20 0x7f5de40c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5dea800640 1 -- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5de4150020 msgr2=0x7f5de40c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5dea800640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5de4150020 0x7f5de40c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5dea800640 1 -- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5de40ada20 msgr2=0x7f5de40c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5dea800640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5de40ada20 0x7f5de40c34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5dea800640 1 -- 172.21.15.31:0/2861174188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de40c39f0 con 0x7f5de4156960 2024-09-11T00:40:55.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5de9e00640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5de40ada20 0x7f5de40c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:40:55.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5dea800640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de4156960 0x7f5de414e2e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5de000cae0 tx=0x7f5de000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5dde000640 1 -- 172.21.15.31:0/2861174188 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0016020 con 0x7f5de4156960 2024-09-11T00:40:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5dde000640 1 -- 172.21.15.31:0/2861174188 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5de000ad30 con 0x7f5de4156960 2024-09-11T00:40:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5deb200640 1 -- 172.21.15.31:0/2861174188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de4150f20 con 0x7f5de4156960 2024-09-11T00:40:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.986+0000 7f5dde000640 1 -- 172.21.15.31:0/2861174188 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0012680 con 0x7f5de4156960 2024-09-11T00:40:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.987+0000 7f5deb200640 1 -- 172.21.15.31:0/2861174188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de40c5740 con 0x7f5de4156960 2024-09-11T00:40:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.988+0000 7f5deb200640 1 -- 172.21.15.31:0/2861174188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de40c9630 con 0x7f5de4156960 2024-09-11T00:40:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.989+0000 7f5dde000640 1 -- 172.21.15.31:0/2861174188 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5de001a3b0 con 0x7f5de4156960 2024-09-11T00:40:55.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.991+0000 7f5dde000640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5dc8077f70 0x7f5dc807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:55.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.991+0000 7f5dde000640 1 -- 172.21.15.31:0/2861174188 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7f5de009a9d0 con 0x7f5de4156960 2024-09-11T00:40:55.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.991+0000 7f5de9e00640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5dc8077f70 0x7f5dc807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:55.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.992+0000 7f5de9e00640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5dc8077f70 0x7f5dc807a430 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80095f0 tx=0x7f5dd8005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:55.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:55.996+0000 7f5dde000640 1 -- 172.21.15.31:0/2861174188 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5de40c9630 con 0x7f5de4156960 2024-09-11T00:40:56.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.253+0000 7f5deb200640 1 -- 172.21.15.31:0/2861174188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5de40ad4e0 con 0x7f5de4156960 2024-09-11T00:40:56.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.254+0000 7f5dde000640 1 -- 172.21.15.31:0/2861174188 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6532 (secure 0 0 0) 0x7f5de0020030 con 0x7f5de4156960 2024-09-11T00:40:56.256 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:56.256 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"btime":"2024-09-11T00:40:50:638638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6087,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:56.256 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2024-09-11T00:40:56.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.256+0000 7f5deb200640 1 -- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5dc8077f70 msgr2=0x7f5dc807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:56.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.256+0000 7f5deb200640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5dc8077f70 0x7f5dc807a430 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80095f0 tx=0x7f5dd8005f90 comp rx=0 tx=0).stop 2024-09-11T00:40:56.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.257+0000 7f5deb200640 1 -- 172.21.15.31:0/2861174188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de4156960 msgr2=0x7f5de414e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:56.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.257+0000 7f5deb200640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de4156960 0x7f5de414e2e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5de000cae0 tx=0x7f5de000a060 comp rx=0 tx=0).stop 2024-09-11T00:40:56.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.257+0000 7f5deb200640 1 -- 172.21.15.31:0/2861174188 shutdown_connections 2024-09-11T00:40:56.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.257+0000 7f5deb200640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5dc8077f70 0x7f5dc807a430 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:56.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.257+0000 7f5deb200640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de4156960 0x7f5de414e2e0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:56.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.257+0000 7f5deb200640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5de4150020 0x7f5de40c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:56.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.257+0000 7f5deb200640 1 --2- 172.21.15.31:0/2861174188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5de40ada20 0x7f5de40c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.257+0000 7f5deb200640 1 -- 172.21.15.31:0/2861174188 >> 172.21.15.31:0/2861174188 conn(0x7f5de4068f10 msgr2=0x7f5de4155510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.257+0000 7f5deb200640 1 -- 172.21.15.31:0/2861174188 shutdown_connections 2024-09-11T00:40:56.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:56.257+0000 7f5deb200640 1 -- 172.21.15.31:0/2861174188 wait complete. 2024-09-11T00:40:56.279 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:40:58.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:40:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.461+0000 7fc994a00640 1 -- 172.21.15.31:0/1753155668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9900a6e20 msgr2=0x7fc9901510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.461+0000 7fc994a00640 1 --2- 172.21.15.31:0/1753155668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9900a6e20 0x7fc9901510c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc9900c5ea0 tx=0x7fc97802a4d0 comp rx=0 tx=0).stop 2024-09-11T00:40:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.461+0000 7fc994a00640 1 -- 172.21.15.31:0/1753155668 shutdown_connections 2024-09-11T00:40:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.461+0000 7fc994a00640 1 --2- 172.21.15.31:0/1753155668 >> v2:172.21.15.204:3300/0 conn(0x7fc990151600 0x7fc9900c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.461+0000 7fc994a00640 1 --2- 172.21.15.31:0/1753155668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9900a6e20 0x7fc9901510c0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.461+0000 7fc994a00640 1 -- 172.21.15.31:0/1753155668 >> 172.21.15.31:0/1753155668 conn(0x7fc99006caf0 msgr2=0x7fc99006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.462+0000 7fc994a00640 1 -- 172.21.15.31:0/1753155668 shutdown_connections 2024-09-11T00:40:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.462+0000 7fc994a00640 1 -- 172.21.15.31:0/1753155668 wait complete. 2024-09-11T00:40:58.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.464+0000 7fc994a00640 1 Processor -- start 2024-09-11T00:40:58.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.464+0000 7fc994a00640 1 -- start start 2024-09-11T00:40:58.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.465+0000 7fc994a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9900a6e20 0x7fc99014da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.465+0000 7fc994a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9900ac2c0 0x7fc990072070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.465+0000 7fc994a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc990151600 0x7fc99014ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.465+0000 7fc994a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc9900acf30 con 0x7fc9900a6e20 2024-09-11T00:40:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.465+0000 7fc994a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc9900acdb0 con 0x7fc9900ac2c0 2024-09-11T00:40:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.465+0000 7fc994a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc9900c66f0 con 0x7fc990151600 2024-09-11T00:40:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.466+0000 7fc986a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9900ac2c0 0x7fc990072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.466+0000 7fc98f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9900a6e20 0x7fc99014da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.466+0000 7fc98fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc990151600 0x7fc99014ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.466+0000 7fc986a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9900ac2c0 0x7fc990072070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:39814/0 (socket says 172.21.15.31:39814) 2024-09-11T00:40:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.466+0000 7fc986a00640 1 -- 172.21.15.31:0/3910858672 learned_addr learned my addr 172.21.15.31:0/3910858672 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:40:58.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.466+0000 7fc986a00640 1 -- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc990151600 msgr2=0x7fc99014ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:58.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.466+0000 7fc986a00640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc990151600 0x7fc99014ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:58.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.466+0000 7fc986a00640 1 -- 172.21.15.31:0/3910858672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9900a6e20 msgr2=0x7fc99014da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.466+0000 7fc986a00640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9900a6e20 0x7fc99014da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.466+0000 7fc986a00640 1 -- 172.21.15.31:0/3910858672 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9901ec660 con 0x7fc9900ac2c0 2024-09-11T00:40:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.467+0000 7fc986a00640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9900ac2c0 0x7fc990072070 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc978009850 tx=0x7fc978003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.467+0000 7fc987e00640 1 -- 172.21.15.31:0/3910858672 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97803f070 con 0x7fc9900ac2c0 2024-09-11T00:40:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.467+0000 7fc987e00640 1 -- 172.21.15.31:0/3910858672 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9780047a0 con 0x7fc9900ac2c0 2024-09-11T00:40:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.467+0000 7fc987e00640 1 -- 172.21.15.31:0/3910858672 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978035030 con 0x7fc9900ac2c0 2024-09-11T00:40:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.467+0000 7fc994a00640 1 -- 172.21.15.31:0/3910858672 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9901ec900 con 0x7fc9900ac2c0 2024-09-11T00:40:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.468+0000 7fc994a00640 1 -- 172.21.15.31:0/3910858672 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc990074900 con 0x7fc9900ac2c0 2024-09-11T00:40:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.469+0000 7fc994a00640 1 -- 172.21.15.31:0/3910858672 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc99014f370 con 0x7fc9900ac2c0 2024-09-11T00:40:58.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.470+0000 7fc987e00640 1 -- 172.21.15.31:0/3910858672 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc978004bb0 con 0x7fc9900ac2c0 2024-09-11T00:40:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.472+0000 7fc987e00640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc96c077fa0 0x7fc96c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:40:58.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.472+0000 7fc987e00640 1 -- 172.21.15.31:0/3910858672 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7fc9780bb4e0 con 0x7fc9900ac2c0 2024-09-11T00:40:58.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.472+0000 7fc98f400640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc96c077fa0 0x7fc96c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:40:58.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.473+0000 7fc98f400640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc96c077fa0 0x7fc96c07a460 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc980001ec0 tx=0x7fc980009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:40:58.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.476+0000 7fc987e00640 1 -- 172.21.15.31:0/3910858672 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc99014f370 con 0x7fc9900ac2c0 2024-09-11T00:40:58.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.735+0000 7fc994a00640 1 -- 172.21.15.31:0/3910858672 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9901510c0 con 0x7fc9900ac2c0 2024-09-11T00:40:58.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.736+0000 7fc987e00640 1 -- 172.21.15.31:0/3910858672 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6532 (secure 0 0 0) 0x7fc9780831e0 con 0x7fc9900ac2c0 2024-09-11T00:40:58.738 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:40:58.738 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"btime":"2024-09-11T00:40:50:638638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6087,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:40:58.738 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2024-09-11T00:40:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.738+0000 7fc994a00640 1 -- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc96c077fa0 msgr2=0x7fc96c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.738+0000 7fc994a00640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc96c077fa0 0x7fc96c07a460 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc980001ec0 tx=0x7fc980009040 comp rx=0 tx=0).stop 2024-09-11T00:40:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.738+0000 7fc994a00640 1 -- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9900ac2c0 msgr2=0x7fc990072070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:40:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.738+0000 7fc994a00640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9900ac2c0 0x7fc990072070 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc978009850 tx=0x7fc978003820 comp rx=0 tx=0).stop 2024-09-11T00:40:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.739+0000 7fc994a00640 1 -- 172.21.15.31:0/3910858672 shutdown_connections 2024-09-11T00:40:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.739+0000 7fc994a00640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc96c077fa0 0x7fc96c07a460 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:58.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.739+0000 7fc994a00640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc990151600 0x7fc99014ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:58.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.739+0000 7fc994a00640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9900ac2c0 0x7fc990072070 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:58.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.739+0000 7fc994a00640 1 --2- 172.21.15.31:0/3910858672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9900a6e20 0x7fc99014da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:40:58.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.739+0000 7fc994a00640 1 -- 172.21.15.31:0/3910858672 >> 172.21.15.31:0/3910858672 conn(0x7fc99006caf0 msgr2=0x7fc99014eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:40:58.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.739+0000 7fc994a00640 1 -- 172.21.15.31:0/3910858672 shutdown_connections 2024-09-11T00:40:58.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:40:58.739+0000 7fc994a00640 1 -- 172.21.15.31:0/3910858672 wait complete. 2024-09-11T00:40:58.760 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:41:00.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:00.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.940+0000 7f3bf0600640 1 -- 172.21.15.31:0/2934025968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3be80c5b10 msgr2=0x7f3be8156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:00.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.940+0000 7f3bf0600640 1 --2- 172.21.15.31:0/2934025968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3be80c5b10 0x7f3be8156ad0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3be806e7f0 tx=0x7f3bdc02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:41:00.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.940+0000 7f3bf0600640 1 -- 172.21.15.31:0/2934025968 shutdown_connections 2024-09-11T00:41:00.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.940+0000 7f3bf0600640 1 --2- 172.21.15.31:0/2934025968 >> v2:172.21.15.204:3300/0 conn(0x7f3be80c5ef0 0x7f3be8153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:00.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.940+0000 7f3bf0600640 1 --2- 172.21.15.31:0/2934025968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3be80c5b10 0x7f3be8156ad0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:00.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.941+0000 7f3bf0600640 1 -- 172.21.15.31:0/2934025968 >> 172.21.15.31:0/2934025968 conn(0x7f3be80a9750 msgr2=0x7f3be80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:00.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.941+0000 7f3bf0600640 1 -- 172.21.15.31:0/2934025968 shutdown_connections 2024-09-11T00:41:00.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.941+0000 7f3bf0600640 1 -- 172.21.15.31:0/2934025968 wait complete. 2024-09-11T00:41:00.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.942+0000 7f3bf0600640 1 Processor -- start 2024-09-11T00:41:00.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.942+0000 7f3bf0600640 1 -- start start 2024-09-11T00:41:00.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.943+0000 7f3bf0600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3be80c5b10 0x7f3be80acef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:00.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.943+0000 7f3bf0600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3be80c5ef0 0x7f3be8151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:00.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.943+0000 7f3bf0600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3be80c6340 0x7f3be814cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:00.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.943+0000 7f3bf0600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81539c0 con 0x7f3be80c5b10 2024-09-11T00:41:00.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.943+0000 7f3bf0600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3be80c3950 con 0x7f3be80c6340 2024-09-11T00:41:00.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.943+0000 7f3bf0600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3be8153b40 con 0x7f3be80c5ef0 2024-09-11T00:41:00.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.943+0000 7f3bedc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3be80c5b10 0x7f3be80acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:00.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.944+0000 7f3bedc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3be80c5b10 0x7f3be80acef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46366/0 (socket says 172.21.15.31:46366) 2024-09-11T00:41:00.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.944+0000 7f3bedc00640 1 -- 172.21.15.31:0/3414492551 learned_addr learned my addr 172.21.15.31:0/3414492551 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:00.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.944+0000 7f3bed200640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3be80c5ef0 0x7f3be8151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:00.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.944+0000 7f3bedc00640 1 -- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3be80c5ef0 msgr2=0x7f3be8151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:00.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.944+0000 7f3bedc00640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3be80c5ef0 0x7f3be8151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:00.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.944+0000 7f3bedc00640 1 -- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3be80c6340 msgr2=0x7f3be814cb10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:41:00.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.944+0000 7f3bedc00640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3be80c6340 0x7f3be814cb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:00.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.944+0000 7f3bedc00640 1 -- 172.21.15.31:0/3414492551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be80748e0 con 0x7f3be80c5b10 2024-09-11T00:41:00.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.944+0000 7f3bed200640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3be80c5ef0 0x7f3be8151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:41:00.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.944+0000 7f3bedc00640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3be80c5b10 0x7f3be80acef0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009850 tx=0x7f3bdc003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:00.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.945+0000 7f3be5600640 1 -- 172.21.15.31:0/3414492551 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc03f070 con 0x7f3be80c5b10 2024-09-11T00:41:00.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.945+0000 7f3be5600640 1 -- 172.21.15.31:0/3414492551 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bdc0047a0 con 0x7f3be80c5b10 2024-09-11T00:41:00.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.945+0000 7f3be5600640 1 -- 172.21.15.31:0/3414492551 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc035030 con 0x7f3be80c5b10 2024-09-11T00:41:00.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.945+0000 7f3bf0600640 1 -- 172.21.15.31:0/3414492551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be81ecf30 con 0x7f3be80c5b10 2024-09-11T00:41:00.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.945+0000 7f3bf0600640 1 -- 172.21.15.31:0/3414492551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be80a70c0 con 0x7f3be80c5b10 2024-09-11T00:41:00.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.946+0000 7f3bf0600640 1 -- 172.21.15.31:0/3414492551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be80ad960 con 0x7f3be80c5b10 2024-09-11T00:41:00.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.948+0000 7f3be5600640 1 -- 172.21.15.31:0/3414492551 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3bdc004bb0 con 0x7f3be80c5b10 2024-09-11T00:41:00.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.949+0000 7f3be5600640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3bc8077f70 0x7f3bc807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:00.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.949+0000 7f3be5600640 1 -- 172.21.15.31:0/3414492551 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7f3bdc0bb740 con 0x7f3be80c5b10 2024-09-11T00:41:00.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.949+0000 7f3bed200640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3bc8077f70 0x7f3bc807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:00.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.950+0000 7f3bed200640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3bc8077f70 0x7f3bc807a430 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40058c0 tx=0x7f3bd4005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:00.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:00.955+0000 7f3be5600640 1 -- 172.21.15.31:0/3414492551 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3be80ad960 con 0x7f3be80c5b10 2024-09-11T00:41:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.237+0000 7f3bf0600640 1 -- 172.21.15.31:0/3414492551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3be8156ad0 con 0x7f3be80c5b10 2024-09-11T00:41:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.237+0000 7f3be5600640 1 -- 172.21.15.31:0/3414492551 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6532 (secure 0 0 0) 0x7f3bdc083a10 con 0x7f3be80c5b10 2024-09-11T00:41:01.240 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:01.240 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"btime":"2024-09-11T00:40:50:638638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6087,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:41:01.240 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2024-09-11T00:41:01.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.240+0000 7f3bf0600640 1 -- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3bc8077f70 msgr2=0x7f3bc807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:01.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.240+0000 7f3bf0600640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3bc8077f70 0x7f3bc807a430 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40058c0 tx=0x7f3bd4005850 comp rx=0 tx=0).stop 2024-09-11T00:41:01.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.240+0000 7f3bf0600640 1 -- 172.21.15.31:0/3414492551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3be80c5b10 msgr2=0x7f3be80acef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:01.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.240+0000 7f3bf0600640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3be80c5b10 0x7f3be80acef0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009850 tx=0x7f3bdc003910 comp rx=0 tx=0).stop 2024-09-11T00:41:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.240+0000 7f3bf0600640 1 -- 172.21.15.31:0/3414492551 shutdown_connections 2024-09-11T00:41:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.240+0000 7f3bf0600640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3bc8077f70 0x7f3bc807a430 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.240+0000 7f3bf0600640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3be80c6340 0x7f3be814cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.240+0000 7f3bf0600640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3be80c5ef0 0x7f3be8151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.240+0000 7f3bf0600640 1 --2- 172.21.15.31:0/3414492551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3be80c5b10 0x7f3be80acef0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.240+0000 7f3bf0600640 1 -- 172.21.15.31:0/3414492551 >> 172.21.15.31:0/3414492551 conn(0x7f3be80a9750 msgr2=0x7f3be8156240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.241+0000 7f3bf0600640 1 -- 172.21.15.31:0/3414492551 shutdown_connections 2024-09-11T00:41:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:01.241+0000 7f3bf0600640 1 -- 172.21.15.31:0/3414492551 wait complete. 2024-09-11T00:41:01.259 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:41:03.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:03.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.428+0000 7fb930c00640 1 -- 172.21.15.31:0/2250337367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb910005210 msgr2=0x7fb91000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:03.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.428+0000 7fb930c00640 1 --2- 172.21.15.31:0/2250337367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb910005210 0x7fb91000b9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb924009ed0 tx=0x7fb910005830 comp rx=0 tx=0).stop 2024-09-11T00:41:03.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.428+0000 7fb930c00640 1 -- 172.21.15.31:0/2250337367 shutdown_connections 2024-09-11T00:41:03.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.428+0000 7fb930c00640 1 --2- 172.21.15.31:0/2250337367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb910005210 0x7fb91000b9c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:03.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.428+0000 7fb930c00640 1 -- 172.21.15.31:0/2250337367 >> 172.21.15.31:0/2250337367 conn(0x7fb92c06caf0 msgr2=0x7fb92c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:03.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.428+0000 7fb930c00640 1 -- 172.21.15.31:0/2250337367 shutdown_connections 2024-09-11T00:41:03.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.428+0000 7fb930c00640 1 -- 172.21.15.31:0/2250337367 wait complete. 2024-09-11T00:41:03.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.430+0000 7fb930c00640 1 Processor -- start 2024-09-11T00:41:03.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.430+0000 7fb930c00640 1 -- start start 2024-09-11T00:41:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb930c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb910005210 0x7fb92c0c0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb930c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c14b2d0 0x7fb92c0729d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb930c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb92c074060 0x7fb92c14a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb930c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c0c5e80 con 0x7fb92c074060 2024-09-11T00:41:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb930c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb92c0c3160 con 0x7fb92c14b2d0 2024-09-11T00:41:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb930c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb92c0c6000 con 0x7fb910005210 2024-09-11T00:41:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb92be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb92c074060 0x7fb92c14a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb92be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb92c074060 0x7fb92c14a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46382/0 (socket says 172.21.15.31:46382) 2024-09-11T00:41:03.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb92be00640 1 -- 172.21.15.31:0/2006377664 learned_addr learned my addr 172.21.15.31:0/2006377664 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb92aa00640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c14b2d0 0x7fb92c0729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.431+0000 7fb92b400640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb910005210 0x7fb92c0c0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.432+0000 7fb92be00640 1 -- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb910005210 msgr2=0x7fb92c0c0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.432+0000 7fb92be00640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb910005210 0x7fb92c0c0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.432+0000 7fb92be00640 1 -- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c14b2d0 msgr2=0x7fb92c0729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.432+0000 7fb92be00640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c14b2d0 0x7fb92c0729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.432+0000 7fb92be00640 1 -- 172.21.15.31:0/2006377664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92c1eca00 con 0x7fb92c074060 2024-09-11T00:41:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.432+0000 7fb92be00640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb92c074060 0x7fb92c14a150 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb92402af20 tx=0x7fb924034ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.432+0000 7fb923400640 1 -- 172.21.15.31:0/2006377664 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924041070 con 0x7fb92c074060 2024-09-11T00:41:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.433+0000 7fb923400640 1 -- 172.21.15.31:0/2006377664 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9240325f0 con 0x7fb92c074060 2024-09-11T00:41:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.433+0000 7fb930c00640 1 -- 172.21.15.31:0/2006377664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92c14af40 con 0x7fb92c074060 2024-09-11T00:41:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.433+0000 7fb923400640 1 -- 172.21.15.31:0/2006377664 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924009ad0 con 0x7fb92c074060 2024-09-11T00:41:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.433+0000 7fb930c00640 1 -- 172.21.15.31:0/2006377664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb92c06f9c0 con 0x7fb92c074060 2024-09-11T00:41:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.434+0000 7fb930c00640 1 -- 172.21.15.31:0/2006377664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb92c050fb0 con 0x7fb92c074060 2024-09-11T00:41:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.435+0000 7fb923400640 1 -- 172.21.15.31:0/2006377664 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb924032a40 con 0x7fb92c074060 2024-09-11T00:41:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.437+0000 7fb923400640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb90c077f70 0x7fb90c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.437+0000 7fb923400640 1 -- 172.21.15.31:0/2006377664 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7fb9240bb110 con 0x7fb92c074060 2024-09-11T00:41:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.437+0000 7fb92b400640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb90c077f70 0x7fb90c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.438+0000 7fb92b400640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb90c077f70 0x7fb90c07a430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb91c001860 tx=0x7fb91c006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.444+0000 7fb923400640 1 -- 172.21.15.31:0/2006377664 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb92c050fb0 con 0x7fb92c074060 2024-09-11T00:41:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.712+0000 7fb930c00640 1 -- 172.21.15.31:0/2006377664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb92c0c7150 con 0x7fb92c074060 2024-09-11T00:41:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.712+0000 7fb923400640 1 -- 172.21.15.31:0/2006377664 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6532 (secure 0 0 0) 0x7fb924082c80 con 0x7fb92c074060 2024-09-11T00:41:03.715 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:03.715 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"btime":"2024-09-11T00:40:50:638638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6087,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:41:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2024-09-11T00:41:03.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 -- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb90c077f70 msgr2=0x7fb90c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:03.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb90c077f70 0x7fb90c07a430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb91c001860 tx=0x7fb91c006040 comp rx=0 tx=0).stop 2024-09-11T00:41:03.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 -- 172.21.15.31:0/2006377664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb92c074060 msgr2=0x7fb92c14a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:03.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb92c074060 0x7fb92c14a150 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb92402af20 tx=0x7fb924034ca0 comp rx=0 tx=0).stop 2024-09-11T00:41:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 -- 172.21.15.31:0/2006377664 shutdown_connections 2024-09-11T00:41:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb90c077f70 0x7fb90c07a430 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb92c074060 0x7fb92c14a150 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb92c14b2d0 0x7fb92c0729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 --2- 172.21.15.31:0/2006377664 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb910005210 0x7fb92c0c0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 -- 172.21.15.31:0/2006377664 >> 172.21.15.31:0/2006377664 conn(0x7fb92c06caf0 msgr2=0x7fb92c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 -- 172.21.15.31:0/2006377664 shutdown_connections 2024-09-11T00:41:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:03.715+0000 7fb930c00640 1 -- 172.21.15.31:0/2006377664 wait complete. 2024-09-11T00:41:03.737 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:41:05.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.912+0000 7fdfd9400640 1 -- 172.21.15.31:0/2061751767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfd40718b0 msgr2=0x7fdfd4074480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.912+0000 7fdfd9400640 1 --2- 172.21.15.31:0/2061751767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfd40718b0 0x7fdfd4074480 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009830 tx=0x7fdfc002a3f0 comp rx=0 tx=0).stop 2024-09-11T00:41:05.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.912+0000 7fdfd9400640 1 -- 172.21.15.31:0/2061751767 shutdown_connections 2024-09-11T00:41:05.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.912+0000 7fdfd9400640 1 --2- 172.21.15.31:0/2061751767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfd40718b0 0x7fdfd4074480 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:05.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.912+0000 7fdfd9400640 1 --2- 172.21.15.31:0/2061751767 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfd40c8cb0 0x7fdfd4070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:05.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.912+0000 7fdfd9400640 1 -- 172.21.15.31:0/2061751767 >> 172.21.15.31:0/2061751767 conn(0x7fdfd406dc00 msgr2=0x7fdfd406e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:05.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.912+0000 7fdfd9400640 1 -- 172.21.15.31:0/2061751767 shutdown_connections 2024-09-11T00:41:05.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.912+0000 7fdfd9400640 1 -- 172.21.15.31:0/2061751767 wait complete. 2024-09-11T00:41:05.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.914+0000 7fdfd9400640 1 Processor -- start 2024-09-11T00:41:05.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.914+0000 7fdfd9400640 1 -- start start 2024-09-11T00:41:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd9400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfd40718b0 0x7fdfd406cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd9400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfd40749c0 0x7fdfd41ddc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd9400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd40c8cb0 0x7fdfd41de1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd9400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd4153980 con 0x7fdfd40c8cb0 2024-09-11T00:41:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd9400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41536a0 con 0x7fdfd40749c0 2024-09-11T00:41:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd9400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd40c9090 con 0x7fdfd40718b0 2024-09-11T00:41:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd40c8cb0 0x7fdfd41de1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:05.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd40c8cb0 0x7fdfd41de1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46388/0 (socket says 172.21.15.31:46388) 2024-09-11T00:41:05.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd3200640 1 -- 172.21.15.31:0/4082316641 learned_addr learned my addr 172.21.15.31:0/4082316641 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:05.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd2800640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfd40718b0 0x7fdfd406cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:05.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.915+0000 7fdfd1e00640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfd40749c0 0x7fdfd41ddc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfd3200640 1 -- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfd40718b0 msgr2=0x7fdfd406cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfd3200640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfd40718b0 0x7fdfd406cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfd3200640 1 -- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfd40749c0 msgr2=0x7fdfd41ddc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfd3200640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfd40749c0 0x7fdfd41ddc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfd3200640 1 -- 172.21.15.31:0/4082316641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd40aa320 con 0x7fdfd40c8cb0 2024-09-11T00:41:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfd1e00640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfd40749c0 0x7fdfd41ddc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:41:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfd3200640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd40c8cb0 0x7fdfd41de1a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00cbc0 tx=0x7fdfcc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:05.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfcaa00640 1 -- 172.21.15.31:0/4082316641 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc016020 con 0x7fdfd40c8cb0 2024-09-11T00:41:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfcaa00640 1 -- 172.21.15.31:0/4082316641 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdfcc00add0 con 0x7fdfd40c8cb0 2024-09-11T00:41:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfcaa00640 1 -- 172.21.15.31:0/4082316641 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc012680 con 0x7fdfd40c8cb0 2024-09-11T00:41:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.916+0000 7fdfd9400640 1 -- 172.21.15.31:0/4082316641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd40aa600 con 0x7fdfd40c8cb0 2024-09-11T00:41:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.917+0000 7fdfd9400640 1 -- 172.21.15.31:0/4082316641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd406d2a0 con 0x7fdfd40c8cb0 2024-09-11T00:41:05.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.918+0000 7fdfd9400640 1 -- 172.21.15.31:0/4082316641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd4074480 con 0x7fdfd40c8cb0 2024-09-11T00:41:05.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.919+0000 7fdfcaa00640 1 -- 172.21.15.31:0/4082316641 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdfcc012820 con 0x7fdfd40c8cb0 2024-09-11T00:41:05.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.920+0000 7fdfcaa00640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfb4077f70 0x7fdfb407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:05.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.920+0000 7fdfcaa00640 1 -- 172.21.15.31:0/4082316641 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7fdfcc09ac50 con 0x7fdfd40c8cb0 2024-09-11T00:41:05.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.921+0000 7fdfd2800640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfb4077f70 0x7fdfb407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:05.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.922+0000 7fdfd2800640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfb4077f70 0x7fdfb407a430 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdfc40095f0 tx=0x7fdfc4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:05.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:05.925+0000 7fdfcaa00640 1 -- 172.21.15.31:0/4082316641 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdfcc020030 con 0x7fdfd40c8cb0 2024-09-11T00:41:06.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.188+0000 7fdfd9400640 1 -- 172.21.15.31:0/4082316641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfd40750c0 con 0x7fdfd40c8cb0 2024-09-11T00:41:06.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.188+0000 7fdfcaa00640 1 -- 172.21.15.31:0/4082316641 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6532 (secure 0 0 0) 0x7fdfcc062e90 con 0x7fdfd40c8cb0 2024-09-11T00:41:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:06.191 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"btime":"2024-09-11T00:40:50:638638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6087,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:41:06.192 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2024-09-11T00:41:06.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 -- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfb4077f70 msgr2=0x7fdfb407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:06.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfb4077f70 0x7fdfb407a430 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdfc40095f0 tx=0x7fdfc4005f90 comp rx=0 tx=0).stop 2024-09-11T00:41:06.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 -- 172.21.15.31:0/4082316641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd40c8cb0 msgr2=0x7fdfd41de1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:06.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd40c8cb0 0x7fdfd41de1a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00cbc0 tx=0x7fdfcc00a430 comp rx=0 tx=0).stop 2024-09-11T00:41:06.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 -- 172.21.15.31:0/4082316641 shutdown_connections 2024-09-11T00:41:06.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfb4077f70 0x7fdfb407a430 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:06.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfd40c8cb0 0x7fdfd41de1a0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfd40749c0 0x7fdfd41ddc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 --2- 172.21.15.31:0/4082316641 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfd40718b0 0x7fdfd406cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 -- 172.21.15.31:0/4082316641 >> 172.21.15.31:0/4082316641 conn(0x7fdfd406dc00 msgr2=0x7fdfd406dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 -- 172.21.15.31:0/4082316641 shutdown_connections 2024-09-11T00:41:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:06.191+0000 7fdfd9400640 1 -- 172.21.15.31:0/4082316641 wait complete. 2024-09-11T00:41:06.210 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:41:08.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:08.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.389+0000 7f5612a00640 1 -- 172.21.15.31:0/26803265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f560c0a7e40 msgr2=0x7f560c0a8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:08.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.389+0000 7f5612a00640 1 --2- 172.21.15.31:0/26803265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f560c0a7e40 0x7f560c0a8220 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009860 tx=0x7f55fc02a540 comp rx=0 tx=0).stop 2024-09-11T00:41:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.389+0000 7f5612a00640 1 -- 172.21.15.31:0/26803265 shutdown_connections 2024-09-11T00:41:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.389+0000 7f5612a00640 1 --2- 172.21.15.31:0/26803265 >> v2:172.21.15.204:3300/0 conn(0x7f560c0719b0 0x7f560c1566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.389+0000 7f5612a00640 1 --2- 172.21.15.31:0/26803265 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f560c0a7e40 0x7f560c0a8220 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.389+0000 7f5612a00640 1 -- 172.21.15.31:0/26803265 >> 172.21.15.31:0/26803265 conn(0x7f560c06dc00 msgr2=0x7f560c06e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.389+0000 7f5612a00640 1 -- 172.21.15.31:0/26803265 shutdown_connections 2024-09-11T00:41:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.389+0000 7f5612a00640 1 -- 172.21.15.31:0/26803265 wait complete. 2024-09-11T00:41:08.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.391+0000 7f5612a00640 1 Processor -- start 2024-09-11T00:41:08.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.391+0000 7f5612a00640 1 -- start start 2024-09-11T00:41:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f5612a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f560c070360 0x7f560c143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f5612a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f560c0719b0 0x7f560c0c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f5612a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f560c0a7e40 0x7f560c1e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f5612a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f560c155490 con 0x7f560c0a7e40 2024-09-11T00:41:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f5612a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f560c06cdb0 con 0x7f560c070360 2024-09-11T00:41:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f5612a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f560c0c2f30 con 0x7f560c0719b0 2024-09-11T00:41:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f5610a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f560c0a7e40 0x7f560c1e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:08.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f5610a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f560c0a7e40 0x7f560c1e9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34988/0 (socket says 172.21.15.31:34988) 2024-09-11T00:41:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f5610a00640 1 -- 172.21.15.31:0/2378830664 learned_addr learned my addr 172.21.15.31:0/2378830664 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f560b400640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f560c0719b0 0x7f560c0c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.392+0000 7f560be00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f560c070360 0x7f560c143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.393+0000 7f5610a00640 1 -- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f560c0719b0 msgr2=0x7f560c0c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.393+0000 7f5610a00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f560c0719b0 0x7f560c0c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.393+0000 7f5610a00640 1 -- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f560c070360 msgr2=0x7f560c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.393+0000 7f5610a00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f560c070360 0x7f560c143a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.393+0000 7f5610a00640 1 -- 172.21.15.31:0/2378830664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f560c1ed2f0 con 0x7f560c0a7e40 2024-09-11T00:41:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.393+0000 7f5610a00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f560c0a7e40 0x7f560c1e9dc0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f560400eb10 tx=0x7f560400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.393+0000 7f5603e00640 1 -- 172.21.15.31:0/2378830664 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604004b20 con 0x7f560c0a7e40 2024-09-11T00:41:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.393+0000 7f5603e00640 1 -- 172.21.15.31:0/2378830664 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5604004ce0 con 0x7f560c0a7e40 2024-09-11T00:41:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.394+0000 7f5612a00640 1 -- 172.21.15.31:0/2378830664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f560c1ed5d0 con 0x7f560c0a7e40 2024-09-11T00:41:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.394+0000 7f5603e00640 1 -- 172.21.15.31:0/2378830664 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604005280 con 0x7f560c0a7e40 2024-09-11T00:41:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.394+0000 7f5612a00640 1 -- 172.21.15.31:0/2378830664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f560c1edb90 con 0x7f560c0a7e40 2024-09-11T00:41:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.395+0000 7f5612a00640 1 -- 172.21.15.31:0/2378830664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f560c0a8220 con 0x7f560c0a7e40 2024-09-11T00:41:08.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.396+0000 7f5603e00640 1 -- 172.21.15.31:0/2378830664 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5604005420 con 0x7f560c0a7e40 2024-09-11T00:41:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.397+0000 7f5603e00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55ec077fa0 0x7f55ec07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.398+0000 7f5603e00640 1 -- 172.21.15.31:0/2378830664 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7784+0+0 (secure 0 0 0) 0x7f560409b230 con 0x7f560c0a7e40 2024-09-11T00:41:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.398+0000 7f560be00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55ec077fa0 0x7f55ec07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:08.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.399+0000 7f560be00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55ec077fa0 0x7f55ec07a460 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f55fc034000 tx=0x7f55fc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:08.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.403+0000 7f5603e00640 1 -- 172.21.15.31:0/2378830664 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5604062e30 con 0x7f560c0a7e40 2024-09-11T00:41:08.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.719+0000 7f5612a00640 1 -- 172.21.15.31:0/2378830664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f560c1445b0 con 0x7f560c0a7e40 2024-09-11T00:41:08.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.719+0000 7f5603e00640 1 -- 172.21.15.31:0/2378830664 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6532 (secure 0 0 0) 0x7f5604067860 con 0x7f560c0a7e40 2024-09-11T00:41:08.721 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:08.722 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"btime":"2024-09-11T00:40:50:638638+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6087,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:40:27.381668+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":33,"addr":"172.21.15.31:6835/3925687919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":3925687919},{"type":"v1","addr":"172.21.15.31:6835","nonce":3925687919}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-11T00:41:08.722 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 77 2024-09-11T00:41:08.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.722+0000 7f5612a00640 1 -- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55ec077fa0 msgr2=0x7f55ec07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:08.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.722+0000 7f5612a00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55ec077fa0 0x7f55ec07a460 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f55fc034000 tx=0x7f55fc004730 comp rx=0 tx=0).stop 2024-09-11T00:41:08.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.722+0000 7f5612a00640 1 -- 172.21.15.31:0/2378830664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f560c0a7e40 msgr2=0x7f560c1e9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:08.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.722+0000 7f5612a00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f560c0a7e40 0x7f560c1e9dc0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f560400eb10 tx=0x7f560400c2d0 comp rx=0 tx=0).stop 2024-09-11T00:41:08.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.722+0000 7f5612a00640 1 -- 172.21.15.31:0/2378830664 shutdown_connections 2024-09-11T00:41:08.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.722+0000 7f5612a00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55ec077fa0 0x7f55ec07a460 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:08.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.722+0000 7f5612a00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f560c0a7e40 0x7f560c1e9dc0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:08.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.722+0000 7f5612a00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f560c0719b0 0x7f560c0c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:08.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.722+0000 7f5612a00640 1 --2- 172.21.15.31:0/2378830664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f560c070360 0x7f560c143a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:08.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.722+0000 7f5612a00640 1 -- 172.21.15.31:0/2378830664 >> 172.21.15.31:0/2378830664 conn(0x7f560c06dc00 msgr2=0x7f560c149120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:08.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.723+0000 7f5612a00640 1 -- 172.21.15.31:0/2378830664 shutdown_connections 2024-09-11T00:41:08.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:08.723+0000 7f5612a00640 1 -- 172.21.15.31:0/2378830664 wait complete. 2024-09-11T00:41:08.742 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:41:10.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.917+0000 7f1c72200640 1 -- 172.21.15.31:0/3237159571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c6c150020 msgr2=0x7f1c6c0ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.917+0000 7f1c72200640 1 --2- 172.21.15.31:0/3237159571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c6c150020 0x7f1c6c0ad4e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c0b1b70 tx=0x7f1c6002a4d0 comp rx=0 tx=0).stop 2024-09-11T00:41:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.917+0000 7f1c72200640 1 -- 172.21.15.31:0/3237159571 shutdown_connections 2024-09-11T00:41:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.917+0000 7f1c72200640 1 --2- 172.21.15.31:0/3237159571 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c156960 0x7f1c6c06e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.917+0000 7f1c72200640 1 --2- 172.21.15.31:0/3237159571 >> v2:172.21.15.204:3300/0 conn(0x7f1c6c0ada20 0x7f1c6c06e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.917+0000 7f1c72200640 1 --2- 172.21.15.31:0/3237159571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c6c150020 0x7f1c6c0ad4e0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.917+0000 7f1c72200640 1 -- 172.21.15.31:0/3237159571 >> 172.21.15.31:0/3237159571 conn(0x7f1c6c068f10 msgr2=0x7f1c6c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.917+0000 7f1c72200640 1 -- 172.21.15.31:0/3237159571 shutdown_connections 2024-09-11T00:41:10.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.917+0000 7f1c72200640 1 -- 172.21.15.31:0/3237159571 wait complete. 2024-09-11T00:41:10.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.919+0000 7f1c72200640 1 Processor -- start 2024-09-11T00:41:10.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.919+0000 7f1c72200640 1 -- start start 2024-09-11T00:41:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.920+0000 7f1c72200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0ada20 0x7f1c6c152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.920+0000 7f1c72200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c6c150020 0x7f1c6c1efba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.920+0000 7f1c72200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c6c156960 0x7f1c6c0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.920+0000 7f1c72200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c157550 con 0x7f1c6c156960 2024-09-11T00:41:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.920+0000 7f1c72200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c1573d0 con 0x7f1c6c150020 2024-09-11T00:41:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.920+0000 7f1c72200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1c6c0cc220 con 0x7f1c6c0ada20 2024-09-11T00:41:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.921+0000 7f1c71800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c6c156960 0x7f1c6c0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:10.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.921+0000 7f1c70e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0ada20 0x7f1c6c152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.921+0000 7f1c70e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0ada20 0x7f1c6c152840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:44470/0 (socket says 172.21.15.31:44470) 2024-09-11T00:41:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.921+0000 7f1c70e00640 1 -- 172.21.15.31:0/3333979933 learned_addr learned my addr 172.21.15.31:0/3333979933 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.921+0000 7f1c6be00640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c6c150020 0x7f1c6c1efba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.921+0000 7f1c71800640 1 -- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0ada20 msgr2=0x7f1c6c152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.921+0000 7f1c71800640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0ada20 0x7f1c6c152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.921+0000 7f1c71800640 1 -- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c6c150020 msgr2=0x7f1c6c1efba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.921+0000 7f1c71800640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c6c150020 0x7f1c6c1efba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:10.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.921+0000 7f1c71800640 1 -- 172.21.15.31:0/3333979933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6c0c3550 con 0x7f1c6c156960 2024-09-11T00:41:10.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.922+0000 7f1c71800640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c6c156960 0x7f1c6c0c53f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400e940 tx=0x7f1c6400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:10.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.922+0000 7f1c68c00640 1 -- 172.21.15.31:0/3333979933 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64004d20 con 0x7f1c6c156960 2024-09-11T00:41:10.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.922+0000 7f1c68c00640 1 -- 172.21.15.31:0/3333979933 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c6400cb20 con 0x7f1c6c156960 2024-09-11T00:41:10.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.922+0000 7f1c68c00640 1 -- 172.21.15.31:0/3333979933 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64005020 con 0x7f1c6c156960 2024-09-11T00:41:10.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.922+0000 7f1c72200640 1 -- 172.21.15.31:0/3333979933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c6c1ecc90 con 0x7f1c6c156960 2024-09-11T00:41:10.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.922+0000 7f1c72200640 1 -- 172.21.15.31:0/3333979933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c6c152d80 con 0x7f1c6c156960 2024-09-11T00:41:10.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.924+0000 7f1c72200640 1 -- 172.21.15.31:0/3333979933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c6c0ad4e0 con 0x7f1c6c156960 2024-09-11T00:41:10.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.924+0000 7f1c68c00640 1 -- 172.21.15.31:0/3333979933 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1c6400f040 con 0x7f1c6c156960 2024-09-11T00:41:10.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.925+0000 7f1c68c00640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c4c077f70 0x7f1c4c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:10.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.925+0000 7f1c68c00640 1 -- 172.21.15.31:0/3333979933 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7f1c6409ab70 con 0x7f1c6c156960 2024-09-11T00:41:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.926+0000 7f1c70e00640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c4c077f70 0x7f1c4c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:10.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.927+0000 7f1c70e00640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c4c077f70 0x7f1c4c07a430 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1c600096c0 tx=0x7f1c60004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:10.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:10.930+0000 7f1c68c00640 1 -- 172.21.15.31:0/3333979933 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1c64016030 con 0x7f1c6c156960 2024-09-11T00:41:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.226+0000 7f1c72200640 1 -- 172.21.15.31:0/3333979933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c6c0cc8e0 con 0x7f1c6c156960 2024-09-11T00:41:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.226+0000 7f1c68c00640 1 -- 172.21.15.31:0/3333979933 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+5702 (secure 0 0 0) 0x7f1c64062710 con 0x7f1c6c156960 2024-09-11T00:41:11.229 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:11.229 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":79,"btime":"2024-09-11T00:41:10:697178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:10.529992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:41:11.229 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 79 2024-09-11T00:41:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 -- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c4c077f70 msgr2=0x7f1c4c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c4c077f70 0x7f1c4c07a430 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1c600096c0 tx=0x7f1c60004730 comp rx=0 tx=0).stop 2024-09-11T00:41:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 -- 172.21.15.31:0/3333979933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c6c156960 msgr2=0x7f1c6c0c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c6c156960 0x7f1c6c0c53f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400e940 tx=0x7f1c6400ee00 comp rx=0 tx=0).stop 2024-09-11T00:41:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 -- 172.21.15.31:0/3333979933 shutdown_connections 2024-09-11T00:41:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c4c077f70 0x7f1c4c07a430 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c6c156960 0x7f1c6c0c53f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c6c150020 0x7f1c6c1efba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 --2- 172.21.15.31:0/3333979933 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c6c0ada20 0x7f1c6c152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 -- 172.21.15.31:0/3333979933 >> 172.21.15.31:0/3333979933 conn(0x7f1c6c068f10 msgr2=0x7f1c6c14f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:11.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 -- 172.21.15.31:0/3333979933 shutdown_connections 2024-09-11T00:41:11.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.229+0000 7f1c72200640 1 -- 172.21.15.31:0/3333979933 wait complete. 2024-09-11T00:41:11.246 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-11T00:41:11.247 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:41:11.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:11.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.429+0000 7fddab200640 1 -- 172.21.15.31:0/1462883571 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdda406cd60 msgr2=0x7fdda40a7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:11.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.429+0000 7fddab200640 1 --2- 172.21.15.31:0/1462883571 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdda406cd60 0x7fdda40a7d20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009830 tx=0x7fdd9802a510 comp rx=0 tx=0).stop 2024-09-11T00:41:11.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.429+0000 7fddab200640 1 -- 172.21.15.31:0/1462883571 shutdown_connections 2024-09-11T00:41:11.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.429+0000 7fddab200640 1 --2- 172.21.15.31:0/1462883571 >> v2:172.21.15.31:3300/0 conn(0x7fdda4156150 0x7fdda4156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.429+0000 7fddab200640 1 --2- 172.21.15.31:0/1462883571 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdda406cd60 0x7fdda40a7d20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.429+0000 7fddab200640 1 -- 172.21.15.31:0/1462883571 >> 172.21.15.31:0/1462883571 conn(0x7fdda40aa760 msgr2=0x7fdda40aab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:11.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.429+0000 7fddab200640 1 -- 172.21.15.31:0/1462883571 shutdown_connections 2024-09-11T00:41:11.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.429+0000 7fddab200640 1 -- 172.21.15.31:0/1462883571 wait complete. 2024-09-11T00:41:11.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.431+0000 7fddab200640 1 Processor -- start 2024-09-11T00:41:11.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.431+0000 7fddab200640 1 -- start start 2024-09-11T00:41:11.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddab200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda406cd60 0x7fdda4155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:11.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddab200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdda40c68b0 0x7fdda4154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:11.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddab200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdda4156150 0x7fdda414dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:11.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddab200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdda4153cb0 con 0x7fdda4156150 2024-09-11T00:41:11.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddab200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdda40a8260 con 0x7fdda406cd60 2024-09-11T00:41:11.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddab200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdda4155ec0 con 0x7fdda40c68b0 2024-09-11T00:41:11.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddaa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdda4156150 0x7fdda414dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:11.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fdda9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda406cd60 0x7fdda4155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:11.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddaa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdda4156150 0x7fdda414dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35010/0 (socket says 172.21.15.31:35010) 2024-09-11T00:41:11.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fdda9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda406cd60 0x7fdda4155980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:38784/0 (socket says 172.21.15.31:38784) 2024-09-11T00:41:11.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddaa800640 1 -- 172.21.15.31:0/3147446166 learned_addr learned my addr 172.21.15.31:0/3147446166 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:11.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fdda9400640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdda40c68b0 0x7fdda4154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:11.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddaa800640 1 -- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdda40c68b0 msgr2=0x7fdda4154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:11.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddaa800640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdda40c68b0 0x7fdda4154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddaa800640 1 -- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda406cd60 msgr2=0x7fdda4155980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:11.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddaa800640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda406cd60 0x7fdda4155980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.433+0000 7fddaa800640 1 -- 172.21.15.31:0/3147446166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda41484f0 con 0x7fdda4156150 2024-09-11T00:41:11.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.434+0000 7fddaa800640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdda4156150 0x7fdda414dab0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fdda000e960 tx=0x7fdda000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:11.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.434+0000 7fdd9e000640 1 -- 172.21.15.31:0/3147446166 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0004d20 con 0x7fdda4156150 2024-09-11T00:41:11.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.434+0000 7fdd9e000640 1 -- 172.21.15.31:0/3147446166 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdda000cb20 con 0x7fdda4156150 2024-09-11T00:41:11.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.434+0000 7fdd9e000640 1 -- 172.21.15.31:0/3147446166 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0005020 con 0x7fdda4156150 2024-09-11T00:41:11.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.434+0000 7fddab200640 1 -- 172.21.15.31:0/3147446166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda41487d0 con 0x7fdda4156150 2024-09-11T00:41:11.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.434+0000 7fddab200640 1 -- 172.21.15.31:0/3147446166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda41492b0 con 0x7fdda4156150 2024-09-11T00:41:11.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.435+0000 7fddab200640 1 -- 172.21.15.31:0/3147446166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdda406d220 con 0x7fdda4156150 2024-09-11T00:41:11.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.437+0000 7fdd9e000640 1 -- 172.21.15.31:0/3147446166 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdda000f040 con 0x7fdda4156150 2024-09-11T00:41:11.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.438+0000 7fdd9e000640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdd88077f70 0x7fdd8807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:11.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.438+0000 7fdd9e000640 1 -- 172.21.15.31:0/3147446166 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7fdda009ab30 con 0x7fdda4156150 2024-09-11T00:41:11.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.438+0000 7fdda9e00640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdd88077f70 0x7fdd8807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:11.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.439+0000 7fdda9e00640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdd88077f70 0x7fdd8807a430 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd98034000 tx=0x7fdd98004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:11.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.442+0000 7fdd9e000640 1 -- 172.21.15.31:0/3147446166 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdda0016030 con 0x7fdda4156150 2024-09-11T00:41:11.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.699+0000 7fddab200640 1 -- 172.21.15.31:0/3147446166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdda4150e10 con 0x7fdda4156150 2024-09-11T00:41:11.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.700+0000 7fdd9e000640 1 -- 172.21.15.31:0/3147446166 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+5702 (secure 0 0 0) 0x7fdda0062750 con 0x7fdda4156150 2024-09-11T00:41:11.702 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:11.702 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":79,"btime":"2024-09-11T00:41:10:697178+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:10.529992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:41:11.702 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 79 2024-09-11T00:41:11.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.702+0000 7fddab200640 1 -- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdd88077f70 msgr2=0x7fdd8807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:11.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.702+0000 7fddab200640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdd88077f70 0x7fdd8807a430 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd98034000 tx=0x7fdd98004730 comp rx=0 tx=0).stop 2024-09-11T00:41:11.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.702+0000 7fddab200640 1 -- 172.21.15.31:0/3147446166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdda4156150 msgr2=0x7fdda414dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.702+0000 7fddab200640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdda4156150 0x7fdda414dab0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fdda000e960 tx=0x7fdda000ee20 comp rx=0 tx=0).stop 2024-09-11T00:41:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.702+0000 7fddab200640 1 -- 172.21.15.31:0/3147446166 shutdown_connections 2024-09-11T00:41:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.702+0000 7fddab200640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdd88077f70 0x7fdd8807a430 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.702+0000 7fddab200640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdda4156150 0x7fdda414dab0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.702+0000 7fddab200640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdda40c68b0 0x7fdda4154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.702+0000 7fddab200640 1 --2- 172.21.15.31:0/3147446166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda406cd60 0x7fdda4155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.702+0000 7fddab200640 1 -- 172.21.15.31:0/3147446166 >> 172.21.15.31:0/3147446166 conn(0x7fdda40aa760 msgr2=0x7fdda40c7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.703+0000 7fddab200640 1 -- 172.21.15.31:0/3147446166 shutdown_connections 2024-09-11T00:41:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:11.703+0000 7fddab200640 1 -- 172.21.15.31:0/3147446166 wait complete. 2024-09-11T00:41:11.726 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:41:10:697178+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 76, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5911": { "addr": "172.21.15.204:6835/98007221", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 98007221, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 98007221, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5911, "incarnation": 73, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:40:42.621929+0000", "qdb_cluster": [ 5911 ], "qdb_leader": 5911, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5911 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6087": { "addr": "172.21.15.31:6834/2909466339", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2909466339, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 2909466339, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6087, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:41:10.529992+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6087 } } } ], "standbys": [ { "addr": "172.21.15.204:6837/3513641573", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 3513641573, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 3513641573, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5773, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:41:13.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:13.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.900+0000 7f976be00640 1 -- 172.21.15.31:0/3822684308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f976c0c5aa0 msgr2=0x7f976c0c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:13.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.900+0000 7f976be00640 1 --2- 172.21.15.31:0/3822684308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f976c0c5aa0 0x7f976c0c7150 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f976c06df10 tx=0x7f975c02a4d0 comp rx=0 tx=0).stop 2024-09-11T00:41:13.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.901+0000 7f976be00640 1 -- 172.21.15.31:0/3822684308 shutdown_connections 2024-09-11T00:41:13.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.901+0000 7f976be00640 1 --2- 172.21.15.31:0/3822684308 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f976c153850 0x7f976c156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:13.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.901+0000 7f976be00640 1 --2- 172.21.15.31:0/3822684308 >> v2:172.21.15.204:3300/0 conn(0x7f976c0c6420 0x7f976c153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:13.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.901+0000 7f976be00640 1 --2- 172.21.15.31:0/3822684308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f976c0c5aa0 0x7f976c0c7150 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:13.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.901+0000 7f976be00640 1 -- 172.21.15.31:0/3822684308 >> 172.21.15.31:0/3822684308 conn(0x7f976c06caf0 msgr2=0x7f976c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.901+0000 7f976be00640 1 -- 172.21.15.31:0/3822684308 shutdown_connections 2024-09-11T00:41:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.901+0000 7f976be00640 1 -- 172.21.15.31:0/3822684308 wait complete. 2024-09-11T00:41:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.902+0000 7f976be00640 1 Processor -- start 2024-09-11T00:41:13.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.902+0000 7f976be00640 1 -- start start 2024-09-11T00:41:13.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.903+0000 7f976be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f976c0c5aa0 0x7f976c148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.903+0000 7f976be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f976c0c6420 0x7f976c149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.903+0000 7f976be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f976c153850 0x7f976c0ac200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.903+0000 7f976be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f976c0c2e80 con 0x7f976c153850 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.903+0000 7f976be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f976c0c2ba0 con 0x7f976c0c5aa0 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.903+0000 7f976be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f976c0c5e80 con 0x7f976c0c6420 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f976c0c5aa0 0x7f976c148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f976c0c6420 0x7f976c149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f976c0c5aa0 0x7f976c148e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:38794/0 (socket says 172.21.15.31:38794) 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976aa00640 1 -- 172.21.15.31:0/1605746568 learned_addr learned my addr 172.21.15.31:0/1605746568 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976b400640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f976c153850 0x7f976c0ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:13.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976a000640 1 -- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f976c0c5aa0 msgr2=0x7f976c148e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:13.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976a000640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f976c0c5aa0 0x7f976c148e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:13.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976a000640 1 -- 172.21.15.31:0/1605746568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f976c153850 msgr2=0x7f976c0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:13.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976a000640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f976c153850 0x7f976c0ac200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:13.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976a000640 1 -- 172.21.15.31:0/1605746568 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976c0aa800 con 0x7f976c0c6420 2024-09-11T00:41:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.904+0000 7f976a000640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f976c0c6420 0x7f976c149460 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f975800ea20 tx=0x7f975800eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.905+0000 7f9762a00640 1 -- 172.21.15.31:0/1605746568 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975800ccc0 con 0x7f976c0c6420 2024-09-11T00:41:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.905+0000 7f9762a00640 1 -- 172.21.15.31:0/1605746568 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9758010430 con 0x7f976c0c6420 2024-09-11T00:41:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.905+0000 7f976be00640 1 -- 172.21.15.31:0/1605746568 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976c0aaae0 con 0x7f976c0c6420 2024-09-11T00:41:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.905+0000 7f9762a00640 1 -- 172.21.15.31:0/1605746568 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758010840 con 0x7f976c0c6420 2024-09-11T00:41:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.905+0000 7f976be00640 1 -- 172.21.15.31:0/1605746568 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f976c14b620 con 0x7f976c0c6420 2024-09-11T00:41:13.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.906+0000 7f976be00640 1 -- 172.21.15.31:0/1605746568 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f976c0aad20 con 0x7f976c0c6420 2024-09-11T00:41:13.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.908+0000 7f9762a00640 1 -- 172.21.15.31:0/1605746568 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f97580109e0 con 0x7f976c0c6420 2024-09-11T00:41:13.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.909+0000 7f9762a00640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f974c077f70 0x7f974c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.909+0000 7f9762a00640 1 -- 172.21.15.31:0/1605746568 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7f975809afe0 con 0x7f976c0c6420 2024-09-11T00:41:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.909+0000 7f976aa00640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f974c077f70 0x7f974c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:13.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.910+0000 7f976aa00640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f974c077f70 0x7f974c07a430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f976c149eb0 tx=0x7f975c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:13.915+0000 7f9762a00640 1 -- 172.21.15.31:0/1605746568 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9758062c20 con 0x7f976c0c6420 2024-09-11T00:41:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.163+0000 7f976be00640 1 -- 172.21.15.31:0/1605746568 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f976c156bb0 con 0x7f976c0c6420 2024-09-11T00:41:14.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.164+0000 7f9762a00640 1 -- 172.21.15.31:0/1605746568 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5706 (secure 0 0 0) 0x7f9758067650 con 0x7f976c0c6420 2024-09-11T00:41:14.166 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:14.166 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"btime":"2024-09-11T00:41:12:709313+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:14.166 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 81 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.166+0000 7f976be00640 1 -- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f974c077f70 msgr2=0x7f974c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.166+0000 7f976be00640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f974c077f70 0x7f974c07a430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f976c149eb0 tx=0x7f975c004730 comp rx=0 tx=0).stop 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.167+0000 7f976be00640 1 -- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f976c0c6420 msgr2=0x7f976c149460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.167+0000 7f976be00640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f976c0c6420 0x7f976c149460 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f975800ea20 tx=0x7f975800eee0 comp rx=0 tx=0).stop 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.167+0000 7f976be00640 1 -- 172.21.15.31:0/1605746568 shutdown_connections 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.167+0000 7f976be00640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f974c077f70 0x7f974c07a430 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.167+0000 7f976be00640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f976c153850 0x7f976c0ac200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.167+0000 7f976be00640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f976c0c6420 0x7f976c149460 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.167+0000 7f976be00640 1 --2- 172.21.15.31:0/1605746568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f976c0c5aa0 0x7f976c148e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.167+0000 7f976be00640 1 -- 172.21.15.31:0/1605746568 >> 172.21.15.31:0/1605746568 conn(0x7f976c06caf0 msgr2=0x7f976c0a9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.167+0000 7f976be00640 1 -- 172.21.15.31:0/1605746568 shutdown_connections 2024-09-11T00:41:14.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:14.167+0000 7f976be00640 1 -- 172.21.15.31:0/1605746568 wait complete. 2024-09-11T00:41:14.188 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5901 2024-09-11T00:41:14.189 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.a 2024-09-11T00:41:15.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:41:15.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:41:15.653 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-11T00:41:15.651+0000 7fc647e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:41:17.190 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-11T00:41:17.190 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-11T00:41:17.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-11T00:41:17.195 INFO:tasks.ceph.mds.a:Started 2024-09-11T00:41:17.195 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:17.352 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-11T00:41:17.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.373+0000 7feaaf200640 1 -- 172.21.15.31:0/2779450462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feab0156350 msgr2=0x7feab01567c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:17.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.373+0000 7feaaf200640 1 --2- 172.21.15.31:0/2779450462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feab0156350 0x7feab01567c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7feaa8005fd0 tx=0x7feaa802a940 comp rx=0 tx=0).stop 2024-09-11T00:41:17.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.373+0000 7feaaf200640 1 -- 172.21.15.31:0/2779450462 shutdown_connections 2024-09-11T00:41:17.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.373+0000 7feaaf200640 1 --2- 172.21.15.31:0/2779450462 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feab0156350 0x7feab01567c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:17.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.373+0000 7feaaf200640 1 -- 172.21.15.31:0/2779450462 >> 172.21.15.31:0/2779450462 conn(0x7feab006caf0 msgr2=0x7feab006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:17.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.374+0000 7feaaf200640 1 -- 172.21.15.31:0/2779450462 shutdown_connections 2024-09-11T00:41:17.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.374+0000 7feaaf200640 1 -- 172.21.15.31:0/2779450462 wait complete. 2024-09-11T00:41:17.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.375+0000 7feaaf200640 1 Processor -- start 2024-09-11T00:41:17.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.375+0000 7feaaf200640 1 -- start start 2024-09-11T00:41:17.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.376+0000 7feaaf200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feab00c5aa0 0x7feab01e0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:17.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.376+0000 7feaaf200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00c6420 0x7feab01df900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:17.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaaf200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feab0156350 0x7feab014df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:17.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaaf200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feab0153a20 con 0x7feab00c6420 2024-09-11T00:41:17.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaaf200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feab00c3160 con 0x7feab00c5aa0 2024-09-11T00:41:17.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaaf200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7feab0153ba0 con 0x7feab0156350 2024-09-11T00:41:17.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaad400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00c6420 0x7feab01df900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:17.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaade00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feab00c5aa0 0x7feab01e0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:17.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaae800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feab0156350 0x7feab014df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:17.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaade00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feab00c5aa0 0x7feab01e0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:38834/0 (socket says 172.21.15.31:38834) 2024-09-11T00:41:17.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaade00640 1 -- 172.21.15.31:0/1187527142 learned_addr learned my addr 172.21.15.31:0/1187527142 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:17.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaade00640 1 -- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feab0156350 msgr2=0x7feab014df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:17.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaade00640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feab0156350 0x7feab014df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:17.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaade00640 1 -- 172.21.15.31:0/1187527142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00c6420 msgr2=0x7feab01df900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:17.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaade00640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00c6420 0x7feab01df900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:17.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaade00640 1 -- 172.21.15.31:0/1187527142 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab00750a0 con 0x7feab00c5aa0 2024-09-11T00:41:17.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaae800640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feab0156350 0x7feab014df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:41:17.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.377+0000 7feaad400640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00c6420 0x7feab01df900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:41:17.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.378+0000 7feaade00640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feab00c5aa0 0x7feab01e0590 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7feaa000c490 tx=0x7feaa000c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:17.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.378+0000 7feaa6000640 1 -- 172.21.15.31:0/1187527142 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0016020 con 0x7feab00c5aa0 2024-09-11T00:41:17.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.378+0000 7feaa6000640 1 -- 172.21.15.31:0/1187527142 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feaa0012940 con 0x7feab00c5aa0 2024-09-11T00:41:17.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.378+0000 7feaa6000640 1 -- 172.21.15.31:0/1187527142 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0012d90 con 0x7feab00c5aa0 2024-09-11T00:41:17.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.378+0000 7feaaf200640 1 -- 172.21.15.31:0/1187527142 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab014ed10 con 0x7feab00c5aa0 2024-09-11T00:41:17.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.378+0000 7feaaf200640 1 -- 172.21.15.31:0/1187527142 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab01e0ad0 con 0x7feab00c5aa0 2024-09-11T00:41:17.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.380+0000 7feaaf200640 1 -- 172.21.15.31:0/1187527142 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab0156de0 con 0x7feab00c5aa0 2024-09-11T00:41:17.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.381+0000 7feaa6000640 1 -- 172.21.15.31:0/1187527142 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7feaa001b400 con 0x7feab00c5aa0 2024-09-11T00:41:17.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.382+0000 7feaa6000640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea90077f70 0x7fea9007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:17.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.382+0000 7feaa6000640 1 -- 172.21.15.31:0/1187527142 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7feaa009a7a0 con 0x7feab00c5aa0 2024-09-11T00:41:17.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.382+0000 7feaad400640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea90077f70 0x7fea9007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:17.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.383+0000 7feaad400640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea90077f70 0x7fea9007a430 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fea9c007fd0 tx=0x7fea9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:17.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.386+0000 7feaa6000640 1 -- 172.21.15.31:0/1187527142 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7feaa0020020 con 0x7feab00c5aa0 2024-09-11T00:41:17.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.683+0000 7feaaf200640 1 -- 172.21.15.31:0/1187527142 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feab014eee0 con 0x7feab00c5aa0 2024-09-11T00:41:17.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.683+0000 7feaa6000640 1 -- 172.21.15.31:0/1187527142 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5706 (secure 0 0 0) 0x7feaa006b070 con 0x7feab00c5aa0 2024-09-11T00:41:17.685 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:17.685 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":81,"btime":"2024-09-11T00:41:12:709313+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:17.686 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 81 2024-09-11T00:41:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 -- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea90077f70 msgr2=0x7fea9007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea90077f70 0x7fea9007a430 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fea9c007fd0 tx=0x7fea9c009040 comp rx=0 tx=0).stop 2024-09-11T00:41:17.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 -- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feab00c5aa0 msgr2=0x7feab01e0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feab00c5aa0 0x7feab01e0590 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7feaa000c490 tx=0x7feaa000c950 comp rx=0 tx=0).stop 2024-09-11T00:41:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 -- 172.21.15.31:0/1187527142 shutdown_connections 2024-09-11T00:41:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea90077f70 0x7fea9007a430 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feab0156350 0x7feab014df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feab00c6420 0x7feab01df900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 --2- 172.21.15.31:0/1187527142 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feab00c5aa0 0x7feab01e0590 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 -- 172.21.15.31:0/1187527142 >> 172.21.15.31:0/1187527142 conn(0x7feab006caf0 msgr2=0x7feab00c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 -- 172.21.15.31:0/1187527142 shutdown_connections 2024-09-11T00:41:17.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:17.686+0000 7feaaf200640 1 -- 172.21.15.31:0/1187527142 wait complete. 2024-09-11T00:41:17.706 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-11T00:41:19.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.883+0000 7fcae5800640 1 -- 172.21.15.31:0/2496025800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae00c67f0 msgr2=0x7fcae00bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.883+0000 7fcae5800640 1 --2- 172.21.15.31:0/2496025800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae00c67f0 0x7fcae00bf600 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009860 tx=0x7fcad002a510 comp rx=0 tx=0).stop 2024-09-11T00:41:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.883+0000 7fcae5800640 1 -- 172.21.15.31:0/2496025800 shutdown_connections 2024-09-11T00:41:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.883+0000 7fcae5800640 1 --2- 172.21.15.31:0/2496025800 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae00c67f0 0x7fcae00bf600 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.883+0000 7fcae5800640 1 -- 172.21.15.31:0/2496025800 >> 172.21.15.31:0/2496025800 conn(0x7fcae00c9af0 msgr2=0x7fcae00c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.883+0000 7fcae5800640 1 -- 172.21.15.31:0/2496025800 shutdown_connections 2024-09-11T00:41:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.884+0000 7fcae5800640 1 -- 172.21.15.31:0/2496025800 wait complete. 2024-09-11T00:41:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.885+0000 7fcae5800640 1 Processor -- start 2024-09-11T00:41:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.885+0000 7fcae5800640 1 -- start start 2024-09-11T00:41:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.886+0000 7fcae5800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae00a8bb0 0x7fcae00a96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.886+0000 7fcae5800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae00c67f0 0x7fcae01ec420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.886+0000 7fcae5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae01578e0 0x7fcae01eb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.886+0000 7fcae5800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcae00a9260 con 0x7fcae01578e0 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.886+0000 7fcae5800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcae00a90e0 con 0x7fcae00a8bb0 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.886+0000 7fcae5800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcae00c5b20 con 0x7fcae00c67f0 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.887+0000 7fcae4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae01578e0 0x7fcae01eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.887+0000 7fcadfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae00a8bb0 0x7fcae00a96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.887+0000 7fcae4e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae01578e0 0x7fcae01eb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42988/0 (socket says 172.21.15.31:42988) 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.887+0000 7fcae4e00640 1 -- 172.21.15.31:0/1083313209 learned_addr learned my addr 172.21.15.31:0/1083313209 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.887+0000 7fcadf400640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae00c67f0 0x7fcae01ec420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.887+0000 7fcae4e00640 1 -- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae00c67f0 msgr2=0x7fcae01ec420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.887+0000 7fcae4e00640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae00c67f0 0x7fcae01ec420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.887+0000 7fcae4e00640 1 -- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae00a8bb0 msgr2=0x7fcae00a96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.887+0000 7fcae4e00640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae00a8bb0 0x7fcae00a96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.887+0000 7fcae4e00640 1 -- 172.21.15.31:0/1083313209 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae00a9c20 con 0x7fcae01578e0 2024-09-11T00:41:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.888+0000 7fcae4e00640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae01578e0 0x7fcae01eb700 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcad800c730 tx=0x7fcad800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.888+0000 7fcad7e00640 1 -- 172.21.15.31:0/1083313209 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad8003b70 con 0x7fcae01578e0 2024-09-11T00:41:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.888+0000 7fcad7e00640 1 -- 172.21.15.31:0/1083313209 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcad8003d10 con 0x7fcae01578e0 2024-09-11T00:41:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.888+0000 7fcae5800640 1 -- 172.21.15.31:0/1083313209 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae01ecc60 con 0x7fcae01578e0 2024-09-11T00:41:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.888+0000 7fcad7e00640 1 -- 172.21.15.31:0/1083313209 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad800aab0 con 0x7fcae01578e0 2024-09-11T00:41:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.888+0000 7fcae5800640 1 -- 172.21.15.31:0/1083313209 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae01ecf00 con 0x7fcae01578e0 2024-09-11T00:41:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.889+0000 7fcae5800640 1 -- 172.21.15.31:0/1083313209 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae00c6bd0 con 0x7fcae01578e0 2024-09-11T00:41:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.891+0000 7fcad7e00640 1 -- 172.21.15.31:0/1083313209 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fcad800ac50 con 0x7fcae01578e0 2024-09-11T00:41:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.892+0000 7fcad7e00640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcac0077f70 0x7fcac007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.892+0000 7fcad7e00640 1 -- 172.21.15.31:0/1083313209 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7fcad809a2b0 con 0x7fcae01578e0 2024-09-11T00:41:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.892+0000 7fcadfe00640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcac0077f70 0x7fcac007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:19.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.893+0000 7fcadfe00640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcac0077f70 0x7fcac007a430 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcad0034000 tx=0x7fcad0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:19.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:19.897+0000 7fcad7e00640 1 -- 172.21.15.31:0/1083313209 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcae00a9c20 con 0x7fcae01578e0 2024-09-11T00:41:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.193+0000 7fcae5800640 1 -- 172.21.15.31:0/1083313209 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcae00c7a60 con 0x7fcae01578e0 2024-09-11T00:41:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.194+0000 7fcad7e00640 1 -- 172.21.15.31:0/1083313209 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6531 (secure 0 0 0) 0x7fcad8061ee0 con 0x7fcae01578e0 2024-09-11T00:41:20.196 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:20.196 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-11T00:41:17:726701+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-11T00:41:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.196+0000 7fcae5800640 1 -- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcac0077f70 msgr2=0x7fcac007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.196+0000 7fcae5800640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcac0077f70 0x7fcac007a430 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcad0034000 tx=0x7fcad0004730 comp rx=0 tx=0).stop 2024-09-11T00:41:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.196+0000 7fcae5800640 1 -- 172.21.15.31:0/1083313209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae01578e0 msgr2=0x7fcae01eb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.196+0000 7fcae5800640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae01578e0 0x7fcae01eb700 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcad800c730 tx=0x7fcad800cbf0 comp rx=0 tx=0).stop 2024-09-11T00:41:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.196+0000 7fcae5800640 1 -- 172.21.15.31:0/1083313209 shutdown_connections 2024-09-11T00:41:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.196+0000 7fcae5800640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcac0077f70 0x7fcac007a430 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.196+0000 7fcae5800640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcae01578e0 0x7fcae01eb700 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.196+0000 7fcae5800640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcae00c67f0 0x7fcae01ec420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.197+0000 7fcae5800640 1 --2- 172.21.15.31:0/1083313209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae00a8bb0 0x7fcae00a96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:20.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.197+0000 7fcae5800640 1 -- 172.21.15.31:0/1083313209 >> 172.21.15.31:0/1083313209 conn(0x7fcae00c9af0 msgr2=0x7fcae00702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:20.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.197+0000 7fcae5800640 1 -- 172.21.15.31:0/1083313209 shutdown_connections 2024-09-11T00:41:20.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:20.197+0000 7fcae5800640 1 -- 172.21.15.31:0/1083313209 wait complete. 2024-09-11T00:41:20.215 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-11T00:41:20.215 INFO:tasks.mds_thrash.fs.[b]:waiting for 111 secs before thrashing 2024-09-11T00:41:36.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:37.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.108+0000 7f45f3200640 1 -- 172.21.15.31:0/2593614772 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45f40c6bb0 msgr2=0x7f45f406e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:37.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.108+0000 7f45f3200640 1 --2- 172.21.15.31:0/2593614772 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45f40c6bb0 0x7f45f406e070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f45e4009830 tx=0x7f45e402a3f0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.108+0000 7f45f3200640 1 -- 172.21.15.31:0/2593614772 shutdown_connections 2024-09-11T00:41:37.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.108+0000 7f45f3200640 1 --2- 172.21.15.31:0/2593614772 >> v2:172.21.15.204:3300/0 conn(0x7f45f40c3280 0x7f45f4153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.108+0000 7f45f3200640 1 --2- 172.21.15.31:0/2593614772 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45f40c6bb0 0x7f45f406e070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.108+0000 7f45f3200640 1 -- 172.21.15.31:0/2593614772 >> 172.21.15.31:0/2593614772 conn(0x7f45f406caf0 msgr2=0x7f45f406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:37.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.108+0000 7f45f3200640 1 -- 172.21.15.31:0/2593614772 shutdown_connections 2024-09-11T00:41:37.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.108+0000 7f45f3200640 1 -- 172.21.15.31:0/2593614772 wait complete. 2024-09-11T00:41:37.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.110+0000 7f45f3200640 1 Processor -- start 2024-09-11T00:41:37.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.110+0000 7f45f3200640 1 -- start start 2024-09-11T00:41:37.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f40c3280 0x7f45f4074970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:37.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f3200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45f40c6bb0 0x7f45f414d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:37.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45f41539b0 0x7f45f40c0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:37.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f45f4152d80 con 0x7f45f40c3280 2024-09-11T00:41:37.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f45f406e5b0 con 0x7f45f41539b0 2024-09-11T00:41:37.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f3200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f45f4152f00 con 0x7f45f40c6bb0 2024-09-11T00:41:37.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f40c3280 0x7f45f4074970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:37.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f40c3280 0x7f45f4074970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41862/0 (socket says 172.21.15.31:41862) 2024-09-11T00:41:37.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f1e00640 1 -- 172.21.15.31:0/4060471577 learned_addr learned my addr 172.21.15.31:0/4060471577 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:37.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f1400640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45f40c6bb0 0x7f45f414d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:37.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.111+0000 7f45f2800640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45f41539b0 0x7f45f40c0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:37.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.112+0000 7f45f1e00640 1 -- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45f40c6bb0 msgr2=0x7f45f414d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:37.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.112+0000 7f45f1e00640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45f40c6bb0 0x7f45f414d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.112+0000 7f45f1e00640 1 -- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45f41539b0 msgr2=0x7f45f40c0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:37.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.112+0000 7f45f1e00640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45f41539b0 0x7f45f40c0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.112+0000 7f45f1e00640 1 -- 172.21.15.31:0/4060471577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f40ab450 con 0x7f45f40c3280 2024-09-11T00:41:37.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.112+0000 7f45f1e00640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f40c3280 0x7f45f4074970 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f45e4000c00 tx=0x7f45e4032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:37.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.112+0000 7f45ea000640 1 -- 172.21.15.31:0/4060471577 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4040070 con 0x7f45f40c3280 2024-09-11T00:41:37.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.113+0000 7f45ea000640 1 -- 172.21.15.31:0/4060471577 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45e40038e0 con 0x7f45f40c3280 2024-09-11T00:41:37.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.113+0000 7f45ea000640 1 -- 172.21.15.31:0/4060471577 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4005c90 con 0x7f45f40c3280 2024-09-11T00:41:37.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.113+0000 7f45f3200640 1 -- 172.21.15.31:0/4060471577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f40ac1d0 con 0x7f45f40c3280 2024-09-11T00:41:37.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.113+0000 7f45f3200640 1 -- 172.21.15.31:0/4060471577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f41ec770 con 0x7f45f40c3280 2024-09-11T00:41:37.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.115+0000 7f45f3200640 1 -- 172.21.15.31:0/4060471577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f40c7100 con 0x7f45f40c3280 2024-09-11T00:41:37.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.116+0000 7f45ea000640 1 -- 172.21.15.31:0/4060471577 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f45e4045460 con 0x7f45f40c3280 2024-09-11T00:41:37.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.117+0000 7f45ea000640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45d4077f70 0x7f45d407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:37.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.117+0000 7f45ea000640 1 -- 172.21.15.31:0/4060471577 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7f45e40bbd90 con 0x7f45f40c3280 2024-09-11T00:41:37.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.117+0000 7f45f1400640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45d4077f70 0x7f45d407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:37.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.118+0000 7f45f1400640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45d4077f70 0x7f45d407a430 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f45e00095f0 tx=0x7f45e0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:37.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.121+0000 7f45ea000640 1 -- 172.21.15.31:0/4060471577 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f45e4083a60 con 0x7f45f40c3280 2024-09-11T00:41:37.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.371+0000 7f45f3200640 1 -- 172.21.15.31:0/4060471577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45f40c4150 con 0x7f45f40c3280 2024-09-11T00:41:37.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.371+0000 7f45ea000640 1 -- 172.21.15.31:0/4060471577 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6531 (secure 0 0 0) 0x7f45e4088490 con 0x7f45f40c3280 2024-09-11T00:41:37.375 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:37.375 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-11T00:41:17:726701+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-11T00:41:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.374+0000 7f45f3200640 1 -- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45d4077f70 msgr2=0x7f45d407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.374+0000 7f45f3200640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45d4077f70 0x7f45d407a430 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f45e00095f0 tx=0x7f45e0005f90 comp rx=0 tx=0).stop 2024-09-11T00:41:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.374+0000 7f45f3200640 1 -- 172.21.15.31:0/4060471577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f40c3280 msgr2=0x7f45f4074970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.374+0000 7f45f3200640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f40c3280 0x7f45f4074970 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f45e4000c00 tx=0x7f45e4032c10 comp rx=0 tx=0).stop 2024-09-11T00:41:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.374+0000 7f45f3200640 1 -- 172.21.15.31:0/4060471577 shutdown_connections 2024-09-11T00:41:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.374+0000 7f45f3200640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45d4077f70 0x7f45d407a430 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.374+0000 7f45f3200640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45f41539b0 0x7f45f40c0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.374+0000 7f45f3200640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45f40c6bb0 0x7f45f414d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.374+0000 7f45f3200640 1 --2- 172.21.15.31:0/4060471577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f40c3280 0x7f45f4074970 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.374+0000 7f45f3200640 1 -- 172.21.15.31:0/4060471577 >> 172.21.15.31:0/4060471577 conn(0x7f45f406caf0 msgr2=0x7f45f4150df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.375+0000 7f45f3200640 1 -- 172.21.15.31:0/4060471577 shutdown_connections 2024-09-11T00:41:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.375+0000 7f45f3200640 1 -- 172.21.15.31:0/4060471577 wait complete. 2024-09-11T00:41:37.394 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-11T00:41:37.394 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:41:37.394 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:41:37.411 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:41:37.412 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:37.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.586+0000 7f90b7400640 1 -- 172.21.15.31:0/4238867653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90b0156300 msgr2=0x7f90b00c5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:37.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.586+0000 7f90b7400640 1 --2- 172.21.15.31:0/4238867653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90b0156300 0x7f90b00c5b40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f90a4009900 tx=0x7f90a402a580 comp rx=0 tx=0).stop 2024-09-11T00:41:37.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.586+0000 7f90b7400640 1 -- 172.21.15.31:0/4238867653 shutdown_connections 2024-09-11T00:41:37.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.586+0000 7f90b7400640 1 --2- 172.21.15.31:0/4238867653 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90b006fa70 0x7f90b01566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.586+0000 7f90b7400640 1 --2- 172.21.15.31:0/4238867653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90b0156300 0x7f90b00c5b40 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.586+0000 7f90b7400640 1 -- 172.21.15.31:0/4238867653 >> 172.21.15.31:0/4238867653 conn(0x7f90b00a9750 msgr2=0x7f90b00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:37.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.586+0000 7f90b7400640 1 -- 172.21.15.31:0/4238867653 shutdown_connections 2024-09-11T00:41:37.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.586+0000 7f90b7400640 1 -- 172.21.15.31:0/4238867653 wait complete. 2024-09-11T00:41:37.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.588+0000 7f90b7400640 1 Processor -- start 2024-09-11T00:41:37.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.588+0000 7f90b7400640 1 -- start start 2024-09-11T00:41:37.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b7400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90b006fa70 0x7f90b00bcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:37.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b7400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90b0156300 0x7f90b014e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:37.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90b0156c20 0x7f90b014abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:37.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b7400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f90b00c77a0 con 0x7f90b0156c20 2024-09-11T00:41:37.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b7400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f90b00c7620 con 0x7f90b0156300 2024-09-11T00:41:37.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b7400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f90b014d040 con 0x7f90b006fa70 2024-09-11T00:41:37.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90b0156c20 0x7f90b014abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:37.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90b0156c20 0x7f90b014abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41870/0 (socket says 172.21.15.31:41870) 2024-09-11T00:41:37.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b5400640 1 -- 172.21.15.31:0/854552572 learned_addr learned my addr 172.21.15.31:0/854552572 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:37.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90afe00640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90b0156300 0x7f90b014e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:37.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b4a00640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90b006fa70 0x7f90b00bcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:37.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b5400640 1 -- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90b006fa70 msgr2=0x7f90b00bcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:37.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b5400640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90b006fa70 0x7f90b00bcfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b5400640 1 -- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90b0156300 msgr2=0x7f90b014e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:37.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b5400640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90b0156300 0x7f90b014e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.589+0000 7f90b5400640 1 -- 172.21.15.31:0/854552572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b014ef00 con 0x7f90b0156c20 2024-09-11T00:41:37.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.590+0000 7f90b5400640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90b0156c20 0x7f90b014abf0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f90a800c760 tx=0x7f90a800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:37.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.590+0000 7f90acc00640 1 -- 172.21.15.31:0/854552572 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8003b70 con 0x7f90b0156c20 2024-09-11T00:41:37.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.590+0000 7f90acc00640 1 -- 172.21.15.31:0/854552572 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90a8003d10 con 0x7f90b0156c20 2024-09-11T00:41:37.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.590+0000 7f90acc00640 1 -- 172.21.15.31:0/854552572 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a80052d0 con 0x7f90b0156c20 2024-09-11T00:41:37.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.590+0000 7f90b7400640 1 -- 172.21.15.31:0/854552572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b014b130 con 0x7f90b0156c20 2024-09-11T00:41:37.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.590+0000 7f90b7400640 1 -- 172.21.15.31:0/854552572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b00aaa90 con 0x7f90b0156c20 2024-09-11T00:41:37.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.591+0000 7f90b7400640 1 -- 172.21.15.31:0/854552572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90b00c2be0 con 0x7f90b0156c20 2024-09-11T00:41:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.592+0000 7f90acc00640 1 -- 172.21.15.31:0/854552572 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f90a8005470 con 0x7f90b0156c20 2024-09-11T00:41:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.593+0000 7f90acc00640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9090077f70 0x7f909007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.593+0000 7f90acc00640 1 -- 172.21.15.31:0/854552572 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7f90a809a2b0 con 0x7f90b0156c20 2024-09-11T00:41:37.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.594+0000 7f90b4a00640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9090077f70 0x7f909007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:37.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.595+0000 7f90b4a00640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9090077f70 0x7f909007a430 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f90a4033000 tx=0x7f90a4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:37.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.598+0000 7f90acc00640 1 -- 172.21.15.31:0/854552572 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f90a8061ee0 con 0x7f90b0156c20 2024-09-11T00:41:37.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.916+0000 7f90b7400640 1 -- 172.21.15.31:0/854552572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90b00c1c20 con 0x7f90b0156c20 2024-09-11T00:41:37.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.917+0000 7f90acc00640 1 -- 172.21.15.31:0/854552572 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6531 (secure 0 0 0) 0x7f90a8066910 con 0x7f90b0156c20 2024-09-11T00:41:37.920 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:37.920 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-11T00:41:17:726701+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:37.921 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-11T00:41:37.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 -- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9090077f70 msgr2=0x7f909007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:37.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9090077f70 0x7f909007a430 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f90a4033000 tx=0x7f90a4004730 comp rx=0 tx=0).stop 2024-09-11T00:41:37.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 -- 172.21.15.31:0/854552572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90b0156c20 msgr2=0x7f90b014abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:37.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90b0156c20 0x7f90b014abf0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f90a800c760 tx=0x7f90a800cb60 comp rx=0 tx=0).stop 2024-09-11T00:41:37.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 -- 172.21.15.31:0/854552572 shutdown_connections 2024-09-11T00:41:37.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9090077f70 0x7f909007a430 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90b0156c20 0x7f90b014abf0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90b0156300 0x7f90b014e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 --2- 172.21.15.31:0/854552572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90b006fa70 0x7f90b00bcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:37.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 -- 172.21.15.31:0/854552572 >> 172.21.15.31:0/854552572 conn(0x7f90b00a9750 msgr2=0x7f90b00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:37.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 -- 172.21.15.31:0/854552572 shutdown_connections 2024-09-11T00:41:37.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:37.920+0000 7f90b7400640 1 -- 172.21.15.31:0/854552572 wait complete. 2024-09-11T00:41:37.939 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:41:39.939 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:40.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.105+0000 7f4f80000640 1 -- 172.21.15.31:0/2220111031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7806ff40 msgr2=0x7f4f780c23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:40.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.105+0000 7f4f80000640 1 --2- 172.21.15.31:0/2220111031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7806ff40 0x7f4f780c23a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4f64009830 tx=0x7f4f6402a530 comp rx=0 tx=0).stop 2024-09-11T00:41:40.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.105+0000 7f4f80000640 1 -- 172.21.15.31:0/2220111031 shutdown_connections 2024-09-11T00:41:40.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.105+0000 7f4f80000640 1 --2- 172.21.15.31:0/2220111031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7806ff40 0x7f4f780c23a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:40.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.105+0000 7f4f80000640 1 -- 172.21.15.31:0/2220111031 >> 172.21.15.31:0/2220111031 conn(0x7f4f780a9750 msgr2=0x7f4f780a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:40.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.105+0000 7f4f80000640 1 -- 172.21.15.31:0/2220111031 shutdown_connections 2024-09-11T00:41:40.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.105+0000 7f4f80000640 1 -- 172.21.15.31:0/2220111031 wait complete. 2024-09-11T00:41:40.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.106+0000 7f4f80000640 1 Processor -- start 2024-09-11T00:41:40.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.107+0000 7f4f80000640 1 -- start start 2024-09-11T00:41:40.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.107+0000 7f4f80000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7806ff40 0x7f4f781eff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:40.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.107+0000 7f4f80000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4f780c3f20 0x7f4f780c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:40.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f80000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f781532c0 0x7f4f781ec060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:40.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f80000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4f780c7650 con 0x7f4f780c3f20 2024-09-11T00:41:40.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f80000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7806f340 con 0x7f4f7806ff40 2024-09-11T00:41:40.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f80000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4f780c77d0 con 0x7f4f781532c0 2024-09-11T00:41:40.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f7cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4f780c3f20 0x7f4f780c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:40.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f7d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7806ff40 0x7f4f781eff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:40.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f7cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4f780c3f20 0x7f4f780c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43240/0 (socket says 172.21.15.31:43240) 2024-09-11T00:41:40.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f7cc00640 1 -- 172.21.15.31:0/1284764630 learned_addr learned my addr 172.21.15.31:0/1284764630 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:40.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f7e000640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f781532c0 0x7f4f781ec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:40.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f7cc00640 1 -- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f781532c0 msgr2=0x7f4f781ec060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:40.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f7cc00640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f781532c0 0x7f4f781ec060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:40.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f7cc00640 1 -- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7806ff40 msgr2=0x7f4f781eff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:40.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f7cc00640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7806ff40 0x7f4f781eff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:40.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.108+0000 7f4f7cc00640 1 -- 172.21.15.31:0/1284764630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f781f04d0 con 0x7f4f780c3f20 2024-09-11T00:41:40.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.109+0000 7f4f7cc00640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4f780c3f20 0x7f4f780c1040 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4f64009950 tx=0x7f4f64032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:40.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.109+0000 7f4f75600640 1 -- 172.21.15.31:0/1284764630 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f64040070 con 0x7f4f780c3f20 2024-09-11T00:41:40.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.109+0000 7f4f75600640 1 -- 172.21.15.31:0/1284764630 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f64005310 con 0x7f4f780c3f20 2024-09-11T00:41:40.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.109+0000 7f4f80000640 1 -- 172.21.15.31:0/1284764630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f781e1030 con 0x7f4f780c3f20 2024-09-11T00:41:40.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.109+0000 7f4f75600640 1 -- 172.21.15.31:0/1284764630 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f640047a0 con 0x7f4f780c3f20 2024-09-11T00:41:40.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.109+0000 7f4f80000640 1 -- 172.21.15.31:0/1284764630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f780c1580 con 0x7f4f780c3f20 2024-09-11T00:41:40.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.110+0000 7f4f80000640 1 -- 172.21.15.31:0/1284764630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f780a7350 con 0x7f4f780c3f20 2024-09-11T00:41:40.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.111+0000 7f4f75600640 1 -- 172.21.15.31:0/1284764630 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4f64004940 con 0x7f4f780c3f20 2024-09-11T00:41:40.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.113+0000 7f4f75600640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4f58077f70 0x7f4f5807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:40.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.113+0000 7f4f75600640 1 -- 172.21.15.31:0/1284764630 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7f4f640bb8e0 con 0x7f4f780c3f20 2024-09-11T00:41:40.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.113+0000 7f4f7d600640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4f58077f70 0x7f4f5807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:40.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.114+0000 7f4f7d600640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4f58077f70 0x7f4f5807a430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c000f60 tx=0x7f4f6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:40.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.117+0000 7f4f75600640 1 -- 172.21.15.31:0/1284764630 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4f64083490 con 0x7f4f780c3f20 2024-09-11T00:41:40.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.357+0000 7f4f80000640 1 -- 172.21.15.31:0/1284764630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f780add30 con 0x7f4f780c3f20 2024-09-11T00:41:40.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.358+0000 7f4f75600640 1 -- 172.21.15.31:0/1284764630 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6531 (secure 0 0 0) 0x7f4f6408c030 con 0x7f4f780c3f20 2024-09-11T00:41:40.361 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:40.361 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-11T00:41:17:726701+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:40.361 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-11T00:41:40.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.360+0000 7f4f80000640 1 -- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4f58077f70 msgr2=0x7f4f5807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.360+0000 7f4f80000640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4f58077f70 0x7f4f5807a430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c000f60 tx=0x7f4f6c009040 comp rx=0 tx=0).stop 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.360+0000 7f4f80000640 1 -- 172.21.15.31:0/1284764630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4f780c3f20 msgr2=0x7f4f780c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.360+0000 7f4f80000640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4f780c3f20 0x7f4f780c1040 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4f64009950 tx=0x7f4f64032b90 comp rx=0 tx=0).stop 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.360+0000 7f4f80000640 1 -- 172.21.15.31:0/1284764630 shutdown_connections 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.360+0000 7f4f80000640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4f58077f70 0x7f4f5807a430 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.360+0000 7f4f80000640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4f781532c0 0x7f4f781ec060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.360+0000 7f4f80000640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4f780c3f20 0x7f4f780c1040 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.360+0000 7f4f80000640 1 --2- 172.21.15.31:0/1284764630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7806ff40 0x7f4f781eff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.360+0000 7f4f80000640 1 -- 172.21.15.31:0/1284764630 >> 172.21.15.31:0/1284764630 conn(0x7f4f780a9750 msgr2=0x7f4f78150d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.361+0000 7f4f80000640 1 -- 172.21.15.31:0/1284764630 shutdown_connections 2024-09-11T00:41:40.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:40.361+0000 7f4f80000640 1 -- 172.21.15.31:0/1284764630 wait complete. 2024-09-11T00:41:40.379 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:41:42.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:42.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.546+0000 7f304b000640 1 -- 172.21.15.31:0/348711059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30440c5aa0 msgr2=0x7f30440c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.546+0000 7f304b000640 1 --2- 172.21.15.31:0/348711059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30440c5aa0 0x7f30440c7150 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3044072000 tx=0x7f303802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:41:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.547+0000 7f304b000640 1 -- 172.21.15.31:0/348711059 shutdown_connections 2024-09-11T00:41:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.547+0000 7f304b000640 1 --2- 172.21.15.31:0/348711059 >> v2:172.21.15.204:3300/0 conn(0x7f3044156350 0x7f30441567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.547+0000 7f304b000640 1 --2- 172.21.15.31:0/348711059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30440c5aa0 0x7f30440c7150 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.547+0000 7f304b000640 1 -- 172.21.15.31:0/348711059 >> 172.21.15.31:0/348711059 conn(0x7f304406caf0 msgr2=0x7f304406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.547+0000 7f304b000640 1 -- 172.21.15.31:0/348711059 shutdown_connections 2024-09-11T00:41:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.547+0000 7f304b000640 1 -- 172.21.15.31:0/348711059 wait complete. 2024-09-11T00:41:42.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.548+0000 7f304b000640 1 Processor -- start 2024-09-11T00:41:42.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.548+0000 7f304b000640 1 -- start start 2024-09-11T00:41:42.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.549+0000 7f304b000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30440c5aa0 0x7f30441dff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.549+0000 7f304b000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30440c6420 0x7f30440aacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.549+0000 7f304b000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3044156350 0x7f304414cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.549+0000 7f304b000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f30440c2e80 con 0x7f30440c5aa0 2024-09-11T00:41:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.549+0000 7f304b000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f30440c2ba0 con 0x7f3044156350 2024-09-11T00:41:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.549+0000 7f304b000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3044153a20 con 0x7f30440c6420 2024-09-11T00:41:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.550+0000 7f3049c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30440c5aa0 0x7f30441dff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.550+0000 7f3049c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30440c5aa0 0x7f30441dff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43250/0 (socket says 172.21.15.31:43250) 2024-09-11T00:41:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.550+0000 7f3049c00640 1 -- 172.21.15.31:0/667093537 learned_addr learned my addr 172.21.15.31:0/667093537 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.550+0000 7f3049200640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30440c6420 0x7f30440aacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:42.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.550+0000 7f304a600640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3044156350 0x7f304414cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.550+0000 7f3049c00640 1 -- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30440c6420 msgr2=0x7f30440aacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.550+0000 7f3049c00640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30440c6420 0x7f30440aacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.550+0000 7f3049c00640 1 -- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3044156350 msgr2=0x7f304414cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.550+0000 7f3049c00640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3044156350 0x7f304414cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.550+0000 7f3049c00640 1 -- 172.21.15.31:0/667093537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3044074c20 con 0x7f30440c5aa0 2024-09-11T00:41:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.551+0000 7f3049c00640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30440c5aa0 0x7f30441dff50 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3038009850 tx=0x7f3038003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.551+0000 7f3041600640 1 -- 172.21.15.31:0/667093537 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303803f070 con 0x7f30440c5aa0 2024-09-11T00:41:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.551+0000 7f3041600640 1 -- 172.21.15.31:0/667093537 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30380047a0 con 0x7f30440c5aa0 2024-09-11T00:41:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.551+0000 7f304b000640 1 -- 172.21.15.31:0/667093537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3044074780 con 0x7f30440c5aa0 2024-09-11T00:41:42.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.551+0000 7f3041600640 1 -- 172.21.15.31:0/667093537 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3038035030 con 0x7f30440c5aa0 2024-09-11T00:41:42.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.551+0000 7f304b000640 1 -- 172.21.15.31:0/667093537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304414d4a0 con 0x7f30440c5aa0 2024-09-11T00:41:42.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.553+0000 7f304b000640 1 -- 172.21.15.31:0/667093537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30440c5e80 con 0x7f30440c5aa0 2024-09-11T00:41:42.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.554+0000 7f3041600640 1 -- 172.21.15.31:0/667093537 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3038004bb0 con 0x7f30440c5aa0 2024-09-11T00:41:42.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.555+0000 7f3041600640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3024077f70 0x7f302407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:42.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.555+0000 7f3041600640 1 -- 172.21.15.31:0/667093537 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7f30380bb5f0 con 0x7f30440c5aa0 2024-09-11T00:41:42.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.555+0000 7f3049200640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3024077f70 0x7f302407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:42.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.556+0000 7f3049200640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3024077f70 0x7f302407a430 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3044155f00 tx=0x7f3030009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:42.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.562+0000 7f3041600640 1 -- 172.21.15.31:0/667093537 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3038083260 con 0x7f30440c5aa0 2024-09-11T00:41:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.794+0000 7f304b000640 1 -- 172.21.15.31:0/667093537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f304406dcf0 con 0x7f30440c5aa0 2024-09-11T00:41:42.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.795+0000 7f3041600640 1 -- 172.21.15.31:0/667093537 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6531 (secure 0 0 0) 0x7f3038088e20 con 0x7f30440c5aa0 2024-09-11T00:41:42.798 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:42.798 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-11T00:41:17:726701+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:42.798 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.797+0000 7f304b000640 1 -- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3024077f70 msgr2=0x7f302407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.797+0000 7f304b000640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3024077f70 0x7f302407a430 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3044155f00 tx=0x7f3030009040 comp rx=0 tx=0).stop 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.798+0000 7f304b000640 1 -- 172.21.15.31:0/667093537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30440c5aa0 msgr2=0x7f30441dff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.798+0000 7f304b000640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30440c5aa0 0x7f30441dff50 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3038009850 tx=0x7f3038003820 comp rx=0 tx=0).stop 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.798+0000 7f304b000640 1 -- 172.21.15.31:0/667093537 shutdown_connections 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.798+0000 7f304b000640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3024077f70 0x7f302407a430 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.798+0000 7f304b000640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3044156350 0x7f304414cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.798+0000 7f304b000640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f30440c6420 0x7f30440aacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.798+0000 7f304b000640 1 --2- 172.21.15.31:0/667093537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f30440c5aa0 0x7f30441dff50 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.798+0000 7f304b000640 1 -- 172.21.15.31:0/667093537 >> 172.21.15.31:0/667093537 conn(0x7f304406caf0 msgr2=0x7f3044150ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.798+0000 7f304b000640 1 -- 172.21.15.31:0/667093537 shutdown_connections 2024-09-11T00:41:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:42.798+0000 7f304b000640 1 -- 172.21.15.31:0/667093537 wait complete. 2024-09-11T00:41:42.817 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:41:44.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:44.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.978+0000 7fc4f8200640 1 -- 172.21.15.31:0/1852788400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f0072480 msgr2=0x7fc4f01511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.978+0000 7fc4f8200640 1 --2- 172.21.15.31:0/1852788400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f0072480 0x7fc4f01511a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0075260 tx=0x7fc4e402a5c0 comp rx=0 tx=0).stop 2024-09-11T00:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.978+0000 7fc4f8200640 1 -- 172.21.15.31:0/1852788400 shutdown_connections 2024-09-11T00:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.978+0000 7fc4f8200640 1 --2- 172.21.15.31:0/1852788400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f0072480 0x7fc4f01511a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.978+0000 7fc4f8200640 1 -- 172.21.15.31:0/1852788400 >> 172.21.15.31:0/1852788400 conn(0x7fc4f00a9750 msgr2=0x7fc4f00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.978+0000 7fc4f8200640 1 -- 172.21.15.31:0/1852788400 shutdown_connections 2024-09-11T00:41:44.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.979+0000 7fc4f8200640 1 -- 172.21.15.31:0/1852788400 wait complete. 2024-09-11T00:41:44.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.980+0000 7fc4f8200640 1 Processor -- start 2024-09-11T00:41:44.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.980+0000 7fc4f8200640 1 -- start start 2024-09-11T00:41:44.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f8200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4f0072480 0x7fc4f01efce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:44.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f8200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f00c23a0 0x7fc4f0157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:44.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f8200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f0153f20 0x7fc4f0148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f8200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f0074200 con 0x7fc4f0072480 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f8200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f0074080 con 0x7fc4f0153f20 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f8200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f0072860 con 0x7fc4f00c23a0 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4f0072480 0x7fc4f01efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4f0072480 0x7fc4f01efce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43262/0 (socket says 172.21.15.31:43262) 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f5800640 1 -- 172.21.15.31:0/3022772575 learned_addr learned my addr 172.21.15.31:0/3022772575 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f4e00640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f00c23a0 0x7fc4f0157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.981+0000 7fc4f6200640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f0153f20 0x7fc4f0148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.982+0000 7fc4f5800640 1 -- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f00c23a0 msgr2=0x7fc4f0157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.982+0000 7fc4f5800640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f00c23a0 0x7fc4f0157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:44.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.982+0000 7fc4f5800640 1 -- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f0153f20 msgr2=0x7fc4f0148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:44.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.982+0000 7fc4f5800640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f0153f20 0x7fc4f0148b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:44.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.982+0000 7fc4f5800640 1 -- 172.21.15.31:0/3022772575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f01558c0 con 0x7fc4f0072480 2024-09-11T00:41:44.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.982+0000 7fc4f5800640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4f0072480 0x7fc4f01efce0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009920 tx=0x7fc4e4032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:44.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.982+0000 7fc4ed600640 1 -- 172.21.15.31:0/3022772575 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4040070 con 0x7fc4f0072480 2024-09-11T00:41:44.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.982+0000 7fc4ed600640 1 -- 172.21.15.31:0/3022772575 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4e4005310 con 0x7fc4f0072480 2024-09-11T00:41:44.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.982+0000 7fc4ed600640 1 -- 172.21.15.31:0/3022772575 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4005c90 con 0x7fc4f0072480 2024-09-11T00:41:44.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.982+0000 7fc4f8200640 1 -- 172.21.15.31:0/3022772575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f0155b60 con 0x7fc4f0072480 2024-09-11T00:41:44.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.983+0000 7fc4f8200640 1 -- 172.21.15.31:0/3022772575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f0156120 con 0x7fc4f0072480 2024-09-11T00:41:44.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.984+0000 7fc4f8200640 1 -- 172.21.15.31:0/3022772575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f0151960 con 0x7fc4f0072480 2024-09-11T00:41:44.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.985+0000 7fc4ed600640 1 -- 172.21.15.31:0/3022772575 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc4e40049b0 con 0x7fc4f0072480 2024-09-11T00:41:44.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.986+0000 7fc4ed600640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc4d0077f70 0x7fc4d007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:44.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.986+0000 7fc4ed600640 1 -- 172.21.15.31:0/3022772575 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7fc4e40bba40 con 0x7fc4f0072480 2024-09-11T00:41:44.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.986+0000 7fc4f4e00640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc4d0077f70 0x7fc4d007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:44.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.987+0000 7fc4f4e00640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc4d0077f70 0x7fc4d007a430 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc007f30 tx=0x7fc4dc007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:44.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:44.990+0000 7fc4ed600640 1 -- 172.21.15.31:0/3022772575 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc4e4083670 con 0x7fc4f0072480 2024-09-11T00:41:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.220+0000 7fc4f8200640 1 -- 172.21.15.31:0/3022772575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4f006a170 con 0x7fc4f0072480 2024-09-11T00:41:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.221+0000 7fc4ed600640 1 -- 172.21.15.31:0/3022772575 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6531 (secure 0 0 0) 0x7fc4e40880a0 con 0x7fc4f0072480 2024-09-11T00:41:45.224 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:45.224 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-11T00:41:17:726701+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.223+0000 7fc4f8200640 1 -- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc4d0077f70 msgr2=0x7fc4d007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.223+0000 7fc4f8200640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc4d0077f70 0x7fc4d007a430 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc007f30 tx=0x7fc4dc007ec0 comp rx=0 tx=0).stop 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.224+0000 7fc4f8200640 1 -- 172.21.15.31:0/3022772575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4f0072480 msgr2=0x7fc4f01efce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.224+0000 7fc4f8200640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4f0072480 0x7fc4f01efce0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009920 tx=0x7fc4e4032c10 comp rx=0 tx=0).stop 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.224+0000 7fc4f8200640 1 -- 172.21.15.31:0/3022772575 shutdown_connections 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.224+0000 7fc4f8200640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc4d0077f70 0x7fc4d007a430 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.224+0000 7fc4f8200640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4f0153f20 0x7fc4f0148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.224+0000 7fc4f8200640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4f00c23a0 0x7fc4f0157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.224+0000 7fc4f8200640 1 --2- 172.21.15.31:0/3022772575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4f0072480 0x7fc4f01efce0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.224+0000 7fc4f8200640 1 -- 172.21.15.31:0/3022772575 >> 172.21.15.31:0/3022772575 conn(0x7fc4f00a9750 msgr2=0x7fc4f014afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:45.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.224+0000 7fc4f8200640 1 -- 172.21.15.31:0/3022772575 shutdown_connections 2024-09-11T00:41:45.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:45.224+0000 7fc4f8200640 1 -- 172.21.15.31:0/3022772575 wait complete. 2024-09-11T00:41:45.245 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:41:45.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:41:45.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:41:47.245 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.416+0000 7f3291a00640 1 -- 172.21.15.31:0/695623500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f328c070360 msgr2=0x7f328c074050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:47.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.416+0000 7f3291a00640 1 --2- 172.21.15.31:0/695623500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f328c070360 0x7f328c074050 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3278005fd0 tx=0x7f327802a580 comp rx=0 tx=0).stop 2024-09-11T00:41:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.417+0000 7f3291a00640 1 -- 172.21.15.31:0/695623500 shutdown_connections 2024-09-11T00:41:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.417+0000 7f3291a00640 1 --2- 172.21.15.31:0/695623500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f328c070360 0x7f328c074050 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.417+0000 7f3291a00640 1 -- 172.21.15.31:0/695623500 >> 172.21.15.31:0/695623500 conn(0x7f328c06dc00 msgr2=0x7f328c06e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:47.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.417+0000 7f3291a00640 1 -- 172.21.15.31:0/695623500 shutdown_connections 2024-09-11T00:41:47.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.417+0000 7f3291a00640 1 -- 172.21.15.31:0/695623500 wait complete. 2024-09-11T00:41:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.419+0000 7f3291a00640 1 Processor -- start 2024-09-11T00:41:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.419+0000 7f3291a00640 1 -- start start 2024-09-11T00:41:47.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.419+0000 7f3291a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f328c06d440 0x7f328c1e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:47.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.419+0000 7f3291a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f328c070360 0x7f328c159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:47.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f3291a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c153b30 0x7f328c159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:47.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f3291a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f328c0c2060 con 0x7f328c06d440 2024-09-11T00:41:47.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f3291a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f328c0c1ee0 con 0x7f328c153b30 2024-09-11T00:41:47.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f3291a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f328c070f50 con 0x7f328c070360 2024-09-11T00:41:47.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f328c06d440 0x7f328c1e1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:47.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f328c06d440 0x7f328c1e1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43282/0 (socket says 172.21.15.31:43282) 2024-09-11T00:41:47.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328aa00640 1 -- 172.21.15.31:0/2240875137 learned_addr learned my addr 172.21.15.31:0/2240875137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:47.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328b400640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c153b30 0x7f328c159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:47.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328a000640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f328c070360 0x7f328c159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:47.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328aa00640 1 -- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f328c070360 msgr2=0x7f328c159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:47.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328aa00640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f328c070360 0x7f328c159a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:47.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328aa00640 1 -- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c153b30 msgr2=0x7f328c159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:47.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328aa00640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c153b30 0x7f328c159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:47.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328aa00640 1 -- 172.21.15.31:0/2240875137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328c14c2f0 con 0x7f328c06d440 2024-09-11T00:41:47.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328b400640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c153b30 0x7f328c159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:41:47.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.420+0000 7f328aa00640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f328c06d440 0x7f328c1e1d90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c3f0 tx=0x7f327c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:47.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.421+0000 7f3282a00640 1 -- 172.21.15.31:0/2240875137 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c017070 con 0x7f328c06d440 2024-09-11T00:41:47.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.421+0000 7f3282a00640 1 -- 172.21.15.31:0/2240875137 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f327c0035f0 con 0x7f328c06d440 2024-09-11T00:41:47.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.421+0000 7f3291a00640 1 -- 172.21.15.31:0/2240875137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328c1ed0f0 con 0x7f328c06d440 2024-09-11T00:41:47.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.421+0000 7f3282a00640 1 -- 172.21.15.31:0/2240875137 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327c003a40 con 0x7f328c06d440 2024-09-11T00:41:47.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.421+0000 7f3291a00640 1 -- 172.21.15.31:0/2240875137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f328c1e22d0 con 0x7f328c06d440 2024-09-11T00:41:47.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.422+0000 7f3291a00640 1 -- 172.21.15.31:0/2240875137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f328c070840 con 0x7f328c06d440 2024-09-11T00:41:47.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.423+0000 7f3282a00640 1 -- 172.21.15.31:0/2240875137 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f327c003be0 con 0x7f328c06d440 2024-09-11T00:41:47.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.424+0000 7f3282a00640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f326c077fa0 0x7f326c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:47.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.424+0000 7f3282a00640 1 -- 172.21.15.31:0/2240875137 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7870+0+0 (secure 0 0 0) 0x7f327c099ba0 con 0x7f328c06d440 2024-09-11T00:41:47.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.424+0000 7f328a000640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f326c077fa0 0x7f326c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:47.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.425+0000 7f328a000640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f326c077fa0 0x7f326c07a460 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3278033000 tx=0x7f3278004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:47.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.428+0000 7f3282a00640 1 -- 172.21.15.31:0/2240875137 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f327c061e20 con 0x7f328c06d440 2024-09-11T00:41:47.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.664+0000 7f3291a00640 1 -- 172.21.15.31:0/2240875137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f328c1ecee0 con 0x7f328c06d440 2024-09-11T00:41:47.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.664+0000 7f3282a00640 1 -- 172.21.15.31:0/2240875137 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6531 (secure 0 0 0) 0x7f327c061720 con 0x7f328c06d440 2024-09-11T00:41:47.667 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:47.668 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"btime":"2024-09-11T00:41:17:726701+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:40:42.621929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5911":{"gid":5911,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6835/98007221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":98007221},{"type":"v1","addr":"172.21.15.204:6835","nonce":98007221}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5911,"qdb_cluster":[5911]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:47.668 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 82 2024-09-11T00:41:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 -- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f326c077fa0 msgr2=0x7f326c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f326c077fa0 0x7f326c07a460 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3278033000 tx=0x7f3278004730 comp rx=0 tx=0).stop 2024-09-11T00:41:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 -- 172.21.15.31:0/2240875137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f328c06d440 msgr2=0x7f328c1e1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f328c06d440 0x7f328c1e1d90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c3f0 tx=0x7f327c00c8b0 comp rx=0 tx=0).stop 2024-09-11T00:41:47.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 -- 172.21.15.31:0/2240875137 shutdown_connections 2024-09-11T00:41:47.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f326c077fa0 0x7f326c07a460 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:47.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f328c153b30 0x7f328c159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:47.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f328c070360 0x7f328c159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:47.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 --2- 172.21.15.31:0/2240875137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f328c06d440 0x7f328c1e1d90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:47.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 -- 172.21.15.31:0/2240875137 >> 172.21.15.31:0/2240875137 conn(0x7f328c06dc00 msgr2=0x7f328c148cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:47.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 -- 172.21.15.31:0/2240875137 shutdown_connections 2024-09-11T00:41:47.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:47.667+0000 7f3291a00640 1 -- 172.21.15.31:0/2240875137 wait complete. 2024-09-11T00:41:47.686 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:41:49.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:49.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.849+0000 7f143ee00640 1 -- 172.21.15.31:0/4099643247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1438072480 msgr2=0x7f1438074a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:49.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.849+0000 7f143ee00640 1 --2- 172.21.15.31:0/4099643247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1438072480 0x7f1438074a90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1428009900 tx=0x7f142802a610 comp rx=0 tx=0).stop 2024-09-11T00:41:49.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.849+0000 7f143ee00640 1 -- 172.21.15.31:0/4099643247 shutdown_connections 2024-09-11T00:41:49.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.849+0000 7f143ee00640 1 --2- 172.21.15.31:0/4099643247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1438072480 0x7f1438074a90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:49.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.849+0000 7f143ee00640 1 -- 172.21.15.31:0/4099643247 >> 172.21.15.31:0/4099643247 conn(0x7f14380a9750 msgr2=0x7f14380a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:49.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.849+0000 7f143ee00640 1 -- 172.21.15.31:0/4099643247 shutdown_connections 2024-09-11T00:41:49.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.849+0000 7f143ee00640 1 -- 172.21.15.31:0/4099643247 wait complete. 2024-09-11T00:41:49.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.851+0000 7f143ee00640 1 Processor -- start 2024-09-11T00:41:49.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.851+0000 7f143ee00640 1 -- start start 2024-09-11T00:41:49.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f143ee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1438072480 0x7f1438148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:49.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f143ee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14381511c0 0x7f1438141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:49.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f143ee00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1438154b60 0x7f14381418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:49.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f143ee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14380c23a0 con 0x7f1438072480 2024-09-11T00:41:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f143ee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1438154420 con 0x7f14381511c0 2024-09-11T00:41:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f143ee00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f14380a7140 con 0x7f1438154b60 2024-09-11T00:41:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f1437e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1438072480 0x7f1438148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f1437e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1438072480 0x7f1438148ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35092/0 (socket says 172.21.15.31:35092) 2024-09-11T00:41:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f1437e00640 1 -- 172.21.15.31:0/2577082082 learned_addr learned my addr 172.21.15.31:0/2577082082 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f1437400640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14381511c0 0x7f1438141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f143ce00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1438154b60 0x7f14381418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f1437e00640 1 -- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1438154b60 msgr2=0x7f14381418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f1437e00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1438154b60 0x7f14381418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f1437e00640 1 -- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14381511c0 msgr2=0x7f1438141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:49.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f1437e00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14381511c0 0x7f1438141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:49.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.852+0000 7f1437e00640 1 -- 172.21.15.31:0/2577082082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14381426a0 con 0x7f1438072480 2024-09-11T00:41:49.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.853+0000 7f1437e00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1438072480 0x7f1438148ce0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1428006010 tx=0x7f14280038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:49.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.853+0000 7f142fe00640 1 -- 172.21.15.31:0/2577082082 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142803f070 con 0x7f1438072480 2024-09-11T00:41:49.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.853+0000 7f142fe00640 1 -- 172.21.15.31:0/2577082082 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14280047a0 con 0x7f1438072480 2024-09-11T00:41:49.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.853+0000 7f142fe00640 1 -- 172.21.15.31:0/2577082082 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1428035030 con 0x7f1438072480 2024-09-11T00:41:49.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.853+0000 7f143ee00640 1 -- 172.21.15.31:0/2577082082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1438142400 con 0x7f1438072480 2024-09-11T00:41:49.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.853+0000 7f143ee00640 1 -- 172.21.15.31:0/2577082082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1438149220 con 0x7f1438072480 2024-09-11T00:41:49.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.854+0000 7f143ee00640 1 -- 172.21.15.31:0/2577082082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1438142160 con 0x7f1438072480 2024-09-11T00:41:49.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.855+0000 7f142fe00640 1 -- 172.21.15.31:0/2577082082 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1428004b90 con 0x7f1438072480 2024-09-11T00:41:49.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.856+0000 7f142fe00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1418077f70 0x7f141807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:49.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.856+0000 7f142fe00640 1 -- 172.21.15.31:0/2577082082 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f14280bb570 con 0x7f1438072480 2024-09-11T00:41:49.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.857+0000 7f1437400640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1418077f70 0x7f141807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:49.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.857+0000 7f1437400640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1418077f70 0x7f141807a430 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f143814ebb0 tx=0x7f1424009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:49.861+0000 7f142fe00640 1 -- 172.21.15.31:0/2577082082 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f14280831e0 con 0x7f1438072480 2024-09-11T00:41:50.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.106+0000 7f143ee00640 1 -- 172.21.15.31:0/2577082082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14380c3740 con 0x7f1438072480 2024-09-11T00:41:50.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.107+0000 7f142fe00640 1 -- 172.21.15.31:0/2577082082 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5702 (secure 0 0 0) 0x7f1428087c10 con 0x7f1438072480 2024-09-11T00:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:50.115 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":83,"btime":"2024-09-11T00:41:49:384128+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:49.384123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:50.117 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 83 2024-09-11T00:41:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.109+0000 7f143ee00640 1 -- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1418077f70 msgr2=0x7f141807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.109+0000 7f143ee00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1418077f70 0x7f141807a430 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f143814ebb0 tx=0x7f1424009040 comp rx=0 tx=0).stop 2024-09-11T00:41:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.109+0000 7f143ee00640 1 -- 172.21.15.31:0/2577082082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1438072480 msgr2=0x7f1438148ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.109+0000 7f143ee00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1438072480 0x7f1438148ce0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1428006010 tx=0x7f14280038a0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.109+0000 7f143ee00640 1 -- 172.21.15.31:0/2577082082 shutdown_connections 2024-09-11T00:41:50.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.109+0000 7f143ee00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1418077f70 0x7f141807a430 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.109+0000 7f143ee00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1438154b60 0x7f14381418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.109+0000 7f143ee00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14381511c0 0x7f1438141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.109+0000 7f143ee00640 1 --2- 172.21.15.31:0/2577082082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1438072480 0x7f1438148ce0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.109+0000 7f143ee00640 1 -- 172.21.15.31:0/2577082082 >> 172.21.15.31:0/2577082082 conn(0x7f14380a9750 msgr2=0x7f14380747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:50.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.110+0000 7f143ee00640 1 -- 172.21.15.31:0/2577082082 shutdown_connections 2024-09-11T00:41:50.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.110+0000 7f143ee00640 1 -- 172.21.15.31:0/2577082082 wait complete. 2024-09-11T00:41:50.128 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-11T00:41:50.128 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:41:50.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:50.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.286+0000 7f14d9200640 1 -- 172.21.15.31:0/892659794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14d414ccc0 msgr2=0x7f14d4156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:50.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.286+0000 7f14d9200640 1 --2- 172.21.15.31:0/892659794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14d414ccc0 0x7f14d4156b00 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f14c0009830 tx=0x7f14c002a510 comp rx=0 tx=0).stop 2024-09-11T00:41:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.287+0000 7f14d9200640 1 -- 172.21.15.31:0/892659794 shutdown_connections 2024-09-11T00:41:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.287+0000 7f14d9200640 1 --2- 172.21.15.31:0/892659794 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14d40c9a10 0x7f14d4159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.287+0000 7f14d9200640 1 --2- 172.21.15.31:0/892659794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14d414ccc0 0x7f14d4156b00 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.287+0000 7f14d9200640 1 -- 172.21.15.31:0/892659794 >> 172.21.15.31:0/892659794 conn(0x7f14d4068ee0 msgr2=0x7f14d40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.287+0000 7f14d9200640 1 -- 172.21.15.31:0/892659794 shutdown_connections 2024-09-11T00:41:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.287+0000 7f14d9200640 1 -- 172.21.15.31:0/892659794 wait complete. 2024-09-11T00:41:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.288+0000 7f14d9200640 1 Processor -- start 2024-09-11T00:41:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.288+0000 7f14d9200640 1 -- start start 2024-09-11T00:41:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.289+0000 7f14d9200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14d40c9a10 0x7f14d41551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.289+0000 7f14d9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14d414c8e0 0x7f14d414bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.289+0000 7f14d9200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14d414ccc0 0x7f14d414c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.289+0000 7f14d9200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14d40c6490 con 0x7f14d414c8e0 2024-09-11T00:41:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.289+0000 7f14d9200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f14d40c6800 con 0x7f14d414ccc0 2024-09-11T00:41:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.289+0000 7f14d9200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f14d40c6c90 con 0x7f14d40c9a10 2024-09-11T00:41:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14d414c8e0 0x7f14d414bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14d40c9a10 0x7f14d41551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14d414c8e0 0x7f14d414bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35124/0 (socket says 172.21.15.31:35124) 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d2a00640 1 -- 172.21.15.31:0/1717974807 learned_addr learned my addr 172.21.15.31:0/1717974807 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d3e00640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14d414ccc0 0x7f14d414c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d2a00640 1 -- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14d40c9a10 msgr2=0x7f14d41551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d2a00640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14d40c9a10 0x7f14d41551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d2a00640 1 -- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14d414ccc0 msgr2=0x7f14d414c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d2a00640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14d414ccc0 0x7f14d414c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d2a00640 1 -- 172.21.15.31:0/1717974807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d414f410 con 0x7f14d414c8e0 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.290+0000 7f14d2a00640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14d414c8e0 0x7f14d414bdb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f14c0000c00 tx=0x7f14c0033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.291+0000 7f14cb400640 1 -- 172.21.15.31:0/1717974807 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c0005c90 con 0x7f14d414c8e0 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.291+0000 7f14cb400640 1 -- 172.21.15.31:0/1717974807 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14c0031e50 con 0x7f14d414c8e0 2024-09-11T00:41:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.291+0000 7f14cb400640 1 -- 172.21.15.31:0/1717974807 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c00332f0 con 0x7f14d414c8e0 2024-09-11T00:41:50.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.291+0000 7f14d9200640 1 -- 172.21.15.31:0/1717974807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d41529b0 con 0x7f14d414c8e0 2024-09-11T00:41:50.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.291+0000 7f14d9200640 1 -- 172.21.15.31:0/1717974807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d4151ba0 con 0x7f14d414c8e0 2024-09-11T00:41:50.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.292+0000 7f14d9200640 1 -- 172.21.15.31:0/1717974807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d40ab9c0 con 0x7f14d414c8e0 2024-09-11T00:41:50.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.293+0000 7f14cb400640 1 -- 172.21.15.31:0/1717974807 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f14c004b030 con 0x7f14d414c8e0 2024-09-11T00:41:50.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.294+0000 7f14cb400640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f14b4077f70 0x7f14b407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:50.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.294+0000 7f14cb400640 1 -- 172.21.15.31:0/1717974807 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f14c00bb5e0 con 0x7f14d414c8e0 2024-09-11T00:41:50.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.295+0000 7f14d3400640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f14b4077f70 0x7f14b407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:50.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.295+0000 7f14d3400640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f14b4077f70 0x7f14b407a430 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f14d4156720 tx=0x7f14c4007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.299+0000 7f14cb400640 1 -- 172.21.15.31:0/1717974807 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f14d40ab9c0 con 0x7f14d414c8e0 2024-09-11T00:41:50.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.545+0000 7f14d9200640 1 -- 172.21.15.31:0/1717974807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14d4156b00 con 0x7f14d414c8e0 2024-09-11T00:41:50.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.546+0000 7f14cb400640 1 -- 172.21.15.31:0/1717974807 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) ==== 94+0+5706 (secure 0 0 0) 0x7f14c003f030 con 0x7f14d414c8e0 2024-09-11T00:41:50.549 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:50.549 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":84,"btime":"2024-09-11T00:41:50:385889+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:49.685812+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:50.549 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 84 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 -- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f14b4077f70 msgr2=0x7f14b407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f14b4077f70 0x7f14b407a430 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f14d4156720 tx=0x7f14c4007e90 comp rx=0 tx=0).stop 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 -- 172.21.15.31:0/1717974807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14d414c8e0 msgr2=0x7f14d414bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14d414c8e0 0x7f14d414bdb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f14c0000c00 tx=0x7f14c0033c90 comp rx=0 tx=0).stop 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 -- 172.21.15.31:0/1717974807 shutdown_connections 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f14b4077f70 0x7f14b407a430 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14d414ccc0 0x7f14d414c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14d414c8e0 0x7f14d414bdb0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 --2- 172.21.15.31:0/1717974807 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14d40c9a10 0x7f14d41551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 -- 172.21.15.31:0/1717974807 >> 172.21.15.31:0/1717974807 conn(0x7f14d4068ee0 msgr2=0x7f14d41593b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:50.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 -- 172.21.15.31:0/1717974807 shutdown_connections 2024-09-11T00:41:50.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:50.549+0000 7f14d9200640 1 -- 172.21.15.31:0/1717974807 wait complete. 2024-09-11T00:41:50.573 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:41:50:385889+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 84, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 84, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6165": { "addr": "172.21.15.31:6835/4205320530", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6833", "nonce": 4205320530, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 4205320530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6165, "incarnation": 83, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:41:49.685812+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6165 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 81, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6087": { "addr": "172.21.15.31:6834/2909466339", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2909466339, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 2909466339, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6087, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:41:12.709307+0000", "qdb_cluster": [ 6087 ], "qdb_leader": 6087, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6087 } } } ], "standbys": [ { "addr": "172.21.15.204:6837/3513641573", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 3513641573, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 3513641573, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5773, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:41:52.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:52.740 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-11T00:41:52.738+0000 7f3a04200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:41:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.739+0000 7f1980600640 1 -- 172.21.15.31:0/1740649796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19780c23a0 msgr2=0x7f19780c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.739+0000 7f1980600640 1 --2- 172.21.15.31:0/1740649796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19780c23a0 0x7f19780c30c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f19780c5500 tx=0x7f196402a680 comp rx=0 tx=0).stop 2024-09-11T00:41:52.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.739+0000 7f1980600640 1 -- 172.21.15.31:0/1740649796 shutdown_connections 2024-09-11T00:41:52.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.739+0000 7f1980600640 1 --2- 172.21.15.31:0/1740649796 >> v2:172.21.15.204:3300/0 conn(0x7f19781518b0 0x7f19780c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:52.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.739+0000 7f1980600640 1 --2- 172.21.15.31:0/1740649796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19780c23a0 0x7f19780c30c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:52.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.739+0000 7f1980600640 1 -- 172.21.15.31:0/1740649796 >> 172.21.15.31:0/1740649796 conn(0x7f19780a9750 msgr2=0x7f19780a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:52.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.740+0000 7f1980600640 1 -- 172.21.15.31:0/1740649796 shutdown_connections 2024-09-11T00:41:52.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.740+0000 7f1980600640 1 -- 172.21.15.31:0/1740649796 wait complete. 2024-09-11T00:41:52.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.741+0000 7f1980600640 1 Processor -- start 2024-09-11T00:41:52.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.741+0000 7f1980600640 1 -- start start 2024-09-11T00:41:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.742+0000 7f1980600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1978072480 0x7f19781ecbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.742+0000 7f1980600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19780c23a0 0x7f197806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.742+0000 7f1980600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19781518b0 0x7f1978146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.742+0000 7f1980600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1978074200 con 0x7f1978072480 2024-09-11T00:41:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.742+0000 7f1980600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1978074080 con 0x7f19781518b0 2024-09-11T00:41:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.742+0000 7f1980600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1978072860 con 0x7f19780c23a0 2024-09-11T00:41:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1978072480 0x7f19781ecbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:52.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197dc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1978072480 0x7f19781ecbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35142/0 (socket says 172.21.15.31:35142) 2024-09-11T00:41:52.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197dc00640 1 -- 172.21.15.31:0/1291108034 learned_addr learned my addr 172.21.15.31:0/1291108034 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:52.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197d200640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19780c23a0 0x7f197806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:52.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197e600640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19781518b0 0x7f1978146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:52.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197dc00640 1 -- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19780c23a0 msgr2=0x7f197806e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:52.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197dc00640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19780c23a0 0x7f197806e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:52.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197dc00640 1 -- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19781518b0 msgr2=0x7f1978146ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:52.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197dc00640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19781518b0 0x7f1978146ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:52.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197dc00640 1 -- 172.21.15.31:0/1291108034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197806eed0 con 0x7f1978072480 2024-09-11T00:41:52.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.743+0000 7f197dc00640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1978072480 0x7f19781ecbd0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f196c00c480 tx=0x7f196c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.744+0000 7f1975600640 1 -- 172.21.15.31:0/1291108034 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c0052d0 con 0x7f1978072480 2024-09-11T00:41:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.744+0000 7f1975600640 1 -- 172.21.15.31:0/1291108034 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f196c005470 con 0x7f1978072480 2024-09-11T00:41:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.744+0000 7f1980600640 1 -- 172.21.15.31:0/1291108034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197806f1b0 con 0x7f1978072480 2024-09-11T00:41:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.744+0000 7f1975600640 1 -- 172.21.15.31:0/1291108034 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c012870 con 0x7f1978072480 2024-09-11T00:41:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.744+0000 7f1980600640 1 -- 172.21.15.31:0/1291108034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197806f770 con 0x7f1978072480 2024-09-11T00:41:52.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.745+0000 7f1980600640 1 -- 172.21.15.31:0/1291108034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1978077240 con 0x7f1978072480 2024-09-11T00:41:52.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.746+0000 7f1975600640 1 -- 172.21.15.31:0/1291108034 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f196c0035f0 con 0x7f1978072480 2024-09-11T00:41:52.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.748+0000 7f1975600640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077f70 0x7f195807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:52.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.748+0000 7f1975600640 1 -- 172.21.15.31:0/1291108034 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f196c09a0a0 con 0x7f1978072480 2024-09-11T00:41:52.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.748+0000 7f197d200640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077f70 0x7f195807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:52.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.749+0000 7f197d200640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077f70 0x7f195807a430 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1964033000 tx=0x7f1964004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:52.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:52.752+0000 7f1975600640 1 -- 172.21.15.31:0/1291108034 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1978077240 con 0x7f1978072480 2024-09-11T00:41:53.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.033+0000 7f1980600640 1 -- 172.21.15.31:0/1291108034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1978153f20 con 0x7f1978072480 2024-09-11T00:41:53.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.033+0000 7f1975600640 1 -- 172.21.15.31:0/1291108034 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5710 (secure 0 0 0) 0x7f196c061c70 con 0x7f1978072480 2024-09-11T00:41:53.036 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:53.036 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":86,"btime":"2024-09-11T00:41:52:395104+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:53.036 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 86 2024-09-11T00:41:53.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 -- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077f70 msgr2=0x7f195807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077f70 0x7f195807a430 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1964033000 tx=0x7f1964004730 comp rx=0 tx=0).stop 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 -- 172.21.15.31:0/1291108034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1978072480 msgr2=0x7f19781ecbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1978072480 0x7f19781ecbd0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f196c00c480 tx=0x7f196c00c940 comp rx=0 tx=0).stop 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 -- 172.21.15.31:0/1291108034 shutdown_connections 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077f70 0x7f195807a430 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19781518b0 0x7f1978146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f19780c23a0 0x7f197806e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 --2- 172.21.15.31:0/1291108034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1978072480 0x7f19781ecbd0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 -- 172.21.15.31:0/1291108034 >> 172.21.15.31:0/1291108034 conn(0x7f19780a9750 msgr2=0x7f197814e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 -- 172.21.15.31:0/1291108034 shutdown_connections 2024-09-11T00:41:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:53.036+0000 7f1980600640 1 -- 172.21.15.31:0/1291108034 wait complete. 2024-09-11T00:41:53.056 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5911 2024-09-11T00:41:53.056 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.d 2024-09-11T00:41:57.057 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-11T00:41:57.057 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:41:57.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:41:57.062 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:41:57.062 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:57.204 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:41:57.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.242+0000 7f601de00640 1 -- 172.21.15.31:0/2098595047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60180c9b30 msgr2=0x7f601806deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:57.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.242+0000 7f601de00640 1 --2- 172.21.15.31:0/2098595047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60180c9b30 0x7f601806deb0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6004005fd0 tx=0x7f600402a540 comp rx=0 tx=0).stop 2024-09-11T00:41:57.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.243+0000 7f601de00640 1 -- 172.21.15.31:0/2098595047 shutdown_connections 2024-09-11T00:41:57.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.243+0000 7f601de00640 1 --2- 172.21.15.31:0/2098595047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60180c9b30 0x7f601806deb0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:57.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.243+0000 7f601de00640 1 --2- 172.21.15.31:0/2098595047 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60180b1160 0x7f601815a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:57.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.243+0000 7f601de00640 1 -- 172.21.15.31:0/2098595047 >> 172.21.15.31:0/2098595047 conn(0x7f60180cc260 msgr2=0x7f60180ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:57.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.243+0000 7f601de00640 1 -- 172.21.15.31:0/2098595047 shutdown_connections 2024-09-11T00:41:57.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.243+0000 7f601de00640 1 -- 172.21.15.31:0/2098595047 wait complete. 2024-09-11T00:41:57.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.244+0000 7f601de00640 1 Processor -- start 2024-09-11T00:41:57.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.244+0000 7f601de00640 1 -- start start 2024-09-11T00:41:57.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.245+0000 7f601de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60180b1160 0x7f60180abf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:57.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.245+0000 7f601de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60180bf2c0 0x7f6018151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:57.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.245+0000 7f601de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60180c9b30 0x7f60181ef720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:57.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.245+0000 7f601de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f60180ae540 con 0x7f60180c9b30 2024-09-11T00:41:57.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.245+0000 7f601de00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f60180b1540 con 0x7f60180b1160 2024-09-11T00:41:57.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.245+0000 7f601de00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f60181524d0 con 0x7f60180bf2c0 2024-09-11T00:41:57.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.245+0000 7f6017400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60180c9b30 0x7f60181ef720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.245+0000 7f6016a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60180b1160 0x7f60180abf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.246+0000 7f6016a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60180b1160 0x7f60180abf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:37592/0 (socket says 172.21.15.31:37592) 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.246+0000 7f6016a00640 1 -- 172.21.15.31:0/288031804 learned_addr learned my addr 172.21.15.31:0/288031804 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.245+0000 7f6017400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60180c9b30 0x7f60181ef720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35158/0 (socket says 172.21.15.31:35158) 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.246+0000 7f6016000640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60180bf2c0 0x7f6018151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.246+0000 7f6016a00640 1 -- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60180bf2c0 msgr2=0x7f6018151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.246+0000 7f6016a00640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60180bf2c0 0x7f6018151ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.246+0000 7f6016a00640 1 -- 172.21.15.31:0/288031804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60180c9b30 msgr2=0x7f60181ef720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.246+0000 7f6016a00640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60180c9b30 0x7f60181ef720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.246+0000 7f6016a00640 1 -- 172.21.15.31:0/288031804 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60181eef60 con 0x7f60180b1160 2024-09-11T00:41:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.246+0000 7f6017400640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60180c9b30 0x7f60181ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:41:57.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.246+0000 7f6016a00640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60180b1160 0x7f60180abf20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f600800e9f0 tx=0x7f600800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:57.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.247+0000 7f600f400640 1 -- 172.21.15.31:0/288031804 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600800cd80 con 0x7f60180b1160 2024-09-11T00:41:57.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.247+0000 7f600f400640 1 -- 172.21.15.31:0/288031804 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6008010430 con 0x7f60180b1160 2024-09-11T00:41:57.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.247+0000 7f600f400640 1 -- 172.21.15.31:0/288031804 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6008010840 con 0x7f60180b1160 2024-09-11T00:41:57.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.247+0000 7f601de00640 1 -- 172.21.15.31:0/288031804 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60181dfdd0 con 0x7f60180b1160 2024-09-11T00:41:57.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.247+0000 7f601de00640 1 -- 172.21.15.31:0/288031804 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60181e0390 con 0x7f60180b1160 2024-09-11T00:41:57.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.248+0000 7f601de00640 1 -- 172.21.15.31:0/288031804 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f601806f300 con 0x7f60180b1160 2024-09-11T00:41:57.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.250+0000 7f600f400640 1 -- 172.21.15.31:0/288031804 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f60080109e0 con 0x7f60180b1160 2024-09-11T00:41:57.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.250+0000 7f600f400640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ff8077f70 0x7f5ff807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:57.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.251+0000 7f600f400640 1 -- 172.21.15.31:0/288031804 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f600809b190 con 0x7f60180b1160 2024-09-11T00:41:57.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.251+0000 7f6016000640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ff8077f70 0x7f5ff807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:57.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.252+0000 7f6016000640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ff8077f70 0x7f5ff807a430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6004033000 tx=0x7f6004004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:57.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.256+0000 7f600f400640 1 -- 172.21.15.31:0/288031804 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6008062d90 con 0x7f60180b1160 2024-09-11T00:41:57.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.544+0000 7f601de00640 1 -- 172.21.15.31:0/288031804 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60180caa50 con 0x7f60180b1160 2024-09-11T00:41:57.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.545+0000 7f600f400640 1 -- 172.21.15.31:0/288031804 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5710 (secure 0 0 0) 0x7f60080677c0 con 0x7f60180b1160 2024-09-11T00:41:57.547 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:41:57.547 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":86,"btime":"2024-09-11T00:41:52:395104+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:41:57.547 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 86 2024-09-11T00:41:57.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 -- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ff8077f70 msgr2=0x7f5ff807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:57.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ff8077f70 0x7f5ff807a430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6004033000 tx=0x7f6004004730 comp rx=0 tx=0).stop 2024-09-11T00:41:57.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 -- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60180b1160 msgr2=0x7f60180abf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:57.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60180b1160 0x7f60180abf20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f600800e9f0 tx=0x7f600800eeb0 comp rx=0 tx=0).stop 2024-09-11T00:41:57.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 -- 172.21.15.31:0/288031804 shutdown_connections 2024-09-11T00:41:57.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ff8077f70 0x7f5ff807a430 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:57.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f60180c9b30 0x7f60181ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:57.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f60180bf2c0 0x7f6018151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:57.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 --2- 172.21.15.31:0/288031804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60180b1160 0x7f60180abf20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:57.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 -- 172.21.15.31:0/288031804 >> 172.21.15.31:0/288031804 conn(0x7f60180cc260 msgr2=0x7f60180c9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:57.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 -- 172.21.15.31:0/288031804 shutdown_connections 2024-09-11T00:41:57.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:57.548+0000 7f601de00640 1 -- 172.21.15.31:0/288031804 wait complete. 2024-09-11T00:41:57.566 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:41:59.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:41:59.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.736+0000 7fadd8a00640 1 -- 172.21.15.31:0/3959677837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40c6bb0 msgr2=0x7fadd406e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:59.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.736+0000 7fadd8a00640 1 --2- 172.21.15.31:0/3959677837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40c6bb0 0x7fadd406e070 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fadc4005fd0 tx=0x7fadc402c6c0 comp rx=0 tx=0).stop 2024-09-11T00:41:59.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.737+0000 7fadd8a00640 1 -- 172.21.15.31:0/3959677837 shutdown_connections 2024-09-11T00:41:59.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.737+0000 7fadd8a00640 1 --2- 172.21.15.31:0/3959677837 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadd41567f0 0x7fadd4156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:59.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.737+0000 7fadd8a00640 1 --2- 172.21.15.31:0/3959677837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40c6bb0 0x7fadd406e070 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:59.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.737+0000 7fadd8a00640 1 -- 172.21.15.31:0/3959677837 >> 172.21.15.31:0/3959677837 conn(0x7fadd406caf0 msgr2=0x7fadd406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:41:59.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.737+0000 7fadd8a00640 1 -- 172.21.15.31:0/3959677837 shutdown_connections 2024-09-11T00:41:59.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.737+0000 7fadd8a00640 1 -- 172.21.15.31:0/3959677837 wait complete. 2024-09-11T00:41:59.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.739+0000 7fadd8a00640 1 Processor -- start 2024-09-11T00:41:59.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.739+0000 7fadd8a00640 1 -- start start 2024-09-11T00:41:59.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadd8a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadd40c3280 0x7fadd4072750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:59.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadd8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40c6bb0 0x7fadd409c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:59.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadd8a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadd41567f0 0x7fadd41ef3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:59.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadd8a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fadd406e5b0 con 0x7fadd40c6bb0 2024-09-11T00:41:59.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadd8a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40c2f60 con 0x7fadd41567f0 2024-09-11T00:41:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadd8a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fadd4152d80 con 0x7fadd40c3280 2024-09-11T00:41:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadcbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40c6bb0 0x7fadd409c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadcbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40c6bb0 0x7fadd409c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50426/0 (socket says 172.21.15.31:50426) 2024-09-11T00:41:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadcbe00640 1 -- 172.21.15.31:0/3276372879 learned_addr learned my addr 172.21.15.31:0/3276372879 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:41:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadd2800640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadd40c3280 0x7fadd4072750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.740+0000 7fadd3200640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadd41567f0 0x7fadd41ef3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.741+0000 7fadcbe00640 1 -- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadd40c3280 msgr2=0x7fadd4072750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.741+0000 7fadcbe00640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadd40c3280 0x7fadd4072750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.741+0000 7fadcbe00640 1 -- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadd41567f0 msgr2=0x7fadd41ef3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:41:59.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.741+0000 7fadcbe00640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadd41567f0 0x7fadd41ef3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:41:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.741+0000 7fadcbe00640 1 -- 172.21.15.31:0/3276372879 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd409cb00 con 0x7fadd40c6bb0 2024-09-11T00:41:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.741+0000 7fadcbe00640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40c6bb0 0x7fadd409c470 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00c450 tx=0x7fadbc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.741+0000 7fadcaa00640 1 -- 172.21.15.31:0/3276372879 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc0052d0 con 0x7fadd40c6bb0 2024-09-11T00:41:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.741+0000 7fadcaa00640 1 -- 172.21.15.31:0/3276372879 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadbc005470 con 0x7fadd40c6bb0 2024-09-11T00:41:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.741+0000 7fadcaa00640 1 -- 172.21.15.31:0/3276372879 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc012870 con 0x7fadd40c6bb0 2024-09-11T00:41:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.741+0000 7fadd8a00640 1 -- 172.21.15.31:0/3276372879 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd4074990 con 0x7fadd40c6bb0 2024-09-11T00:41:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.742+0000 7fadd8a00640 1 -- 172.21.15.31:0/3276372879 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd41560a0 con 0x7fadd40c6bb0 2024-09-11T00:41:59.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.743+0000 7fadd8a00640 1 -- 172.21.15.31:0/3276372879 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd4072050 con 0x7fadd40c6bb0 2024-09-11T00:41:59.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.744+0000 7fadcaa00640 1 -- 172.21.15.31:0/3276372879 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fadbc0035f0 con 0x7fadd40c6bb0 2024-09-11T00:41:59.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.745+0000 7fadcaa00640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fadb0077fa0 0x7fadb007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:41:59.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.745+0000 7fadcaa00640 1 -- 172.21.15.31:0/3276372879 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7fadbc099ff0 con 0x7fadd40c6bb0 2024-09-11T00:41:59.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.746+0000 7fadd2800640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fadb0077fa0 0x7fadb007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:41:59.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.746+0000 7fadd2800640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fadb0077fa0 0x7fadb007a460 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fadc4035000 tx=0x7fadc4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:41:59.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:41:59.750+0000 7fadcaa00640 1 -- 172.21.15.31:0/3276372879 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fadd4072050 con 0x7fadd40c6bb0 2024-09-11T00:42:00.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.027+0000 7fadd8a00640 1 -- 172.21.15.31:0/3276372879 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadd4074eb0 con 0x7fadd40c6bb0 2024-09-11T00:42:00.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.028+0000 7fadcaa00640 1 -- 172.21.15.31:0/3276372879 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7fadbc061be0 con 0x7fadd40c6bb0 2024-09-11T00:42:00.030 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:42:00.030 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:42:00.030 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:42:00.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 -- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fadb0077fa0 msgr2=0x7fadb007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:42:00.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fadb0077fa0 0x7fadb007a460 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fadc4035000 tx=0x7fadc4004730 comp rx=0 tx=0).stop 2024-09-11T00:42:00.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 -- 172.21.15.31:0/3276372879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40c6bb0 msgr2=0x7fadd409c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:42:00.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40c6bb0 0x7fadd409c470 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00c450 tx=0x7fadbc00c910 comp rx=0 tx=0).stop 2024-09-11T00:42:00.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 -- 172.21.15.31:0/3276372879 shutdown_connections 2024-09-11T00:42:00.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fadb0077fa0 0x7fadb007a460 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:42:00.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadd41567f0 0x7fadd41ef3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:42:00.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fadd40c6bb0 0x7fadd409c470 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:42:00.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 --2- 172.21.15.31:0/3276372879 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fadd40c3280 0x7fadd4072750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:42:00.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 -- 172.21.15.31:0/3276372879 >> 172.21.15.31:0/3276372879 conn(0x7fadd406caf0 msgr2=0x7fadd4152280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:42:00.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 -- 172.21.15.31:0/3276372879 shutdown_connections 2024-09-11T00:42:00.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:42:00.030+0000 7fadd8a00640 1 -- 172.21.15.31:0/3276372879 wait complete. 2024-09-11T00:42:00.048 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-11T00:42:00.049 INFO:tasks.mds_thrash.fs.[a]:waiting for 63 secs before thrashing 2024-09-11T00:42:15.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:42:15.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:42:45.765 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:42:45.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:43:03.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:03.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.237+0000 7f2fa0a00640 1 -- 172.21.15.31:0/1970739307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f9c0a6e20 msgr2=0x7f2f9c1510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:03.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.237+0000 7f2fa0a00640 1 --2- 172.21.15.31:0/1970739307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f9c0a6e20 0x7f2f9c1510c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0c5ea0 tx=0x7f2f8802a4d0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.237+0000 7f2fa0a00640 1 -- 172.21.15.31:0/1970739307 shutdown_connections 2024-09-11T00:43:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.237+0000 7f2fa0a00640 1 --2- 172.21.15.31:0/1970739307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f9c0a6e20 0x7f2f9c1510c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.237+0000 7f2fa0a00640 1 --2- 172.21.15.31:0/1970739307 >> v2:172.21.15.204:3300/0 conn(0x7f2f9c0ac2c0 0x7f2f9c0ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.237+0000 7f2fa0a00640 1 -- 172.21.15.31:0/1970739307 >> 172.21.15.31:0/1970739307 conn(0x7f2f9c06caf0 msgr2=0x7f2f9c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.237+0000 7f2fa0a00640 1 -- 172.21.15.31:0/1970739307 shutdown_connections 2024-09-11T00:43:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.237+0000 7f2fa0a00640 1 -- 172.21.15.31:0/1970739307 wait complete. 2024-09-11T00:43:03.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.239+0000 7f2fa0a00640 1 Processor -- start 2024-09-11T00:43:03.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.239+0000 7f2fa0a00640 1 -- start start 2024-09-11T00:43:03.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2fa0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f9c0a6e20 0x7f2f9c0c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:03.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2fa0a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f9c0ac2c0 0x7f2f9c14a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:03.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2fa0a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f9c0acbe0 0x7f2f9c14aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:03.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2fa0a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c151780 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2fa0a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c151600 con 0x7f2f9c0ac2c0 2024-09-11T00:43:03.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2fa0a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9c151900 con 0x7f2f9c0acbe0 2024-09-11T00:43:03.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2f9a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f9c0a6e20 0x7f2f9c0c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2f9a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f9c0a6e20 0x7f2f9c0c3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33726/0 (socket says 172.21.15.31:33726) 2024-09-11T00:43:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2f9a800640 1 -- 172.21.15.31:0/2702510002 learned_addr learned my addr 172.21.15.31:0/2702510002 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2f9b200640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f9c0acbe0 0x7f2f9c14aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2f99e00640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f9c0ac2c0 0x7f2f9c14a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2f9a800640 1 -- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f9c0acbe0 msgr2=0x7f2f9c14aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2f9a800640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f9c0acbe0 0x7f2f9c14aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2f9a800640 1 -- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f9c0ac2c0 msgr2=0x7f2f9c14a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2f9a800640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f9c0ac2c0 0x7f2f9c14a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.241+0000 7f2f9a800640 1 -- 172.21.15.31:0/2702510002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c074f30 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.242+0000 7f2f9a800640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f9c0a6e20 0x7f2f9c0c3a90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00e9f0 tx=0x7f2f8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:03.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.242+0000 7f2f92a00640 1 -- 172.21.15.31:0/2702510002 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c00cd80 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.242+0000 7f2f92a00640 1 -- 172.21.15.31:0/2702510002 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f8c010430 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.242+0000 7f2fa0a00640 1 -- 172.21.15.31:0/2702510002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9c0a7dd0 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.242+0000 7f2f92a00640 1 -- 172.21.15.31:0/2702510002 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c010840 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.242+0000 7f2fa0a00640 1 -- 172.21.15.31:0/2702510002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9c0c3fd0 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.243+0000 7f2fa0a00640 1 -- 172.21.15.31:0/2702510002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f9c1510c0 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.244+0000 7f2f92a00640 1 -- 172.21.15.31:0/2702510002 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2f8c0109e0 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.245+0000 7f2f92a00640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2f7c077f70 0x7f2f7c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:03.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.246+0000 7f2f92a00640 1 -- 172.21.15.31:0/2702510002 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f2f8c09b1d0 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.246+0000 7f2f99e00640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2f7c077f70 0x7f2f7c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:03.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.247+0000 7f2f99e00640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2f7c077f70 0x7f2f7c07a430 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2f88033000 tx=0x7f2f88004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:03.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.250+0000 7f2f92a00640 1 -- 172.21.15.31:0/2702510002 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2f8c062dd0 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.523+0000 7f2fa0a00640 1 -- 172.21.15.31:0/2702510002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f9c06ebf0 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.523+0000 7f2f92a00640 1 -- 172.21.15.31:0/2702510002 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7f2f8c09a090 con 0x7f2f9c0a6e20 2024-09-11T00:43:03.525 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:03.525 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 -- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2f7c077f70 msgr2=0x7f2f7c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2f7c077f70 0x7f2f7c07a430 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2f88033000 tx=0x7f2f88004730 comp rx=0 tx=0).stop 2024-09-11T00:43:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 -- 172.21.15.31:0/2702510002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f9c0a6e20 msgr2=0x7f2f9c0c3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f9c0a6e20 0x7f2f9c0c3a90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00e9f0 tx=0x7f2f8c00eeb0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 -- 172.21.15.31:0/2702510002 shutdown_connections 2024-09-11T00:43:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2f7c077f70 0x7f2f7c07a430 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f9c0acbe0 0x7f2f9c14aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f9c0ac2c0 0x7f2f9c14a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 --2- 172.21.15.31:0/2702510002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f9c0a6e20 0x7f2f9c0c3a90 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 -- 172.21.15.31:0/2702510002 >> 172.21.15.31:0/2702510002 conn(0x7f2f9c06caf0 msgr2=0x7f2f9c14f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 -- 172.21.15.31:0/2702510002 shutdown_connections 2024-09-11T00:43:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.526+0000 7f2fa0a00640 1 -- 172.21.15.31:0/2702510002 wait complete. 2024-09-11T00:43:03.546 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-11T00:43:03.546 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-11T00:43:03.546 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:43:03.562 INFO:tasks.ceph.mds.a:Stopped 2024-09-11T00:43:03.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:03.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.792+0000 7fecee000640 1 -- 172.21.15.31:0/2292641157 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece80c6420 msgr2=0x7fece8153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:03.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.792+0000 7fecee000640 1 --2- 172.21.15.31:0/2292641157 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece80c6420 0x7fece8153310 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fecd4009860 tx=0x7fecd402a5a0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.792+0000 7fecee000640 1 -- 172.21.15.31:0/2292641157 shutdown_connections 2024-09-11T00:43:03.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.792+0000 7fecee000640 1 --2- 172.21.15.31:0/2292641157 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece80c6420 0x7fece8153310 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.792+0000 7fecee000640 1 -- 172.21.15.31:0/2292641157 >> 172.21.15.31:0/2292641157 conn(0x7fece806caf0 msgr2=0x7fece806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:03.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.792+0000 7fecee000640 1 -- 172.21.15.31:0/2292641157 shutdown_connections 2024-09-11T00:43:03.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.792+0000 7fecee000640 1 -- 172.21.15.31:0/2292641157 wait complete. 2024-09-11T00:43:03.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.794+0000 7fecee000640 1 Processor -- start 2024-09-11T00:43:03.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.795+0000 7fecee000640 1 -- start start 2024-09-11T00:43:03.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fecee000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece80c5aa0 0x7fece81ef4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:03.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fecee000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece80c6420 0x7fece814eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:03.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fecee000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece8153850 0x7fece814f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:03.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fecee000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fece80c5e80 con 0x7fece80c5aa0 2024-09-11T00:43:03.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fecee000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fece80c3160 con 0x7fece80c6420 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fecee000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fece80c6000 con 0x7fece8153850 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fececc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece80c5aa0 0x7fece81ef4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fececc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece80c5aa0 0x7fece81ef4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33728/0 (socket says 172.21.15.31:33728) 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fececc00640 1 -- 172.21.15.31:0/1291428202 learned_addr learned my addr 172.21.15.31:0/1291428202 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fece7e00640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece80c6420 0x7fece814eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7feced600640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece8153850 0x7fece814f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.796+0000 7fececc00640 1 -- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece8153850 msgr2=0x7fece814f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.797+0000 7fececc00640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece8153850 0x7fece814f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.797+0000 7fececc00640 1 -- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece80c6420 msgr2=0x7fece814eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.797+0000 7fececc00640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece80c6420 0x7fece814eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.797+0000 7fececc00640 1 -- 172.21.15.31:0/1291428202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece81589b0 con 0x7fece80c5aa0 2024-09-11T00:43:03.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.797+0000 7fececc00640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece80c5aa0 0x7fece81ef4d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00c490 tx=0x7fecdc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:03.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.797+0000 7fece4c00640 1 -- 172.21.15.31:0/1291428202 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc016020 con 0x7fece80c5aa0 2024-09-11T00:43:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.797+0000 7fece4c00640 1 -- 172.21.15.31:0/1291428202 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecdc0129e0 con 0x7fece80c5aa0 2024-09-11T00:43:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.797+0000 7fece4c00640 1 -- 172.21.15.31:0/1291428202 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc012e30 con 0x7fece80c5aa0 2024-09-11T00:43:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.797+0000 7fecee000640 1 -- 172.21.15.31:0/1291428202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece81eff50 con 0x7fece80c5aa0 2024-09-11T00:43:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.797+0000 7fecee000640 1 -- 172.21.15.31:0/1291428202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece8157130 con 0x7fece80c5aa0 2024-09-11T00:43:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.799+0000 7fecee000640 1 -- 172.21.15.31:0/1291428202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece8153310 con 0x7fece80c5aa0 2024-09-11T00:43:03.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.800+0000 7fece4c00640 1 -- 172.21.15.31:0/1291428202 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fecdc01a3b0 con 0x7fece80c5aa0 2024-09-11T00:43:03.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.801+0000 7fece4c00640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fecc8077f70 0x7fecc807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:03.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.801+0000 7fece4c00640 1 -- 172.21.15.31:0/1291428202 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7fecdc09a2b0 con 0x7fece80c5aa0 2024-09-11T00:43:03.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.802+0000 7fece7e00640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fecc8077f70 0x7fecc807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:03.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.802+0000 7fece7e00640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fecc8077f70 0x7fecc807a430 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fecd4034000 tx=0x7fecd4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:03.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:03.807+0000 7fece4c00640 1 -- 172.21.15.31:0/1291428202 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fecdc020070 con 0x7fece80c5aa0 2024-09-11T00:43:04.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.109+0000 7fecee000640 1 -- 172.21.15.31:0/1291428202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fece806a200 con 0x7fece80c5aa0 2024-09-11T00:43:04.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.110+0000 7fece4c00640 1 -- 172.21.15.31:0/1291428202 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7fecdc061df0 con 0x7fece80c5aa0 2024-09-11T00:43:04.111 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:04.111 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:04.112 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:04.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 -- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fecc8077f70 msgr2=0x7fecc807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:04.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fecc8077f70 0x7fecc807a430 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fecd4034000 tx=0x7fecd4004730 comp rx=0 tx=0).stop 2024-09-11T00:43:04.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 -- 172.21.15.31:0/1291428202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece80c5aa0 msgr2=0x7fece81ef4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:04.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece80c5aa0 0x7fece81ef4d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00c490 tx=0x7fecdc00c950 comp rx=0 tx=0).stop 2024-09-11T00:43:04.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 -- 172.21.15.31:0/1291428202 shutdown_connections 2024-09-11T00:43:04.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fecc8077f70 0x7fecc807a430 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:04.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fece8153850 0x7fece814f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:04.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fece80c6420 0x7fece814eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:04.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 --2- 172.21.15.31:0/1291428202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fece80c5aa0 0x7fece81ef4d0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:04.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 -- 172.21.15.31:0/1291428202 >> 172.21.15.31:0/1291428202 conn(0x7fece806caf0 msgr2=0x7fece80c6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:04.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 -- 172.21.15.31:0/1291428202 shutdown_connections 2024-09-11T00:43:04.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:04.112+0000 7fecee000640 1 -- 172.21.15.31:0/1291428202 wait complete. 2024-09-11T00:43:04.132 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:43:06.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.296+0000 7fa086a00640 1 -- 172.21.15.31:0/1940485867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa080072480 msgr2=0x7fa0801511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.296+0000 7fa086a00640 1 --2- 172.21.15.31:0/1940485867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa080072480 0x7fa0801511a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fa080075260 tx=0x7fa07002a5c0 comp rx=0 tx=0).stop 2024-09-11T00:43:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.296+0000 7fa086a00640 1 -- 172.21.15.31:0/1940485867 shutdown_connections 2024-09-11T00:43:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.296+0000 7fa086a00640 1 --2- 172.21.15.31:0/1940485867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa080072480 0x7fa0801511a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.296+0000 7fa086a00640 1 -- 172.21.15.31:0/1940485867 >> 172.21.15.31:0/1940485867 conn(0x7fa0800a9750 msgr2=0x7fa0800a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.296+0000 7fa086a00640 1 -- 172.21.15.31:0/1940485867 shutdown_connections 2024-09-11T00:43:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.297+0000 7fa086a00640 1 -- 172.21.15.31:0/1940485867 wait complete. 2024-09-11T00:43:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.298+0000 7fa086a00640 1 Processor -- start 2024-09-11T00:43:06.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.298+0000 7fa086a00640 1 -- start start 2024-09-11T00:43:06.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa086a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa080072480 0x7fa0800c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:06.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa086a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0800c23a0 0x7fa0801eae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa086a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa080153f20 0x7fa0801eb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa086a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa080074200 con 0x7fa0800c23a0 2024-09-11T00:43:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa086a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa080074080 con 0x7fa080072480 2024-09-11T00:43:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa086a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa080072860 con 0x7fa080153f20 2024-09-11T00:43:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa077e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0800c23a0 0x7fa0801eae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa07fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa080072480 0x7fa0800c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa07fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa080072480 0x7fa0800c41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:51144/0 (socket says 172.21.15.31:51144) 2024-09-11T00:43:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa077e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0800c23a0 0x7fa0801eae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33750/0 (socket says 172.21.15.31:33750) 2024-09-11T00:43:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa07fe00640 1 -- 172.21.15.31:0/2610935246 learned_addr learned my addr 172.21.15.31:0/2610935246 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.299+0000 7fa084a00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa080153f20 0x7fa0801eb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa077e00640 1 -- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa080153f20 msgr2=0x7fa0801eb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa077e00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa080153f20 0x7fa0801eb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa077e00640 1 -- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa080072480 msgr2=0x7fa0800c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa077e00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa080072480 0x7fa0800c41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa077e00640 1 -- 172.21.15.31:0/2610935246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08014a240 con 0x7fa0800c23a0 2024-09-11T00:43:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa077e00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0800c23a0 0x7fa0801eae20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fa06800c3c0 tx=0x7fa06800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa07cc00640 1 -- 172.21.15.31:0/2610935246 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068011790 con 0x7fa0800c23a0 2024-09-11T00:43:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa07cc00640 1 -- 172.21.15.31:0/2610935246 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa068017070 con 0x7fa0800c23a0 2024-09-11T00:43:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa07cc00640 1 -- 172.21.15.31:0/2610935246 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068011ce0 con 0x7fa0800c23a0 2024-09-11T00:43:06.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa086a00640 1 -- 172.21.15.31:0/2610935246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08014a520 con 0x7fa0800c23a0 2024-09-11T00:43:06.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.300+0000 7fa086a00640 1 -- 172.21.15.31:0/2610935246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa08014aae0 con 0x7fa0800c23a0 2024-09-11T00:43:06.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.301+0000 7fa086a00640 1 -- 172.21.15.31:0/2610935246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa080077240 con 0x7fa0800c23a0 2024-09-11T00:43:06.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.303+0000 7fa07cc00640 1 -- 172.21.15.31:0/2610935246 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa06801c460 con 0x7fa0800c23a0 2024-09-11T00:43:06.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.303+0000 7fa07cc00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa05c077f70 0x7fa05c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:06.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.304+0000 7fa07cc00640 1 -- 172.21.15.31:0/2610935246 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7fa06809a240 con 0x7fa0800c23a0 2024-09-11T00:43:06.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.304+0000 7fa07fe00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa05c077f70 0x7fa05c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:06.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.304+0000 7fa07fe00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa05c077f70 0x7fa05c07a430 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa0700095f0 tx=0x7fa070004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:06.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.308+0000 7fa07cc00640 1 -- 172.21.15.31:0/2610935246 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa080077240 con 0x7fa0800c23a0 2024-09-11T00:43:06.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.549+0000 7fa086a00640 1 -- 172.21.15.31:0/2610935246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0801511a0 con 0x7fa0800c23a0 2024-09-11T00:43:06.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.549+0000 7fa07cc00640 1 -- 172.21.15.31:0/2610935246 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7fa068061e10 con 0x7fa0800c23a0 2024-09-11T00:43:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:06.552 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:06.552 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:06.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 -- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa05c077f70 msgr2=0x7fa05c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:06.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa05c077f70 0x7fa05c07a430 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa0700095f0 tx=0x7fa070004730 comp rx=0 tx=0).stop 2024-09-11T00:43:06.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 -- 172.21.15.31:0/2610935246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0800c23a0 msgr2=0x7fa0801eae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:06.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0800c23a0 0x7fa0801eae20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fa06800c3c0 tx=0x7fa06800c880 comp rx=0 tx=0).stop 2024-09-11T00:43:06.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 -- 172.21.15.31:0/2610935246 shutdown_connections 2024-09-11T00:43:06.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa05c077f70 0x7fa05c07a430 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:06.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa080153f20 0x7fa0801eb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:06.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0800c23a0 0x7fa0801eae20 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:06.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 --2- 172.21.15.31:0/2610935246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa080072480 0x7fa0800c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:06.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 -- 172.21.15.31:0/2610935246 >> 172.21.15.31:0/2610935246 conn(0x7fa0800a9750 msgr2=0x7fa08014c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:06.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 -- 172.21.15.31:0/2610935246 shutdown_connections 2024-09-11T00:43:06.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:06.552+0000 7fa086a00640 1 -- 172.21.15.31:0/2610935246 wait complete. 2024-09-11T00:43:06.571 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:43:08.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.742+0000 7f5179c00640 1 -- 172.21.15.31:0/1213793942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51741539c0 msgr2=0x7f5174159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.742+0000 7f5179c00640 1 --2- 172.21.15.31:0/1213793942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51741539c0 0x7f5174159670 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f516c005fd0 tx=0x7f516c02a8a0 comp rx=0 tx=0).stop 2024-09-11T00:43:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.743+0000 7f5179c00640 1 -- 172.21.15.31:0/1213793942 shutdown_connections 2024-09-11T00:43:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.743+0000 7f5179c00640 1 --2- 172.21.15.31:0/1213793942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51741539c0 0x7f5174159670 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.743+0000 7f5179c00640 1 --2- 172.21.15.31:0/1213793942 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5174156ad0 0x7f5174153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.743+0000 7f5179c00640 1 -- 172.21.15.31:0/1213793942 >> 172.21.15.31:0/1213793942 conn(0x7f51740a9750 msgr2=0x7f51740a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.743+0000 7f5179c00640 1 -- 172.21.15.31:0/1213793942 shutdown_connections 2024-09-11T00:43:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.743+0000 7f5179c00640 1 -- 172.21.15.31:0/1213793942 wait complete. 2024-09-11T00:43:08.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.744+0000 7f5179c00640 1 Processor -- start 2024-09-11T00:43:08.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.744+0000 7f5179c00640 1 -- start start 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.745+0000 7f5179c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51740c5b10 0x7f517414ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.745+0000 7f5179c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51741539c0 0x7f5174154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.745+0000 7f5179c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5174156ad0 0x7f517414aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.745+0000 7f5179c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5174156ef0 con 0x7f51740c5b10 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.745+0000 7f5179c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f51740c7320 con 0x7f51741539c0 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.745+0000 7f5179c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f51740c3950 con 0x7f5174156ad0 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.745+0000 7f5172a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51740c5b10 0x7f517414ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.745+0000 7f5172a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51740c5b10 0x7f517414ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35224/0 (socket says 172.21.15.31:35224) 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.745+0000 7f5172a00640 1 -- 172.21.15.31:0/432572537 learned_addr learned my addr 172.21.15.31:0/432572537 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.745+0000 7f516a000640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51741539c0 0x7f5174154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.746+0000 7f5173400640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5174156ad0 0x7f517414aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.746+0000 7f5172a00640 1 -- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5174156ad0 msgr2=0x7f517414aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.746+0000 7f5172a00640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5174156ad0 0x7f517414aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.746+0000 7f5172a00640 1 -- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51741539c0 msgr2=0x7f5174154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.746+0000 7f5172a00640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51741539c0 0x7f5174154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.746+0000 7f5172a00640 1 -- 172.21.15.31:0/432572537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5174155c10 con 0x7f51740c5b10 2024-09-11T00:43:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.746+0000 7f5172a00640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51740c5b10 0x7f517414ed50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f516400c450 tx=0x7f516400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.746+0000 7f516b400640 1 -- 172.21.15.31:0/432572537 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164016020 con 0x7f51740c5b10 2024-09-11T00:43:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.747+0000 7f516b400640 1 -- 172.21.15.31:0/432572537 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5164012920 con 0x7f51740c5b10 2024-09-11T00:43:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.747+0000 7f516b400640 1 -- 172.21.15.31:0/432572537 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164012d70 con 0x7f51740c5b10 2024-09-11T00:43:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.747+0000 7f5179c00640 1 -- 172.21.15.31:0/432572537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5174073250 con 0x7f51740c5b10 2024-09-11T00:43:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.747+0000 7f5179c00640 1 -- 172.21.15.31:0/432572537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5174073810 con 0x7f51740c5b10 2024-09-11T00:43:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.748+0000 7f5179c00640 1 -- 172.21.15.31:0/432572537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5174077240 con 0x7f51740c5b10 2024-09-11T00:43:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.749+0000 7f516b400640 1 -- 172.21.15.31:0/432572537 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f516401b400 con 0x7f51740c5b10 2024-09-11T00:43:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.750+0000 7f516b400640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5150077f70 0x7f515007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.750+0000 7f516b400640 1 -- 172.21.15.31:0/432572537 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f516409a7c0 con 0x7f51740c5b10 2024-09-11T00:43:08.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.750+0000 7f516a000640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5150077f70 0x7f515007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:08.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.751+0000 7f516a000640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5150077f70 0x7f515007a430 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5174153100 tx=0x7f515c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:08.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:08.754+0000 7f516b400640 1 -- 172.21.15.31:0/432572537 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5174077240 con 0x7f51740c5b10 2024-09-11T00:43:09.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.013+0000 7f5179c00640 1 -- 172.21.15.31:0/432572537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51740c1e70 con 0x7f51740c5b10 2024-09-11T00:43:09.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.014+0000 7f516b400640 1 -- 172.21.15.31:0/432572537 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7f5164020020 con 0x7f51740c5b10 2024-09-11T00:43:09.016 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:09.016 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:09.016 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 -- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5150077f70 msgr2=0x7f515007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5150077f70 0x7f515007a430 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5174153100 tx=0x7f515c005f90 comp rx=0 tx=0).stop 2024-09-11T00:43:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 -- 172.21.15.31:0/432572537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51740c5b10 msgr2=0x7f517414ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51740c5b10 0x7f517414ed50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f516400c450 tx=0x7f516400c910 comp rx=0 tx=0).stop 2024-09-11T00:43:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 -- 172.21.15.31:0/432572537 shutdown_connections 2024-09-11T00:43:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5150077f70 0x7f515007a430 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5174156ad0 0x7f517414aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51741539c0 0x7f5174154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 --2- 172.21.15.31:0/432572537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51740c5b10 0x7f517414ed50 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:09.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 -- 172.21.15.31:0/432572537 >> 172.21.15.31:0/432572537 conn(0x7f51740a9750 msgr2=0x7f51740c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:09.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 -- 172.21.15.31:0/432572537 shutdown_connections 2024-09-11T00:43:09.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:09.016+0000 7f5179c00640 1 -- 172.21.15.31:0/432572537 wait complete. 2024-09-11T00:43:09.033 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:43:11.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:11.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:11.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.218+0000 7f48cfe00640 1 -- 172.21.15.31:0/2523017567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48d008c9b0 msgr2=0x7f48d00c47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.218+0000 7f48cfe00640 1 --2- 172.21.15.31:0/2523017567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48d008c9b0 0x7f48d00c47a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f48d00ad0b0 tx=0x7f48b802c640 comp rx=0 tx=0).stop 2024-09-11T00:43:11.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.218+0000 7f48cfe00640 1 -- 172.21.15.31:0/2523017567 shutdown_connections 2024-09-11T00:43:11.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.218+0000 7f48cfe00640 1 --2- 172.21.15.31:0/2523017567 >> v2:172.21.15.204:3300/0 conn(0x7f48d006f8e0 0x7f48d006fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.218+0000 7f48cfe00640 1 --2- 172.21.15.31:0/2523017567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48d008c9b0 0x7f48d00c47a0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.218+0000 7f48cfe00640 1 --2- 172.21.15.31:0/2523017567 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48d00c43c0 0x7f48d006dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.218+0000 7f48cfe00640 1 -- 172.21.15.31:0/2523017567 >> 172.21.15.31:0/2523017567 conn(0x7f48d006caf0 msgr2=0x7f48d006ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:11.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.218+0000 7f48cfe00640 1 -- 172.21.15.31:0/2523017567 shutdown_connections 2024-09-11T00:43:11.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.218+0000 7f48cfe00640 1 -- 172.21.15.31:0/2523017567 wait complete. 2024-09-11T00:43:11.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.220+0000 7f48cfe00640 1 Processor -- start 2024-09-11T00:43:11.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.221+0000 7f48cfe00640 1 -- start start 2024-09-11T00:43:11.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48cfe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48d006f8e0 0x7f48d01fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48cfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48d008c9b0 0x7f48d00a80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48cfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48d00c43c0 0x7f48d01efc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48cfe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f48d0162d80 con 0x7f48d00c43c0 2024-09-11T00:43:11.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48cfe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f48d008c690 con 0x7f48d008c9b0 2024-09-11T00:43:11.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48cfe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f48d0163310 con 0x7f48d006f8e0 2024-09-11T00:43:11.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48cea00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48d006f8e0 0x7f48d01fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48c7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48d008c9b0 0x7f48d00a80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48cea00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48d006f8e0 0x7f48d01fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:60218/0 (socket says 172.21.15.31:60218) 2024-09-11T00:43:11.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48cf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48d00c43c0 0x7f48d01efc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48cea00640 1 -- 172.21.15.31:0/2850841625 learned_addr learned my addr 172.21.15.31:0/2850841625 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.222+0000 7f48c7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48d008c9b0 0x7f48d00a80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:39324/0 (socket says 172.21.15.31:39324) 2024-09-11T00:43:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48cea00640 1 -- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48d008c9b0 msgr2=0x7f48d00a80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48cea00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48d008c9b0 0x7f48d00a80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48cea00640 1 -- 172.21.15.31:0/2850841625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48d00c43c0 msgr2=0x7f48d01efc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48cea00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48d00c43c0 0x7f48d01efc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48cea00640 1 -- 172.21.15.31:0/2850841625 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d01ec300 con 0x7f48d006f8e0 2024-09-11T00:43:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48c7e00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48d008c9b0 0x7f48d00a80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:43:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48cf400640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48d00c43c0 0x7f48d01efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:43:11.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48cea00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48d006f8e0 0x7f48d01fc590 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f48c000e7c0 tx=0x7f48c000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48c6a00640 1 -- 172.21.15.31:0/2850841625 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c000ccc0 con 0x7f48d006f8e0 2024-09-11T00:43:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48c6a00640 1 -- 172.21.15.31:0/2850841625 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48c0010430 con 0x7f48d006f8e0 2024-09-11T00:43:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48c6a00640 1 -- 172.21.15.31:0/2850841625 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c0010860 con 0x7f48d006f8e0 2024-09-11T00:43:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.223+0000 7f48cfe00640 1 -- 172.21.15.31:0/2850841625 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d015dc40 con 0x7f48d006f8e0 2024-09-11T00:43:11.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.224+0000 7f48cfe00640 1 -- 172.21.15.31:0/2850841625 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d01f0160 con 0x7f48d006f8e0 2024-09-11T00:43:11.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.225+0000 7f48cfe00640 1 -- 172.21.15.31:0/2850841625 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d006fd90 con 0x7f48d006f8e0 2024-09-11T00:43:11.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.226+0000 7f48c6a00640 1 -- 172.21.15.31:0/2850841625 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f48c0010a00 con 0x7f48d006f8e0 2024-09-11T00:43:11.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.227+0000 7f48c6a00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f48ac077fa0 0x7f48ac07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.227+0000 7f48c6a00640 1 -- 172.21.15.31:0/2850841625 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f48c009af10 con 0x7f48d006f8e0 2024-09-11T00:43:11.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.227+0000 7f48c7e00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f48ac077fa0 0x7f48ac07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.228+0000 7f48c7e00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f48ac077fa0 0x7f48ac07a460 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f48b8034000 tx=0x7f48b8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:11.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.232+0000 7f48c6a00640 1 -- 172.21.15.31:0/2850841625 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f48c0062ae0 con 0x7f48d006f8e0 2024-09-11T00:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.384+0000 7f70ac000640 1 -- 172.21.15.31:0/3569442852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70a4072480 msgr2=0x7f70a41511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.384+0000 7f70ac000640 1 --2- 172.21.15.31:0/3569442852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70a4072480 0x7f70a41511a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7098005fd0 tx=0x7f709802a670 comp rx=0 tx=0).stop 2024-09-11T00:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.385+0000 7f70ac000640 1 -- 172.21.15.31:0/3569442852 shutdown_connections 2024-09-11T00:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.385+0000 7f70ac000640 1 --2- 172.21.15.31:0/3569442852 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70a40c23a0 0x7f70a41516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.385+0000 7f70ac000640 1 --2- 172.21.15.31:0/3569442852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70a4072480 0x7f70a41511a0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.385+0000 7f70ac000640 1 -- 172.21.15.31:0/3569442852 >> 172.21.15.31:0/3569442852 conn(0x7f70a40a9750 msgr2=0x7f70a40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.385+0000 7f70ac000640 1 -- 172.21.15.31:0/3569442852 shutdown_connections 2024-09-11T00:43:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.385+0000 7f70ac000640 1 -- 172.21.15.31:0/3569442852 wait complete. 2024-09-11T00:43:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.386+0000 7f70ac000640 1 Processor -- start 2024-09-11T00:43:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.386+0000 7f70ac000640 1 -- start start 2024-09-11T00:43:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.387+0000 7f70ac000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70a4072480 0x7f70a41e9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.387+0000 7f70ac000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70a40c23a0 0x7f70a40c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.387+0000 7f70ac000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70a4153f20 0x7f70a41eda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.387+0000 7f70ac000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4074200 con 0x7f70a4072480 2024-09-11T00:43:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.387+0000 7f70ac000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4074080 con 0x7f70a40c23a0 2024-09-11T00:43:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.387+0000 7f70ac000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f70a4072860 con 0x7f70a4153f20 2024-09-11T00:43:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70a9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70a4072480 0x7f70a41e9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70a9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70a4072480 0x7f70a41e9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35266/0 (socket says 172.21.15.31:35266) 2024-09-11T00:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70a9600640 1 -- 172.21.15.31:0/2475046093 learned_addr learned my addr 172.21.15.31:0/2475046093 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70aa000640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70a4153f20 0x7f70a41eda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70a8c00640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70a40c23a0 0x7f70a40c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70a9600640 1 -- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70a4153f20 msgr2=0x7f70a41eda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70a9600640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70a4153f20 0x7f70a41eda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70a9600640 1 -- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70a40c23a0 msgr2=0x7f70a40c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70a9600640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70a40c23a0 0x7f70a40c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70a9600640 1 -- 172.21.15.31:0/2475046093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a41ea0c0 con 0x7f70a4072480 2024-09-11T00:43:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.388+0000 7f70a9600640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70a4072480 0x7f70a41e9b80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7098009850 tx=0x7f70980038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.389+0000 7f70a1600640 1 -- 172.21.15.31:0/2475046093 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709803f070 con 0x7f70a4072480 2024-09-11T00:43:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.389+0000 7f70a1600640 1 -- 172.21.15.31:0/2475046093 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70980047a0 con 0x7f70a4072480 2024-09-11T00:43:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.389+0000 7f70ac000640 1 -- 172.21.15.31:0/2475046093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a4144210 con 0x7f70a4072480 2024-09-11T00:43:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.389+0000 7f70a1600640 1 -- 172.21.15.31:0/2475046093 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098035030 con 0x7f70a4072480 2024-09-11T00:43:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.389+0000 7f70ac000640 1 -- 172.21.15.31:0/2475046093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a4143a80 con 0x7f70a4072480 2024-09-11T00:43:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.390+0000 7f70ac000640 1 -- 172.21.15.31:0/2475046093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a41ed660 con 0x7f70a4072480 2024-09-11T00:43:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.391+0000 7f70a1600640 1 -- 172.21.15.31:0/2475046093 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7098004b90 con 0x7f70a4072480 2024-09-11T00:43:11.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.392+0000 7f70a1600640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7084077f70 0x7f708407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.392+0000 7f70a1600640 1 -- 172.21.15.31:0/2475046093 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f70980bb6e0 con 0x7f70a4072480 2024-09-11T00:43:11.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.392+0000 7f70a8c00640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7084077f70 0x7f708407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.393+0000 7f70a8c00640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7084077f70 0x7f708407a430 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f70a4148250 tx=0x7f7090005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:11.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.401+0000 7f70a1600640 1 -- 172.21.15.31:0/2475046093 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f70980832b0 con 0x7f70a4072480 2024-09-11T00:43:11.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.478+0000 7f48cfe00640 1 -- 172.21.15.31:0/2850841625 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48d00c1bd0 con 0x7f48d006f8e0 2024-09-11T00:43:11.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.479+0000 7f48c6a00640 1 -- 172.21.15.31:0/2850841625 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7f48c0067510 con 0x7f48d006f8e0 2024-09-11T00:43:11.481 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:11.481 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 -- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f48ac077fa0 msgr2=0x7f48ac07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f48ac077fa0 0x7f48ac07a460 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f48b8034000 tx=0x7f48b8004730 comp rx=0 tx=0).stop 2024-09-11T00:43:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 -- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48d006f8e0 msgr2=0x7f48d01fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48d006f8e0 0x7f48d01fc590 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f48c000e7c0 tx=0x7f48c000ec80 comp rx=0 tx=0).stop 2024-09-11T00:43:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 -- 172.21.15.31:0/2850841625 shutdown_connections 2024-09-11T00:43:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f48ac077fa0 0x7f48ac07a460 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48d00c43c0 0x7f48d01efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48d008c9b0 0x7f48d00a80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 --2- 172.21.15.31:0/2850841625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48d006f8e0 0x7f48d01fc590 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 -- 172.21.15.31:0/2850841625 >> 172.21.15.31:0/2850841625 conn(0x7f48d006caf0 msgr2=0x7f48d0160bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 -- 172.21.15.31:0/2850841625 shutdown_connections 2024-09-11T00:43:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.482+0000 7f48cfe00640 1 -- 172.21.15.31:0/2850841625 wait complete. 2024-09-11T00:43:11.500 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:43:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.637+0000 7f70ac000640 1 -- 172.21.15.31:0/2475046093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70a41511a0 con 0x7f70a4072480 2024-09-11T00:43:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.637+0000 7f70a1600640 1 -- 172.21.15.31:0/2475046093 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7f7098087ce0 con 0x7f70a4072480 2024-09-11T00:43:11.639 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:11.640 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:11.640 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.639+0000 7f70ac000640 1 -- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7084077f70 msgr2=0x7f708407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7084077f70 0x7f708407a430 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f70a4148250 tx=0x7f7090005f90 comp rx=0 tx=0).stop 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 -- 172.21.15.31:0/2475046093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70a4072480 msgr2=0x7f70a41e9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70a4072480 0x7f70a41e9b80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7098009850 tx=0x7f70980038a0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 -- 172.21.15.31:0/2475046093 shutdown_connections 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7084077f70 0x7f708407a430 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70a4153f20 0x7f70a41eda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f70a40c23a0 0x7f70a40c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 --2- 172.21.15.31:0/2475046093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70a4072480 0x7f70a41e9b80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 -- 172.21.15.31:0/2475046093 >> 172.21.15.31:0/2475046093 conn(0x7f70a40a9750 msgr2=0x7f70a414fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 -- 172.21.15.31:0/2475046093 shutdown_connections 2024-09-11T00:43:11.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.640+0000 7f70ac000640 1 -- 172.21.15.31:0/2475046093 wait complete. 2024-09-11T00:43:11.657 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-11T00:43:11.657 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-11T00:43:11.657 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:43:11.673 INFO:tasks.ceph.mds.c:Stopped 2024-09-11T00:43:11.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:11.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.873+0000 7fb27ce00640 1 -- 172.21.15.31:0/3767310864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2781518b0 msgr2=0x7fb2780c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.873+0000 7fb27ce00640 1 --2- 172.21.15.31:0/3767310864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2781518b0 0x7fb2780c3600 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb2780add10 tx=0x7fb27002ab40 comp rx=0 tx=0).stop 2024-09-11T00:43:11.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.873+0000 7fb27ce00640 1 -- 172.21.15.31:0/3767310864 shutdown_connections 2024-09-11T00:43:11.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.873+0000 7fb27ce00640 1 --2- 172.21.15.31:0/3767310864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2781518b0 0x7fb2780c3600 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.873+0000 7fb27ce00640 1 --2- 172.21.15.31:0/3767310864 >> v2:172.21.15.204:3300/0 conn(0x7fb2780c23a0 0x7fb2780c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.873+0000 7fb27ce00640 1 -- 172.21.15.31:0/3767310864 >> 172.21.15.31:0/3767310864 conn(0x7fb2780a9750 msgr2=0x7fb2780a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:11.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.873+0000 7fb27ce00640 1 -- 172.21.15.31:0/3767310864 shutdown_connections 2024-09-11T00:43:11.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.873+0000 7fb27ce00640 1 -- 172.21.15.31:0/3767310864 wait complete. 2024-09-11T00:43:11.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.874+0000 7fb27ce00640 1 Processor -- start 2024-09-11T00:43:11.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.874+0000 7fb27ce00640 1 -- start start 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.875+0000 7fb27ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb278072480 0x7fb2780c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.875+0000 7fb27ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2780c23a0 0x7fb278146c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.875+0000 7fb27ce00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2781518b0 0x7fb278147180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.875+0000 7fb27ce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb278074200 con 0x7fb2780c23a0 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.875+0000 7fb27ce00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb278074080 con 0x7fb278072480 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.875+0000 7fb27ce00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb278072860 con 0x7fb2781518b0 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.875+0000 7fb275400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2780c23a0 0x7fb278146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb275400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2780c23a0 0x7fb278146c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35286/0 (socket says 172.21.15.31:35286) 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb275400640 1 -- 172.21.15.31:0/2532200438 learned_addr learned my addr 172.21.15.31:0/2532200438 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb276800640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2781518b0 0x7fb278147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb275e00640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb278072480 0x7fb2780c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb275400640 1 -- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2781518b0 msgr2=0x7fb278147180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb275400640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2781518b0 0x7fb278147180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb275400640 1 -- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb278072480 msgr2=0x7fb2780c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:11.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb275400640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb278072480 0x7fb2780c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:11.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb275400640 1 -- 172.21.15.31:0/2532200438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27814d190 con 0x7fb2780c23a0 2024-09-11T00:43:11.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb275e00640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb278072480 0x7fb2780c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:43:11.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb275400640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2780c23a0 0x7fb278146c40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb26400e9c0 tx=0x7fb26400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.876+0000 7fb26e000640 1 -- 172.21.15.31:0/2532200438 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26400cd80 con 0x7fb2780c23a0 2024-09-11T00:43:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.877+0000 7fb26e000640 1 -- 172.21.15.31:0/2532200438 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb264005020 con 0x7fb2780c23a0 2024-09-11T00:43:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.877+0000 7fb26e000640 1 -- 172.21.15.31:0/2532200438 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb264005470 con 0x7fb2780c23a0 2024-09-11T00:43:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.877+0000 7fb27ce00640 1 -- 172.21.15.31:0/2532200438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2781e1590 con 0x7fb2780c23a0 2024-09-11T00:43:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.877+0000 7fb27ce00640 1 -- 172.21.15.31:0/2532200438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2781e0ab0 con 0x7fb2780c23a0 2024-09-11T00:43:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.878+0000 7fb27ce00640 1 -- 172.21.15.31:0/2532200438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb278153f20 con 0x7fb2780c23a0 2024-09-11T00:43:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.879+0000 7fb26e000640 1 -- 172.21.15.31:0/2532200438 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb2640047a0 con 0x7fb2780c23a0 2024-09-11T00:43:11.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.880+0000 7fb26e000640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb258077f70 0x7fb25807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.880+0000 7fb26e000640 1 -- 172.21.15.31:0/2532200438 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7fb26409b110 con 0x7fb2780c23a0 2024-09-11T00:43:11.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.880+0000 7fb275e00640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb258077f70 0x7fb25807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:11.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.881+0000 7fb275e00640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb258077f70 0x7fb25807a430 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb268000f60 tx=0x7fb268009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:11.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:11.884+0000 7fb26e000640 1 -- 172.21.15.31:0/2532200438 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb27814d190 con 0x7fb2780c23a0 2024-09-11T00:43:12.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.115+0000 7fb27ce00640 1 -- 172.21.15.31:0/2532200438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb27814b520 con 0x7fb2780c23a0 2024-09-11T00:43:12.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.116+0000 7fb26e000640 1 -- 172.21.15.31:0/2532200438 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7fb2640632a0 con 0x7fb2780c23a0 2024-09-11T00:43:12.118 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:12.118 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:12.118 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.118+0000 7fb27ce00640 1 -- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb258077f70 msgr2=0x7fb25807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.118+0000 7fb27ce00640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb258077f70 0x7fb25807a430 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb268000f60 tx=0x7fb268009040 comp rx=0 tx=0).stop 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.118+0000 7fb27ce00640 1 -- 172.21.15.31:0/2532200438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2780c23a0 msgr2=0x7fb278146c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.118+0000 7fb27ce00640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2780c23a0 0x7fb278146c40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb26400e9c0 tx=0x7fb26400ee80 comp rx=0 tx=0).stop 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.118+0000 7fb27ce00640 1 -- 172.21.15.31:0/2532200438 shutdown_connections 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.118+0000 7fb27ce00640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb258077f70 0x7fb25807a430 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.118+0000 7fb27ce00640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2781518b0 0x7fb278147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.118+0000 7fb27ce00640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2780c23a0 0x7fb278146c40 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.118+0000 7fb27ce00640 1 --2- 172.21.15.31:0/2532200438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb278072480 0x7fb2780c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.118+0000 7fb27ce00640 1 -- 172.21.15.31:0/2532200438 >> 172.21.15.31:0/2532200438 conn(0x7fb2780a9750 msgr2=0x7fb2780a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.119+0000 7fb27ce00640 1 -- 172.21.15.31:0/2532200438 shutdown_connections 2024-09-11T00:43:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:12.119+0000 7fb27ce00640 1 -- 172.21.15.31:0/2532200438 wait complete. 2024-09-11T00:43:12.136 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:43:13.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.671+0000 7f255ba00640 1 -- 172.21.15.31:0/3413817827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f253c004d90 msgr2=0x7f253c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.671+0000 7f255ba00640 1 --2- 172.21.15.31:0/3413817827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f253c004d90 0x7f253c005210 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f254c005fd0 tx=0x7f253c005830 comp rx=0 tx=0).stop 2024-09-11T00:43:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.671+0000 7f2559a00640 1 -- 172.21.15.31:0/3413817827 reap_dead start 2024-09-11T00:43:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.671+0000 7f255ba00640 1 -- 172.21.15.31:0/3413817827 shutdown_connections 2024-09-11T00:43:13.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.671+0000 7f255ba00640 1 -- 172.21.15.31:0/3413817827 >> 172.21.15.31:0/3413817827 conn(0x7f25540a9750 msgr2=0x7f25540a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.671+0000 7f255ba00640 1 -- 172.21.15.31:0/3413817827 shutdown_connections 2024-09-11T00:43:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.671+0000 7f255ba00640 1 -- 172.21.15.31:0/3413817827 wait complete. 2024-09-11T00:43:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.672+0000 7f255ba00640 1 Processor -- start 2024-09-11T00:43:13.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.672+0000 7f255ba00640 1 -- start start 2024-09-11T00:43:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.673+0000 7f255ba00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f253c004d90 0x7f25541dedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.673+0000 7f255ba00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25541556e0 0x7f25541df760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.673+0000 7f255ba00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25541581b0 0x7f25540c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.673+0000 7f255ba00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2554074200 con 0x7f25541581b0 2024-09-11T00:43:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.673+0000 7f255ba00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2554074080 con 0x7f253c004d90 2024-09-11T00:43:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.673+0000 7f255ba00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2554072860 con 0x7f25541556e0 2024-09-11T00:43:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2559a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25541581b0 0x7f25540c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2559a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25541581b0 0x7f25540c30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35304/0 (socket says 172.21.15.31:35304) 2024-09-11T00:43:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2559a00640 1 -- 172.21.15.31:0/3809259449 learned_addr learned my addr 172.21.15.31:0/3809259449 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:13.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2553e00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25541556e0 0x7f25541df760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2559000640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f253c004d90 0x7f25541dedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2559a00640 1 -- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25541556e0 msgr2=0x7f25541df760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2559a00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25541556e0 0x7f25541df760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2559a00640 1 -- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f253c004d90 msgr2=0x7f25541dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2559a00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f253c004d90 0x7f25541dedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2559a00640 1 -- 172.21.15.31:0/3809259449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25540c4fd0 con 0x7f25541581b0 2024-09-11T00:43:13.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.674+0000 7f2559a00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25541581b0 0x7f25540c30c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f254c036040 tx=0x7f254c02bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.675+0000 7f2550c00640 1 -- 172.21.15.31:0/3809259449 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c042070 con 0x7f25541581b0 2024-09-11T00:43:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.675+0000 7f2550c00640 1 -- 172.21.15.31:0/3809259449 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f254c033770 con 0x7f25541581b0 2024-09-11T00:43:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.675+0000 7f2550c00640 1 -- 172.21.15.31:0/3809259449 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c038040 con 0x7f25541581b0 2024-09-11T00:43:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.675+0000 7f255ba00640 1 -- 172.21.15.31:0/3809259449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25540c52b0 con 0x7f25541581b0 2024-09-11T00:43:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.675+0000 7f255ba00640 1 -- 172.21.15.31:0/3809259449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25541de5d0 con 0x7f25541581b0 2024-09-11T00:43:13.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.676+0000 7f255ba00640 1 -- 172.21.15.31:0/3809259449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2554154090 con 0x7f25541581b0 2024-09-11T00:43:13.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.677+0000 7f2550c00640 1 -- 172.21.15.31:0/3809259449 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f254c03b070 con 0x7f25541581b0 2024-09-11T00:43:13.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.678+0000 7f2550c00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 0x7f253407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.678+0000 7f2550c00640 1 -- 172.21.15.31:0/3809259449 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f254c0b9ec0 con 0x7f25541581b0 2024-09-11T00:43:13.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.678+0000 7f2559000640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 0x7f253407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:13.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.679+0000 7f2559000640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 0x7f253407a430 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2548006bf0 tx=0x7f25480067a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:13.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.682+0000 7f2550c00640 1 -- 172.21.15.31:0/3809259449 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f254c081970 con 0x7f25541581b0 2024-09-11T00:43:13.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.914+0000 7f255ba00640 1 -- 172.21.15.31:0/3809259449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2554151320 con 0x7f25541581b0 2024-09-11T00:43:13.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.915+0000 7f2550c00640 1 -- 172.21.15.31:0/3809259449 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7f254c0863a0 con 0x7f25541581b0 2024-09-11T00:43:13.917 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:13.917 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:13.918 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 -- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 msgr2=0x7f253407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 0x7f253407a430 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2548006bf0 tx=0x7f25480067a0 comp rx=0 tx=0).stop 2024-09-11T00:43:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 -- 172.21.15.31:0/3809259449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25541581b0 msgr2=0x7f25540c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25541581b0 0x7f25540c30c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f254c036040 tx=0x7f254c02bf90 comp rx=0 tx=0).stop 2024-09-11T00:43:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 -- 172.21.15.31:0/3809259449 shutdown_connections 2024-09-11T00:43:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 0x7f253407a430 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25541581b0 0x7f25540c30c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25541556e0 0x7f25541df760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 --2- 172.21.15.31:0/3809259449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f253c004d90 0x7f25541dedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:13.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 -- 172.21.15.31:0/3809259449 >> 172.21.15.31:0/3809259449 conn(0x7f25540a9750 msgr2=0x7f255414c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:13.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 -- 172.21.15.31:0/3809259449 shutdown_connections 2024-09-11T00:43:13.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:13.918+0000 7f255ba00640 1 -- 172.21.15.31:0/3809259449 wait complete. 2024-09-11T00:43:13.936 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:43:14.137 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:14.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.295+0000 7f3e7e200640 1 -- 172.21.15.31:0/329796266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e780a6e20 msgr2=0x7f3e781510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:14.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.295+0000 7f3e7e200640 1 --2- 172.21.15.31:0/329796266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e780a6e20 0x7f3e781510c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f3e780c5ea0 tx=0x7f3e6002a4d0 comp rx=0 tx=0).stop 2024-09-11T00:43:14.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.295+0000 7f3e7e200640 1 -- 172.21.15.31:0/329796266 shutdown_connections 2024-09-11T00:43:14.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.295+0000 7f3e7e200640 1 --2- 172.21.15.31:0/329796266 >> v2:172.21.15.204:3300/0 conn(0x7f3e78151600 0x7f3e780c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:14.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.295+0000 7f3e7e200640 1 --2- 172.21.15.31:0/329796266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e780a6e20 0x7f3e781510c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:14.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.295+0000 7f3e7e200640 1 -- 172.21.15.31:0/329796266 >> 172.21.15.31:0/329796266 conn(0x7f3e7806caf0 msgr2=0x7f3e7806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:14.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.296+0000 7f3e7e200640 1 -- 172.21.15.31:0/329796266 shutdown_connections 2024-09-11T00:43:14.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.296+0000 7f3e7e200640 1 -- 172.21.15.31:0/329796266 wait complete. 2024-09-11T00:43:14.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.297+0000 7f3e7e200640 1 Processor -- start 2024-09-11T00:43:14.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.297+0000 7f3e7e200640 1 -- start start 2024-09-11T00:43:14.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.298+0000 7f3e7e200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e780a6e20 0x7f3e78157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:14.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.298+0000 7f3e7e200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780ac2c0 0x7f3e7814a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:14.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.298+0000 7f3e7e200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e78151600 0x7f3e780a8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:14.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.298+0000 7f3e7e200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3e780acf30 con 0x7f3e78151600 2024-09-11T00:43:14.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.298+0000 7f3e7e200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3e780acdb0 con 0x7f3e780ac2c0 2024-09-11T00:43:14.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.298+0000 7f3e7e200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3e780c66f0 con 0x7f3e780a6e20 2024-09-11T00:43:14.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.298+0000 7f3e7d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e78151600 0x7f3e780a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:14.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.298+0000 7f3e7d800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e78151600 0x7f3e780a8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35326/0 (socket says 172.21.15.31:35326) 2024-09-11T00:43:14.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.298+0000 7f3e7d800640 1 -- 172.21.15.31:0/2666933993 learned_addr learned my addr 172.21.15.31:0/2666933993 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:14.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e7ce00640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e780a6e20 0x7f3e78157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:14.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e6fe00640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780ac2c0 0x7f3e7814a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e7d800640 1 -- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e780a6e20 msgr2=0x7f3e78157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e7d800640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e780a6e20 0x7f3e78157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e7d800640 1 -- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780ac2c0 msgr2=0x7f3e7814a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e7d800640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780ac2c0 0x7f3e7814a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e7d800640 1 -- 172.21.15.31:0/2666933993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7814a9c0 con 0x7f3e78151600 2024-09-11T00:43:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e7d800640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e78151600 0x7f3e780a8430 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000e960 tx=0x7f3e7000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e75600640 1 -- 172.21.15.31:0/2666933993 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70004d20 con 0x7f3e78151600 2024-09-11T00:43:14.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e75600640 1 -- 172.21.15.31:0/2666933993 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e7000cb20 con 0x7f3e78151600 2024-09-11T00:43:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e75600640 1 -- 172.21.15.31:0/2666933993 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e70005020 con 0x7f3e78151600 2024-09-11T00:43:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.299+0000 7f3e7e200640 1 -- 172.21.15.31:0/2666933993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e780a9ee0 con 0x7f3e78151600 2024-09-11T00:43:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.300+0000 7f3e7e200640 1 -- 172.21.15.31:0/2666933993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e78155df0 con 0x7f3e78151600 2024-09-11T00:43:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.301+0000 7f3e7e200640 1 -- 172.21.15.31:0/2666933993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e781510c0 con 0x7f3e78151600 2024-09-11T00:43:14.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.302+0000 7f3e75600640 1 -- 172.21.15.31:0/2666933993 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3e7000f040 con 0x7f3e78151600 2024-09-11T00:43:14.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.303+0000 7f3e75600640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e54077f70 0x7f3e5407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:14.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.303+0000 7f3e75600640 1 -- 172.21.15.31:0/2666933993 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f3e7009abb0 con 0x7f3e78151600 2024-09-11T00:43:14.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.303+0000 7f3e7ce00640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e54077f70 0x7f3e5407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:14.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.304+0000 7f3e7ce00640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e54077f70 0x7f3e5407a430 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3e78153220 tx=0x7f3e68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:14.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.307+0000 7f3e75600640 1 -- 172.21.15.31:0/2666933993 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3e70016030 con 0x7f3e78151600 2024-09-11T00:43:14.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.536+0000 7f3e7e200640 1 -- 172.21.15.31:0/2666933993 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e78155fc0 con 0x7f3e78151600 2024-09-11T00:43:14.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.536+0000 7f3e75600640 1 -- 172.21.15.31:0/2666933993 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7f3e700626f0 con 0x7f3e78151600 2024-09-11T00:43:14.538 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:14.538 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:14.539 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:14.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.538+0000 7f3e7e200640 1 -- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e54077f70 msgr2=0x7f3e5407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.538+0000 7f3e7e200640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e54077f70 0x7f3e5407a430 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3e78153220 tx=0x7f3e68009040 comp rx=0 tx=0).stop 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.538+0000 7f3e7e200640 1 -- 172.21.15.31:0/2666933993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e78151600 msgr2=0x7f3e780a8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.538+0000 7f3e7e200640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e78151600 0x7f3e780a8430 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000e960 tx=0x7f3e7000ee20 comp rx=0 tx=0).stop 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.538+0000 7f3e7e200640 1 -- 172.21.15.31:0/2666933993 shutdown_connections 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.538+0000 7f3e7e200640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e54077f70 0x7f3e5407a430 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.538+0000 7f3e7e200640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e78151600 0x7f3e780a8430 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.538+0000 7f3e7e200640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e780ac2c0 0x7f3e7814a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.538+0000 7f3e7e200640 1 --2- 172.21.15.31:0/2666933993 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e780a6e20 0x7f3e78157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.538+0000 7f3e7e200640 1 -- 172.21.15.31:0/2666933993 >> 172.21.15.31:0/2666933993 conn(0x7f3e7806caf0 msgr2=0x7f3e7814ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.539+0000 7f3e7e200640 1 -- 172.21.15.31:0/2666933993 shutdown_connections 2024-09-11T00:43:14.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:14.539+0000 7f3e7e200640 1 -- 172.21.15.31:0/2666933993 wait complete. 2024-09-11T00:43:14.556 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:43:15.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:43:15.897 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:43:15.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:16.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.102+0000 7f40ed200640 1 -- 172.21.15.31:0/2419666981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40e80c84e0 msgr2=0x7f40e80c69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:16.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.102+0000 7f40ed200640 1 --2- 172.21.15.31:0/2419666981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40e80c84e0 0x7f40e80c69b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f40e806dbd0 tx=0x7f40e002aac0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.102+0000 7f40ed200640 1 -- 172.21.15.31:0/2419666981 shutdown_connections 2024-09-11T00:43:16.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.102+0000 7f40ed200640 1 --2- 172.21.15.31:0/2419666981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40e80c84e0 0x7f40e80c69b0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.102+0000 7f40ed200640 1 --2- 172.21.15.31:0/2419666981 >> v2:172.21.15.204:3300/0 conn(0x7f40e80c4cb0 0x7f40e8074230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.102+0000 7f40ed200640 1 -- 172.21.15.31:0/2419666981 >> 172.21.15.31:0/2419666981 conn(0x7f40e80aa760 msgr2=0x7f40e80aab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:16.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.102+0000 7f40ed200640 1 -- 172.21.15.31:0/2419666981 shutdown_connections 2024-09-11T00:43:16.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.102+0000 7f40ed200640 1 -- 172.21.15.31:0/2419666981 wait complete. 2024-09-11T00:43:16.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.104+0000 7f40ed200640 1 Processor -- start 2024-09-11T00:43:16.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.104+0000 7f40ed200640 1 -- start start 2024-09-11T00:43:16.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.105+0000 7f40ed200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40e80c4cb0 0x7f40e814bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:16.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.105+0000 7f40ed200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40e80c84e0 0x7f40e81ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:16.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.105+0000 7f40ed200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40e81e1120 0x7f40e8157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:16.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.105+0000 7f40ed200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f40e806d380 con 0x7f40e80c4cb0 2024-09-11T00:43:16.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.105+0000 7f40ed200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f40e806d200 con 0x7f40e80c84e0 2024-09-11T00:43:16.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.105+0000 7f40ed200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f40e80c6ef0 con 0x7f40e81e1120 2024-09-11T00:43:16.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.105+0000 7f40e7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40e80c4cb0 0x7f40e814bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:16.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.105+0000 7f40e7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40e80c4cb0 0x7f40e814bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35336/0 (socket says 172.21.15.31:35336) 2024-09-11T00:43:16.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.105+0000 7f40e7400640 1 -- 172.21.15.31:0/3208235540 learned_addr learned my addr 172.21.15.31:0/3208235540 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:16.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40dea00640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40e80c84e0 0x7f40e81ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:16.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40e7400640 1 -- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40e81e1120 msgr2=0x7f40e8157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:16.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40e7400640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40e81e1120 0x7f40e8157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40e7400640 1 -- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40e80c84e0 msgr2=0x7f40e81ef7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:16.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40e7400640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40e80c84e0 0x7f40e81ef7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40e7400640 1 -- 172.21.15.31:0/3208235540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e8155b70 con 0x7f40e80c4cb0 2024-09-11T00:43:16.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40dea00640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40e80c84e0 0x7f40e81ef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:43:16.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40e7400640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40e80c4cb0 0x7f40e814bfc0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f40d800d950 tx=0x7f40d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:16.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40dfe00640 1 -- 172.21.15.31:0/3208235540 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8017070 con 0x7f40e80c4cb0 2024-09-11T00:43:16.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40dfe00640 1 -- 172.21.15.31:0/3208235540 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40d8009d20 con 0x7f40e80c4cb0 2024-09-11T00:43:16.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40dfe00640 1 -- 172.21.15.31:0/3208235540 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d80106c0 con 0x7f40e80c4cb0 2024-09-11T00:43:16.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40ed200640 1 -- 172.21.15.31:0/3208235540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e81e15b0 con 0x7f40e80c4cb0 2024-09-11T00:43:16.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.106+0000 7f40ed200640 1 -- 172.21.15.31:0/3208235540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e81560c0 con 0x7f40e80c4cb0 2024-09-11T00:43:16.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.107+0000 7f40ed200640 1 -- 172.21.15.31:0/3208235540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40e80c7350 con 0x7f40e80c4cb0 2024-09-11T00:43:16.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.108+0000 7f40dfe00640 1 -- 172.21.15.31:0/3208235540 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f40d8010860 con 0x7f40e80c4cb0 2024-09-11T00:43:16.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.109+0000 7f40dfe00640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40c4077f70 0x7f40c407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:16.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.109+0000 7f40dfe00640 1 -- 172.21.15.31:0/3208235540 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f40d809aa80 con 0x7f40e80c4cb0 2024-09-11T00:43:16.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.110+0000 7f40dea00640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40c4077f70 0x7f40c407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:16.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.110+0000 7f40dea00640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40c4077f70 0x7f40c407a430 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f40e8150800 tx=0x7f40d0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:16.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.114+0000 7f40dfe00640 1 -- 172.21.15.31:0/3208235540 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f40d809ce20 con 0x7f40e80c4cb0 2024-09-11T00:43:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.342+0000 7f40ed200640 1 -- 172.21.15.31:0/3208235540 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40e80aed40 con 0x7f40e80c4cb0 2024-09-11T00:43:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.343+0000 7f40dfe00640 1 -- 172.21.15.31:0/3208235540 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7f40d8062ca0 con 0x7f40e80c4cb0 2024-09-11T00:43:16.345 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:16.345 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.345+0000 7f40ed200640 1 -- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40c4077f70 msgr2=0x7f40c407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.345+0000 7f40ed200640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40c4077f70 0x7f40c407a430 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f40e8150800 tx=0x7f40d0009040 comp rx=0 tx=0).stop 2024-09-11T00:43:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.345+0000 7f40ed200640 1 -- 172.21.15.31:0/3208235540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40e80c4cb0 msgr2=0x7f40e814bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.345+0000 7f40ed200640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40e80c4cb0 0x7f40e814bfc0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f40d800d950 tx=0x7f40d800de10 comp rx=0 tx=0).stop 2024-09-11T00:43:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.345+0000 7f40ed200640 1 -- 172.21.15.31:0/3208235540 shutdown_connections 2024-09-11T00:43:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.345+0000 7f40ed200640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40c4077f70 0x7f40c407a430 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.345+0000 7f40ed200640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40e81e1120 0x7f40e8157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.345+0000 7f40ed200640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40e80c84e0 0x7f40e81ef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.345+0000 7f40ed200640 1 --2- 172.21.15.31:0/3208235540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40e80c4cb0 0x7f40e814bfc0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.345+0000 7f40ed200640 1 -- 172.21.15.31:0/3208235540 >> 172.21.15.31:0/3208235540 conn(0x7f40e80aa760 msgr2=0x7f40e80a9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.346+0000 7f40ed200640 1 -- 172.21.15.31:0/3208235540 shutdown_connections 2024-09-11T00:43:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.346+0000 7f40ed200640 1 -- 172.21.15.31:0/3208235540 wait complete. 2024-09-11T00:43:16.364 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:43:16.557 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.717+0000 7fe33cc00640 1 -- 172.21.15.31:0/936745262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe31c003a50 msgr2=0x7fe31c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.717+0000 7fe33cc00640 1 --2- 172.21.15.31:0/936745262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe31c003a50 0x7fe31c003e70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe3280016c0 tx=0x7fe31c005830 comp rx=0 tx=0).stop 2024-09-11T00:43:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.718+0000 7fe33cc00640 1 -- 172.21.15.31:0/936745262 shutdown_connections 2024-09-11T00:43:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.718+0000 7fe33cc00640 1 --2- 172.21.15.31:0/936745262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe31c003a50 0x7fe31c003e70 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.718+0000 7fe33cc00640 1 -- 172.21.15.31:0/936745262 >> 172.21.15.31:0/936745262 conn(0x7fe33806caf0 msgr2=0x7fe33806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.718+0000 7fe33cc00640 1 -- 172.21.15.31:0/936745262 shutdown_connections 2024-09-11T00:43:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.718+0000 7fe33cc00640 1 -- 172.21.15.31:0/936745262 wait complete. 2024-09-11T00:43:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.719+0000 7fe33cc00640 1 Processor -- start 2024-09-11T00:43:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.719+0000 7fe33cc00640 1 -- start start 2024-09-11T00:43:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.720+0000 7fe33cc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe31c003a50 0x7fe3380a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.720+0000 7fe33cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3380a4260 0x7fe3380a9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.720+0000 7fe33cc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3380aa000 0x7fe3380a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.720+0000 7fe33cc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe338151450 con 0x7fe3380a4260 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.720+0000 7fe33cc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe3381512d0 con 0x7fe3380aa000 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.720+0000 7fe33cc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe3380c5d20 con 0x7fe31c003a50 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe336a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3380a4260 0x7fe3380a9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe336a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3380a4260 0x7fe3380a9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35362/0 (socket says 172.21.15.31:35362) 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe336a00640 1 -- 172.21.15.31:0/422028378 learned_addr learned my addr 172.21.15.31:0/422028378 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe337400640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe31c003a50 0x7fe3380a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe337e00640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3380aa000 0x7fe3380a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe336a00640 1 -- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe31c003a50 msgr2=0x7fe3380a65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe336a00640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe31c003a50 0x7fe3380a65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe336a00640 1 -- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3380aa000 msgr2=0x7fe3380a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe336a00640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3380aa000 0x7fe3380a99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe336a00640 1 -- 172.21.15.31:0/422028378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3381f08f0 con 0x7fe3380a4260 2024-09-11T00:43:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.721+0000 7fe336a00640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3380a4260 0x7fe3380a9480 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe324011780 tx=0x7fe324011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.722+0000 7fe32f400640 1 -- 172.21.15.31:0/422028378 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32401f070 con 0x7fe3380a4260 2024-09-11T00:43:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.722+0000 7fe32f400640 1 -- 172.21.15.31:0/422028378 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe324011e90 con 0x7fe3380a4260 2024-09-11T00:43:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.722+0000 7fe32f400640 1 -- 172.21.15.31:0/422028378 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3240126b0 con 0x7fe3380a4260 2024-09-11T00:43:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.722+0000 7fe33cc00640 1 -- 172.21.15.31:0/422028378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3380a8c80 con 0x7fe3380a4260 2024-09-11T00:43:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.722+0000 7fe33cc00640 1 -- 172.21.15.31:0/422028378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3380a6b00 con 0x7fe3380a4260 2024-09-11T00:43:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.723+0000 7fe33cc00640 1 -- 172.21.15.31:0/422028378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe33814d5a0 con 0x7fe3380a4260 2024-09-11T00:43:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.724+0000 7fe32f400640 1 -- 172.21.15.31:0/422028378 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe324012850 con 0x7fe3380a4260 2024-09-11T00:43:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.725+0000 7fe32f400640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe318077f70 0x7fe31807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:16.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.725+0000 7fe32f400640 1 -- 172.21.15.31:0/422028378 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7fe32409b1b0 con 0x7fe3380a4260 2024-09-11T00:43:16.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.725+0000 7fe337400640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe318077f70 0x7fe31807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:16.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.726+0000 7fe337400640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe318077f70 0x7fe31807a430 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe328032000 tx=0x7fe328001190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:16.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.729+0000 7fe32f400640 1 -- 172.21.15.31:0/422028378 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe33814d5a0 con 0x7fe3380a4260 2024-09-11T00:43:16.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.966+0000 7fe33cc00640 1 -- 172.21.15.31:0/422028378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3380a8a50 con 0x7fe3380a4260 2024-09-11T00:43:16.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.967+0000 7fe32f400640 1 -- 172.21.15.31:0/422028378 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7fe324062d80 con 0x7fe3380a4260 2024-09-11T00:43:16.969 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:16.969 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:16.969 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 -- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe318077f70 msgr2=0x7fe31807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe318077f70 0x7fe31807a430 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe328032000 tx=0x7fe328001190 comp rx=0 tx=0).stop 2024-09-11T00:43:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 -- 172.21.15.31:0/422028378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3380a4260 msgr2=0x7fe3380a9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3380a4260 0x7fe3380a9480 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe324011780 tx=0x7fe324011c40 comp rx=0 tx=0).stop 2024-09-11T00:43:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 -- 172.21.15.31:0/422028378 shutdown_connections 2024-09-11T00:43:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe318077f70 0x7fe31807a430 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3380aa000 0x7fe3380a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3380a4260 0x7fe3380a9480 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 --2- 172.21.15.31:0/422028378 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe31c003a50 0x7fe3380a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 -- 172.21.15.31:0/422028378 >> 172.21.15.31:0/422028378 conn(0x7fe33806caf0 msgr2=0x7fe338151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.969+0000 7fe33cc00640 1 -- 172.21.15.31:0/422028378 shutdown_connections 2024-09-11T00:43:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:16.970+0000 7fe33cc00640 1 -- 172.21.15.31:0/422028378 wait complete. 2024-09-11T00:43:16.987 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:43:18.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:18.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.534+0000 7f28b8400640 1 -- 172.21.15.31:0/3110386857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b0153f20 msgr2=0x7f28b00c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.534+0000 7f28b8400640 1 --2- 172.21.15.31:0/3110386857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b0153f20 0x7f28b00c30c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f28b00c4400 tx=0x7f28a802aac0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.534+0000 7f28b8400640 1 -- 172.21.15.31:0/3110386857 shutdown_connections 2024-09-11T00:43:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.534+0000 7f28b8400640 1 --2- 172.21.15.31:0/3110386857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b0153f20 0x7f28b00c30c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.534+0000 7f28b8400640 1 --2- 172.21.15.31:0/3110386857 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28b00c23a0 0x7f28b01516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.534+0000 7f28b8400640 1 --2- 172.21.15.31:0/3110386857 >> v2:172.21.15.204:3300/0 conn(0x7f28b0072480 0x7f28b01511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.534+0000 7f28b8400640 1 -- 172.21.15.31:0/3110386857 >> 172.21.15.31:0/3110386857 conn(0x7f28b00a9750 msgr2=0x7f28b00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.534+0000 7f28b8400640 1 -- 172.21.15.31:0/3110386857 shutdown_connections 2024-09-11T00:43:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.534+0000 7f28b8400640 1 -- 172.21.15.31:0/3110386857 wait complete. 2024-09-11T00:43:18.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.536+0000 7f28b8400640 1 Processor -- start 2024-09-11T00:43:18.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.536+0000 7f28b8400640 1 -- start start 2024-09-11T00:43:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b8400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b0072480 0x7f28b01debe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b8400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28b00c23a0 0x7f28b00c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b8400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28b0153f20 0x7f28b01dd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b8400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28b0074200 con 0x7f28b0072480 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b8400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28b0074080 con 0x7f28b00c23a0 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b8400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f28b0072860 con 0x7f28b0153f20 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b0072480 0x7f28b01debe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b0072480 0x7f28b01debe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45740/0 (socket says 172.21.15.31:45740) 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b5a00640 1 -- 172.21.15.31:0/1553642350 learned_addr learned my addr 172.21.15.31:0/1553642350 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b5000640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28b00c23a0 0x7f28b00c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b6400640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28b0153f20 0x7f28b01dd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b5a00640 1 -- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28b0153f20 msgr2=0x7f28b01dd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b5a00640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28b0153f20 0x7f28b01dd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b5a00640 1 -- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28b00c23a0 msgr2=0x7f28b00c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b5a00640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28b00c23a0 0x7f28b00c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.537+0000 7f28b5a00640 1 -- 172.21.15.31:0/1553642350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b01de200 con 0x7f28b0072480 2024-09-11T00:43:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.538+0000 7f28b5a00640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b0072480 0x7f28b01debe0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f28a400ea20 tx=0x7f28a400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.538+0000 7f28ad600640 1 -- 172.21.15.31:0/1553642350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a400cd80 con 0x7f28b0072480 2024-09-11T00:43:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.538+0000 7f28ad600640 1 -- 172.21.15.31:0/1553642350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28a4010430 con 0x7f28b0072480 2024-09-11T00:43:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.538+0000 7f28ad600640 1 -- 172.21.15.31:0/1553642350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a4010840 con 0x7f28b0072480 2024-09-11T00:43:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.538+0000 7f28b8400640 1 -- 172.21.15.31:0/1553642350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b01de4a0 con 0x7f28b0072480 2024-09-11T00:43:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.538+0000 7f28b8400640 1 -- 172.21.15.31:0/1553642350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b01ead10 con 0x7f28b0072480 2024-09-11T00:43:18.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.539+0000 7f28b8400640 1 -- 172.21.15.31:0/1553642350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b01511a0 con 0x7f28b0072480 2024-09-11T00:43:18.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.540+0000 7f28ad600640 1 -- 172.21.15.31:0/1553642350 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f28a40109e0 con 0x7f28b0072480 2024-09-11T00:43:18.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.542+0000 7f28ad600640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2890077f70 0x7f289007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:18.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.542+0000 7f28ad600640 1 -- 172.21.15.31:0/1553642350 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f28a409b0d0 con 0x7f28b0072480 2024-09-11T00:43:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.542+0000 7f28b5000640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2890077f70 0x7f289007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.543+0000 7f28b5000640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2890077f70 0x7f289007a430 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f28b00c4d80 tx=0x7f289c00b410 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:18.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.546+0000 7f28ad600640 1 -- 172.21.15.31:0/1553642350 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f28a4062cb0 con 0x7f28b0072480 2024-09-11T00:43:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.782+0000 7f28b8400640 1 -- 172.21.15.31:0/1553642350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28b00c2120 con 0x7f28b0072480 2024-09-11T00:43:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.782+0000 7f28ad600640 1 -- 172.21.15.31:0/1553642350 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7f28a40676e0 con 0x7f28b0072480 2024-09-11T00:43:18.784 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:18.785 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:18.785 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 -- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2890077f70 msgr2=0x7f289007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2890077f70 0x7f289007a430 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f28b00c4d80 tx=0x7f289c00b410 comp rx=0 tx=0).stop 2024-09-11T00:43:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 -- 172.21.15.31:0/1553642350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b0072480 msgr2=0x7f28b01debe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b0072480 0x7f28b01debe0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f28a400ea20 tx=0x7f28a400eee0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 -- 172.21.15.31:0/1553642350 shutdown_connections 2024-09-11T00:43:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2890077f70 0x7f289007a430 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28b0153f20 0x7f28b01dd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28b00c23a0 0x7f28b00c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 --2- 172.21.15.31:0/1553642350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28b0072480 0x7f28b01debe0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 -- 172.21.15.31:0/1553642350 >> 172.21.15.31:0/1553642350 conn(0x7f28b00a9750 msgr2=0x7f28b014f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 -- 172.21.15.31:0/1553642350 shutdown_connections 2024-09-11T00:43:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:18.785+0000 7f28b8400640 1 -- 172.21.15.31:0/1553642350 wait complete. 2024-09-11T00:43:18.803 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:43:18.988 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:19.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.150+0000 7f90e7e00640 1 -- 172.21.15.31:0/446834746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90c80052e0 msgr2=0x7f90c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:19.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.150+0000 7f90e7e00640 1 --2- 172.21.15.31:0/446834746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90c80052e0 0x7f90c800b9c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f90d8009e20 tx=0x7f90c8005830 comp rx=0 tx=0).stop 2024-09-11T00:43:19.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.150+0000 7f90e7e00640 1 -- 172.21.15.31:0/446834746 shutdown_connections 2024-09-11T00:43:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.150+0000 7f90e7e00640 1 --2- 172.21.15.31:0/446834746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90c80052e0 0x7f90c800b9c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.150+0000 7f90e7e00640 1 -- 172.21.15.31:0/446834746 >> 172.21.15.31:0/446834746 conn(0x7f90e00a9750 msgr2=0x7f90e00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.150+0000 7f90e7e00640 1 -- 172.21.15.31:0/446834746 shutdown_connections 2024-09-11T00:43:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.150+0000 7f90e7e00640 1 -- 172.21.15.31:0/446834746 wait complete. 2024-09-11T00:43:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.152+0000 7f90e7e00640 1 Processor -- start 2024-09-11T00:43:19.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.152+0000 7f90e7e00640 1 -- start start 2024-09-11T00:43:19.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90c80052e0 0x7f90e01efe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90e014ed50 0x7f90e014cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90e014d0e0 0x7f90e01ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f90e00c5ef0 con 0x7f90c80052e0 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e7e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f90e00c7320 con 0x7f90e014d0e0 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e7e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f90e00c6070 con 0x7f90e014ed50 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90c80052e0 0x7f90e01efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90c80052e0 0x7f90e01efe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45758/0 (socket says 172.21.15.31:45758) 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e5400640 1 -- 172.21.15.31:0/3668020604 learned_addr learned my addr 172.21.15.31:0/3668020604 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e4a00640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90e014ed50 0x7f90e014cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e5e00640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90e014d0e0 0x7f90e01ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:19.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e5400640 1 -- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90e014ed50 msgr2=0x7f90e014cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e5400640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90e014ed50 0x7f90e014cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e5400640 1 -- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90e014d0e0 msgr2=0x7f90e01ebed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e5400640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90e014d0e0 0x7f90e01ebed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.153+0000 7f90e5400640 1 -- 172.21.15.31:0/3668020604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e00722b0 con 0x7f90c80052e0 2024-09-11T00:43:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.154+0000 7f90e5400640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90c80052e0 0x7f90e01efe50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f90d40066b0 tx=0x7f90d40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.154+0000 7f90dd600640 1 -- 172.21.15.31:0/3668020604 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d4015070 con 0x7f90c80052e0 2024-09-11T00:43:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.154+0000 7f90dd600640 1 -- 172.21.15.31:0/3668020604 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90d4011900 con 0x7f90c80052e0 2024-09-11T00:43:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.154+0000 7f90e7e00640 1 -- 172.21.15.31:0/3668020604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e01555f0 con 0x7f90c80052e0 2024-09-11T00:43:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.154+0000 7f90dd600640 1 -- 172.21.15.31:0/3668020604 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d4011d50 con 0x7f90c80052e0 2024-09-11T00:43:19.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.154+0000 7f90e7e00640 1 -- 172.21.15.31:0/3668020604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e01e1060 con 0x7f90c80052e0 2024-09-11T00:43:19.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.155+0000 7f90e7e00640 1 -- 172.21.15.31:0/3668020604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e00c6560 con 0x7f90c80052e0 2024-09-11T00:43:19.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.156+0000 7f90dd600640 1 -- 172.21.15.31:0/3668020604 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f90d40103c0 con 0x7f90c80052e0 2024-09-11T00:43:19.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.157+0000 7f90dd600640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90c0077fa0 0x7f90c007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:19.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.157+0000 7f90dd600640 1 -- 172.21.15.31:0/3668020604 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7956+0+0 (secure 0 0 0) 0x7f90d409b640 con 0x7f90c80052e0 2024-09-11T00:43:19.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.158+0000 7f90e4a00640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90c0077fa0 0x7f90c007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:19.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.158+0000 7f90e4a00640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90c0077fa0 0x7f90c007a460 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f90e01e0ee0 tx=0x7f90cc006b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:19.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.161+0000 7f90dd600640 1 -- 172.21.15.31:0/3668020604 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f90d40637e0 con 0x7f90c80052e0 2024-09-11T00:43:19.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.394+0000 7f90e7e00640 1 -- 172.21.15.31:0/3668020604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90e00c1e70 con 0x7f90c80052e0 2024-09-11T00:43:19.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.395+0000 7f90dd600640 1 -- 172.21.15.31:0/3668020604 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6538 (secure 0 0 0) 0x7f90d40630e0 con 0x7f90c80052e0 2024-09-11T00:43:19.397 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:19.397 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":87,"btime":"2024-09-11T00:41:57:808116+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87},{"gid":5773,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:41:52.395095+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":12,"addr":"172.21.15.31:6835/4205320530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6833","nonce":4205320530},{"type":"v1","addr":"172.21.15.31:6835","nonce":4205320530}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6165,"qdb_cluster":[6165]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:19.397 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 87 2024-09-11T00:43:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 -- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90c0077fa0 msgr2=0x7f90c007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90c0077fa0 0x7f90c007a460 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f90e01e0ee0 tx=0x7f90cc006b70 comp rx=0 tx=0).stop 2024-09-11T00:43:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 -- 172.21.15.31:0/3668020604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90c80052e0 msgr2=0x7f90e01efe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90c80052e0 0x7f90e01efe50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f90d40066b0 tx=0x7f90d40039c0 comp rx=0 tx=0).stop 2024-09-11T00:43:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 -- 172.21.15.31:0/3668020604 shutdown_connections 2024-09-11T00:43:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90c0077fa0 0x7f90c007a460 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90e014d0e0 0x7f90e01ebed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f90e014ed50 0x7f90e014cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 --2- 172.21.15.31:0/3668020604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90c80052e0 0x7f90e01efe50 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:19.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 -- 172.21.15.31:0/3668020604 >> 172.21.15.31:0/3668020604 conn(0x7f90e00a9750 msgr2=0x7f90e006ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:19.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.397+0000 7f90e7e00640 1 -- 172.21.15.31:0/3668020604 shutdown_connections 2024-09-11T00:43:19.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:19.398+0000 7f90e7e00640 1 -- 172.21.15.31:0/3668020604 wait complete. 2024-09-11T00:43:19.421 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:43:20.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:20.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.973+0000 7f22d1c00640 1 -- 172.21.15.31:0/179737970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22cc1518b0 msgr2=0x7f22cc0c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:20.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.973+0000 7f22d1c00640 1 --2- 172.21.15.31:0/179737970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22cc1518b0 0x7f22cc0c3600 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009ca0 tx=0x7f22c402ab50 comp rx=0 tx=0).stop 2024-09-11T00:43:20.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.973+0000 7f22d1c00640 1 -- 172.21.15.31:0/179737970 shutdown_connections 2024-09-11T00:43:20.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.973+0000 7f22d1c00640 1 --2- 172.21.15.31:0/179737970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22cc1518b0 0x7f22cc0c3600 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:20.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.973+0000 7f22d1c00640 1 --2- 172.21.15.31:0/179737970 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22cc0c23a0 0x7f22cc0c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:20.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.973+0000 7f22d1c00640 1 -- 172.21.15.31:0/179737970 >> 172.21.15.31:0/179737970 conn(0x7f22cc0a9750 msgr2=0x7f22cc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:20.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.973+0000 7f22d1c00640 1 -- 172.21.15.31:0/179737970 shutdown_connections 2024-09-11T00:43:20.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.973+0000 7f22d1c00640 1 -- 172.21.15.31:0/179737970 wait complete. 2024-09-11T00:43:20.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.974+0000 7f22d1c00640 1 Processor -- start 2024-09-11T00:43:20.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.975+0000 7f22d1c00640 1 -- start start 2024-09-11T00:43:20.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.975+0000 7f22d1c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22cc072480 0x7f22cc155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.975+0000 7f22d1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22cc0c23a0 0x7f22cc0c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.975+0000 7f22d1c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22cc1518b0 0x7f22cc1ebfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22d1c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc072860 con 0x7f22cc0c23a0 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22d1c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc074290 con 0x7f22cc072480 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22d1c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f22cc0729e0 con 0x7f22cc1518b0 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22ca000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22cc0c23a0 0x7f22cc0c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22ca000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22cc0c23a0 0x7f22cc0c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45768/0 (socket says 172.21.15.31:45768) 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22ca000640 1 -- 172.21.15.31:0/330834257 learned_addr learned my addr 172.21.15.31:0/330834257 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22caa00640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22cc072480 0x7f22cc155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22cb400640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22cc1518b0 0x7f22cc1ebfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22ca000640 1 -- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22cc1518b0 msgr2=0x7f22cc1ebfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22ca000640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22cc1518b0 0x7f22cc1ebfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:20.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22ca000640 1 -- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22cc072480 msgr2=0x7f22cc155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:20.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22ca000640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22cc072480 0x7f22cc155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:20.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22ca000640 1 -- 172.21.15.31:0/330834257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22cc1dff30 con 0x7f22cc0c23a0 2024-09-11T00:43:20.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22cb400640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22cc1518b0 0x7f22cc1ebfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:43:20.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.976+0000 7f22ca000640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22cc0c23a0 0x7f22cc0c1040 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f22b800d950 tx=0x7f22b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:20.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.977+0000 7f22c2a00640 1 -- 172.21.15.31:0/330834257 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b8017070 con 0x7f22cc0c23a0 2024-09-11T00:43:20.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.977+0000 7f22c2a00640 1 -- 172.21.15.31:0/330834257 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22b8009d40 con 0x7f22cc0c23a0 2024-09-11T00:43:20.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.977+0000 7f22c2a00640 1 -- 172.21.15.31:0/330834257 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b80106c0 con 0x7f22cc0c23a0 2024-09-11T00:43:20.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.977+0000 7f22d1c00640 1 -- 172.21.15.31:0/330834257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22cc1e0210 con 0x7f22cc0c23a0 2024-09-11T00:43:20.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.977+0000 7f22d1c00640 1 -- 172.21.15.31:0/330834257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22cc1f0190 con 0x7f22cc0c23a0 2024-09-11T00:43:20.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.978+0000 7f22d1c00640 1 -- 172.21.15.31:0/330834257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22cc0c3cb0 con 0x7f22cc0c23a0 2024-09-11T00:43:20.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.979+0000 7f22c2a00640 1 -- 172.21.15.31:0/330834257 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f22b8010860 con 0x7f22cc0c23a0 2024-09-11T00:43:20.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.980+0000 7f22c2a00640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f22ac077f70 0x7f22ac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:20.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.980+0000 7f22c2a00640 1 -- 172.21.15.31:0/330834257 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8042+0+0 (secure 0 0 0) 0x7f22b809ab80 con 0x7f22cc0c23a0 2024-09-11T00:43:20.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.980+0000 7f22caa00640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f22ac077f70 0x7f22ac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:20.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.981+0000 7f22caa00640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f22ac077f70 0x7f22ac07a430 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0017c0 tx=0x7f22bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:20.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:20.984+0000 7f22c2a00640 1 -- 172.21.15.31:0/330834257 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f22b8062670 con 0x7f22cc0c23a0 2024-09-11T00:43:21.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.216+0000 7f22d1c00640 1 -- 172.21.15.31:0/330834257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22cc0c7780 con 0x7f22cc0c23a0 2024-09-11T00:43:21.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.217+0000 7f22c2a00640 1 -- 172.21.15.31:0/330834257 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+5709 (secure 0 0 0) 0x7f22b80670a0 con 0x7f22cc0c23a0 2024-09-11T00:43:21.219 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:21.219 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"btime":"2024-09-11T00:43:20:713893+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:19.894440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:reconnect","state_seq":75,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:21.219 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 89 2024-09-11T00:43:21.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.219+0000 7f22d1c00640 1 -- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f22ac077f70 msgr2=0x7f22ac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.219+0000 7f22d1c00640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f22ac077f70 0x7f22ac07a430 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0017c0 tx=0x7f22bc009040 comp rx=0 tx=0).stop 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.219+0000 7f22d1c00640 1 -- 172.21.15.31:0/330834257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22cc0c23a0 msgr2=0x7f22cc0c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.219+0000 7f22d1c00640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22cc0c23a0 0x7f22cc0c1040 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f22b800d950 tx=0x7f22b800de10 comp rx=0 tx=0).stop 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.220+0000 7f22d1c00640 1 -- 172.21.15.31:0/330834257 shutdown_connections 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.220+0000 7f22d1c00640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f22ac077f70 0x7f22ac07a430 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.220+0000 7f22d1c00640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f22cc1518b0 0x7f22cc1ebfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.220+0000 7f22d1c00640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22cc0c23a0 0x7f22cc0c1040 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.220+0000 7f22d1c00640 1 --2- 172.21.15.31:0/330834257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22cc072480 0x7f22cc155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.220+0000 7f22d1c00640 1 -- 172.21.15.31:0/330834257 >> 172.21.15.31:0/330834257 conn(0x7f22cc0a9750 msgr2=0x7f22cc150700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.220+0000 7f22d1c00640 1 -- 172.21.15.31:0/330834257 shutdown_connections 2024-09-11T00:43:21.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.220+0000 7f22d1c00640 1 -- 172.21.15.31:0/330834257 wait complete. 2024-09-11T00:43:21.238 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-11T00:43:21.238 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:43:21.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.397+0000 7fd100400640 1 -- 172.21.15.31:0/972699558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e0003a00 msgr2=0x7fd0e0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.397+0000 7fd100400640 1 --2- 172.21.15.31:0/972699558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e0003a00 0x7fd0e0003e20 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00bb20 tx=0x7fd0e0005830 comp rx=0 tx=0).stop 2024-09-11T00:43:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.398+0000 7fd0fe400640 1 -- 172.21.15.31:0/972699558 reap_dead start 2024-09-11T00:43:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.398+0000 7fd100400640 1 -- 172.21.15.31:0/972699558 shutdown_connections 2024-09-11T00:43:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.398+0000 7fd100400640 1 -- 172.21.15.31:0/972699558 >> 172.21.15.31:0/972699558 conn(0x7fd0f80a9750 msgr2=0x7fd0f80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.398+0000 7fd100400640 1 -- 172.21.15.31:0/972699558 shutdown_connections 2024-09-11T00:43:21.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.398+0000 7fd100400640 1 -- 172.21.15.31:0/972699558 wait complete. 2024-09-11T00:43:21.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.399+0000 7fd100400640 1 Processor -- start 2024-09-11T00:43:21.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.399+0000 7fd100400640 1 -- start start 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.400+0000 7fd100400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0e0003a00 0x7fd0f814f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.400+0000 7fd100400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0f80aaa60 0x7fd0f8155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.400+0000 7fd100400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0f80aaeb0 0x7fd0f814ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.400+0000 7fd100400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f80c77a0 con 0x7fd0f80aaa60 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.400+0000 7fd100400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f80c7620 con 0x7fd0f80aaeb0 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.400+0000 7fd100400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f8156df0 con 0x7fd0e0003a00 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.400+0000 7fd0fd000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0f80aaa60 0x7fd0f8155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.400+0000 7fd0fd000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0f80aaa60 0x7fd0f8155660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45792/0 (socket says 172.21.15.31:45792) 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.401+0000 7fd0fd000640 1 -- 172.21.15.31:0/2753425348 learned_addr learned my addr 172.21.15.31:0/2753425348 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.401+0000 7fd0fe400640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0f80aaeb0 0x7fd0f814ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.401+0000 7fd0fda00640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0e0003a00 0x7fd0f814f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:21.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.401+0000 7fd0fd000640 1 -- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0e0003a00 msgr2=0x7fd0f814f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.401+0000 7fd0fd000640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0e0003a00 0x7fd0f814f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.401+0000 7fd0fd000640 1 -- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0f80aaeb0 msgr2=0x7fd0f814ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.401+0000 7fd0fd000640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0f80aaeb0 0x7fd0f814ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.401+0000 7fd0fd000640 1 -- 172.21.15.31:0/2753425348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f80adcd0 con 0x7fd0f80aaa60 2024-09-11T00:43:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.401+0000 7fd0fd000640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0f80aaa60 0x7fd0f8155660 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4001050 tx=0x7fd0e40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.401+0000 7fd0f5600640 1 -- 172.21.15.31:0/2753425348 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4015070 con 0x7fd0f80aaa60 2024-09-11T00:43:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.402+0000 7fd0f5600640 1 -- 172.21.15.31:0/2753425348 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0e4011920 con 0x7fd0f80aaa60 2024-09-11T00:43:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.402+0000 7fd100400640 1 -- 172.21.15.31:0/2753425348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f814f9d0 con 0x7fd0f80aaa60 2024-09-11T00:43:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.402+0000 7fd0f5600640 1 -- 172.21.15.31:0/2753425348 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4011d70 con 0x7fd0f80aaa60 2024-09-11T00:43:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.402+0000 7fd100400640 1 -- 172.21.15.31:0/2753425348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f8150300 con 0x7fd0f80aaa60 2024-09-11T00:43:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.403+0000 7fd100400640 1 -- 172.21.15.31:0/2753425348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f8150540 con 0x7fd0f80aaa60 2024-09-11T00:43:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.404+0000 7fd0f5600640 1 -- 172.21.15.31:0/2753425348 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd0e4010420 con 0x7fd0f80aaa60 2024-09-11T00:43:21.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.405+0000 7fd0f5600640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0d8077f70 0x7fd0d807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:21.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.405+0000 7fd0f5600640 1 -- 172.21.15.31:0/2753425348 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8042+0+0 (secure 0 0 0) 0x7fd0e409b820 con 0x7fd0f80aaa60 2024-09-11T00:43:21.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.405+0000 7fd0fda00640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0d8077f70 0x7fd0d807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:21.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.406+0000 7fd0fda00640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0d8077f70 0x7fd0d807a430 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec036000 tx=0x7fd0ec0046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:21.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.409+0000 7fd0f5600640 1 -- 172.21.15.31:0/2753425348 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd0f8150540 con 0x7fd0f80aaa60 2024-09-11T00:43:21.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:21.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.584+0000 7f2c3b800640 1 -- 172.21.15.31:0/1663273315 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c34156300 msgr2=0x7f2c340c5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.584+0000 7f2c3b800640 1 --2- 172.21.15.31:0/1663273315 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c34156300 0x7f2c340c5b40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2c24005fd0 tx=0x7f2c2402a540 comp rx=0 tx=0).stop 2024-09-11T00:43:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.584+0000 7f2c3b800640 1 -- 172.21.15.31:0/1663273315 shutdown_connections 2024-09-11T00:43:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.584+0000 7f2c3b800640 1 --2- 172.21.15.31:0/1663273315 >> v2:172.21.15.204:3300/0 conn(0x7f2c340c23a0 0x7f2c340c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.585+0000 7f2c3b800640 1 --2- 172.21.15.31:0/1663273315 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c34156300 0x7f2c340c5b40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.585+0000 7f2c3b800640 1 -- 172.21.15.31:0/1663273315 >> 172.21.15.31:0/1663273315 conn(0x7f2c340a9750 msgr2=0x7f2c340a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.585+0000 7f2c3b800640 1 -- 172.21.15.31:0/1663273315 shutdown_connections 2024-09-11T00:43:21.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.585+0000 7f2c3b800640 1 -- 172.21.15.31:0/1663273315 wait complete. 2024-09-11T00:43:21.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.586+0000 7f2c3b800640 1 Processor -- start 2024-09-11T00:43:21.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.586+0000 7f2c3b800640 1 -- start start 2024-09-11T00:43:21.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.587+0000 7f2c3b800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c3406fa70 0x7f2c341efe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:21.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.587+0000 7f2c3b800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c340c23a0 0x7f2c341f0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:21.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.587+0000 7f2c3b800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c34156300 0x7f2c341499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:21.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.587+0000 7f2c3b800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2c340c77a0 con 0x7f2c340c23a0 2024-09-11T00:43:21.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.587+0000 7f2c3b800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2c340c7620 con 0x7f2c3406fa70 2024-09-11T00:43:21.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.587+0000 7f2c3b800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2c34156df0 con 0x7f2c34156300 2024-09-11T00:43:21.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.588+0000 7f2c2be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c340c23a0 0x7f2c341f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:21.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.588+0000 7f2c2be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c340c23a0 0x7f2c341f0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45806/0 (socket says 172.21.15.31:45806) 2024-09-11T00:43:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.588+0000 7f2c2be00640 1 -- 172.21.15.31:0/3318601099 learned_addr learned my addr 172.21.15.31:0/3318601099 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.588+0000 7f2c39800640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c34156300 0x7f2c341499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.588+0000 7f2c38e00640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c3406fa70 0x7f2c341efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.588+0000 7f2c2be00640 1 -- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c34156300 msgr2=0x7f2c341499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.588+0000 7f2c2be00640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c34156300 0x7f2c341499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.588+0000 7f2c2be00640 1 -- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c3406fa70 msgr2=0x7f2c341efe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.588+0000 7f2c2be00640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c3406fa70 0x7f2c341efe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.588+0000 7f2c2be00640 1 -- 172.21.15.31:0/3318601099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c341ef960 con 0x7f2c340c23a0 2024-09-11T00:43:21.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.589+0000 7f2c2be00640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c340c23a0 0x7f2c341f0380 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00c3c0 tx=0x7f2c1c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.589+0000 7f2c31600640 1 -- 172.21.15.31:0/3318601099 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c017070 con 0x7f2c340c23a0 2024-09-11T00:43:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.589+0000 7f2c31600640 1 -- 172.21.15.31:0/3318601099 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c1c0053c0 con 0x7f2c340c23a0 2024-09-11T00:43:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.589+0000 7f2c31600640 1 -- 172.21.15.31:0/3318601099 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c003850 con 0x7f2c340c23a0 2024-09-11T00:43:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.589+0000 7f2c3b800640 1 -- 172.21.15.31:0/3318601099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c341544f0 con 0x7f2c340c23a0 2024-09-11T00:43:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.589+0000 7f2c3b800640 1 -- 172.21.15.31:0/3318601099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c341ed250 con 0x7f2c340c23a0 2024-09-11T00:43:21.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.590+0000 7f2c3b800640 1 -- 172.21.15.31:0/3318601099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c34075190 con 0x7f2c340c23a0 2024-09-11T00:43:21.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.591+0000 7f2c31600640 1 -- 172.21.15.31:0/3318601099 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2c1c0039f0 con 0x7f2c340c23a0 2024-09-11T00:43:21.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.592+0000 7f2c31600640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c10077f70 0x7f2c1007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:21.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.592+0000 7f2c31600640 1 -- 172.21.15.31:0/3318601099 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 8042+0+0 (secure 0 0 0) 0x7f2c1c099be0 con 0x7f2c340c23a0 2024-09-11T00:43:21.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.593+0000 7f2c38e00640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c10077f70 0x7f2c1007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:21.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.593+0000 7f2c38e00640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c10077f70 0x7f2c1007a430 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2c24033000 tx=0x7f2c24004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.596+0000 7f2c31600640 1 -- 172.21.15.31:0/3318601099 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2c1c061760 con 0x7f2c340c23a0 2024-09-11T00:43:21.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.648+0000 7fd100400640 1 -- 172.21.15.31:0/2753425348 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0f8152080 con 0x7fd0f80aaa60 2024-09-11T00:43:21.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.649+0000 7fd0f5600640 1 -- 172.21.15.31:0/2753425348 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+5709 (secure 0 0 0) 0x7fd0f8152080 con 0x7fd0f80aaa60 2024-09-11T00:43:21.651 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:21.651 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"btime":"2024-09-11T00:43:20:713893+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:19.894440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:reconnect","state_seq":75,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:21.651 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 89 2024-09-11T00:43:21.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 -- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0d8077f70 msgr2=0x7fd0d807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0d8077f70 0x7fd0d807a430 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec036000 tx=0x7fd0ec0046c0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 -- 172.21.15.31:0/2753425348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0f80aaa60 msgr2=0x7fd0f8155660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0f80aaa60 0x7fd0f8155660 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4001050 tx=0x7fd0e40039c0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 -- 172.21.15.31:0/2753425348 shutdown_connections 2024-09-11T00:43:21.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0d8077f70 0x7fd0d807a430 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0f80aaeb0 0x7fd0f814ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0f80aaa60 0x7fd0f8155660 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 --2- 172.21.15.31:0/2753425348 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0e0003a00 0x7fd0f814f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 -- 172.21.15.31:0/2753425348 >> 172.21.15.31:0/2753425348 conn(0x7fd0f80a9750 msgr2=0x7fd0f80c3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:21.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 -- 172.21.15.31:0/2753425348 shutdown_connections 2024-09-11T00:43:21.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.651+0000 7fd100400640 1 -- 172.21.15.31:0/2753425348 wait complete. 2024-09-11T00:43:21.672 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:43:20:713893+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5773": { "addr": "172.21.15.204:6837/3513641573", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 3513641573, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 3513641573, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5773, "incarnation": 88, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 75 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:43:19.894440+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5773 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 81, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6087": { "addr": "172.21.15.31:6834/2909466339", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2909466339, "type": "v2" }, { "addr": "172.21.15.31:6834", "nonce": 2909466339, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6087, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:41:12.709307+0000", "qdb_cluster": [ 6087 ], "qdb_leader": 6087, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6087 } } } ], "standbys": [ { "addr": "172.21.15.204:6835/1575320768", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1575320768, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 1575320768, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 87, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5594, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:43:21.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.836+0000 7f2c3b800640 1 -- 172.21.15.31:0/3318601099 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c341efb30 con 0x7f2c340c23a0 2024-09-11T00:43:21.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.837+0000 7f2c31600640 1 -- 172.21.15.31:0/3318601099 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5706 (secure 0 0 0) 0x7f2c1c066190 con 0x7f2c340c23a0 2024-09-11T00:43:21.839 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:21.839 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"btime":"2024-09-11T00:43:21:719772+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5594,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:20.722568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":76,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:41:12.709307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6087},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6087":{"gid":6087,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":8,"addr":"172.21.15.31:6834/2909466339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2909466339},{"type":"v1","addr":"172.21.15.31:6834","nonce":2909466339}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6087,"qdb_cluster":[6087]},"id":2}]} 2024-09-11T00:43:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 90 2024-09-11T00:43:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 -- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c10077f70 msgr2=0x7f2c1007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c10077f70 0x7f2c1007a430 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2c24033000 tx=0x7f2c24004730 comp rx=0 tx=0).stop 2024-09-11T00:43:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 -- 172.21.15.31:0/3318601099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c340c23a0 msgr2=0x7f2c341f0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c340c23a0 0x7f2c341f0380 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00c3c0 tx=0x7f2c1c00c880 comp rx=0 tx=0).stop 2024-09-11T00:43:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 -- 172.21.15.31:0/3318601099 shutdown_connections 2024-09-11T00:43:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c10077f70 0x7f2c1007a430 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c34156300 0x7f2c341499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c340c23a0 0x7f2c341f0380 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 --2- 172.21.15.31:0/3318601099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c3406fa70 0x7f2c341efe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 -- 172.21.15.31:0/3318601099 >> 172.21.15.31:0/3318601099 conn(0x7f2c340a9750 msgr2=0x7f2c340c38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 -- 172.21.15.31:0/3318601099 shutdown_connections 2024-09-11T00:43:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:21.839+0000 7f2c3b800640 1 -- 172.21.15.31:0/3318601099 wait complete. 2024-09-11T00:43:21.857 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:43:23.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:23.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.842+0000 7f362a800640 1 -- 172.21.15.31:0/2614196203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240c71c0 msgr2=0x7f36240c5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:23.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.842+0000 7f362a800640 1 --2- 172.21.15.31:0/2614196203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240c71c0 0x7f36240c5920 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3610005fd0 tx=0x7f361002a540 comp rx=0 tx=0).stop 2024-09-11T00:43:23.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.842+0000 7f362a800640 1 -- 172.21.15.31:0/2614196203 shutdown_connections 2024-09-11T00:43:23.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.842+0000 7f362a800640 1 --2- 172.21.15.31:0/2614196203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f36240c8ed0 0x7f36240c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:23.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.842+0000 7f362a800640 1 --2- 172.21.15.31:0/2614196203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240c71c0 0x7f36240c5920 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:23.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.842+0000 7f362a800640 1 -- 172.21.15.31:0/2614196203 >> 172.21.15.31:0/2614196203 conn(0x7f36240688e0 msgr2=0x7f3624068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:23.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.842+0000 7f362a800640 1 -- 172.21.15.31:0/2614196203 shutdown_connections 2024-09-11T00:43:23.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.842+0000 7f362a800640 1 -- 172.21.15.31:0/2614196203 wait complete. 2024-09-11T00:43:23.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.844+0000 7f362a800640 1 Processor -- start 2024-09-11T00:43:23.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.844+0000 7f362a800640 1 -- start start 2024-09-11T00:43:23.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.845+0000 7f362a800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36240c71c0 0x7f36240a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:23.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.845+0000 7f362a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240c8ed0 0x7f36240acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:23.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.845+0000 7f362a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3624158260 0x7f362406de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:23.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.845+0000 7f362a800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3624158640 con 0x7f36240c8ed0 2024-09-11T00:43:23.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.845+0000 7f362a800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f36240c44c0 con 0x7f36240c71c0 2024-09-11T00:43:23.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.845+0000 7f362a800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f36240c98c0 con 0x7f3624158260 2024-09-11T00:43:23.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3622a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240c8ed0 0x7f36240acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:23.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3622a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240c8ed0 0x7f36240acfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45820/0 (socket says 172.21.15.31:45820) 2024-09-11T00:43:23.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3622a00640 1 -- 172.21.15.31:0/3683397366 learned_addr learned my addr 172.21.15.31:0/3683397366 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:23.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3623400640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36240c71c0 0x7f36240a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:23.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3623e00640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3624158260 0x7f362406de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:23.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3622a00640 1 -- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3624158260 msgr2=0x7f362406de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:23.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3622a00640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3624158260 0x7f362406de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:23.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3622a00640 1 -- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36240c71c0 msgr2=0x7f36240a85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:23.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3622a00640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36240c71c0 0x7f36240a85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:23.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3622a00640 1 -- 172.21.15.31:0/3683397366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3624152f20 con 0x7f36240c8ed0 2024-09-11T00:43:23.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.846+0000 7f3622a00640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240c8ed0 0x7f36240acfe0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3610006010 tx=0x7f36100038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:23.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.847+0000 7f361b400640 1 -- 172.21.15.31:0/3683397366 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361003f070 con 0x7f36240c8ed0 2024-09-11T00:43:23.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.847+0000 7f361b400640 1 -- 172.21.15.31:0/3683397366 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36100047a0 con 0x7f36240c8ed0 2024-09-11T00:43:23.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.847+0000 7f361b400640 1 -- 172.21.15.31:0/3683397366 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610035030 con 0x7f36240c8ed0 2024-09-11T00:43:23.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.847+0000 7f362a800640 1 -- 172.21.15.31:0/3683397366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f362406ead0 con 0x7f36240c8ed0 2024-09-11T00:43:23.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.847+0000 7f362a800640 1 -- 172.21.15.31:0/3683397366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36241ecc50 con 0x7f36240c8ed0 2024-09-11T00:43:23.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.848+0000 7f362a800640 1 -- 172.21.15.31:0/3683397366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36240c9380 con 0x7f36240c8ed0 2024-09-11T00:43:23.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.850+0000 7f361b400640 1 -- 172.21.15.31:0/3683397366 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3610004b90 con 0x7f36240c8ed0 2024-09-11T00:43:23.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.850+0000 7f361b400640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3604077fa0 0x7f360407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:23.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.851+0000 7f361b400640 1 -- 172.21.15.31:0/3683397366 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7f36100bbb20 con 0x7f36240c8ed0 2024-09-11T00:43:23.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.851+0000 7f3623400640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3604077fa0 0x7f360407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:23.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.852+0000 7f3623400640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3604077fa0 0x7f360407a460 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3614000f60 tx=0x7f3614009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:23.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:23.854+0000 7f361b400640 1 -- 172.21.15.31:0/3683397366 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f36100836f0 con 0x7f36240c8ed0 2024-09-11T00:43:23.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:24.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.049+0000 7f5e05e00640 1 -- 172.21.15.31:0/2369134957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5e000ada20 msgr2=0x7f5e0006e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.049+0000 7f5e05e00640 1 --2- 172.21.15.31:0/2369134957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5e000ada20 0x7f5e0006e140 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5dec005fd0 tx=0x7f5dec02a5d0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.050+0000 7f5e05e00640 1 -- 172.21.15.31:0/2369134957 shutdown_connections 2024-09-11T00:43:24.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.050+0000 7f5e05e00640 1 --2- 172.21.15.31:0/2369134957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5e000ada20 0x7f5e0006e140 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.050+0000 7f5e05e00640 1 --2- 172.21.15.31:0/2369134957 >> v2:172.21.15.204:3300/0 conn(0x7f5e00150020 0x7f5e000ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.050+0000 7f5e05e00640 1 -- 172.21.15.31:0/2369134957 >> 172.21.15.31:0/2369134957 conn(0x7f5e00068f10 msgr2=0x7f5e000cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:24.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.050+0000 7f5e05e00640 1 -- 172.21.15.31:0/2369134957 shutdown_connections 2024-09-11T00:43:24.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.050+0000 7f5e05e00640 1 -- 172.21.15.31:0/2369134957 wait complete. 2024-09-11T00:43:24.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.052+0000 7f5e05e00640 1 Processor -- start 2024-09-11T00:43:24.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.052+0000 7f5e05e00640 1 -- start start 2024-09-11T00:43:24.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5e05e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e000ada20 0x7f5e000c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:24.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5e05e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5e00150020 0x7f5e000c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:24.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5e05e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e00156960 0x7f5e000c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:24.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5e05e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5e00157550 con 0x7f5e000ada20 2024-09-11T00:43:24.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5e05e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5e001573d0 con 0x7f5e00156960 2024-09-11T00:43:24.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5e05e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5e000cc220 con 0x7f5e00150020 2024-09-11T00:43:24.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5e04a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e000ada20 0x7f5e000c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:24.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5e04a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e000ada20 0x7f5e000c5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45822/0 (socket says 172.21.15.31:45822) 2024-09-11T00:43:24.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5dffe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5e00150020 0x7f5e000c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:24.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5e04a00640 1 -- 172.21.15.31:0/1071995082 learned_addr learned my addr 172.21.15.31:0/1071995082 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:24.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.053+0000 7f5e05400640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e00156960 0x7f5e000c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:24.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5e04a00640 1 -- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5e00150020 msgr2=0x7f5e000c2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5e04a00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5e00150020 0x7f5e000c2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5e04a00640 1 -- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e00156960 msgr2=0x7f5e000c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5e04a00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e00156960 0x7f5e000c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5e04a00640 1 -- 172.21.15.31:0/1071995082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e00150f40 con 0x7f5e000ada20 2024-09-11T00:43:24.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5dffe00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5e00150020 0x7f5e000c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:43:24.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5e05400640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e00156960 0x7f5e000c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:43:24.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5e04a00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e000ada20 0x7f5e000c5a40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5df400e9f0 tx=0x7f5df400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:24.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5dfcc00640 1 -- 172.21.15.31:0/1071995082 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df400cd80 con 0x7f5e000ada20 2024-09-11T00:43:24.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5dfcc00640 1 -- 172.21.15.31:0/1071995082 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5df4010430 con 0x7f5e000ada20 2024-09-11T00:43:24.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5e05e00640 1 -- 172.21.15.31:0/1071995082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e000c5240 con 0x7f5e000ada20 2024-09-11T00:43:24.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5dfcc00640 1 -- 172.21.15.31:0/1071995082 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4010840 con 0x7f5e000ada20 2024-09-11T00:43:24.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.054+0000 7f5e05e00640 1 -- 172.21.15.31:0/1071995082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e00151220 con 0x7f5e000ada20 2024-09-11T00:43:24.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.056+0000 7f5e05e00640 1 -- 172.21.15.31:0/1071995082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e000cbd60 con 0x7f5e000ada20 2024-09-11T00:43:24.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.057+0000 7f5dfcc00640 1 -- 172.21.15.31:0/1071995082 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5df40109e0 con 0x7f5e000ada20 2024-09-11T00:43:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.058+0000 7f5dfcc00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5de0077f70 0x7f5de007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.059+0000 7f5dfcc00640 1 -- 172.21.15.31:0/1071995082 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7f5df409b110 con 0x7f5e000ada20 2024-09-11T00:43:24.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.059+0000 7f5dffe00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5de0077f70 0x7f5de007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:24.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.060+0000 7f5dffe00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5de0077f70 0x7f5de007a430 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5dec033000 tx=0x7f5dec004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.067+0000 7f5dfcc00640 1 -- 172.21.15.31:0/1071995082 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5df4062da0 con 0x7f5e000ada20 2024-09-11T00:43:24.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.152+0000 7f362a800640 1 -- 172.21.15.31:0/3683397366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36240c8310 con 0x7f36240c8ed0 2024-09-11T00:43:24.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.153+0000 7f361b400640 1 -- 172.21.15.31:0/3683397366 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+4886 (secure 0 0 0) 0x7f3610088120 con 0x7f36240c8ed0 2024-09-11T00:43:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:24.155 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"btime":"2024-09-11T00:43:23:741212+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:23.048179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:43:24.155 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 93 2024-09-11T00:43:24.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 -- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3604077fa0 msgr2=0x7f360407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3604077fa0 0x7f360407a460 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3614000f60 tx=0x7f3614009040 comp rx=0 tx=0).stop 2024-09-11T00:43:24.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 -- 172.21.15.31:0/3683397366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240c8ed0 msgr2=0x7f36240acfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240c8ed0 0x7f36240acfe0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3610006010 tx=0x7f36100038a0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 -- 172.21.15.31:0/3683397366 shutdown_connections 2024-09-11T00:43:24.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3604077fa0 0x7f360407a460 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3624158260 0x7f362406de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36240c8ed0 0x7f36240acfe0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 --2- 172.21.15.31:0/3683397366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36240c71c0 0x7f36240a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 -- 172.21.15.31:0/3683397366 >> 172.21.15.31:0/3683397366 conn(0x7f36240688e0 msgr2=0x7f3624154d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:24.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 -- 172.21.15.31:0/3683397366 shutdown_connections 2024-09-11T00:43:24.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.156+0000 7f362a800640 1 -- 172.21.15.31:0/3683397366 wait complete. 2024-09-11T00:43:24.176 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=6165 2024-09-11T00:43:24.176 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.a 2024-09-11T00:43:24.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.369+0000 7f5e05e00640 1 -- 172.21.15.31:0/1071995082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e000cc8e0 con 0x7f5e000ada20 2024-09-11T00:43:24.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.370+0000 7f5dfcc00640 1 -- 172.21.15.31:0/1071995082 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+4886 (secure 0 0 0) 0x7f5df40677d0 con 0x7f5e000ada20 2024-09-11T00:43:24.372 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:24.372 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"btime":"2024-09-11T00:43:23:741212+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:23.048179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:43:24.372 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 93 2024-09-11T00:43:24.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 -- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5de0077f70 msgr2=0x7f5de007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5de0077f70 0x7f5de007a430 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5dec033000 tx=0x7f5dec004730 comp rx=0 tx=0).stop 2024-09-11T00:43:24.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 -- 172.21.15.31:0/1071995082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e000ada20 msgr2=0x7f5e000c5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e000ada20 0x7f5e000c5a40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5df400e9f0 tx=0x7f5df400eeb0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 -- 172.21.15.31:0/1071995082 shutdown_connections 2024-09-11T00:43:24.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5de0077f70 0x7f5de007a430 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e00156960 0x7f5e000c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5e00150020 0x7f5e000c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 --2- 172.21.15.31:0/1071995082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e000ada20 0x7f5e000c5a40 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 -- 172.21.15.31:0/1071995082 >> 172.21.15.31:0/1071995082 conn(0x7f5e00068f10 msgr2=0x7f5e001549d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:24.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 -- 172.21.15.31:0/1071995082 shutdown_connections 2024-09-11T00:43:24.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.372+0000 7f5e05e00640 1 -- 172.21.15.31:0/1071995082 wait complete. 2024-09-11T00:43:24.391 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-11T00:43:24.391 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:43:24.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.568+0000 7f2c65400640 1 -- 172.21.15.31:0/2015181038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c600a6e20 msgr2=0x7f2c601510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.568+0000 7f2c65400640 1 --2- 172.21.15.31:0/2015181038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c600a6e20 0x7f2c601510c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c005fd0 tx=0x7f2c4c02a560 comp rx=0 tx=0).stop 2024-09-11T00:43:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.568+0000 7f2c65400640 1 -- 172.21.15.31:0/2015181038 shutdown_connections 2024-09-11T00:43:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.568+0000 7f2c65400640 1 --2- 172.21.15.31:0/2015181038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c600a6e20 0x7f2c601510c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.568+0000 7f2c65400640 1 -- 172.21.15.31:0/2015181038 >> 172.21.15.31:0/2015181038 conn(0x7f2c6006caf0 msgr2=0x7f2c6006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.568+0000 7f2c65400640 1 -- 172.21.15.31:0/2015181038 shutdown_connections 2024-09-11T00:43:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.568+0000 7f2c65400640 1 -- 172.21.15.31:0/2015181038 wait complete. 2024-09-11T00:43:24.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.570+0000 7f2c65400640 1 Processor -- start 2024-09-11T00:43:24.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.570+0000 7f2c65400640 1 -- start start 2024-09-11T00:43:24.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c65400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c600a6e20 0x7f2c6014c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:24.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c65400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c600ac2c0 0x7f2c60146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:24.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c65400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c60151600 0x7f2c60146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:24.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c65400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2c600c68c0 con 0x7f2c600a6e20 2024-09-11T00:43:24.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c65400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2c600c5d20 con 0x7f2c600ac2c0 2024-09-11T00:43:24.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c65400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2c600c6a40 con 0x7f2c60151600 2024-09-11T00:43:24.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c5fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c600a6e20 0x7f2c6014c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c5fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c600a6e20 0x7f2c6014c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45830/0 (socket says 172.21.15.31:45830) 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c5fe00640 1 -- 172.21.15.31:0/4269414081 learned_addr learned my addr 172.21.15.31:0/4269414081 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c5f400640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c600ac2c0 0x7f2c60146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c64a00640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c60151600 0x7f2c60146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c5fe00640 1 -- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c60151600 msgr2=0x7f2c60146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c5fe00640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c60151600 0x7f2c60146880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c5fe00640 1 -- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c600ac2c0 msgr2=0x7f2c60146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c5fe00640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c600ac2c0 0x7f2c60146340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c5fe00640 1 -- 172.21.15.31:0/4269414081 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c60149a60 con 0x7f2c600a6e20 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c64a00640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c60151600 0x7f2c60146880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.571+0000 7f2c5f400640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c600ac2c0 0x7f2c60146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.572+0000 7f2c5fe00640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c600a6e20 0x7f2c6014c6e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000c450 tx=0x7f2c5000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.572+0000 7f2c57e00640 1 -- 172.21.15.31:0/4269414081 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c500052d0 con 0x7f2c600a6e20 2024-09-11T00:43:24.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.572+0000 7f2c57e00640 1 -- 172.21.15.31:0/4269414081 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c50005470 con 0x7f2c600a6e20 2024-09-11T00:43:24.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.572+0000 7f2c65400640 1 -- 172.21.15.31:0/4269414081 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c60149280 con 0x7f2c600a6e20 2024-09-11T00:43:24.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.572+0000 7f2c57e00640 1 -- 172.21.15.31:0/4269414081 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c50012870 con 0x7f2c600a6e20 2024-09-11T00:43:24.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.572+0000 7f2c65400640 1 -- 172.21.15.31:0/4269414081 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c60148d70 con 0x7f2c600a6e20 2024-09-11T00:43:24.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.574+0000 7f2c65400640 1 -- 172.21.15.31:0/4269414081 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c600c6e30 con 0x7f2c600a6e20 2024-09-11T00:43:24.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.576+0000 7f2c57e00640 1 -- 172.21.15.31:0/4269414081 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2c500035f0 con 0x7f2c600a6e20 2024-09-11T00:43:24.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.577+0000 7f2c57e00640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c40077f70 0x7f2c4007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:24.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.577+0000 7f2c57e00640 1 -- 172.21.15.31:0/4269414081 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7f2c5009a320 con 0x7f2c600a6e20 2024-09-11T00:43:24.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.577+0000 7f2c5f400640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c40077f70 0x7f2c4007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:24.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.578+0000 7f2c5f400640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c40077f70 0x7f2c4007a430 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c033000 tx=0x7f2c4c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:24.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.581+0000 7f2c57e00640 1 -- 172.21.15.31:0/4269414081 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2c50062420 con 0x7f2c600a6e20 2024-09-11T00:43:24.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.875+0000 7f2c65400640 1 -- 172.21.15.31:0/4269414081 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c6006d070 con 0x7f2c600a6e20 2024-09-11T00:43:24.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.875+0000 7f2c57e00640 1 -- 172.21.15.31:0/4269414081 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+4883 (secure 0 0 0) 0x7f2c50061d20 con 0x7f2c600a6e20 2024-09-11T00:43:24.877 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:24.878 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":94,"btime":"2024-09-11T00:43:24:744753+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:23.747852+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:43:24.878 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 94 2024-09-11T00:43:24.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.878+0000 7f2c65400640 1 -- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c40077f70 msgr2=0x7f2c4007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.878+0000 7f2c65400640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c40077f70 0x7f2c4007a430 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c033000 tx=0x7f2c4c004730 comp rx=0 tx=0).stop 2024-09-11T00:43:24.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.878+0000 7f2c65400640 1 -- 172.21.15.31:0/4269414081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c600a6e20 msgr2=0x7f2c6014c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:24.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.878+0000 7f2c65400640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c600a6e20 0x7f2c6014c6e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000c450 tx=0x7f2c5000c910 comp rx=0 tx=0).stop 2024-09-11T00:43:24.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.878+0000 7f2c65400640 1 -- 172.21.15.31:0/4269414081 shutdown_connections 2024-09-11T00:43:24.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.878+0000 7f2c65400640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c40077f70 0x7f2c4007a430 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.878+0000 7f2c65400640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c60151600 0x7f2c60146880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.878+0000 7f2c65400640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c600ac2c0 0x7f2c60146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.878+0000 7f2c65400640 1 --2- 172.21.15.31:0/4269414081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c600a6e20 0x7f2c6014c6e0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:24.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.878+0000 7f2c65400640 1 -- 172.21.15.31:0/4269414081 >> 172.21.15.31:0/4269414081 conn(0x7f2c6006caf0 msgr2=0x7f2c6014eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:24.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.879+0000 7f2c65400640 1 -- 172.21.15.31:0/4269414081 shutdown_connections 2024-09-11T00:43:24.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:24.879+0000 7f2c65400640 1 -- 172.21.15.31:0/4269414081 wait complete. 2024-09-11T00:43:24.896 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:43:24:744753+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5773": { "addr": "172.21.15.204:6837/3513641573", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 3513641573, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 3513641573, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5773, "incarnation": 88, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 77 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:43:22.726458+0000", "qdb_cluster": [ 5773 ], "qdb_leader": 5773, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5773 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5594": { "addr": "172.21.15.204:6835/1575320768", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1575320768, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 1575320768, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5594, "incarnation": 92, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:43:23.747852+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5594 } } } ], "standbys": [] } 2024-09-11T00:43:25.406 INFO:tasks.ceph.mds.b.smithi204.stderr:2024-09-11T00:43:25.403+0000 7f8a3a600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:43:26.177 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-11T00:43:26.178 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-11T00:43:26.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-11T00:43:26.182 INFO:tasks.ceph.mds.a:Started 2024-09-11T00:43:26.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:26.334 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-11T00:43:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.364+0000 7f9963200640 1 -- 172.21.15.31:0/1705319193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9964156d00 msgr2=0x7f99640b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.364+0000 7f9963200640 1 --2- 172.21.15.31:0/1705319193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9964156d00 0x7f99640b0b40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f99640ae630 tx=0x7f995c02a980 comp rx=0 tx=0).stop 2024-09-11T00:43:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.364+0000 7f9963200640 1 -- 172.21.15.31:0/1705319193 shutdown_connections 2024-09-11T00:43:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.364+0000 7f9963200640 1 --2- 172.21.15.31:0/1705319193 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9964156d00 0x7f99640b0b40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.364+0000 7f9963200640 1 --2- 172.21.15.31:0/1705319193 >> v2:172.21.15.31:3300/0 conn(0x7f99640c9eb0 0x7f99641567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.364+0000 7f9963200640 1 --2- 172.21.15.31:0/1705319193 >> v2:172.21.15.204:3300/0 conn(0x7f99640c6690 0x7f996406dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.364+0000 7f9963200640 1 -- 172.21.15.31:0/1705319193 >> 172.21.15.31:0/1705319193 conn(0x7f99640cbec0 msgr2=0x7f99640cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.364+0000 7f9963200640 1 -- 172.21.15.31:0/1705319193 shutdown_connections 2024-09-11T00:43:26.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.364+0000 7f9963200640 1 -- 172.21.15.31:0/1705319193 wait complete. 2024-09-11T00:43:26.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.366+0000 7f9963200640 1 Processor -- start 2024-09-11T00:43:26.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.367+0000 7f9963200640 1 -- start start 2024-09-11T00:43:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9963200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99640c6690 0x7f99640c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9963200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996414cdc0 0x7f996414d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9963200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99641ef0b0 0x7f99641ef530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9963200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f99640c6bf0 con 0x7f99640c6690 2024-09-11T00:43:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9963200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f99640c6a70 con 0x7f996414cdc0 2024-09-11T00:43:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9963200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f99640bf1a0 con 0x7f99641ef0b0 2024-09-11T00:43:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9961e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99640c6690 0x7f99640c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9962800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99641ef0b0 0x7f99641ef530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:26.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9961400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996414cdc0 0x7f996414d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:26.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9962800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99641ef0b0 0x7f99641ef530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:50660/0 (socket says 172.21.15.31:50660) 2024-09-11T00:43:26.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.368+0000 7f9962800640 1 -- 172.21.15.31:0/3733415617 learned_addr learned my addr 172.21.15.31:0/3733415617 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:26.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.369+0000 7f9962800640 1 -- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996414cdc0 msgr2=0x7f996414d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:26.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.369+0000 7f9962800640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996414cdc0 0x7f996414d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:26.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.369+0000 7f9962800640 1 -- 172.21.15.31:0/3733415617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99640c6690 msgr2=0x7f99640c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:26.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.369+0000 7f9962800640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99640c6690 0x7f99640c3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:26.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.369+0000 7f9962800640 1 -- 172.21.15.31:0/3733415617 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99641e06e0 con 0x7f99641ef0b0 2024-09-11T00:43:26.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.369+0000 7f9962800640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99641ef0b0 0x7f99641ef530 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f995c02af70 tx=0x7f995c003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:26.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.370+0000 7f995a000640 1 -- 172.21.15.31:0/3733415617 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c034040 con 0x7f99641ef0b0 2024-09-11T00:43:26.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.370+0000 7f995a000640 1 -- 172.21.15.31:0/3733415617 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f995c004500 con 0x7f99641ef0b0 2024-09-11T00:43:26.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.370+0000 7f995a000640 1 -- 172.21.15.31:0/3733415617 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c045480 con 0x7f99641ef0b0 2024-09-11T00:43:26.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.370+0000 7f9963200640 1 -- 172.21.15.31:0/3733415617 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99640c57e0 con 0x7f99641ef0b0 2024-09-11T00:43:26.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.370+0000 7f9963200640 1 -- 172.21.15.31:0/3733415617 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f99641e2350 con 0x7f99641ef0b0 2024-09-11T00:43:26.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.371+0000 7f9963200640 1 -- 172.21.15.31:0/3733415617 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99640c99a0 con 0x7f99641ef0b0 2024-09-11T00:43:26.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.372+0000 7f995a000640 1 -- 172.21.15.31:0/3733415617 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f995c03a430 con 0x7f99641ef0b0 2024-09-11T00:43:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.373+0000 7f995a000640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9944077f70 0x7f994407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.373+0000 7f995a000640 1 -- 172.21.15.31:0/3733415617 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7f995c0bbfd0 con 0x7f99641ef0b0 2024-09-11T00:43:26.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.374+0000 7f9961e00640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9944077f70 0x7f994407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:26.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.374+0000 7f9961e00640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9944077f70 0x7f994407a430 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f99540095f0 tx=0x7f9954005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:26.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.378+0000 7f995a000640 1 -- 172.21.15.31:0/3733415617 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f995c083aa0 con 0x7f99641ef0b0 2024-09-11T00:43:26.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.696+0000 7f9963200640 1 -- 172.21.15.31:0/3733415617 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99640cc620 con 0x7f99641ef0b0 2024-09-11T00:43:26.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.696+0000 7f995a000640 1 -- 172.21.15.31:0/3733415617 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+4890 (secure 0 0 0) 0x7f995c0884d0 con 0x7f99641ef0b0 2024-09-11T00:43:26.698 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:26.698 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":95,"btime":"2024-09-11T00:43:25:749837+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:43:26.699 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 95 2024-09-11T00:43:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 -- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9944077f70 msgr2=0x7f994407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9944077f70 0x7f994407a430 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f99540095f0 tx=0x7f9954005f90 comp rx=0 tx=0).stop 2024-09-11T00:43:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 -- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99641ef0b0 msgr2=0x7f99641ef530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99641ef0b0 0x7f99641ef530 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f995c02af70 tx=0x7f995c003f10 comp rx=0 tx=0).stop 2024-09-11T00:43:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 -- 172.21.15.31:0/3733415617 shutdown_connections 2024-09-11T00:43:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9944077f70 0x7f994407a430 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99641ef0b0 0x7f99641ef530 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f996414cdc0 0x7f996414d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 --2- 172.21.15.31:0/3733415617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99640c6690 0x7f99640c3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:26.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 -- 172.21.15.31:0/3733415617 >> 172.21.15.31:0/3733415617 conn(0x7f99640cbec0 msgr2=0x7f99641521d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:26.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 -- 172.21.15.31:0/3733415617 shutdown_connections 2024-09-11T00:43:26.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:26.699+0000 7f9963200640 1 -- 172.21.15.31:0/3733415617 wait complete. 2024-09-11T00:43:26.718 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-11T00:43:26.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:27.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.082+0000 7f0a99200640 1 -- 172.21.15.31:0/3330044230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a940c6340 msgr2=0x7f0a94159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.082+0000 7f0a99200640 1 --2- 172.21.15.31:0/3330044230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a940c6340 0x7f0a94159670 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0a94070470 tx=0x7f0a8c02aac0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.082+0000 7f0a99200640 1 -- 172.21.15.31:0/3330044230 shutdown_connections 2024-09-11T00:43:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.082+0000 7f0a99200640 1 --2- 172.21.15.31:0/3330044230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a940c6340 0x7f0a94159670 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.082+0000 7f0a99200640 1 --2- 172.21.15.31:0/3330044230 >> v2:172.21.15.204:3300/0 conn(0x7f0a940c5b10 0x7f0a94156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.082+0000 7f0a99200640 1 -- 172.21.15.31:0/3330044230 >> 172.21.15.31:0/3330044230 conn(0x7f0a940a9750 msgr2=0x7f0a940a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.082+0000 7f0a99200640 1 -- 172.21.15.31:0/3330044230 shutdown_connections 2024-09-11T00:43:27.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.082+0000 7f0a99200640 1 -- 172.21.15.31:0/3330044230 wait complete. 2024-09-11T00:43:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.084+0000 7f0a99200640 1 Processor -- start 2024-09-11T00:43:27.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.084+0000 7f0a99200640 1 -- start start 2024-09-11T00:43:27.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a99200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a940c5b10 0x7f0a9414f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:27.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a99200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a940c5ef0 0x7f0a94150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:27.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a99200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a940c6340 0x7f0a9414e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:27.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a99200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a94153b90 con 0x7f0a940c5ef0 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a99200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0a940c3950 con 0x7f0a940c5b10 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a99200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0a94153d10 con 0x7f0a940c6340 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a91400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a940c5ef0 0x7f0a94150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a91e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a940c5b10 0x7f0a9414f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a92800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a940c6340 0x7f0a9414e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a91e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a940c5b10 0x7f0a9414f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:50822/0 (socket says 172.21.15.31:50822) 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a91e00640 1 -- 172.21.15.31:0/1716332079 learned_addr learned my addr 172.21.15.31:0/1716332079 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.085+0000 7f0a91400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a940c5ef0 0x7f0a94150030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45874/0 (socket says 172.21.15.31:45874) 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a91400640 1 -- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a940c6340 msgr2=0x7f0a9414e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a91400640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a940c6340 0x7f0a9414e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a91400640 1 -- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a940c5b10 msgr2=0x7f0a9414f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a91400640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a940c5b10 0x7f0a9414f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a91400640 1 -- 172.21.15.31:0/1716332079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a94073630 con 0x7f0a940c5ef0 2024-09-11T00:43:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a91400640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a940c5ef0 0x7f0a94150030 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000c3f0 tx=0x7f0a8000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a8a000640 1 -- 172.21.15.31:0/1716332079 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80017070 con 0x7f0a940c5ef0 2024-09-11T00:43:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a8a000640 1 -- 172.21.15.31:0/1716332079 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a800053c0 con 0x7f0a940c5ef0 2024-09-11T00:43:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a99200640 1 -- 172.21.15.31:0/1716332079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a941549d0 con 0x7f0a940c5ef0 2024-09-11T00:43:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a8a000640 1 -- 172.21.15.31:0/1716332079 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80003850 con 0x7f0a940c5ef0 2024-09-11T00:43:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.086+0000 7f0a99200640 1 -- 172.21.15.31:0/1716332079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a941546f0 con 0x7f0a940c5ef0 2024-09-11T00:43:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.088+0000 7f0a99200640 1 -- 172.21.15.31:0/1716332079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a941568a0 con 0x7f0a940c5ef0 2024-09-11T00:43:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.090+0000 7f0a8a000640 1 -- 172.21.15.31:0/1716332079 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0a800039f0 con 0x7f0a940c5ef0 2024-09-11T00:43:27.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.090+0000 7f0a8a000640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a74077f70 0x7f0a7407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:27.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.091+0000 7f0a8a000640 1 -- 172.21.15.31:0/1716332079 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7f0a80099cd0 con 0x7f0a940c5ef0 2024-09-11T00:43:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.091+0000 7f0a91e00640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a74077f70 0x7f0a7407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.092+0000 7f0a91e00640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a74077f70 0x7f0a7407a430 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0a940c4df0 tx=0x7f0a8400b4a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:27.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.095+0000 7f0a8a000640 1 -- 172.21.15.31:0/1716332079 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0a941568a0 con 0x7f0a940c5ef0 2024-09-11T00:43:27.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.381+0000 7f0a99200640 1 -- 172.21.15.31:0/1716332079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a940c1e70 con 0x7f0a940c5ef0 2024-09-11T00:43:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.382+0000 7f0a8a000640 1 -- 172.21.15.31:0/1716332079 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5714 (secure 0 0 0) 0x7f0a80061820 con 0x7f0a940c5ef0 2024-09-11T00:43:27.384 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:27.384 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":96,"btime":"2024-09-11T00:43:26:755763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:43:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 96 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.384+0000 7f0a99200640 1 -- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a74077f70 msgr2=0x7f0a7407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a74077f70 0x7f0a7407a430 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0a940c4df0 tx=0x7f0a8400b4a0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 -- 172.21.15.31:0/1716332079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a940c5ef0 msgr2=0x7f0a94150030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a940c5ef0 0x7f0a94150030 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000c3f0 tx=0x7f0a8000c8b0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 -- 172.21.15.31:0/1716332079 shutdown_connections 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a74077f70 0x7f0a7407a430 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a940c6340 0x7f0a9414e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a940c5ef0 0x7f0a94150030 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 --2- 172.21.15.31:0/1716332079 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a940c5b10 0x7f0a9414f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 -- 172.21.15.31:0/1716332079 >> 172.21.15.31:0/1716332079 conn(0x7f0a940a9750 msgr2=0x7f0a940c42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:27.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 -- 172.21.15.31:0/1716332079 shutdown_connections 2024-09-11T00:43:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:27.385+0000 7f0a99200640 1 -- 172.21.15.31:0/1716332079 wait complete. 2024-09-11T00:43:27.405 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=6087 2024-09-11T00:43:27.405 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.c 2024-09-11T00:43:27.826 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:43:27.823+0000 7f71df800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:43:28.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.897+0000 7fdc14e00640 1 -- 172.21.15.31:0/3122271007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc100c64b0 msgr2=0x7fdc1006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.897+0000 7fdc14e00640 1 --2- 172.21.15.31:0/3122271007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc100c64b0 0x7fdc1006dc60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009ca0 tx=0x7fdc0802aae0 comp rx=0 tx=0).stop 2024-09-11T00:43:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.897+0000 7fdc14e00640 1 -- 172.21.15.31:0/3122271007 shutdown_connections 2024-09-11T00:43:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.897+0000 7fdc14e00640 1 --2- 172.21.15.31:0/3122271007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc100c64b0 0x7fdc1006dc60 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.897+0000 7fdc14e00640 1 -- 172.21.15.31:0/3122271007 >> 172.21.15.31:0/3122271007 conn(0x7fdc1006caf0 msgr2=0x7fdc1006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.897+0000 7fdc14e00640 1 -- 172.21.15.31:0/3122271007 shutdown_connections 2024-09-11T00:43:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.897+0000 7fdc14e00640 1 -- 172.21.15.31:0/3122271007 wait complete. 2024-09-11T00:43:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.899+0000 7fdc14e00640 1 Processor -- start 2024-09-11T00:43:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.900+0000 7fdc14e00640 1 -- start start 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc14e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc10070410 0x7fdc100ab230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc14e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc100acaf0 0x7fdc1014cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc14e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc100c64b0 0x7fdc1014d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc14e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdc100c4ad0 con 0x7fdc100c64b0 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc14e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc100700f0 con 0x7fdc100acaf0 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc14e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdc100c4c50 con 0x7fdc10070410 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc0fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc100c64b0 0x7fdc1014d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc0ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc100acaf0 0x7fdc1014cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc0fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc100c64b0 0x7fdc1014d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40238/0 (socket says 172.21.15.31:40238) 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc0fe00640 1 -- 172.21.15.31:0/2056494506 learned_addr learned my addr 172.21.15.31:0/2056494506 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc0f400640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc10070410 0x7fdc100ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc0fe00640 1 -- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc10070410 msgr2=0x7fdc100ab230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc0fe00640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc10070410 0x7fdc100ab230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc0fe00640 1 -- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc100acaf0 msgr2=0x7fdc1014cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc0fe00640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc100acaf0 0x7fdc1014cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.901+0000 7fdc0fe00640 1 -- 172.21.15.31:0/2056494506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc10074a10 con 0x7fdc100c64b0 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.902+0000 7fdc0ea00640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc100acaf0 0x7fdc1014cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.902+0000 7fdc0fe00640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc100c64b0 0x7fdc1014d170 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009ca0 tx=0x7fdc08003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.902+0000 7fdc07400640 1 -- 172.21.15.31:0/2056494506 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc08044070 con 0x7fdc100c64b0 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.902+0000 7fdc07400640 1 -- 172.21.15.31:0/2056494506 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc08004250 con 0x7fdc100c64b0 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.902+0000 7fdc14e00640 1 -- 172.21.15.31:0/2056494506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc10072070 con 0x7fdc100c64b0 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.902+0000 7fdc07400640 1 -- 172.21.15.31:0/2056494506 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc08004680 con 0x7fdc100c64b0 2024-09-11T00:43:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.902+0000 7fdc14e00640 1 -- 172.21.15.31:0/2056494506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc10152b40 con 0x7fdc100c64b0 2024-09-11T00:43:28.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.904+0000 7fdc14e00640 1 -- 172.21.15.31:0/2056494506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc1006dc60 con 0x7fdc100c64b0 2024-09-11T00:43:28.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.905+0000 7fdc07400640 1 -- 172.21.15.31:0/2056494506 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdc0802e980 con 0x7fdc100c64b0 2024-09-11T00:43:28.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.906+0000 7fdc07400640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdbf0077fa0 0x7fdbf007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:28.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.906+0000 7fdc07400640 1 -- 172.21.15.31:0/2056494506 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7fdc080bb8c0 con 0x7fdc100c64b0 2024-09-11T00:43:28.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.906+0000 7fdc0f400640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdbf0077fa0 0x7fdbf007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:28.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.907+0000 7fdc0f400640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdbf0077fa0 0x7fdbf007a460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdc000017c0 tx=0x7fdc00009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:28.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:28.911+0000 7fdc07400640 1 -- 172.21.15.31:0/2056494506 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdc08032030 con 0x7fdc100c64b0 2024-09-11T00:43:29.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.200+0000 7fdc14e00640 1 -- 172.21.15.31:0/2056494506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc100c3e20 con 0x7fdc100c64b0 2024-09-11T00:43:29.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.201+0000 7fdc07400640 1 -- 172.21.15.31:0/2056494506 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5714 (secure 0 0 0) 0x7fdc08004820 con 0x7fdc100c64b0 2024-09-11T00:43:29.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:29.203 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":96,"btime":"2024-09-11T00:43:26:755763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:43:29.203 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 96 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 -- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdbf0077fa0 msgr2=0x7fdbf007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdbf0077fa0 0x7fdbf007a460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdc000017c0 tx=0x7fdc00009040 comp rx=0 tx=0).stop 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 -- 172.21.15.31:0/2056494506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc100c64b0 msgr2=0x7fdc1014d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc100c64b0 0x7fdc1014d170 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009ca0 tx=0x7fdc08003f10 comp rx=0 tx=0).stop 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 -- 172.21.15.31:0/2056494506 shutdown_connections 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdbf0077fa0 0x7fdbf007a460 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc100c64b0 0x7fdc1014d170 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc100acaf0 0x7fdc1014cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 --2- 172.21.15.31:0/2056494506 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc10070410 0x7fdc100ab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 -- 172.21.15.31:0/2056494506 >> 172.21.15.31:0/2056494506 conn(0x7fdc1006caf0 msgr2=0x7fdc100aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 -- 172.21.15.31:0/2056494506 shutdown_connections 2024-09-11T00:43:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:29.204+0000 7fdc14e00640 1 -- 172.21.15.31:0/2056494506 wait complete. 2024-09-11T00:43:29.222 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-11T00:43:29.223 INFO:tasks.mds_thrash.fs.[a]:waiting for 101 secs before thrashing 2024-09-11T00:43:34.407 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-11T00:43:34.407 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-11T00:43:34.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-11T00:43:34.412 INFO:tasks.ceph.mds.c:Started 2024-09-11T00:43:34.412 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:34.561 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-11T00:43:34.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.604+0000 7efdcb200640 1 -- 172.21.15.31:0/3970888196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdb00037a0 msgr2=0x7efdb0003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:34.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.604+0000 7efdcb200640 1 --2- 172.21.15.31:0/3970888196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdb00037a0 0x7efdb0003c20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7efdb8006eb0 tx=0x7efdb0005830 comp rx=0 tx=0).stop 2024-09-11T00:43:34.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.604+0000 7efdcb200640 1 -- 172.21.15.31:0/3970888196 shutdown_connections 2024-09-11T00:43:34.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.604+0000 7efdcb200640 1 --2- 172.21.15.31:0/3970888196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdb00037a0 0x7efdb0003c20 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:34.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.604+0000 7efdcb200640 1 -- 172.21.15.31:0/3970888196 >> 172.21.15.31:0/3970888196 conn(0x7efdcc0c9af0 msgr2=0x7efdcc0c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:34.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.604+0000 7efdcb200640 1 -- 172.21.15.31:0/3970888196 shutdown_connections 2024-09-11T00:43:34.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.604+0000 7efdcb200640 1 -- 172.21.15.31:0/3970888196 wait complete. 2024-09-11T00:43:34.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.605+0000 7efdcb200640 1 Processor -- start 2024-09-11T00:43:34.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.606+0000 7efdcb200640 1 -- start start 2024-09-11T00:43:34.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.606+0000 7efdcb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efdb00037a0 0x7efdcc1ec040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.606+0000 7efdcb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdcc1f0230 0x7efdcc1ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdcb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc0a86c0 0x7efdcc0adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdcb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc0a9260 con 0x7efdcc0a86c0 2024-09-11T00:43:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdcb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc0a90e0 con 0x7efdb00037a0 2024-09-11T00:43:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdcb200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efdcc0c5b20 con 0x7efdcc1f0230 2024-09-11T00:43:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdc9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efdb00037a0 0x7efdcc1ec040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc0a86c0 0x7efdcc0adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdc9e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efdb00037a0 0x7efdcc1ec040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:40240/0 (socket says 172.21.15.31:40240) 2024-09-11T00:43:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdc9e00640 1 -- 172.21.15.31:0/3113415356 learned_addr learned my addr 172.21.15.31:0/3113415356 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:34.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdc9400640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdcc1f0230 0x7efdcc1ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdc9e00640 1 -- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdcc1f0230 msgr2=0x7efdcc1ec580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdc9e00640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdcc1f0230 0x7efdcc1ec580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdc9e00640 1 -- 172.21.15.31:0/3113415356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc0a86c0 msgr2=0x7efdcc0adc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdc9e00640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc0a86c0 0x7efdcc0adc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:34.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdc9e00640 1 -- 172.21.15.31:0/3113415356 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc06f380 con 0x7efdb00037a0 2024-09-11T00:43:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.607+0000 7efdca800640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc0a86c0 0x7efdcc0adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:43:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.608+0000 7efdc9e00640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efdb00037a0 0x7efdcc1ec040 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efdbc006600 tx=0x7efdbc0039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.608+0000 7efdc2000640 1 -- 172.21.15.31:0/3113415356 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc015070 con 0x7efdb00037a0 2024-09-11T00:43:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.608+0000 7efdc2000640 1 -- 172.21.15.31:0/3113415356 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efdbc011860 con 0x7efdb00037a0 2024-09-11T00:43:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.608+0000 7efdc2000640 1 -- 172.21.15.31:0/3113415356 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc011cb0 con 0x7efdb00037a0 2024-09-11T00:43:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.608+0000 7efdcb200640 1 -- 172.21.15.31:0/3113415356 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdcc06f0a0 con 0x7efdb00037a0 2024-09-11T00:43:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.608+0000 7efdcb200640 1 -- 172.21.15.31:0/3113415356 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdcc0a8170 con 0x7efdb00037a0 2024-09-11T00:43:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.609+0000 7efdcb200640 1 -- 172.21.15.31:0/3113415356 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdcc0c6bd0 con 0x7efdb00037a0 2024-09-11T00:43:34.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.611+0000 7efdc2000640 1 -- 172.21.15.31:0/3113415356 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7efdbc0103c0 con 0x7efdb00037a0 2024-09-11T00:43:34.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.612+0000 7efdc2000640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efdac077fa0 0x7efdac07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:34.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.612+0000 7efdc2000640 1 -- 172.21.15.31:0/3113415356 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7efdbc09b5a0 con 0x7efdb00037a0 2024-09-11T00:43:34.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.612+0000 7efdc9400640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efdac077fa0 0x7efdac07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:34.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.613+0000 7efdc9400640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efdac077fa0 0x7efdac07a460 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7efdb8006e80 tx=0x7efdb802d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:34.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.618+0000 7efdc2000640 1 -- 172.21.15.31:0/3113415356 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efdbc0630f0 con 0x7efdb00037a0 2024-09-11T00:43:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.914+0000 7efdcb200640 1 -- 172.21.15.31:0/3113415356 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efdcc0c7a60 con 0x7efdb00037a0 2024-09-11T00:43:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.915+0000 7efdc2000640 1 -- 172.21.15.31:0/3113415356 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6539 (secure 0 0 0) 0x7efdbc067b20 con 0x7efdb00037a0 2024-09-11T00:43:34.917 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:34.917 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"btime":"2024-09-11T00:43:34:776882+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6402,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:43:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 97 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.917+0000 7efdcb200640 1 -- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efdac077fa0 msgr2=0x7efdac07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.917+0000 7efdcb200640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efdac077fa0 0x7efdac07a460 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7efdb8006e80 tx=0x7efdb802d000 comp rx=0 tx=0).stop 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.918+0000 7efdcb200640 1 -- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efdb00037a0 msgr2=0x7efdcc1ec040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.918+0000 7efdcb200640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efdb00037a0 0x7efdcc1ec040 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efdbc006600 tx=0x7efdbc0039c0 comp rx=0 tx=0).stop 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.918+0000 7efdcb200640 1 -- 172.21.15.31:0/3113415356 shutdown_connections 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.918+0000 7efdcb200640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efdac077fa0 0x7efdac07a460 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.918+0000 7efdcb200640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc0a86c0 0x7efdcc0adc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.918+0000 7efdcb200640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efdcc1f0230 0x7efdcc1ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.918+0000 7efdcb200640 1 --2- 172.21.15.31:0/3113415356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efdb00037a0 0x7efdcc1ec040 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.918+0000 7efdcb200640 1 -- 172.21.15.31:0/3113415356 >> 172.21.15.31:0/3113415356 conn(0x7efdcc0c9af0 msgr2=0x7efdcc157920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:34.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.918+0000 7efdcb200640 1 -- 172.21.15.31:0/3113415356 shutdown_connections 2024-09-11T00:43:34.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:34.918+0000 7efdcb200640 1 -- 172.21.15.31:0/3113415356 wait complete. 2024-09-11T00:43:34.936 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-11T00:43:34.936 INFO:tasks.mds_thrash.fs.[b]:waiting for 22 secs before thrashing 2024-09-11T00:43:45.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:43:45.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:43:56.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:57.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.122+0000 7fbe92a00640 1 -- 172.21.15.31:0/4291054443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe8c0c3280 msgr2=0x7fbe8c153470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:57.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.122+0000 7fbe92a00640 1 --2- 172.21.15.31:0/4291054443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe8c0c3280 0x7fbe8c153470 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0aaa50 tx=0x7fbe7802a4d0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.122+0000 7fbe92a00640 1 -- 172.21.15.31:0/4291054443 shutdown_connections 2024-09-11T00:43:57.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.122+0000 7fbe92a00640 1 --2- 172.21.15.31:0/4291054443 >> v2:172.21.15.204:3300/0 conn(0x7fbe8c1539b0 0x7fbe8c1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.122+0000 7fbe92a00640 1 --2- 172.21.15.31:0/4291054443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe8c0c3280 0x7fbe8c153470 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.122+0000 7fbe92a00640 1 -- 172.21.15.31:0/4291054443 >> 172.21.15.31:0/4291054443 conn(0x7fbe8c06caf0 msgr2=0x7fbe8c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:57.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.123+0000 7fbe92a00640 1 -- 172.21.15.31:0/4291054443 shutdown_connections 2024-09-11T00:43:57.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.123+0000 7fbe92a00640 1 -- 172.21.15.31:0/4291054443 wait complete. 2024-09-11T00:43:57.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.125+0000 7fbe92a00640 1 Processor -- start 2024-09-11T00:43:57.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.125+0000 7fbe92a00640 1 -- start start 2024-09-11T00:43:57.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.126+0000 7fbe92a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe8c0c3280 0x7fbe8c14fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:57.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.126+0000 7fbe92a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe8c0c6bb0 0x7fbe8c0728c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:57.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.126+0000 7fbe92a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe8c1539b0 0x7fbe8c151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:57.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.126+0000 7fbe92a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c152f00 con 0x7fbe8c0c3280 2024-09-11T00:43:57.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.126+0000 7fbe92a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c152d80 con 0x7fbe8c0c6bb0 2024-09-11T00:43:57.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.126+0000 7fbe92a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbe8c0c6f90 con 0x7fbe8c1539b0 2024-09-11T00:43:57.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.127+0000 7fbe92000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe8c1539b0 0x7fbe8c151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:57.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.127+0000 7fbe90c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe8c0c6bb0 0x7fbe8c0728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:57.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.127+0000 7fbe91600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe8c0c3280 0x7fbe8c14fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:57.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.127+0000 7fbe92000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe8c1539b0 0x7fbe8c151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:33758/0 (socket says 172.21.15.31:33758) 2024-09-11T00:43:57.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.127+0000 7fbe90c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe8c0c6bb0 0x7fbe8c0728c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:53914/0 (socket says 172.21.15.31:53914) 2024-09-11T00:43:57.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.127+0000 7fbe92000640 1 -- 172.21.15.31:0/1391108345 learned_addr learned my addr 172.21.15.31:0/1391108345 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:57.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.128+0000 7fbe92000640 1 -- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe8c0c6bb0 msgr2=0x7fbe8c0728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:57.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.128+0000 7fbe92000640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe8c0c6bb0 0x7fbe8c0728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.128+0000 7fbe92000640 1 -- 172.21.15.31:0/1391108345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe8c0c3280 msgr2=0x7fbe8c14fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:57.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.128+0000 7fbe92000640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe8c0c3280 0x7fbe8c14fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.128+0000 7fbe92000640 1 -- 172.21.15.31:0/1391108345 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8c073e80 con 0x7fbe8c1539b0 2024-09-11T00:43:57.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.128+0000 7fbe90c00640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe8c0c6bb0 0x7fbe8c0728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:43:57.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.128+0000 7fbe92000640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe8c1539b0 0x7fbe8c151190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400eab0 tx=0x7fbe8400ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:57.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.128+0000 7fbe89600640 1 -- 172.21.15.31:0/1391108345 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8400cea0 con 0x7fbe8c1539b0 2024-09-11T00:43:57.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.128+0000 7fbe89600640 1 -- 172.21.15.31:0/1391108345 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe84004b20 con 0x7fbe8c1539b0 2024-09-11T00:43:57.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.128+0000 7fbe89600640 1 -- 172.21.15.31:0/1391108345 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84005020 con 0x7fbe8c1539b0 2024-09-11T00:43:57.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.129+0000 7fbe92a00640 1 -- 172.21.15.31:0/1391108345 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe8c073ba0 con 0x7fbe8c1539b0 2024-09-11T00:43:57.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.129+0000 7fbe92a00640 1 -- 172.21.15.31:0/1391108345 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe8c1f0150 con 0x7fbe8c1539b0 2024-09-11T00:43:57.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.131+0000 7fbe92a00640 1 -- 172.21.15.31:0/1391108345 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe8c153470 con 0x7fbe8c1539b0 2024-09-11T00:43:57.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.132+0000 7fbe89600640 1 -- 172.21.15.31:0/1391108345 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbe840051c0 con 0x7fbe8c1539b0 2024-09-11T00:43:57.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.134+0000 7fbe89600640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe6c077fa0 0x7fbe6c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:57.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.134+0000 7fbe89600640 1 -- 172.21.15.31:0/1391108345 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7fbe8409b180 con 0x7fbe8c1539b0 2024-09-11T00:43:57.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.134+0000 7fbe91600640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe6c077fa0 0x7fbe6c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:57.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.135+0000 7fbe91600640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe6c077fa0 0x7fbe6c07a460 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbe800017c0 tx=0x7fbe80009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:57.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.140+0000 7fbe89600640 1 -- 172.21.15.31:0/1391108345 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbe84062d50 con 0x7fbe8c1539b0 2024-09-11T00:43:57.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.407+0000 7fbe92a00640 1 -- 172.21.15.31:0/1391108345 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe8c0726a0 con 0x7fbe8c1539b0 2024-09-11T00:43:57.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.408+0000 7fbe89600640 1 -- 172.21.15.31:0/1391108345 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6539 (secure 0 0 0) 0x7fbe84067780 con 0x7fbe8c1539b0 2024-09-11T00:43:57.410 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:57.411 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"btime":"2024-09-11T00:43:34:776882+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6402,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:43:57.411 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 97 2024-09-11T00:43:57.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 -- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe6c077fa0 msgr2=0x7fbe6c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:57.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe6c077fa0 0x7fbe6c07a460 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbe800017c0 tx=0x7fbe80009040 comp rx=0 tx=0).stop 2024-09-11T00:43:57.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 -- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe8c1539b0 msgr2=0x7fbe8c151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:57.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe8c1539b0 0x7fbe8c151190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400eab0 tx=0x7fbe8400ef70 comp rx=0 tx=0).stop 2024-09-11T00:43:57.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 -- 172.21.15.31:0/1391108345 shutdown_connections 2024-09-11T00:43:57.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe6c077fa0 0x7fbe6c07a460 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe8c1539b0 0x7fbe8c151190 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe8c0c6bb0 0x7fbe8c0728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 --2- 172.21.15.31:0/1391108345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe8c0c3280 0x7fbe8c14fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 -- 172.21.15.31:0/1391108345 >> 172.21.15.31:0/1391108345 conn(0x7fbe8c06caf0 msgr2=0x7fbe8c0abaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:57.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 -- 172.21.15.31:0/1391108345 shutdown_connections 2024-09-11T00:43:57.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.410+0000 7fbe92a00640 1 -- 172.21.15.31:0/1391108345 wait complete. 2024-09-11T00:43:57.429 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-11T00:43:57.429 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:43:57.429 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:43:57.445 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:43:57.446 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:43:57.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.622+0000 7fa885800640 1 -- 172.21.15.31:0/3973524547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa880150020 msgr2=0x7fa8800ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:57.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.622+0000 7fa885800640 1 --2- 172.21.15.31:0/3973524547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa880150020 0x7fa8800ad4e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa870009830 tx=0x7fa87002a5c0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.622+0000 7fa885800640 1 -- 172.21.15.31:0/3973524547 shutdown_connections 2024-09-11T00:43:57.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.622+0000 7fa885800640 1 --2- 172.21.15.31:0/3973524547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa880150020 0x7fa8800ad4e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.622+0000 7fa885800640 1 -- 172.21.15.31:0/3973524547 >> 172.21.15.31:0/3973524547 conn(0x7fa880068f10 msgr2=0x7fa8800cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:57.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.622+0000 7fa885800640 1 -- 172.21.15.31:0/3973524547 shutdown_connections 2024-09-11T00:43:57.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.622+0000 7fa885800640 1 -- 172.21.15.31:0/3973524547 wait complete. 2024-09-11T00:43:57.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.624+0000 7fa885800640 1 Processor -- start 2024-09-11T00:43:57.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.624+0000 7fa885800640 1 -- start start 2024-09-11T00:43:57.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa885800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa88006e850 0x7fa8800ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:57.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa885800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8800ada20 0x7fa8800c5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:57.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa885800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa880150020 0x7fa88014be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:57.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa885800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa880157550 con 0x7fa880150020 2024-09-11T00:43:57.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa885800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa8801573d0 con 0x7fa88006e850 2024-09-11T00:43:57.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa885800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa880156960 con 0x7fa8800ada20 2024-09-11T00:43:57.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa884e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa880150020 0x7fa88014be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:57.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa87fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa88006e850 0x7fa8800ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:57.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa87fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa88006e850 0x7fa8800ac780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:53928/0 (socket says 172.21.15.31:53928) 2024-09-11T00:43:57.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa87f400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8800ada20 0x7fa8800c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:57.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.625+0000 7fa87fe00640 1 -- 172.21.15.31:0/2784365444 learned_addr learned my addr 172.21.15.31:0/2784365444 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:43:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa884e00640 1 -- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8800ada20 msgr2=0x7fa8800c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa884e00640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8800ada20 0x7fa8800c5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa884e00640 1 -- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa88006e850 msgr2=0x7fa8800ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa884e00640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa88006e850 0x7fa8800ac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa884e00640 1 -- 172.21.15.31:0/2784365444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8801532d0 con 0x7fa880150020 2024-09-11T00:43:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa87fe00640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa88006e850 0x7fa8800ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:43:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa884e00640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa880150020 0x7fa88014be80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa87800ca00 tx=0x7fa87800cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa877e00640 1 -- 172.21.15.31:0/2784365444 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878016020 con 0x7fa880150020 2024-09-11T00:43:57.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa877e00640 1 -- 172.21.15.31:0/2784365444 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa87800ac60 con 0x7fa880150020 2024-09-11T00:43:57.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa877e00640 1 -- 172.21.15.31:0/2784365444 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878012680 con 0x7fa880150020 2024-09-11T00:43:57.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.626+0000 7fa885800640 1 -- 172.21.15.31:0/2784365444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8801e1c50 con 0x7fa880150020 2024-09-11T00:43:57.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.627+0000 7fa885800640 1 -- 172.21.15.31:0/2784365444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8800accc0 con 0x7fa880150020 2024-09-11T00:43:57.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.628+0000 7fa885800640 1 -- 172.21.15.31:0/2784365444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa880152de0 con 0x7fa880150020 2024-09-11T00:43:57.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.629+0000 7fa877e00640 1 -- 172.21.15.31:0/2784365444 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa878012820 con 0x7fa880150020 2024-09-11T00:43:57.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.630+0000 7fa877e00640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa860077f70 0x7fa86007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:43:57.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.630+0000 7fa877e00640 1 -- 172.21.15.31:0/2784365444 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7fa87809aaf0 con 0x7fa880150020 2024-09-11T00:43:57.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.630+0000 7fa87fe00640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa860077f70 0x7fa86007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:43:57.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.631+0000 7fa87fe00640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa860077f70 0x7fa86007a430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa870009830 tx=0x7fa870004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:43:57.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.634+0000 7fa877e00640 1 -- 172.21.15.31:0/2784365444 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa878020030 con 0x7fa880150020 2024-09-11T00:43:57.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.889+0000 7fa885800640 1 -- 172.21.15.31:0/2784365444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8800cc220 con 0x7fa880150020 2024-09-11T00:43:57.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.889+0000 7fa877e00640 1 -- 172.21.15.31:0/2784365444 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6539 (secure 0 0 0) 0x7fa8780625b0 con 0x7fa880150020 2024-09-11T00:43:57.893 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:43:57.893 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"btime":"2024-09-11T00:43:34:776882+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6402,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:43:57.893 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 97 2024-09-11T00:43:57.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.892+0000 7fa885800640 1 -- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa860077f70 msgr2=0x7fa86007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:57.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.892+0000 7fa885800640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa860077f70 0x7fa86007a430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa870009830 tx=0x7fa870004730 comp rx=0 tx=0).stop 2024-09-11T00:43:57.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.892+0000 7fa885800640 1 -- 172.21.15.31:0/2784365444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa880150020 msgr2=0x7fa88014be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:43:57.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.892+0000 7fa885800640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa880150020 0x7fa88014be80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa87800ca00 tx=0x7fa87800cec0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.892+0000 7fa885800640 1 -- 172.21.15.31:0/2784365444 shutdown_connections 2024-09-11T00:43:57.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.892+0000 7fa885800640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa860077f70 0x7fa86007a430 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.892+0000 7fa885800640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa880150020 0x7fa88014be80 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.892+0000 7fa885800640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa8800ada20 0x7fa8800c5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.892+0000 7fa885800640 1 --2- 172.21.15.31:0/2784365444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa88006e850 0x7fa8800ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:43:57.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.892+0000 7fa885800640 1 -- 172.21.15.31:0/2784365444 >> 172.21.15.31:0/2784365444 conn(0x7fa880068f10 msgr2=0x7fa880158350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:43:57.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.893+0000 7fa885800640 1 -- 172.21.15.31:0/2784365444 shutdown_connections 2024-09-11T00:43:57.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:43:57.893+0000 7fa885800640 1 -- 172.21.15.31:0/2784365444 wait complete. 2024-09-11T00:43:57.913 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:43:59.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:00.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.094+0000 7fb69b200640 1 -- 172.21.15.31:0/3960153507 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb69c159b60 msgr2=0x7fb69c1617f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:00.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.094+0000 7fb69b200640 1 --2- 172.21.15.31:0/3960153507 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb69c159b60 0x7fb69c1617f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb694009be0 tx=0x7fb69402a920 comp rx=0 tx=0).stop 2024-09-11T00:44:00.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.094+0000 7fb69b200640 1 -- 172.21.15.31:0/3960153507 shutdown_connections 2024-09-11T00:44:00.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.094+0000 7fb69b200640 1 --2- 172.21.15.31:0/3960153507 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb69c159b60 0x7fb69c1617f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:00.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.094+0000 7fb69b200640 1 -- 172.21.15.31:0/3960153507 >> 172.21.15.31:0/3960153507 conn(0x7fb69c0adb00 msgr2=0x7fb69c0adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:00.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.094+0000 7fb69b200640 1 -- 172.21.15.31:0/3960153507 shutdown_connections 2024-09-11T00:44:00.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.094+0000 7fb69b200640 1 -- 172.21.15.31:0/3960153507 wait complete. 2024-09-11T00:44:00.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.096+0000 7fb69b200640 1 Processor -- start 2024-09-11T00:44:00.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.096+0000 7fb69b200640 1 -- start start 2024-09-11T00:44:00.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb69b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb69c06cc10 0x7fb69c158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:00.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb69b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb69c14c880 0x7fb69c14a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:00.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb69b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb69c159b60 0x7fb69c14a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:00.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb69b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c14cc60 con 0x7fb69c06cc10 2024-09-11T00:44:00.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb69b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c156810 con 0x7fb69c14c880 2024-09-11T00:44:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb69b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb69c06edc0 con 0x7fb69c159b60 2024-09-11T00:44:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb699e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb69c06cc10 0x7fb69c158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb699e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb69c06cc10 0x7fb69c158b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46506/0 (socket says 172.21.15.31:46506) 2024-09-11T00:44:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb699e00640 1 -- 172.21.15.31:0/1189025608 learned_addr learned my addr 172.21.15.31:0/1189025608 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb69a800640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb69c159b60 0x7fb69c14a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.097+0000 7fb699400640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb69c14c880 0x7fb69c14a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:00.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.098+0000 7fb699e00640 1 -- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb69c159b60 msgr2=0x7fb69c14a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:00.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.098+0000 7fb699e00640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb69c159b60 0x7fb69c14a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:00.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.098+0000 7fb699e00640 1 -- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb69c14c880 msgr2=0x7fb69c14a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:00.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.098+0000 7fb699e00640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb69c14c880 0x7fb69c14a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:00.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.098+0000 7fb699e00640 1 -- 172.21.15.31:0/1189025608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69c0aced0 con 0x7fb69c06cc10 2024-09-11T00:44:00.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.098+0000 7fb699e00640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb69c06cc10 0x7fb69c158b90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00c490 tx=0x7fb68c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:00.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.099+0000 7fb692000640 1 -- 172.21.15.31:0/1189025608 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0052d0 con 0x7fb69c06cc10 2024-09-11T00:44:00.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.099+0000 7fb692000640 1 -- 172.21.15.31:0/1189025608 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb68c005490 con 0x7fb69c06cc10 2024-09-11T00:44:00.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.099+0000 7fb69b200640 1 -- 172.21.15.31:0/1189025608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb69c1eca90 con 0x7fb69c06cc10 2024-09-11T00:44:00.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.099+0000 7fb692000640 1 -- 172.21.15.31:0/1189025608 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c012890 con 0x7fb69c06cc10 2024-09-11T00:44:00.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.099+0000 7fb69b200640 1 -- 172.21.15.31:0/1189025608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb69c0ca8c0 con 0x7fb69c06cc10 2024-09-11T00:44:00.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.101+0000 7fb69b200640 1 -- 172.21.15.31:0/1189025608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb69c0c59a0 con 0x7fb69c06cc10 2024-09-11T00:44:00.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.102+0000 7fb692000640 1 -- 172.21.15.31:0/1189025608 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb68c012a30 con 0x7fb69c06cc10 2024-09-11T00:44:00.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.103+0000 7fb692000640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb67c077fa0 0x7fb67c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:00.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.103+0000 7fb692000640 1 -- 172.21.15.31:0/1189025608 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7fb68c09a120 con 0x7fb69c06cc10 2024-09-11T00:44:00.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.104+0000 7fb699400640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb67c077fa0 0x7fb67c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:00.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.104+0000 7fb699400640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb67c077fa0 0x7fb67c07a460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb688007f20 tx=0x7fb688007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:00.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.108+0000 7fb692000640 1 -- 172.21.15.31:0/1189025608 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb68c061bf0 con 0x7fb69c06cc10 2024-09-11T00:44:00.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.386+0000 7fb69b200640 1 -- 172.21.15.31:0/1189025608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb69c0c92b0 con 0x7fb69c06cc10 2024-09-11T00:44:00.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.387+0000 7fb692000640 1 -- 172.21.15.31:0/1189025608 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6539 (secure 0 0 0) 0x7fb68c066620 con 0x7fb69c06cc10 2024-09-11T00:44:00.389 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:00.390 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"btime":"2024-09-11T00:43:34:776882+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6402,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:44:00.390 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 97 2024-09-11T00:44:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.389+0000 7fb69b200640 1 -- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb67c077fa0 msgr2=0x7fb67c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.389+0000 7fb69b200640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb67c077fa0 0x7fb67c07a460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb688007f20 tx=0x7fb688007eb0 comp rx=0 tx=0).stop 2024-09-11T00:44:00.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.389+0000 7fb69b200640 1 -- 172.21.15.31:0/1189025608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb69c06cc10 msgr2=0x7fb69c158b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.389+0000 7fb69b200640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb69c06cc10 0x7fb69c158b90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00c490 tx=0x7fb68c00c950 comp rx=0 tx=0).stop 2024-09-11T00:44:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.390+0000 7fb69b200640 1 -- 172.21.15.31:0/1189025608 shutdown_connections 2024-09-11T00:44:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.390+0000 7fb69b200640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb67c077fa0 0x7fb67c07a460 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.390+0000 7fb69b200640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb69c159b60 0x7fb69c14a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.390+0000 7fb69b200640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb69c14c880 0x7fb69c14a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.390+0000 7fb69b200640 1 --2- 172.21.15.31:0/1189025608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb69c06cc10 0x7fb69c158b90 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.390+0000 7fb69b200640 1 -- 172.21.15.31:0/1189025608 >> 172.21.15.31:0/1189025608 conn(0x7fb69c0adb00 msgr2=0x7fb69c1e21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.390+0000 7fb69b200640 1 -- 172.21.15.31:0/1189025608 shutdown_connections 2024-09-11T00:44:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:00.390+0000 7fb69b200640 1 -- 172.21.15.31:0/1189025608 wait complete. 2024-09-11T00:44:00.410 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:44:02.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.590+0000 7f9ccd200640 1 -- 172.21.15.31:0/1204371716 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9cc8153b30 msgr2=0x7f9cc8070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.591+0000 7f9ccd200640 1 --2- 172.21.15.31:0/1204371716 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9cc8153b30 0x7f9cc8070840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009890 tx=0x7f9cb802a5d0 comp rx=0 tx=0).stop 2024-09-11T00:44:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.591+0000 7f9ccd200640 1 -- 172.21.15.31:0/1204371716 shutdown_connections 2024-09-11T00:44:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.591+0000 7f9ccd200640 1 --2- 172.21.15.31:0/1204371716 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9cc8153b30 0x7f9cc8070840 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.591+0000 7f9ccd200640 1 -- 172.21.15.31:0/1204371716 >> 172.21.15.31:0/1204371716 conn(0x7f9cc806dc00 msgr2=0x7f9cc806e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.591+0000 7f9ccd200640 1 -- 172.21.15.31:0/1204371716 shutdown_connections 2024-09-11T00:44:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.591+0000 7f9ccd200640 1 -- 172.21.15.31:0/1204371716 wait complete. 2024-09-11T00:44:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.593+0000 7f9ccd200640 1 Processor -- start 2024-09-11T00:44:02.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.593+0000 7f9ccd200640 1 -- start start 2024-09-11T00:44:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9ccd200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc806d440 0x7f9cc81e1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9ccd200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9cc8070360 0x7f9cc80a9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9ccd200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9cc8153b30 0x7f9cc80a98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9ccd200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc8070f00 con 0x7f9cc806d440 2024-09-11T00:44:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9ccd200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc8070d80 con 0x7f9cc8153b30 2024-09-11T00:44:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9ccd200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc8071080 con 0x7f9cc8070360 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9cc5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc806d440 0x7f9cc81e1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9cc5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc806d440 0x7f9cc81e1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46514/0 (socket says 172.21.15.31:46514) 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9cc5e00640 1 -- 172.21.15.31:0/3485368697 learned_addr learned my addr 172.21.15.31:0/3485368697 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9cc5400640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9cc8070360 0x7f9cc80a9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.594+0000 7f9cc6800640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9cc8153b30 0x7f9cc80a98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.595+0000 7f9cc5e00640 1 -- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9cc8070360 msgr2=0x7f9cc80a9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.595+0000 7f9cc5e00640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9cc8070360 0x7f9cc80a9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.595+0000 7f9cc5e00640 1 -- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9cc8153b30 msgr2=0x7f9cc80a98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.595+0000 7f9cc5e00640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9cc8153b30 0x7f9cc80a98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.595+0000 7f9cc5e00640 1 -- 172.21.15.31:0/3485368697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc8148d70 con 0x7f9cc806d440 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.595+0000 7f9cc5e00640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc806d440 0x7f9cc81e1e60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80099b0 tx=0x7f9cb8033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.595+0000 7f9cbe000640 1 -- 172.21.15.31:0/3485368697 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8031b50 con 0x7f9cc806d440 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.595+0000 7f9cbe000640 1 -- 172.21.15.31:0/3485368697 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9cb8031de0 con 0x7f9cc806d440 2024-09-11T00:44:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.595+0000 7f9cbe000640 1 -- 172.21.15.31:0/3485368697 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8005280 con 0x7f9cc806d440 2024-09-11T00:44:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.595+0000 7f9ccd200640 1 -- 172.21.15.31:0/3485368697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc80a8890 con 0x7f9cc806d440 2024-09-11T00:44:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.596+0000 7f9ccd200640 1 -- 172.21.15.31:0/3485368697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc80a8e50 con 0x7f9cc806d440 2024-09-11T00:44:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.597+0000 7f9ccd200640 1 -- 172.21.15.31:0/3485368697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc8070840 con 0x7f9cc806d440 2024-09-11T00:44:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.598+0000 7f9cbe000640 1 -- 172.21.15.31:0/3485368697 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9cb8005420 con 0x7f9cc806d440 2024-09-11T00:44:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.599+0000 7f9cbe000640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ca8077f70 0x7f9ca807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.599+0000 7f9cbe000640 1 -- 172.21.15.31:0/3485368697 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7f9cb80bb9d0 con 0x7f9cc806d440 2024-09-11T00:44:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.600+0000 7f9cc5400640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ca8077f70 0x7f9ca807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.600+0000 7f9cc5400640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ca8077f70 0x7f9ca807a430 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8149320 tx=0x7f9cb4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:02.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.604+0000 7f9cbe000640 1 -- 172.21.15.31:0/3485368697 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9cb803f030 con 0x7f9cc806d440 2024-09-11T00:44:02.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.889+0000 7f9ccd200640 1 -- 172.21.15.31:0/3485368697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cc80a9090 con 0x7f9cc806d440 2024-09-11T00:44:02.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.889+0000 7f9cbe000640 1 -- 172.21.15.31:0/3485368697 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6539 (secure 0 0 0) 0x7f9cb8083b70 con 0x7f9cc806d440 2024-09-11T00:44:02.891 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:02.891 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"btime":"2024-09-11T00:43:34:776882+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6402,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:44:02.892 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 97 2024-09-11T00:44:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 -- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ca8077f70 msgr2=0x7f9ca807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ca8077f70 0x7f9ca807a430 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8149320 tx=0x7f9cb4009040 comp rx=0 tx=0).stop 2024-09-11T00:44:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 -- 172.21.15.31:0/3485368697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc806d440 msgr2=0x7f9cc81e1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc806d440 0x7f9cc81e1e60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80099b0 tx=0x7f9cb8033ba0 comp rx=0 tx=0).stop 2024-09-11T00:44:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 -- 172.21.15.31:0/3485368697 shutdown_connections 2024-09-11T00:44:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9ca8077f70 0x7f9ca807a430 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9cc8153b30 0x7f9cc80a98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9cc8070360 0x7f9cc80a9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 --2- 172.21.15.31:0/3485368697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9cc806d440 0x7f9cc81e1e60 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 -- 172.21.15.31:0/3485368697 >> 172.21.15.31:0/3485368697 conn(0x7f9cc806dc00 msgr2=0x7f9cc806dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.892+0000 7f9ccd200640 1 -- 172.21.15.31:0/3485368697 shutdown_connections 2024-09-11T00:44:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:02.893+0000 7f9ccd200640 1 -- 172.21.15.31:0/3485368697 wait complete. 2024-09-11T00:44:02.914 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:44:04.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.090+0000 7f86d0e00640 1 -- 172.21.15.31:0/2485240761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86cc1539b0 msgr2=0x7f86cc1567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.090+0000 7f86d0e00640 1 --2- 172.21.15.31:0/2485240761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86cc1539b0 0x7f86cc1567f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0719b0 tx=0x7f86c402a910 comp rx=0 tx=0).stop 2024-09-11T00:44:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.090+0000 7f86d0e00640 1 -- 172.21.15.31:0/2485240761 shutdown_connections 2024-09-11T00:44:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.090+0000 7f86d0e00640 1 --2- 172.21.15.31:0/2485240761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86cc1539b0 0x7f86cc1567f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.090+0000 7f86d0e00640 1 --2- 172.21.15.31:0/2485240761 >> v2:172.21.15.204:3300/0 conn(0x7f86cc0c3280 0x7f86cc153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.090+0000 7f86d0e00640 1 --2- 172.21.15.31:0/2485240761 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc0c6bb0 0x7f86cc06e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.090+0000 7f86d0e00640 1 -- 172.21.15.31:0/2485240761 >> 172.21.15.31:0/2485240761 conn(0x7f86cc06caf0 msgr2=0x7f86cc06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.090+0000 7f86d0e00640 1 -- 172.21.15.31:0/2485240761 shutdown_connections 2024-09-11T00:44:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.090+0000 7f86d0e00640 1 -- 172.21.15.31:0/2485240761 wait complete. 2024-09-11T00:44:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.092+0000 7f86d0e00640 1 Processor -- start 2024-09-11T00:44:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.092+0000 7f86d0e00640 1 -- start start 2024-09-11T00:44:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86d0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86cc0c3280 0x7f86cc152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86d0e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc0c6bb0 0x7f86cc1e0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86d0e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc1539b0 0x7f86cc1e1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86d0e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86cc152d80 con 0x7f86cc0c3280 2024-09-11T00:44:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86d0e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f86cc06e5b0 con 0x7f86cc0c6bb0 2024-09-11T00:44:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86d0e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f86cc152f00 con 0x7f86cc1539b0 2024-09-11T00:44:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86cc0c3280 0x7f86cc152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86cb400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86cc0c3280 0x7f86cc152530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46528/0 (socket says 172.21.15.31:46528) 2024-09-11T00:44:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86cb400640 1 -- 172.21.15.31:0/3609530464 learned_addr learned my addr 172.21.15.31:0/3609530464 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86c3e00640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc0c6bb0 0x7f86cc1e0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:05.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.093+0000 7f86cbe00640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc1539b0 0x7f86cc1e1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86cb400640 1 -- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc1539b0 msgr2=0x7f86cc1e1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86cb400640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc1539b0 0x7f86cc1e1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86cb400640 1 -- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc0c6bb0 msgr2=0x7f86cc1e0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86cb400640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc0c6bb0 0x7f86cc1e0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86cb400640 1 -- 172.21.15.31:0/3609530464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86cc1ebb40 con 0x7f86cc0c3280 2024-09-11T00:44:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86cb400640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86cc0c3280 0x7f86cc152530 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00e9f0 tx=0x7f86bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86c3400640 1 -- 172.21.15.31:0/3609530464 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86bc00cd80 con 0x7f86cc0c3280 2024-09-11T00:44:05.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86c3400640 1 -- 172.21.15.31:0/3609530464 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86bc005020 con 0x7f86cc0c3280 2024-09-11T00:44:05.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86c3400640 1 -- 172.21.15.31:0/3609530464 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86bc005450 con 0x7f86cc0c3280 2024-09-11T00:44:05.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86d0e00640 1 -- 172.21.15.31:0/3609530464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86cc14c1a0 con 0x7f86cc0c3280 2024-09-11T00:44:05.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.094+0000 7f86d0e00640 1 -- 172.21.15.31:0/3609530464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86cc1ec0e0 con 0x7f86cc0c3280 2024-09-11T00:44:05.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.096+0000 7f86d0e00640 1 -- 172.21.15.31:0/3609530464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86cc0c70c0 con 0x7f86cc0c3280 2024-09-11T00:44:05.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.097+0000 7f86c3400640 1 -- 172.21.15.31:0/3609530464 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f86bc0047a0 con 0x7f86cc0c3280 2024-09-11T00:44:05.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.098+0000 7f86c3400640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f86a8077f70 0x7f86a807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:05.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.098+0000 7f86c3400640 1 -- 172.21.15.31:0/3609530464 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7f86bc09afe0 con 0x7f86cc0c3280 2024-09-11T00:44:05.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.098+0000 7f86c3e00640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f86a8077f70 0x7f86a807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:05.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.099+0000 7f86c3e00640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f86a8077f70 0x7f86a807a430 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0c6380 tx=0x7f86b4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:05.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.103+0000 7f86c3400640 1 -- 172.21.15.31:0/3609530464 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f86bc062b50 con 0x7f86cc0c3280 2024-09-11T00:44:05.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.427+0000 7f86d0e00640 1 -- 172.21.15.31:0/3609530464 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86cc0c4150 con 0x7f86cc0c3280 2024-09-11T00:44:05.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.427+0000 7f86c3400640 1 -- 172.21.15.31:0/3609530464 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6539 (secure 0 0 0) 0x7f86bc067580 con 0x7f86cc0c3280 2024-09-11T00:44:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:05.429 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"btime":"2024-09-11T00:43:34:776882+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6402,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:44:05.430 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 97 2024-09-11T00:44:05.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 -- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f86a8077f70 msgr2=0x7f86a807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:05.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f86a8077f70 0x7f86a807a430 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0c6380 tx=0x7f86b4005f90 comp rx=0 tx=0).stop 2024-09-11T00:44:05.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 -- 172.21.15.31:0/3609530464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86cc0c3280 msgr2=0x7f86cc152530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:05.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86cc0c3280 0x7f86cc152530 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00e9f0 tx=0x7f86bc00edf0 comp rx=0 tx=0).stop 2024-09-11T00:44:05.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 -- 172.21.15.31:0/3609530464 shutdown_connections 2024-09-11T00:44:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f86a8077f70 0x7f86a807a430 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f86cc1539b0 0x7f86cc1e1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86cc0c6bb0 0x7f86cc1e0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 --2- 172.21.15.31:0/3609530464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86cc0c3280 0x7f86cc152530 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 -- 172.21.15.31:0/3609530464 >> 172.21.15.31:0/3609530464 conn(0x7f86cc06caf0 msgr2=0x7f86cc0ab6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 -- 172.21.15.31:0/3609530464 shutdown_connections 2024-09-11T00:44:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:05.430+0000 7f86d0e00640 1 -- 172.21.15.31:0/3609530464 wait complete. 2024-09-11T00:44:05.453 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:44:07.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:07.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.628+0000 7f6f86400640 1 -- 172.21.15.31:0/2903256493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f68004800 msgr2=0x7f6f68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:07.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.628+0000 7f6f86400640 1 --2- 172.21.15.31:0/2903256493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f68004800 0x7f6f68004c80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c4a0 tx=0x7f6f68005830 comp rx=0 tx=0).stop 2024-09-11T00:44:07.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.628+0000 7f6f85a00640 1 -- 172.21.15.31:0/2903256493 reap_dead start 2024-09-11T00:44:07.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.628+0000 7f6f86400640 1 -- 172.21.15.31:0/2903256493 shutdown_connections 2024-09-11T00:44:07.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.628+0000 7f6f86400640 1 -- 172.21.15.31:0/2903256493 >> 172.21.15.31:0/2903256493 conn(0x7f6f8006caf0 msgr2=0x7f6f8006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:07.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.628+0000 7f6f86400640 1 -- 172.21.15.31:0/2903256493 shutdown_connections 2024-09-11T00:44:07.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.628+0000 7f6f86400640 1 -- 172.21.15.31:0/2903256493 wait complete. 2024-09-11T00:44:07.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.629+0000 7f6f86400640 1 Processor -- start 2024-09-11T00:44:07.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.630+0000 7f6f86400640 1 -- start start 2024-09-11T00:44:07.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.630+0000 7f6f86400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f68004800 0x7f6f800722e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.630+0000 7f6f86400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f801ef8e0 0x7f6f8014e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.630+0000 7f6f86400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f800aca50 0x7f6f8014ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.630+0000 7f6f86400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6f800c2e80 con 0x7f6f68004800 2024-09-11T00:44:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.630+0000 7f6f86400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6f800c2ba0 con 0x7f6f801ef8e0 2024-09-11T00:44:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.630+0000 7f6f86400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6f80153a20 con 0x7f6f800aca50 2024-09-11T00:44:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f85000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f68004800 0x7f6f800722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f85000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f68004800 0x7f6f800722e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46532/0 (socket says 172.21.15.31:46532) 2024-09-11T00:44:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f7fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f801ef8e0 0x7f6f8014e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f85000640 1 -- 172.21.15.31:0/541238923 learned_addr learned my addr 172.21.15.31:0/541238923 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:07.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f85a00640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f800aca50 0x7f6f8014ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:07.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f85000640 1 -- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f800aca50 msgr2=0x7f6f8014ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:07.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f85000640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f800aca50 0x7f6f8014ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f85000640 1 -- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f801ef8e0 msgr2=0x7f6f8014e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f85000640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f801ef8e0 0x7f6f8014e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f85000640 1 -- 172.21.15.31:0/541238923 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8006f350 con 0x7f6f68004800 2024-09-11T00:44:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.631+0000 7f6f7fe00640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f801ef8e0 0x7f6f8014e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:44:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.632+0000 7f6f85000640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f68004800 0x7f6f800722e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6f74011a80 tx=0x7f6f74011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.632+0000 7f6f7cc00640 1 -- 172.21.15.31:0/541238923 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7401f070 con 0x7f6f68004800 2024-09-11T00:44:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.632+0000 7f6f7cc00640 1 -- 172.21.15.31:0/541238923 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f74012450 con 0x7f6f68004800 2024-09-11T00:44:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.632+0000 7f6f86400640 1 -- 172.21.15.31:0/541238923 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f801efda0 con 0x7f6f68004800 2024-09-11T00:44:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.632+0000 7f6f7cc00640 1 -- 172.21.15.31:0/541238923 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f74012840 con 0x7f6f68004800 2024-09-11T00:44:07.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.632+0000 7f6f86400640 1 -- 172.21.15.31:0/541238923 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f800abcd0 con 0x7f6f68004800 2024-09-11T00:44:07.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.634+0000 7f6f86400640 1 -- 172.21.15.31:0/541238923 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f800c5e80 con 0x7f6f68004800 2024-09-11T00:44:07.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.634+0000 7f6f7cc00640 1 -- 172.21.15.31:0/541238923 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6f740129e0 con 0x7f6f68004800 2024-09-11T00:44:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.635+0000 7f6f7cc00640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f60077f70 0x7f6f6007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.636+0000 7f6f7cc00640 1 -- 172.21.15.31:0/541238923 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7f6f7409b6a0 con 0x7f6f68004800 2024-09-11T00:44:07.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.636+0000 7f6f7fe00640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f60077f70 0x7f6f6007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:07.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.637+0000 7f6f7fe00640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f60077f70 0x7f6f6007a430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c470 tx=0x7f6f6c038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:07.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.641+0000 7f6f7cc00640 1 -- 172.21.15.31:0/541238923 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6f74063160 con 0x7f6f68004800 2024-09-11T00:44:07.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.918+0000 7f6f86400640 1 -- 172.21.15.31:0/541238923 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f801511f0 con 0x7f6f68004800 2024-09-11T00:44:07.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.919+0000 7f6f7cc00640 1 -- 172.21.15.31:0/541238923 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6539 (secure 0 0 0) 0x7f6f74067b90 con 0x7f6f68004800 2024-09-11T00:44:07.921 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:07.921 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"btime":"2024-09-11T00:43:34:776882+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6402,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:44:07.921 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 97 2024-09-11T00:44:07.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.921+0000 7f6f86400640 1 -- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f60077f70 msgr2=0x7f6f6007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:07.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.921+0000 7f6f86400640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f60077f70 0x7f6f6007a430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c470 tx=0x7f6f6c038040 comp rx=0 tx=0).stop 2024-09-11T00:44:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.921+0000 7f6f86400640 1 -- 172.21.15.31:0/541238923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f68004800 msgr2=0x7f6f800722e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.921+0000 7f6f86400640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f68004800 0x7f6f800722e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6f74011a80 tx=0x7f6f74011f40 comp rx=0 tx=0).stop 2024-09-11T00:44:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.922+0000 7f6f86400640 1 -- 172.21.15.31:0/541238923 shutdown_connections 2024-09-11T00:44:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.922+0000 7f6f86400640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6f60077f70 0x7f6f6007a430 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.922+0000 7f6f86400640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6f800aca50 0x7f6f8014ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.922+0000 7f6f86400640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6f801ef8e0 0x7f6f8014e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.922+0000 7f6f86400640 1 --2- 172.21.15.31:0/541238923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f68004800 0x7f6f800722e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.922+0000 7f6f86400640 1 -- 172.21.15.31:0/541238923 >> 172.21.15.31:0/541238923 conn(0x7f6f8006caf0 msgr2=0x7f6f800c0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.922+0000 7f6f86400640 1 -- 172.21.15.31:0/541238923 shutdown_connections 2024-09-11T00:44:07.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:07.922+0000 7f6f86400640 1 -- 172.21.15.31:0/541238923 wait complete. 2024-09-11T00:44:07.939 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:44:09.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:10.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.123+0000 7f83a1c00640 1 -- 172.21.15.31:0/1453131991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f839c0c71c0 msgr2=0x7f839c0c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:10.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.123+0000 7f83a1c00640 1 --2- 172.21.15.31:0/1453131991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f839c0c71c0 0x7f839c0c95e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8394005fd0 tx=0x7f839402a820 comp rx=0 tx=0).stop 2024-09-11T00:44:10.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.123+0000 7f83a1c00640 1 -- 172.21.15.31:0/1453131991 shutdown_connections 2024-09-11T00:44:10.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.123+0000 7f83a1c00640 1 --2- 172.21.15.31:0/1453131991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f839c0c71c0 0x7f839c0c95e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:10.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.123+0000 7f83a1c00640 1 --2- 172.21.15.31:0/1453131991 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f839c158260 0x7f839c0c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:10.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.123+0000 7f83a1c00640 1 -- 172.21.15.31:0/1453131991 >> 172.21.15.31:0/1453131991 conn(0x7f839c0688e0 msgr2=0x7f839c068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:10.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.123+0000 7f83a1c00640 1 -- 172.21.15.31:0/1453131991 shutdown_connections 2024-09-11T00:44:10.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.123+0000 7f83a1c00640 1 -- 172.21.15.31:0/1453131991 wait complete. 2024-09-11T00:44:10.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.125+0000 7f83a1c00640 1 Processor -- start 2024-09-11T00:44:10.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.125+0000 7f83a1c00640 1 -- start start 2024-09-11T00:44:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f83a1c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f839c0c5750 0x7f839c1ef610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f83a1c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f839c0c71c0 0x7f839c153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f83a1c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c158260 0x7f839c1544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f83a1c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f839c0c44c0 con 0x7f839c0c5750 2024-09-11T00:44:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f83a1c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f839c0c9b20 con 0x7f839c158260 2024-09-11T00:44:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f83a1c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f839c158640 con 0x7f839c0c71c0 2024-09-11T00:44:10.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f839c0c5750 0x7f839c1ef610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f839c0c5750 0x7f839c1ef610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38742/0 (socket says 172.21.15.31:38742) 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839aa00640 1 -- 172.21.15.31:0/1445236346 learned_addr learned my addr 172.21.15.31:0/1445236346 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839a000640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f839c0c71c0 0x7f839c153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839b400640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c158260 0x7f839c1544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839aa00640 1 -- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f839c0c71c0 msgr2=0x7f839c153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839aa00640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f839c0c71c0 0x7f839c153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839aa00640 1 -- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c158260 msgr2=0x7f839c1544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839aa00640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c158260 0x7f839c1544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839aa00640 1 -- 172.21.15.31:0/1445236346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c154c90 con 0x7f839c0c5750 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.127+0000 7f839b400640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c158260 0x7f839c1544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.128+0000 7f839aa00640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f839c0c5750 0x7f839c1ef610 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f838c00e7c0 tx=0x7f838c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.128+0000 7f8392a00640 1 -- 172.21.15.31:0/1445236346 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c00ccc0 con 0x7f839c0c5750 2024-09-11T00:44:10.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.128+0000 7f8392a00640 1 -- 172.21.15.31:0/1445236346 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f838c010430 con 0x7f839c0c5750 2024-09-11T00:44:10.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.128+0000 7f83a1c00640 1 -- 172.21.15.31:0/1445236346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839c0a9810 con 0x7f839c0c5750 2024-09-11T00:44:10.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.128+0000 7f8392a00640 1 -- 172.21.15.31:0/1445236346 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c010860 con 0x7f839c0c5750 2024-09-11T00:44:10.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.128+0000 7f83a1c00640 1 -- 172.21.15.31:0/1445236346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839c1efb50 con 0x7f839c0c5750 2024-09-11T00:44:10.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.129+0000 7f83a1c00640 1 -- 172.21.15.31:0/1445236346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f839c0c5b90 con 0x7f839c0c5750 2024-09-11T00:44:10.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.131+0000 7f8392a00640 1 -- 172.21.15.31:0/1445236346 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f838c010a00 con 0x7f839c0c5750 2024-09-11T00:44:10.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.132+0000 7f8392a00640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f837c077fa0 0x7f837c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:10.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.132+0000 7f8392a00640 1 -- 172.21.15.31:0/1445236346 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7f838c09aff0 con 0x7f839c0c5750 2024-09-11T00:44:10.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.133+0000 7f839a000640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f837c077fa0 0x7f837c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:10.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.133+0000 7f839a000640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f837c077fa0 0x7f837c07a460 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8388007fd0 tx=0x7f8388009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.138+0000 7f8392a00640 1 -- 172.21.15.31:0/1445236346 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f838c062af0 con 0x7f839c0c5750 2024-09-11T00:44:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.394+0000 7f83a1c00640 1 -- 172.21.15.31:0/1445236346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f839c0a99e0 con 0x7f839c0c5750 2024-09-11T00:44:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.395+0000 7f8392a00640 1 -- 172.21.15.31:0/1445236346 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6539 (secure 0 0 0) 0x7f838c067520 con 0x7f839c0c5750 2024-09-11T00:44:10.397 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:10.397 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"btime":"2024-09-11T00:43:34:776882+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6402,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:44:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 97 2024-09-11T00:44:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 -- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f837c077fa0 msgr2=0x7f837c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f837c077fa0 0x7f837c07a460 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8388007fd0 tx=0x7f8388009040 comp rx=0 tx=0).stop 2024-09-11T00:44:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 -- 172.21.15.31:0/1445236346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f839c0c5750 msgr2=0x7f839c1ef610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f839c0c5750 0x7f839c1ef610 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f838c00e7c0 tx=0x7f838c00ec80 comp rx=0 tx=0).stop 2024-09-11T00:44:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 -- 172.21.15.31:0/1445236346 shutdown_connections 2024-09-11T00:44:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f837c077fa0 0x7f837c07a460 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f839c158260 0x7f839c1544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f839c0c71c0 0x7f839c153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 --2- 172.21.15.31:0/1445236346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f839c0c5750 0x7f839c1ef610 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 -- 172.21.15.31:0/1445236346 >> 172.21.15.31:0/1445236346 conn(0x7f839c0688e0 msgr2=0x7f839c1e0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 -- 172.21.15.31:0/1445236346 shutdown_connections 2024-09-11T00:44:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:10.397+0000 7f83a1c00640 1 -- 172.21.15.31:0/1445236346 wait complete. 2024-09-11T00:44:10.417 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:44:12.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:12.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.606+0000 7f18eca00640 1 -- 172.21.15.31:0/1275075102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18e80a6e20 msgr2=0x7f18e81510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:12.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.606+0000 7f18eca00640 1 --2- 172.21.15.31:0/1275075102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18e80a6e20 0x7f18e81510c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f18d4005fd0 tx=0x7f18d402a4b0 comp rx=0 tx=0).stop 2024-09-11T00:44:12.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.606+0000 7f18eca00640 1 -- 172.21.15.31:0/1275075102 shutdown_connections 2024-09-11T00:44:12.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.606+0000 7f18eca00640 1 --2- 172.21.15.31:0/1275075102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18e80a6e20 0x7f18e81510c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:12.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.606+0000 7f18eca00640 1 --2- 172.21.15.31:0/1275075102 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18e80ac2c0 0x7f18e80ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:12.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.606+0000 7f18eca00640 1 -- 172.21.15.31:0/1275075102 >> 172.21.15.31:0/1275075102 conn(0x7f18e806caf0 msgr2=0x7f18e806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:12.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.606+0000 7f18eca00640 1 -- 172.21.15.31:0/1275075102 shutdown_connections 2024-09-11T00:44:12.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.606+0000 7f18eca00640 1 -- 172.21.15.31:0/1275075102 wait complete. 2024-09-11T00:44:12.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.608+0000 7f18eca00640 1 Processor -- start 2024-09-11T00:44:12.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.608+0000 7f18eca00640 1 -- start start 2024-09-11T00:44:12.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.609+0000 7f18eca00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18e80a6e20 0x7f18e81e1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:12.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.609+0000 7f18eca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18e80ac2c0 0x7f18e814c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:12.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.609+0000 7f18eca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18e80acbe0 0x7f18e8073470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:12.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.609+0000 7f18eca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f18e8151780 con 0x7f18e80ac2c0 2024-09-11T00:44:12.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.609+0000 7f18eca00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18e8151600 con 0x7f18e80acbe0 2024-09-11T00:44:12.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.609+0000 7f18eca00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f18e8151900 con 0x7f18e80a6e20 2024-09-11T00:44:12.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.609+0000 7f18e6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18e80ac2c0 0x7f18e814c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:12.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.609+0000 7f18e6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18e80ac2c0 0x7f18e814c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38766/0 (socket says 172.21.15.31:38766) 2024-09-11T00:44:12.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.609+0000 7f18e6a00640 1 -- 172.21.15.31:0/3203075468 learned_addr learned my addr 172.21.15.31:0/3203075468 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:12.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.609+0000 7f18e7400640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18e80a6e20 0x7f18e81e1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:12.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.610+0000 7f18e7e00640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18e80acbe0 0x7f18e8073470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:12.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.610+0000 7f18e6a00640 1 -- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18e80a6e20 msgr2=0x7f18e81e1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:12.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.610+0000 7f18e6a00640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18e80a6e20 0x7f18e81e1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:12.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.610+0000 7f18e6a00640 1 -- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18e80acbe0 msgr2=0x7f18e8073470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:12.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.610+0000 7f18e6a00640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18e80acbe0 0x7f18e8073470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:12.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.610+0000 7f18e6a00640 1 -- 172.21.15.31:0/3203075468 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e8074f50 con 0x7f18e80ac2c0 2024-09-11T00:44:12.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.610+0000 7f18e6a00640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18e80ac2c0 0x7f18e814c3c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f18d40099f0 tx=0x7f18d4003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:12.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.611+0000 7f18df400640 1 -- 172.21.15.31:0/3203075468 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d403f070 con 0x7f18e80ac2c0 2024-09-11T00:44:12.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.611+0000 7f18df400640 1 -- 172.21.15.31:0/3203075468 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18d40047a0 con 0x7f18e80ac2c0 2024-09-11T00:44:12.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.611+0000 7f18eca00640 1 -- 172.21.15.31:0/3203075468 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e814b8b0 con 0x7f18e80ac2c0 2024-09-11T00:44:12.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.611+0000 7f18df400640 1 -- 172.21.15.31:0/3203075468 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d4035030 con 0x7f18e80ac2c0 2024-09-11T00:44:12.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.611+0000 7f18eca00640 1 -- 172.21.15.31:0/3203075468 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e8155de0 con 0x7f18e80ac2c0 2024-09-11T00:44:12.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.612+0000 7f18eca00640 1 -- 172.21.15.31:0/3203075468 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18e806a290 con 0x7f18e80ac2c0 2024-09-11T00:44:12.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.613+0000 7f18df400640 1 -- 172.21.15.31:0/3203075468 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f18d4004c10 con 0x7f18e80ac2c0 2024-09-11T00:44:12.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.614+0000 7f18df400640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18c8077fa0 0x7f18c807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:12.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.615+0000 7f18df400640 1 -- 172.21.15.31:0/3203075468 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8128+0+0 (secure 0 0 0) 0x7f18d40bbbe0 con 0x7f18e80ac2c0 2024-09-11T00:44:12.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.615+0000 7f18e7400640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18c8077fa0 0x7f18c807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:12.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.616+0000 7f18e7400640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18c8077fa0 0x7f18c807a460 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f18e814f080 tx=0x7f18d800b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:12.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.620+0000 7f18df400640 1 -- 172.21.15.31:0/3203075468 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f18e806a290 con 0x7f18e80ac2c0 2024-09-11T00:44:12.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.912+0000 7f18eca00640 1 -- 172.21.15.31:0/3203075468 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18e81510c0 con 0x7f18e80ac2c0 2024-09-11T00:44:12.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.912+0000 7f18df400640 1 -- 172.21.15.31:0/3203075468 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6539 (secure 0 0 0) 0x7f18d4083620 con 0x7f18e80ac2c0 2024-09-11T00:44:12.914 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:12.914 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":97,"btime":"2024-09-11T00:43:34:776882+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6402,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:43:25.749830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5594},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5594":{"gid":5594,"name":"d","rank":0,"incarnation":92,"state":"up:active","state_seq":25,"addr":"172.21.15.204:6835/1575320768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1575320768},{"type":"v1","addr":"172.21.15.204:6835","nonce":1575320768}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5594,"qdb_cluster":[5594]},"id":2}]} 2024-09-11T00:44:12.915 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 97 2024-09-11T00:44:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 -- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18c8077fa0 msgr2=0x7f18c807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18c8077fa0 0x7f18c807a460 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f18e814f080 tx=0x7f18d800b390 comp rx=0 tx=0).stop 2024-09-11T00:44:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 -- 172.21.15.31:0/3203075468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18e80ac2c0 msgr2=0x7f18e814c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18e80ac2c0 0x7f18e814c3c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f18d40099f0 tx=0x7f18d4003910 comp rx=0 tx=0).stop 2024-09-11T00:44:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 -- 172.21.15.31:0/3203075468 shutdown_connections 2024-09-11T00:44:12.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f18c8077fa0 0x7f18c807a460 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18e80acbe0 0x7f18e8073470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18e80ac2c0 0x7f18e814c3c0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 --2- 172.21.15.31:0/3203075468 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18e80a6e20 0x7f18e81e1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 -- 172.21.15.31:0/3203075468 >> 172.21.15.31:0/3203075468 conn(0x7f18e806caf0 msgr2=0x7f18e8153300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 -- 172.21.15.31:0/3203075468 shutdown_connections 2024-09-11T00:44:12.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:12.915+0000 7f18eca00640 1 -- 172.21.15.31:0/3203075468 wait complete. 2024-09-11T00:44:12.936 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:44:14.937 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:15.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.112+0000 7fb40fe00640 1 -- 172.21.15.31:0/574002861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4080c2d20 msgr2=0x7fb4080c37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:15.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.112+0000 7fb40fe00640 1 --2- 172.21.15.31:0/574002861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4080c2d20 0x7fb4080c37f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb400005fd0 tx=0x7fb40002a8b0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.112+0000 7fb40fe00640 1 -- 172.21.15.31:0/574002861 shutdown_connections 2024-09-11T00:44:15.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.112+0000 7fb40fe00640 1 --2- 172.21.15.31:0/574002861 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4080c2d20 0x7fb4080c37f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.112+0000 7fb40fe00640 1 --2- 172.21.15.31:0/574002861 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4080abde0 0x7fb40806db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.112+0000 7fb40fe00640 1 -- 172.21.15.31:0/574002861 >> 172.21.15.31:0/574002861 conn(0x7fb4080c8320 msgr2=0x7fb4080c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:15.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.112+0000 7fb40fe00640 1 -- 172.21.15.31:0/574002861 shutdown_connections 2024-09-11T00:44:15.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.112+0000 7fb40fe00640 1 -- 172.21.15.31:0/574002861 wait complete. 2024-09-11T00:44:15.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.114+0000 7fb40fe00640 1 Processor -- start 2024-09-11T00:44:15.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.114+0000 7fb40fe00640 1 -- start start 2024-09-11T00:44:15.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.115+0000 7fb40fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb408074630 0x7fb40814c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:15.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.115+0000 7fb40fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4080abde0 0x7fb40813ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:15.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.115+0000 7fb40fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4080c2d20 0x7fb40813f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:15.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.115+0000 7fb40fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb408153b70 con 0x7fb4080abde0 2024-09-11T00:44:15.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.115+0000 7fb40fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb408152d60 con 0x7fb4080c2d20 2024-09-11T00:44:15.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb408141060 con 0x7fb408074630 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4080abde0 0x7fb40813ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40d400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb408074630 0x7fb40814c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40d400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb408074630 0x7fb40814c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:58908/0 (socket says 172.21.15.31:58908) 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4080abde0 0x7fb40813ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38780/0 (socket says 172.21.15.31:38780) 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40d400640 1 -- 172.21.15.31:0/366624105 learned_addr learned my addr 172.21.15.31:0/366624105 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40de00640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4080c2d20 0x7fb40813f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40d400640 1 -- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4080c2d20 msgr2=0x7fb40813f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40d400640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4080c2d20 0x7fb40813f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40d400640 1 -- 172.21.15.31:0/366624105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4080abde0 msgr2=0x7fb40813ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40d400640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4080abde0 0x7fb40813ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.116+0000 7fb40d400640 1 -- 172.21.15.31:0/366624105 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40813f840 con 0x7fb408074630 2024-09-11T00:44:15.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.117+0000 7fb40d400640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb408074630 0x7fb40814c300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c430 tx=0x7fb3fc00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:15.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.117+0000 7fb405600640 1 -- 172.21.15.31:0/366624105 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc017070 con 0x7fb408074630 2024-09-11T00:44:15.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.117+0000 7fb405600640 1 -- 172.21.15.31:0/366624105 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3fc0053e0 con 0x7fb408074630 2024-09-11T00:44:15.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.117+0000 7fb405600640 1 -- 172.21.15.31:0/366624105 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc003850 con 0x7fb408074630 2024-09-11T00:44:15.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.117+0000 7fb40fe00640 1 -- 172.21.15.31:0/366624105 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4081eadf0 con 0x7fb408074630 2024-09-11T00:44:15.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.117+0000 7fb40fe00640 1 -- 172.21.15.31:0/366624105 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb40813fae0 con 0x7fb408074630 2024-09-11T00:44:15.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.119+0000 7fb40fe00640 1 -- 172.21.15.31:0/366624105 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb40813fec0 con 0x7fb408074630 2024-09-11T00:44:15.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.120+0000 7fb405600640 1 -- 172.21.15.31:0/366624105 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb3fc0039f0 con 0x7fb408074630 2024-09-11T00:44:15.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.121+0000 7fb405600640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3e807c380 0x7fb3e807e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:15.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.121+0000 7fb405600640 1 -- 172.21.15.31:0/366624105 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7fb3fc099d20 con 0x7fb408074630 2024-09-11T00:44:15.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.122+0000 7fb40ca00640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3e807c380 0x7fb3e807e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:15.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.122+0000 7fb40ca00640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3e807c380 0x7fb3e807e840 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb4080c2620 tx=0x7fb3f4009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:15.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.126+0000 7fb405600640 1 -- 172.21.15.31:0/366624105 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb40813fec0 con 0x7fb408074630 2024-09-11T00:44:15.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.417+0000 7fb40fe00640 1 -- 172.21.15.31:0/366624105 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb40806e2e0 con 0x7fb408074630 2024-09-11T00:44:15.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.418+0000 7fb405600640 1 -- 172.21.15.31:0/366624105 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5707 (secure 0 0 0) 0x7fb3fc0617e0 con 0x7fb408074630 2024-09-11T00:44:15.420 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:15.421 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":99,"btime":"2024-09-11T00:44:15:410415+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:14.759910+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:44:15.421 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 99 2024-09-11T00:44:15.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 -- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3e807c380 msgr2=0x7fb3e807e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:15.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3e807c380 0x7fb3e807e840 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb4080c2620 tx=0x7fb3f4009260 comp rx=0 tx=0).stop 2024-09-11T00:44:15.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 -- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb408074630 msgr2=0x7fb40814c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:15.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb408074630 0x7fb40814c300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c430 tx=0x7fb3fc00c8f0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 -- 172.21.15.31:0/366624105 shutdown_connections 2024-09-11T00:44:15.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb3e807c380 0x7fb3e807e840 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4080c2d20 0x7fb40813f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4080abde0 0x7fb40813ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 --2- 172.21.15.31:0/366624105 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb408074630 0x7fb40814c300 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 -- 172.21.15.31:0/366624105 >> 172.21.15.31:0/366624105 conn(0x7fb4080c8320 msgr2=0x7fb40806caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:15.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 -- 172.21.15.31:0/366624105 shutdown_connections 2024-09-11T00:44:15.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.421+0000 7fb40fe00640 1 -- 172.21.15.31:0/366624105 wait complete. 2024-09-11T00:44:15.441 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-11T00:44:15.441 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:44:15.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.629+0000 7f02a5a00640 1 -- 172.21.15.31:0/2811569782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a00c5aa0 msgr2=0x7f02a00c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:15.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.629+0000 7f02a5a00640 1 --2- 172.21.15.31:0/2811569782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a00c5aa0 0x7f02a00c7150 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f02a0072000 tx=0x7f029002a5f0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.629+0000 7f02a5a00640 1 -- 172.21.15.31:0/2811569782 shutdown_connections 2024-09-11T00:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.629+0000 7f02a5a00640 1 --2- 172.21.15.31:0/2811569782 >> v2:172.21.15.204:3300/0 conn(0x7f02a0156350 0x7f02a01567c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.629+0000 7f02a5a00640 1 --2- 172.21.15.31:0/2811569782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a00c5aa0 0x7f02a00c7150 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.629+0000 7f02a5a00640 1 -- 172.21.15.31:0/2811569782 >> 172.21.15.31:0/2811569782 conn(0x7f02a006caf0 msgr2=0x7f02a006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.629+0000 7f02a5a00640 1 -- 172.21.15.31:0/2811569782 shutdown_connections 2024-09-11T00:44:15.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.629+0000 7f02a5a00640 1 -- 172.21.15.31:0/2811569782 wait complete. 2024-09-11T00:44:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.631+0000 7f02a5a00640 1 Processor -- start 2024-09-11T00:44:15.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.631+0000 7f02a5a00640 1 -- start start 2024-09-11T00:44:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f02a5a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02a00c5aa0 0x7f02a01e0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:15.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f02a5a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02a00c6420 0x7f02a01ef190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f02a5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a0156350 0x7f02a01ef6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f02a5a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f02a00c2e80 con 0x7f02a0156350 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f02a5a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f02a00c2ba0 con 0x7f02a00c6420 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f02a5a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f02a0153a20 con 0x7f02a00c5aa0 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f02a5000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a0156350 0x7f02a01ef6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f02a5000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a0156350 0x7f02a01ef6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38792/0 (socket says 172.21.15.31:38792) 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f02a5000640 1 -- 172.21.15.31:0/472451745 learned_addr learned my addr 172.21.15.31:0/472451745 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f029fe00640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02a00c5aa0 0x7f02a01e0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.632+0000 7f029f400640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02a00c6420 0x7f02a01ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.633+0000 7f02a5000640 1 -- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02a00c5aa0 msgr2=0x7f02a01e0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.633+0000 7f02a5000640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02a00c5aa0 0x7f02a01e0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.633+0000 7f02a5000640 1 -- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02a00c6420 msgr2=0x7f02a01ef190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:15.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.633+0000 7f02a5000640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02a00c6420 0x7f02a01ef190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.633+0000 7f02a5000640 1 -- 172.21.15.31:0/472451745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a00ac4e0 con 0x7f02a0156350 2024-09-11T00:44:15.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.633+0000 7f02a5000640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a0156350 0x7f02a01ef6d0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f029800ea80 tx=0x7f029800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.633+0000 7f0297e00640 1 -- 172.21.15.31:0/472451745 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298004b20 con 0x7f02a0156350 2024-09-11T00:44:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.633+0000 7f0297e00640 1 -- 172.21.15.31:0/472451745 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0298004ce0 con 0x7f02a0156350 2024-09-11T00:44:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.633+0000 7f0297e00640 1 -- 172.21.15.31:0/472451745 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298005280 con 0x7f02a0156350 2024-09-11T00:44:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.633+0000 7f02a5a00640 1 -- 172.21.15.31:0/472451745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a00accb0 con 0x7f02a0156350 2024-09-11T00:44:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.634+0000 7f02a5a00640 1 -- 172.21.15.31:0/472451745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a01eff80 con 0x7f02a0156350 2024-09-11T00:44:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.635+0000 7f02a5a00640 1 -- 172.21.15.31:0/472451745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02a00acaa0 con 0x7f02a0156350 2024-09-11T00:44:15.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.636+0000 7f0297e00640 1 -- 172.21.15.31:0/472451745 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0298005420 con 0x7f02a0156350 2024-09-11T00:44:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.638+0000 7f0297e00640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0280077fa0 0x7f028007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.638+0000 7f0297e00640 1 -- 172.21.15.31:0/472451745 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7f029809b240 con 0x7f02a0156350 2024-09-11T00:44:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.638+0000 7f029fe00640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0280077fa0 0x7f028007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.639+0000 7f029fe00640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0280077fa0 0x7f028007a460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0290033000 tx=0x7f0290004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:15.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.642+0000 7f0297e00640 1 -- 172.21.15.31:0/472451745 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0298062d00 con 0x7f02a0156350 2024-09-11T00:44:15.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.891+0000 7f02a5a00640 1 -- 172.21.15.31:0/472451745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02a00c5e80 con 0x7f02a0156350 2024-09-11T00:44:15.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.891+0000 7f0297e00640 1 -- 172.21.15.31:0/472451745 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5707 (secure 0 0 0) 0x7f0298067730 con 0x7f02a0156350 2024-09-11T00:44:15.893 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:15.894 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":99,"btime":"2024-09-11T00:44:15:410415+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:14.759910+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:44:15.894 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 99 2024-09-11T00:44:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 -- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0280077fa0 msgr2=0x7f028007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0280077fa0 0x7f028007a460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0290033000 tx=0x7f0290004730 comp rx=0 tx=0).stop 2024-09-11T00:44:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 -- 172.21.15.31:0/472451745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a0156350 msgr2=0x7f02a01ef6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a0156350 0x7f02a01ef6d0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f029800ea80 tx=0x7f029800ef40 comp rx=0 tx=0).stop 2024-09-11T00:44:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 -- 172.21.15.31:0/472451745 shutdown_connections 2024-09-11T00:44:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0280077fa0 0x7f028007a460 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02a0156350 0x7f02a01ef6d0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02a00c6420 0x7f02a01ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 --2- 172.21.15.31:0/472451745 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f02a00c5aa0 0x7f02a01e0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:15.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 -- 172.21.15.31:0/472451745 >> 172.21.15.31:0/472451745 conn(0x7f02a006caf0 msgr2=0x7f02a00c6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:15.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 -- 172.21.15.31:0/472451745 shutdown_connections 2024-09-11T00:44:15.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:15.894+0000 7f02a5a00640 1 -- 172.21.15.31:0/472451745 wait complete. 2024-09-11T00:44:15.915 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:44:15:410415+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 99, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5773": { "addr": "172.21.15.204:6837/3513641573", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 3513641573, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 3513641573, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5773, "incarnation": 88, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 77 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:43:22.726458+0000", "qdb_cluster": [ 5773 ], "qdb_leader": 5773, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5773 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6402": { "addr": "172.21.15.31:6835/2323279595", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 2323279595, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 2323279595, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6402, "incarnation": 98, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:44:14.759910+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6402 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:44:16.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:44:16.097 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:44:17.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.099+0000 7f82c2e00640 1 -- 172.21.15.31:0/3519670364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc0c6730 msgr2=0x7f82bc1596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.099+0000 7f82c2e00640 1 --2- 172.21.15.31:0/3519670364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc0c6730 0x7f82bc1596d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009830 tx=0x7f82b002a470 comp rx=0 tx=0).stop 2024-09-11T00:44:18.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.099+0000 7f82c2e00640 1 -- 172.21.15.31:0/3519670364 shutdown_connections 2024-09-11T00:44:18.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.099+0000 7f82c2e00640 1 --2- 172.21.15.31:0/3519670364 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc0bf240 0x7f82bc0b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:18.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.099+0000 7f82c2e00640 1 --2- 172.21.15.31:0/3519670364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc0c6730 0x7f82bc1596d0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:18.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.099+0000 7f82c2e00640 1 -- 172.21.15.31:0/3519670364 >> 172.21.15.31:0/3519670364 conn(0x7f82bc068ac0 msgr2=0x7f82bc068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:18.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.099+0000 7f82c2e00640 1 -- 172.21.15.31:0/3519670364 shutdown_connections 2024-09-11T00:44:18.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.099+0000 7f82c2e00640 1 -- 172.21.15.31:0/3519670364 wait complete. 2024-09-11T00:44:18.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.101+0000 7f82c2e00640 1 Processor -- start 2024-09-11T00:44:18.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.101+0000 7f82c2e00640 1 -- start start 2024-09-11T00:44:18.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.103+0000 7f82c2e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc0bf240 0x7f82bc0adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:18.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.103+0000 7f82c2e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc0c6730 0x7f82bc0c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:18.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.103+0000 7f82c2e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82bc156e40 0x7f82bc159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:18.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.103+0000 7f82c2e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc0c6c90 con 0x7f82bc0c6730 2024-09-11T00:44:18.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.103+0000 7f82c2e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc0c6b10 con 0x7f82bc156e40 2024-09-11T00:44:18.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.103+0000 7f82c2e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f82bc06e850 con 0x7f82bc0bf240 2024-09-11T00:44:18.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.104+0000 7f82c1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc0c6730 0x7f82bc0c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:18.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.104+0000 7f82c1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc0c6730 0x7f82bc0c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57772/0 (socket says 172.21.15.31:57772) 2024-09-11T00:44:18.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.104+0000 7f82c1000640 1 -- 172.21.15.31:0/2836492446 learned_addr learned my addr 172.21.15.31:0/2836492446 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:18.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.104+0000 7f82c1a00640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc0bf240 0x7f82bc0adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:18.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.104+0000 7f82c2400640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82bc156e40 0x7f82bc159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:18.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.104+0000 7f82c1000640 1 -- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc0bf240 msgr2=0x7f82bc0adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:18.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.104+0000 7f82c1000640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc0bf240 0x7f82bc0adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:18.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.104+0000 7f82c1000640 1 -- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82bc156e40 msgr2=0x7f82bc159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:18.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.104+0000 7f82c1000640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82bc156e40 0x7f82bc159cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:18.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.104+0000 7f82c1000640 1 -- 172.21.15.31:0/2836492446 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82bc0bf710 con 0x7f82bc0c6730 2024-09-11T00:44:18.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.105+0000 7f82c1000640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc0c6730 0x7f82bc0c5340 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f82a800e9f0 tx=0x7f82a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:18.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.105+0000 7f82b9600640 1 -- 172.21.15.31:0/2836492446 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a800cd80 con 0x7f82bc0c6730 2024-09-11T00:44:18.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.105+0000 7f82b9600640 1 -- 172.21.15.31:0/2836492446 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82a8010430 con 0x7f82bc0c6730 2024-09-11T00:44:18.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.105+0000 7f82b9600640 1 -- 172.21.15.31:0/2836492446 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a8010840 con 0x7f82bc0c6730 2024-09-11T00:44:18.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.105+0000 7f82c2e00640 1 -- 172.21.15.31:0/2836492446 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82bc0bf9f0 con 0x7f82bc0c6730 2024-09-11T00:44:18.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.105+0000 7f82c2e00640 1 -- 172.21.15.31:0/2836492446 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82bc0c1df0 con 0x7f82bc0c6730 2024-09-11T00:44:18.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.106+0000 7f82c2e00640 1 -- 172.21.15.31:0/2836492446 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82bc0c9a40 con 0x7f82bc0c6730 2024-09-11T00:44:18.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.109+0000 7f82b9600640 1 -- 172.21.15.31:0/2836492446 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f82a80109e0 con 0x7f82bc0c6730 2024-09-11T00:44:18.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.109+0000 7f82b9600640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c077f70 0x7f829c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:18.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.110+0000 7f82b9600640 1 -- 172.21.15.31:0/2836492446 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7f82a809b330 con 0x7f82bc0c6730 2024-09-11T00:44:18.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.110+0000 7f82c1a00640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c077f70 0x7f829c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:18.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.111+0000 7f82c1a00640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c077f70 0x7f829c07a430 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f82b0034000 tx=0x7f82b0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:18.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.116+0000 7f82b9600640 1 -- 172.21.15.31:0/2836492446 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f82a8062d60 con 0x7f82bc0c6730 2024-09-11T00:44:18.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.376+0000 7f82c2e00640 1 -- 172.21.15.31:0/2836492446 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82bc153cb0 con 0x7f82bc0c6730 2024-09-11T00:44:18.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.377+0000 7f82b9600640 1 -- 172.21.15.31:0/2836492446 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+5713 (secure 0 0 0) 0x7f82a8067790 con 0x7f82bc0c6730 2024-09-11T00:44:18.379 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:18.379 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":101,"btime":"2024-09-11T00:44:17:423172+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:17.423168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":2}]} 2024-09-11T00:44:18.379 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 101 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.379+0000 7f82c2e00640 1 -- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c077f70 msgr2=0x7f829c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.379+0000 7f82c2e00640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c077f70 0x7f829c07a430 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f82b0034000 tx=0x7f82b0004730 comp rx=0 tx=0).stop 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.379+0000 7f82c2e00640 1 -- 172.21.15.31:0/2836492446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc0c6730 msgr2=0x7f82bc0c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.379+0000 7f82c2e00640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc0c6730 0x7f82bc0c5340 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f82a800e9f0 tx=0x7f82a800eeb0 comp rx=0 tx=0).stop 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.379+0000 7f82c2e00640 1 -- 172.21.15.31:0/2836492446 shutdown_connections 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.380+0000 7f82c2e00640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f829c077f70 0x7f829c07a430 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.380+0000 7f82c2e00640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82bc156e40 0x7f82bc159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.380+0000 7f82c2e00640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82bc0c6730 0x7f82bc0c5340 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.380+0000 7f82c2e00640 1 --2- 172.21.15.31:0/2836492446 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82bc0bf240 0x7f82bc0adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.380+0000 7f82c2e00640 1 -- 172.21.15.31:0/2836492446 >> 172.21.15.31:0/2836492446 conn(0x7f82bc068ac0 msgr2=0x7f82bc0af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.380+0000 7f82c2e00640 1 -- 172.21.15.31:0/2836492446 shutdown_connections 2024-09-11T00:44:18.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:18.380+0000 7f82c2e00640 1 -- 172.21.15.31:0/2836492446 wait complete. 2024-09-11T00:44:18.400 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5594 2024-09-11T00:44:18.400 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.d 2024-09-11T00:44:20.788 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-11T00:44:20.786+0000 7f1496600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:44:26.402 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-11T00:44:26.402 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:44:26.402 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:44:26.406 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:44:26.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:26.554 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:44:26.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.578+0000 7fd59c800640 1 -- 172.21.15.31:0/2160763468 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5940b1230 msgr2=0x7fd5941596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:26.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.578+0000 7fd59c800640 1 --2- 172.21.15.31:0/2160763468 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5940b1230 0x7fd5941596b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd588009860 tx=0x7fd58802a3d0 comp rx=0 tx=0).stop 2024-09-11T00:44:26.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.578+0000 7fd59c800640 1 -- 172.21.15.31:0/2160763468 shutdown_connections 2024-09-11T00:44:26.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.578+0000 7fd59c800640 1 --2- 172.21.15.31:0/2160763468 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5940b1230 0x7fd5941596b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:26.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.578+0000 7fd59c800640 1 -- 172.21.15.31:0/2160763468 >> 172.21.15.31:0/2160763468 conn(0x7fd594068f10 msgr2=0x7fd5940cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:26.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.578+0000 7fd59c800640 1 -- 172.21.15.31:0/2160763468 shutdown_connections 2024-09-11T00:44:26.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.578+0000 7fd59c800640 1 -- 172.21.15.31:0/2160763468 wait complete. 2024-09-11T00:44:26.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.580+0000 7fd59c800640 1 Processor -- start 2024-09-11T00:44:26.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.580+0000 7fd59c800640 1 -- start start 2024-09-11T00:44:26.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd59c800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5940b1230 0x7fd5941df720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:26.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd59c800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5940c9b00 0x7fd594158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:26.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd59c800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5940c9fe0 0x7fd5940c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:26.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd59c800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd594156540 con 0x7fd5940b1230 2024-09-11T00:44:26.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd59c800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd594156260 con 0x7fd5940c9b00 2024-09-11T00:44:26.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd59c800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd5940cca20 con 0x7fd5940c9fe0 2024-09-11T00:44:26.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd599e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5940b1230 0x7fd5941df720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:26.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd599e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5940b1230 0x7fd5941df720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57778/0 (socket says 172.21.15.31:57778) 2024-09-11T00:44:26.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd599e00640 1 -- 172.21.15.31:0/2488034616 learned_addr learned my addr 172.21.15.31:0/2488034616 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:26.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd599400640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5940c9b00 0x7fd594158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:26.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.581+0000 7fd59a800640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5940c9fe0 0x7fd5940c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd599e00640 1 -- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5940c9fe0 msgr2=0x7fd5940c5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd599e00640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5940c9fe0 0x7fd5940c5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd599e00640 1 -- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5940c9b00 msgr2=0x7fd594158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd599e00640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5940c9b00 0x7fd594158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd599e00640 1 -- 172.21.15.31:0/2488034616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5940c2e00 con 0x7fd5940b1230 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd599400640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5940c9b00 0x7fd594158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd599e00640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5940b1230 0x7fd5941df720 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fd58802a9c0 tx=0x7fd588032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd58e000640 1 -- 172.21.15.31:0/2488034616 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588040070 con 0x7fd5940b1230 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd58e000640 1 -- 172.21.15.31:0/2488034616 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd588005310 con 0x7fd5940b1230 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd58e000640 1 -- 172.21.15.31:0/2488034616 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5880047a0 con 0x7fd5940b1230 2024-09-11T00:44:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.582+0000 7fd59c800640 1 -- 172.21.15.31:0/2488034616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5940c49a0 con 0x7fd5940b1230 2024-09-11T00:44:26.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.583+0000 7fd59c800640 1 -- 172.21.15.31:0/2488034616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5940c4f60 con 0x7fd5940b1230 2024-09-11T00:44:26.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.584+0000 7fd59c800640 1 -- 172.21.15.31:0/2488034616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5940766d0 con 0x7fd5940b1230 2024-09-11T00:44:26.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.585+0000 7fd58e000640 1 -- 172.21.15.31:0/2488034616 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd588004b50 con 0x7fd5940b1230 2024-09-11T00:44:26.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.586+0000 7fd58e000640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd578077fa0 0x7fd57807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:26.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.587+0000 7fd58e000640 1 -- 172.21.15.31:0/2488034616 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7fd5880bbb80 con 0x7fd5940b1230 2024-09-11T00:44:26.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.587+0000 7fd599400640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd578077fa0 0x7fd57807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:26.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.588+0000 7fd599400640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd578077fa0 0x7fd57807a460 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd584007fd0 tx=0x7fd584009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:26.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.593+0000 7fd58e000640 1 -- 172.21.15.31:0/2488034616 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd588083640 con 0x7fd5940b1230 2024-09-11T00:44:26.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.864+0000 7fd59c800640 1 -- 172.21.15.31:0/2488034616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5940bf390 con 0x7fd5940b1230 2024-09-11T00:44:26.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.864+0000 7fd58e000640 1 -- 172.21.15.31:0/2488034616 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6542 (secure 0 0 0) 0x7fd588088070 con 0x7fd5940b1230 2024-09-11T00:44:26.866 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:26.866 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":102,"btime":"2024-09-11T00:44:26:658257+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:17.423168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":2}]} 2024-09-11T00:44:26.867 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 102 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 -- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd578077fa0 msgr2=0x7fd57807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd578077fa0 0x7fd57807a460 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd584007fd0 tx=0x7fd584009040 comp rx=0 tx=0).stop 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 -- 172.21.15.31:0/2488034616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5940b1230 msgr2=0x7fd5941df720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5940b1230 0x7fd5941df720 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fd58802a9c0 tx=0x7fd588032c10 comp rx=0 tx=0).stop 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 -- 172.21.15.31:0/2488034616 shutdown_connections 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd578077fa0 0x7fd57807a460 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5940c9fe0 0x7fd5940c5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5940c9b00 0x7fd594158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 --2- 172.21.15.31:0/2488034616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5940b1230 0x7fd5941df720 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 -- 172.21.15.31:0/2488034616 >> 172.21.15.31:0/2488034616 conn(0x7fd594068f10 msgr2=0x7fd5941569d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 -- 172.21.15.31:0/2488034616 shutdown_connections 2024-09-11T00:44:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:26.867+0000 7fd59c800640 1 -- 172.21.15.31:0/2488034616 wait complete. 2024-09-11T00:44:26.885 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-11T00:44:26.885 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before thrashing 2024-09-11T00:44:30.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:31.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.069+0000 7f7058c00640 1 -- 172.21.15.31:0/2466284141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70540c6420 msgr2=0x7f7054153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:31.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.069+0000 7f7058c00640 1 --2- 172.21.15.31:0/2466284141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70540c6420 0x7f7054153310 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7054072000 tx=0x7f704002a560 comp rx=0 tx=0).stop 2024-09-11T00:44:31.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.069+0000 7f7058c00640 1 -- 172.21.15.31:0/2466284141 shutdown_connections 2024-09-11T00:44:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.069+0000 7f7058c00640 1 --2- 172.21.15.31:0/2466284141 >> v2:172.21.15.204:3300/0 conn(0x7f7054153850 0x7f7054156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.069+0000 7f7058c00640 1 --2- 172.21.15.31:0/2466284141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70540c6420 0x7f7054153310 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.069+0000 7f7058c00640 1 -- 172.21.15.31:0/2466284141 >> 172.21.15.31:0/2466284141 conn(0x7f705406caf0 msgr2=0x7f705406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.069+0000 7f7058c00640 1 -- 172.21.15.31:0/2466284141 shutdown_connections 2024-09-11T00:44:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.069+0000 7f7058c00640 1 -- 172.21.15.31:0/2466284141 wait complete. 2024-09-11T00:44:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.071+0000 7f7058c00640 1 Processor -- start 2024-09-11T00:44:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.071+0000 7f7058c00640 1 -- start start 2024-09-11T00:44:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7058c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70540c5aa0 0x7f70541dff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7058c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70540c6420 0x7f705414e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7058c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7054153850 0x7f70541ef660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7058c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f70540c5e80 con 0x7f70540c5aa0 2024-09-11T00:44:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7058c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f70540c2e80 con 0x7f7054153850 2024-09-11T00:44:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7058c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f70540c6000 con 0x7f70540c6420 2024-09-11T00:44:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7053400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70540c5aa0 0x7f70541dff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7053400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70540c5aa0 0x7f70541dff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45672/0 (socket says 172.21.15.31:45672) 2024-09-11T00:44:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7053400640 1 -- 172.21.15.31:0/1397423292 learned_addr learned my addr 172.21.15.31:0/1397423292 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7052a00640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70540c6420 0x7f705414e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.072+0000 7f7053e00640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7054153850 0x7f70541ef660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.073+0000 7f7053400640 1 -- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70540c6420 msgr2=0x7f705414e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.073+0000 7f7053400640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70540c6420 0x7f705414e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.073+0000 7f7053400640 1 -- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7054153850 msgr2=0x7f70541ef660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.073+0000 7f7053400640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7054153850 0x7f70541ef660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.073+0000 7f7053400640 1 -- 172.21.15.31:0/1397423292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70541e1600 con 0x7f70540c5aa0 2024-09-11T00:44:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.073+0000 7f7053400640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70540c5aa0 0x7f70541dff30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f704400c430 tx=0x7f704400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.073+0000 7f704b400640 1 -- 172.21.15.31:0/1397423292 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044011790 con 0x7f70540c5aa0 2024-09-11T00:44:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.073+0000 7f704b400640 1 -- 172.21.15.31:0/1397423292 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7044017070 con 0x7f70540c5aa0 2024-09-11T00:44:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.073+0000 7f7058c00640 1 -- 172.21.15.31:0/1397423292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70541eba50 con 0x7f70540c5aa0 2024-09-11T00:44:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.073+0000 7f704b400640 1 -- 172.21.15.31:0/1397423292 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044011d00 con 0x7f70540c5aa0 2024-09-11T00:44:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.074+0000 7f7058c00640 1 -- 172.21.15.31:0/1397423292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7054155240 con 0x7f70540c5aa0 2024-09-11T00:44:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.075+0000 7f7058c00640 1 -- 172.21.15.31:0/1397423292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7054153310 con 0x7f70540c5aa0 2024-09-11T00:44:31.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.076+0000 7f704b400640 1 -- 172.21.15.31:0/1397423292 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f704401c460 con 0x7f70540c5aa0 2024-09-11T00:44:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.077+0000 7f704b400640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7034077f70 0x7f703407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.078+0000 7f704b400640 1 -- 172.21.15.31:0/1397423292 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7f704409a5e0 con 0x7f70540c5aa0 2024-09-11T00:44:31.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.078+0000 7f7052a00640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7034077f70 0x7f703407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:31.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.079+0000 7f7052a00640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7034077f70 0x7f703407a430 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7040033000 tx=0x7f7040004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:31.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.083+0000 7f704b400640 1 -- 172.21.15.31:0/1397423292 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7044061f90 con 0x7f70540c5aa0 2024-09-11T00:44:31.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.347+0000 7f7058c00640 1 -- 172.21.15.31:0/1397423292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70540aa0e0 con 0x7f70540c5aa0 2024-09-11T00:44:31.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.347+0000 7f704b400640 1 -- 172.21.15.31:0/1397423292 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6542 (secure 0 0 0) 0x7f70440669c0 con 0x7f70540c5aa0 2024-09-11T00:44:31.349 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:31.349 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":102,"btime":"2024-09-11T00:44:26:658257+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:17.423168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":2}]} 2024-09-11T00:44:31.350 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 102 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 -- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7034077f70 msgr2=0x7f703407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7034077f70 0x7f703407a430 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7040033000 tx=0x7f7040004730 comp rx=0 tx=0).stop 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 -- 172.21.15.31:0/1397423292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70540c5aa0 msgr2=0x7f70541dff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70540c5aa0 0x7f70541dff30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f704400c430 tx=0x7f704400c8f0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 -- 172.21.15.31:0/1397423292 shutdown_connections 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7034077f70 0x7f703407a430 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7054153850 0x7f70541ef660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f70540c6420 0x7f705414e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 --2- 172.21.15.31:0/1397423292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70540c5aa0 0x7f70541dff30 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 -- 172.21.15.31:0/1397423292 >> 172.21.15.31:0/1397423292 conn(0x7f705406caf0 msgr2=0x7f70540a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:31.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.350+0000 7f7058c00640 1 -- 172.21.15.31:0/1397423292 shutdown_connections 2024-09-11T00:44:31.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.351+0000 7f7058c00640 1 -- 172.21.15.31:0/1397423292 wait complete. 2024-09-11T00:44:31.370 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-11T00:44:31.371 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-11T00:44:31.371 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:44:31.390 INFO:tasks.ceph.mds.c:Stopped 2024-09-11T00:44:31.390 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.616+0000 7fb2dd000640 1 -- 172.21.15.31:0/754975334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2d80bf2c0 msgr2=0x7fb2d815a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.616+0000 7fb2dd000640 1 --2- 172.21.15.31:0/754975334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2d80bf2c0 0x7fb2d815a1a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0005fd0 tx=0x7fb2d002a8b0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.616+0000 7fb2dd000640 1 -- 172.21.15.31:0/754975334 shutdown_connections 2024-09-11T00:44:31.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.616+0000 7fb2dd000640 1 --2- 172.21.15.31:0/754975334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2d80bf2c0 0x7fb2d815a1a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.616+0000 7fb2dd000640 1 -- 172.21.15.31:0/754975334 >> 172.21.15.31:0/754975334 conn(0x7fb2d80cc260 msgr2=0x7fb2d80ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:31.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.616+0000 7fb2dd000640 1 -- 172.21.15.31:0/754975334 shutdown_connections 2024-09-11T00:44:31.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.616+0000 7fb2dd000640 1 -- 172.21.15.31:0/754975334 wait complete. 2024-09-11T00:44:31.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.619+0000 7fb2dd000640 1 Processor -- start 2024-09-11T00:44:31.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.619+0000 7fb2dd000640 1 -- start start 2024-09-11T00:44:31.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.620+0000 7fb2dd000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2d80b1160 0x7fb2d80aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:31.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2dd000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2d80bf2c0 0x7fb2d81e1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:31.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2dd000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2d80c9b30 0x7fb2d8149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:31.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2dd000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d80b1540 con 0x7fb2d80c9b30 2024-09-11T00:44:31.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2dd000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d806e5c0 con 0x7fb2d80b1160 2024-09-11T00:44:31.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2dd000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb2d80ae540 con 0x7fb2d80bf2c0 2024-09-11T00:44:31.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2d5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2d80b1160 0x7fb2d80aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:31.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2d5400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2d80bf2c0 0x7fb2d81e1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:31.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2d5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2d80b1160 0x7fb2d80aee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:52304/0 (socket says 172.21.15.31:52304) 2024-09-11T00:44:31.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2d5e00640 1 -- 172.21.15.31:0/1446338057 learned_addr learned my addr 172.21.15.31:0/1446338057 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:31.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2d6800640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2d80c9b30 0x7fb2d8149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:31.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.621+0000 7fb2d5e00640 1 -- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2d80bf2c0 msgr2=0x7fb2d81e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.622+0000 7fb2d5e00640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2d80bf2c0 0x7fb2d81e1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.622+0000 7fb2d5e00640 1 -- 172.21.15.31:0/1446338057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2d80c9b30 msgr2=0x7fb2d8149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.622+0000 7fb2d5e00640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2d80c9b30 0x7fb2d8149f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.622+0000 7fb2d5e00640 1 -- 172.21.15.31:0/1446338057 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d80ac290 con 0x7fb2d80b1160 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.622+0000 7fb2d6800640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2d80c9b30 0x7fb2d8149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.622+0000 7fb2d5e00640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2d80b1160 0x7fb2d80aee90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800c3e0 tx=0x7fb2c800c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.622+0000 7fb2ce000640 1 -- 172.21.15.31:0/1446338057 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8011790 con 0x7fb2d80b1160 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.623+0000 7fb2ce000640 1 -- 172.21.15.31:0/1446338057 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2c8017070 con 0x7fb2d80b1160 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.623+0000 7fb2dd000640 1 -- 172.21.15.31:0/1446338057 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d80af820 con 0x7fb2d80b1160 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.623+0000 7fb2ce000640 1 -- 172.21.15.31:0/1446338057 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8011c80 con 0x7fb2d80b1160 2024-09-11T00:44:31.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.623+0000 7fb2dd000640 1 -- 172.21.15.31:0/1446338057 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d80ad5a0 con 0x7fb2d80b1160 2024-09-11T00:44:31.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.624+0000 7fb2dd000640 1 -- 172.21.15.31:0/1446338057 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2d806f300 con 0x7fb2d80b1160 2024-09-11T00:44:31.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.625+0000 7fb2ce000640 1 -- 172.21.15.31:0/1446338057 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb2c801c460 con 0x7fb2d80b1160 2024-09-11T00:44:31.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.627+0000 7fb2ce000640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2b8077f70 0x7fb2b807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:31.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.627+0000 7fb2ce000640 1 -- 172.21.15.31:0/1446338057 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7fb2c809a450 con 0x7fb2d80b1160 2024-09-11T00:44:31.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.628+0000 7fb2d5400640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2b8077f70 0x7fb2b807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:31.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.628+0000 7fb2d5400640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2b8077f70 0x7fb2b807a430 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4007fd0 tx=0x7fb2c4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:31.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.631+0000 7fb2ce000640 1 -- 172.21.15.31:0/1446338057 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb2c8011e20 con 0x7fb2d80b1160 2024-09-11T00:44:31.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.910+0000 7fb2dd000640 1 -- 172.21.15.31:0/1446338057 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2d80b0920 con 0x7fb2d80b1160 2024-09-11T00:44:31.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.910+0000 7fb2ce000640 1 -- 172.21.15.31:0/1446338057 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6542 (secure 0 0 0) 0x7fb2c8062560 con 0x7fb2d80b1160 2024-09-11T00:44:31.912 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:31.912 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":102,"btime":"2024-09-11T00:44:26:658257+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:17.423168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":2}]} 2024-09-11T00:44:31.913 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 102 2024-09-11T00:44:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 -- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2b8077f70 msgr2=0x7fb2b807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2b8077f70 0x7fb2b807a430 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4007fd0 tx=0x7fb2c4009040 comp rx=0 tx=0).stop 2024-09-11T00:44:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 -- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2d80b1160 msgr2=0x7fb2d80aee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2d80b1160 0x7fb2d80aee90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800c3e0 tx=0x7fb2c800c8a0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 -- 172.21.15.31:0/1446338057 shutdown_connections 2024-09-11T00:44:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2b8077f70 0x7fb2b807a430 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2d80c9b30 0x7fb2d8149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb2d80bf2c0 0x7fb2d81e1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 --2- 172.21.15.31:0/1446338057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2d80b1160 0x7fb2d80aee90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:31.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 -- 172.21.15.31:0/1446338057 >> 172.21.15.31:0/1446338057 conn(0x7fb2d80cc260 msgr2=0x7fb2d8158770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:31.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 -- 172.21.15.31:0/1446338057 shutdown_connections 2024-09-11T00:44:31.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:31.913+0000 7fb2dd000640 1 -- 172.21.15.31:0/1446338057 wait complete. 2024-09-11T00:44:31.935 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:44:33.936 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:34.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.106+0000 7f4257200640 1 -- 172.21.15.31:0/3918812009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42580acbe0 msgr2=0x7f42580c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:34.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.106+0000 7f4257200640 1 --2- 172.21.15.31:0/3918812009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42580acbe0 0x7f42580c61b0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f4250005fd0 tx=0x7f425002a980 comp rx=0 tx=0).stop 2024-09-11T00:44:34.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.106+0000 7f4257200640 1 -- 172.21.15.31:0/3918812009 shutdown_connections 2024-09-11T00:44:34.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.106+0000 7f4257200640 1 --2- 172.21.15.31:0/3918812009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42580acbe0 0x7f42580c61b0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:34.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.106+0000 7f4257200640 1 --2- 172.21.15.31:0/3918812009 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f42580ac2c0 0x7f42580ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:34.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.106+0000 7f4257200640 1 -- 172.21.15.31:0/3918812009 >> 172.21.15.31:0/3918812009 conn(0x7f425806caf0 msgr2=0x7f425806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:34.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.106+0000 7f4257200640 1 -- 172.21.15.31:0/3918812009 shutdown_connections 2024-09-11T00:44:34.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.106+0000 7f4257200640 1 -- 172.21.15.31:0/3918812009 wait complete. 2024-09-11T00:44:34.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.108+0000 7f4257200640 1 Processor -- start 2024-09-11T00:44:34.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.108+0000 7f4257200640 1 -- start start 2024-09-11T00:44:34.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.109+0000 7f4257200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f42580a6e20 0x7f42581ec390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:34.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.109+0000 7f4257200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42580ac2c0 0x7f42580c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:34.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.109+0000 7f4257200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42580acbe0 0x7f42580748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:34.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.109+0000 7f4257200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4258151950 con 0x7f42580ac2c0 2024-09-11T00:44:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.109+0000 7f4257200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f42581517d0 con 0x7f42580acbe0 2024-09-11T00:44:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.109+0000 7f4257200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f42580c66f0 con 0x7f42580a6e20 2024-09-11T00:44:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.109+0000 7f4255400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42580ac2c0 0x7f42580c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.109+0000 7f4255400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42580ac2c0 0x7f42580c0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45704/0 (socket says 172.21.15.31:45704) 2024-09-11T00:44:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.109+0000 7f4255400640 1 -- 172.21.15.31:0/4095312210 learned_addr learned my addr 172.21.15.31:0/4095312210 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.110+0000 7f4256800640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42580acbe0 0x7f42580748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.110+0000 7f4255e00640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f42580a6e20 0x7f42581ec390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:34.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.110+0000 7f4255400640 1 -- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f42580a6e20 msgr2=0x7f42581ec390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:34.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.110+0000 7f4255400640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f42580a6e20 0x7f42581ec390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:34.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.110+0000 7f4255400640 1 -- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42580acbe0 msgr2=0x7f42580748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.110+0000 7f4255400640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42580acbe0 0x7f42580748f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.110+0000 7f4255400640 1 -- 172.21.15.31:0/4095312210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42581e95a0 con 0x7f42580ac2c0 2024-09-11T00:44:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.110+0000 7f4255400640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42580ac2c0 0x7f42580c0d60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f424400c460 tx=0x7f424400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.110+0000 7f424e000640 1 -- 172.21.15.31:0/4095312210 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244017070 con 0x7f42580ac2c0 2024-09-11T00:44:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.111+0000 7f424e000640 1 -- 172.21.15.31:0/4095312210 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42440035f0 con 0x7f42580ac2c0 2024-09-11T00:44:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.111+0000 7f4257200640 1 -- 172.21.15.31:0/4095312210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f425806f2d0 con 0x7f42580ac2c0 2024-09-11T00:44:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.111+0000 7f424e000640 1 -- 172.21.15.31:0/4095312210 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244003a40 con 0x7f42580ac2c0 2024-09-11T00:44:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.111+0000 7f4257200640 1 -- 172.21.15.31:0/4095312210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f425806f850 con 0x7f42580ac2c0 2024-09-11T00:44:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.112+0000 7f4257200640 1 -- 172.21.15.31:0/4095312210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42580c6300 con 0x7f42580ac2c0 2024-09-11T00:44:34.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.113+0000 7f424e000640 1 -- 172.21.15.31:0/4095312210 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4244003be0 con 0x7f42580ac2c0 2024-09-11T00:44:34.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.114+0000 7f424e000640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4238077f70 0x7f423807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:34.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.114+0000 7f424e000640 1 -- 172.21.15.31:0/4095312210 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7f4244099de0 con 0x7f42580ac2c0 2024-09-11T00:44:34.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.115+0000 7f4255e00640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4238077f70 0x7f423807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:34.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.116+0000 7f4255e00640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4238077f70 0x7f423807a430 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4248009bd0 tx=0x7f4248005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:34.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.119+0000 7f424e000640 1 -- 172.21.15.31:0/4095312210 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f42580c6300 con 0x7f42580ac2c0 2024-09-11T00:44:34.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.361+0000 7f4257200640 1 -- 172.21.15.31:0/4095312210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f425814d560 con 0x7f42580ac2c0 2024-09-11T00:44:34.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.362+0000 7f424e000640 1 -- 172.21.15.31:0/4095312210 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6542 (secure 0 0 0) 0x7f42581e95a0 con 0x7f42580ac2c0 2024-09-11T00:44:34.364 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:34.364 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":102,"btime":"2024-09-11T00:44:26:658257+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:17.423168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":2}]} 2024-09-11T00:44:34.364 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 102 2024-09-11T00:44:34.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 -- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4238077f70 msgr2=0x7f423807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:34.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4238077f70 0x7f423807a430 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4248009bd0 tx=0x7f4248005f90 comp rx=0 tx=0).stop 2024-09-11T00:44:34.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 -- 172.21.15.31:0/4095312210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42580ac2c0 msgr2=0x7f42580c0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:34.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42580ac2c0 0x7f42580c0d60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f424400c460 tx=0x7f424400c920 comp rx=0 tx=0).stop 2024-09-11T00:44:34.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 -- 172.21.15.31:0/4095312210 shutdown_connections 2024-09-11T00:44:34.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4238077f70 0x7f423807a430 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:34.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42580acbe0 0x7f42580748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:34.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42580ac2c0 0x7f42580c0d60 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:34.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 --2- 172.21.15.31:0/4095312210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f42580a6e20 0x7f42581ec390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:34.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 -- 172.21.15.31:0/4095312210 >> 172.21.15.31:0/4095312210 conn(0x7f425806caf0 msgr2=0x7f425806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:34.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 -- 172.21.15.31:0/4095312210 shutdown_connections 2024-09-11T00:44:34.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:34.364+0000 7f4257200640 1 -- 172.21.15.31:0/4095312210 wait complete. 2024-09-11T00:44:34.383 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:44:36.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:36.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.570+0000 7ff42e800640 1 -- 172.21.15.31:0/1637033446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff428072480 msgr2=0x7ff428074a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:36.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.570+0000 7ff42e800640 1 --2- 172.21.15.31:0/1637033446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff428072480 0x7ff428074a90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff418009900 tx=0x7ff41802a580 comp rx=0 tx=0).stop 2024-09-11T00:44:36.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.570+0000 7ff42e800640 1 -- 172.21.15.31:0/1637033446 shutdown_connections 2024-09-11T00:44:36.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.570+0000 7ff42e800640 1 --2- 172.21.15.31:0/1637033446 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff4281511c0 0x7ff428151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:36.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.570+0000 7ff42e800640 1 --2- 172.21.15.31:0/1637033446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff428072480 0x7ff428074a90 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:36.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.570+0000 7ff42e800640 1 -- 172.21.15.31:0/1637033446 >> 172.21.15.31:0/1637033446 conn(0x7ff4280a9750 msgr2=0x7ff4280a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:36.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.570+0000 7ff42e800640 1 -- 172.21.15.31:0/1637033446 shutdown_connections 2024-09-11T00:44:36.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.570+0000 7ff42e800640 1 -- 172.21.15.31:0/1637033446 wait complete. 2024-09-11T00:44:36.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.573+0000 7ff42e800640 1 Processor -- start 2024-09-11T00:44:36.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.573+0000 7ff42e800640 1 -- start start 2024-09-11T00:44:36.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.574+0000 7ff42e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff428072480 0x7ff42806f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:36.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.574+0000 7ff42e800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff4281511c0 0x7ff42806ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:36.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.574+0000 7ff42e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff428154b60 0x7ff4280c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:36.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.574+0000 7ff42e800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff4280c23a0 con 0x7ff428154b60 2024-09-11T00:44:36.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.574+0000 7ff42e800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff428154640 con 0x7ff428072480 2024-09-11T00:44:36.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.574+0000 7ff42e800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff4280a7140 con 0x7ff4281511c0 2024-09-11T00:44:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.574+0000 7ff427e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff428154b60 0x7ff4280c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.574+0000 7ff427e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff428154b60 0x7ff4280c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45728/0 (socket says 172.21.15.31:45728) 2024-09-11T00:44:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.574+0000 7ff427e00640 1 -- 172.21.15.31:0/2356255532 learned_addr learned my addr 172.21.15.31:0/2356255532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.575+0000 7ff426a00640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff4281511c0 0x7ff42806ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.575+0000 7ff427e00640 1 -- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff4281511c0 msgr2=0x7ff42806ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.575+0000 7ff427e00640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff4281511c0 0x7ff42806ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.575+0000 7ff427e00640 1 -- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff428072480 msgr2=0x7ff42806f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.575+0000 7ff427e00640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff428072480 0x7ff42806f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.575+0000 7ff427e00640 1 -- 172.21.15.31:0/2356255532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4280c62c0 con 0x7ff428154b60 2024-09-11T00:44:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.576+0000 7ff427e00640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff428154b60 0x7ff4280c67c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff42000cad0 tx=0x7ff42000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.576+0000 7ff41f400640 1 -- 172.21.15.31:0/2356255532 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42000adc0 con 0x7ff428154b60 2024-09-11T00:44:36.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.576+0000 7ff41f400640 1 -- 172.21.15.31:0/2356255532 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff420004b20 con 0x7ff428154b60 2024-09-11T00:44:36.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.576+0000 7ff41f400640 1 -- 172.21.15.31:0/2356255532 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420005020 con 0x7ff428154b60 2024-09-11T00:44:36.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.576+0000 7ff42e800640 1 -- 172.21.15.31:0/2356255532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4280c5920 con 0x7ff428154b60 2024-09-11T00:44:36.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.576+0000 7ff42e800640 1 -- 172.21.15.31:0/2356255532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff42806fe30 con 0x7ff428154b60 2024-09-11T00:44:36.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.578+0000 7ff42e800640 1 -- 172.21.15.31:0/2356255532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff428072860 con 0x7ff428154b60 2024-09-11T00:44:36.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.580+0000 7ff41f400640 1 -- 172.21.15.31:0/2356255532 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff4200051c0 con 0x7ff428154b60 2024-09-11T00:44:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.581+0000 7ff41f400640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff408077f70 0x7ff40807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.581+0000 7ff41f400640 1 -- 172.21.15.31:0/2356255532 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7ff42009aec0 con 0x7ff428154b60 2024-09-11T00:44:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.582+0000 7ff427400640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff408077f70 0x7ff40807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:36.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.583+0000 7ff427400640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff408077f70 0x7ff40807a430 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff418033000 tx=0x7ff4180046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.589+0000 7ff41f400640 1 -- 172.21.15.31:0/2356255532 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff420014030 con 0x7ff428154b60 2024-09-11T00:44:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.873+0000 7ff42e800640 1 -- 172.21.15.31:0/2356255532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4280a6f20 con 0x7ff428154b60 2024-09-11T00:44:36.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.874+0000 7ff41f400640 1 -- 172.21.15.31:0/2356255532 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6542 (secure 0 0 0) 0x7ff420063030 con 0x7ff428154b60 2024-09-11T00:44:36.875 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:36.876 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":102,"btime":"2024-09-11T00:44:26:658257+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:17.423168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":2}]} 2024-09-11T00:44:36.876 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 102 2024-09-11T00:44:36.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.876+0000 7ff42e800640 1 -- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff408077f70 msgr2=0x7ff40807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:36.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.876+0000 7ff42e800640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff408077f70 0x7ff40807a430 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff418033000 tx=0x7ff4180046c0 comp rx=0 tx=0).stop 2024-09-11T00:44:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.876+0000 7ff42e800640 1 -- 172.21.15.31:0/2356255532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff428154b60 msgr2=0x7ff4280c67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.876+0000 7ff42e800640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff428154b60 0x7ff4280c67c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff42000cad0 tx=0x7ff42000cf90 comp rx=0 tx=0).stop 2024-09-11T00:44:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.876+0000 7ff42e800640 1 -- 172.21.15.31:0/2356255532 shutdown_connections 2024-09-11T00:44:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.876+0000 7ff42e800640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff408077f70 0x7ff40807a430 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.876+0000 7ff42e800640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff428154b60 0x7ff4280c67c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.876+0000 7ff42e800640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff4281511c0 0x7ff42806ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.876+0000 7ff42e800640 1 --2- 172.21.15.31:0/2356255532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff428072480 0x7ff42806f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.876+0000 7ff42e800640 1 -- 172.21.15.31:0/2356255532 >> 172.21.15.31:0/2356255532 conn(0x7ff4280a9750 msgr2=0x7ff42814ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.877+0000 7ff42e800640 1 -- 172.21.15.31:0/2356255532 shutdown_connections 2024-09-11T00:44:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:36.877+0000 7ff42e800640 1 -- 172.21.15.31:0/2356255532 wait complete. 2024-09-11T00:44:36.896 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:44:38.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:39.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.077+0000 7f0973e00640 1 -- 172.21.15.31:0/3121675724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c0c5b10 msgr2=0x7f096c156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:39.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.077+0000 7f0973e00640 1 --2- 172.21.15.31:0/3121675724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c0c5b10 0x7f096c156ad0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f096c06de50 tx=0x7f096002a560 comp rx=0 tx=0).stop 2024-09-11T00:44:39.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.077+0000 7f0973e00640 1 -- 172.21.15.31:0/3121675724 shutdown_connections 2024-09-11T00:44:39.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.077+0000 7f0973e00640 1 --2- 172.21.15.31:0/3121675724 >> v2:172.21.15.204:3300/0 conn(0x7f096c0c6310 0x7f096c1539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:39.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.077+0000 7f0973e00640 1 --2- 172.21.15.31:0/3121675724 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f096c0c5ef0 0x7f096c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.077+0000 7f0973e00640 1 --2- 172.21.15.31:0/3121675724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c0c5b10 0x7f096c156ad0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.077+0000 7f0973e00640 1 -- 172.21.15.31:0/3121675724 >> 172.21.15.31:0/3121675724 conn(0x7f096c0a9750 msgr2=0x7f096c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.077+0000 7f0973e00640 1 -- 172.21.15.31:0/3121675724 shutdown_connections 2024-09-11T00:44:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.077+0000 7f0973e00640 1 -- 172.21.15.31:0/3121675724 wait complete. 2024-09-11T00:44:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.078+0000 7f0973e00640 1 Processor -- start 2024-09-11T00:44:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.079+0000 7f0973e00640 1 -- start start 2024-09-11T00:44:39.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.079+0000 7f0973e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f096c0c5b10 0x7f096c14bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:39.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.079+0000 7f0973e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c0c5ef0 0x7f096c14ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:39.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.079+0000 7f0973e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f096c0c6310 0x7f096c14b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:39.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.079+0000 7f0973e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1597f0 con 0x7f096c0c5ef0 2024-09-11T00:44:39.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.079+0000 7f0973e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f096c159670 con 0x7f096c0c5b10 2024-09-11T00:44:39.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.079+0000 7f0973e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f096c159970 con 0x7f096c0c6310 2024-09-11T00:44:39.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.080+0000 7f0970a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c0c5ef0 0x7f096c14ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:39.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.080+0000 7f0970a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c0c5ef0 0x7f096c14ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60804/0 (socket says 172.21.15.31:60804) 2024-09-11T00:44:39.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.080+0000 7f0971400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f096c0c5b10 0x7f096c14bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:39.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.080+0000 7f0970a00640 1 -- 172.21.15.31:0/1525192229 learned_addr learned my addr 172.21.15.31:0/1525192229 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:39.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.080+0000 7f0971e00640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f096c0c6310 0x7f096c14b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:39.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.080+0000 7f0970a00640 1 -- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f096c0c6310 msgr2=0x7f096c14b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:39.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.080+0000 7f0970a00640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f096c0c6310 0x7f096c14b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:39.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.080+0000 7f0970a00640 1 -- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f096c0c5b10 msgr2=0x7f096c14bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:39.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.080+0000 7f0970a00640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f096c0c5b10 0x7f096c14bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:39.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.080+0000 7f0970a00640 1 -- 172.21.15.31:0/1525192229 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096c1df750 con 0x7f096c0c5ef0 2024-09-11T00:44:39.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.081+0000 7f0970a00640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c0c5ef0 0x7f096c14ab80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f095800d950 tx=0x7f095800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:39.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.081+0000 7f0969600640 1 -- 172.21.15.31:0/1525192229 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0958017070 con 0x7f096c0c5ef0 2024-09-11T00:44:39.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.081+0000 7f0969600640 1 -- 172.21.15.31:0/1525192229 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0958009d20 con 0x7f096c0c5ef0 2024-09-11T00:44:39.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.081+0000 7f0969600640 1 -- 172.21.15.31:0/1525192229 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09580106c0 con 0x7f096c0c5ef0 2024-09-11T00:44:39.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.081+0000 7f0973e00640 1 -- 172.21.15.31:0/1525192229 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096c1ef380 con 0x7f096c0c5ef0 2024-09-11T00:44:39.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.081+0000 7f0973e00640 1 -- 172.21.15.31:0/1525192229 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f096c1ef940 con 0x7f096c0c5ef0 2024-09-11T00:44:39.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.082+0000 7f0973e00640 1 -- 172.21.15.31:0/1525192229 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f096c156ad0 con 0x7f096c0c5ef0 2024-09-11T00:44:39.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.083+0000 7f0969600640 1 -- 172.21.15.31:0/1525192229 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0958010860 con 0x7f096c0c5ef0 2024-09-11T00:44:39.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.084+0000 7f0969600640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f094c077f70 0x7f094c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:39.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.085+0000 7f0969600640 1 -- 172.21.15.31:0/1525192229 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7f095809aba0 con 0x7f096c0c5ef0 2024-09-11T00:44:39.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.085+0000 7f0971400640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f094c077f70 0x7f094c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:39.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.086+0000 7f0971400640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f094c077f70 0x7f094c07a430 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0960033000 tx=0x7f0960004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:39.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.089+0000 7f0969600640 1 -- 172.21.15.31:0/1525192229 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0958062660 con 0x7f096c0c5ef0 2024-09-11T00:44:39.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.375+0000 7f0973e00640 1 -- 172.21.15.31:0/1525192229 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f096c06fde0 con 0x7f096c0c5ef0 2024-09-11T00:44:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.375+0000 7f0969600640 1 -- 172.21.15.31:0/1525192229 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6542 (secure 0 0 0) 0x7f0958067090 con 0x7f096c0c5ef0 2024-09-11T00:44:39.377 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":102,"btime":"2024-09-11T00:44:26:658257+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:17.423168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":2}]} 2024-09-11T00:44:39.377 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 102 2024-09-11T00:44:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 -- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f094c077f70 msgr2=0x7f094c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f094c077f70 0x7f094c07a430 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0960033000 tx=0x7f0960004730 comp rx=0 tx=0).stop 2024-09-11T00:44:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 -- 172.21.15.31:0/1525192229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c0c5ef0 msgr2=0x7f096c14ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c0c5ef0 0x7f096c14ab80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f095800d950 tx=0x7f095800de10 comp rx=0 tx=0).stop 2024-09-11T00:44:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 -- 172.21.15.31:0/1525192229 shutdown_connections 2024-09-11T00:44:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f094c077f70 0x7f094c07a430 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f096c0c6310 0x7f096c14b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f096c0c5ef0 0x7f096c14ab80 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:39.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 --2- 172.21.15.31:0/1525192229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f096c0c5b10 0x7f096c14bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:39.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 -- 172.21.15.31:0/1525192229 >> 172.21.15.31:0/1525192229 conn(0x7f096c0a9750 msgr2=0x7f096c0a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:39.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 -- 172.21.15.31:0/1525192229 shutdown_connections 2024-09-11T00:44:39.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:39.378+0000 7f0973e00640 1 -- 172.21.15.31:0/1525192229 wait complete. 2024-09-11T00:44:39.398 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:44:41.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:41.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.579+0000 7f6934e00640 1 -- 172.21.15.31:0/1422568175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f69301534f0 msgr2=0x7f6930156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.579+0000 7f6934e00640 1 --2- 172.21.15.31:0/1422568175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f69301534f0 0x7f6930156ae0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6920009860 tx=0x7f692002a510 comp rx=0 tx=0).stop 2024-09-11T00:44:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.579+0000 7f6934e00640 1 -- 172.21.15.31:0/1422568175 shutdown_connections 2024-09-11T00:44:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.579+0000 7f6934e00640 1 --2- 172.21.15.31:0/1422568175 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f69301534f0 0x7f6930156ae0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.579+0000 7f6934e00640 1 -- 172.21.15.31:0/1422568175 >> 172.21.15.31:0/1422568175 conn(0x7f6930068f10 msgr2=0x7f69300cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.580+0000 7f6934e00640 1 -- 172.21.15.31:0/1422568175 shutdown_connections 2024-09-11T00:44:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.580+0000 7f6934e00640 1 -- 172.21.15.31:0/1422568175 wait complete. 2024-09-11T00:44:41.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.581+0000 7f6934e00640 1 Processor -- start 2024-09-11T00:44:41.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.581+0000 7f6934e00640 1 -- start start 2024-09-11T00:44:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f6934e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69300c9b40 0x7f69301ec980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f6934e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69301534f0 0x7f69301584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f6934e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6930159650 0x7f69300ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f6934e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f69300cca60 con 0x7f69300c9b40 2024-09-11T00:44:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f6934e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f69300c6cb0 con 0x7f69301534f0 2024-09-11T00:44:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f6934e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6930147000 con 0x7f6930159650 2024-09-11T00:44:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f692de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69300c9b40 0x7f69301ec980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f692de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69300c9b40 0x7f69301ec980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60812/0 (socket says 172.21.15.31:60812) 2024-09-11T00:44:41.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f692de00640 1 -- 172.21.15.31:0/1415668884 learned_addr learned my addr 172.21.15.31:0/1415668884 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:41.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f692d400640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69301534f0 0x7f69301584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:41.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f692e800640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6930159650 0x7f69300ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f692de00640 1 -- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6930159650 msgr2=0x7f69300ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f692de00640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6930159650 0x7f69300ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f692de00640 1 -- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69301534f0 msgr2=0x7f69301584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.583+0000 7f692de00640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69301534f0 0x7f69301584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.584+0000 7f692de00640 1 -- 172.21.15.31:0/1415668884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6930147660 con 0x7f69300c9b40 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.584+0000 7f692de00640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69300c9b40 0x7f69301ec980 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6920009980 tx=0x7f6920032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.584+0000 7f6926000640 1 -- 172.21.15.31:0/1415668884 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920040070 con 0x7f69300c9b40 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.584+0000 7f6926000640 1 -- 172.21.15.31:0/1415668884 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69200038e0 con 0x7f69300c9b40 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.584+0000 7f6926000640 1 -- 172.21.15.31:0/1415668884 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920003d30 con 0x7f69300c9b40 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.584+0000 7f6934e00640 1 -- 172.21.15.31:0/1415668884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69301ec180 con 0x7f69300c9b40 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.584+0000 7f6934e00640 1 -- 172.21.15.31:0/1415668884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69301e2120 con 0x7f69300c9b40 2024-09-11T00:44:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.586+0000 7f6934e00640 1 -- 172.21.15.31:0/1415668884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69301538d0 con 0x7f69300c9b40 2024-09-11T00:44:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.587+0000 7f6926000640 1 -- 172.21.15.31:0/1415668884 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6920045460 con 0x7f69300c9b40 2024-09-11T00:44:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.588+0000 7f6926000640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6910077f70 0x7f691007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.588+0000 7f6926000640 1 -- 172.21.15.31:0/1415668884 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7f69200bbd60 con 0x7f69300c9b40 2024-09-11T00:44:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.588+0000 7f692d400640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6910077f70 0x7f691007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:41.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.589+0000 7f692d400640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6910077f70 0x7f691007a430 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f69301f06a0 tx=0x7f691c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:41.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.593+0000 7f6926000640 1 -- 172.21.15.31:0/1415668884 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f69200837f0 con 0x7f69300c9b40 2024-09-11T00:44:41.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.869+0000 7f6934e00640 1 -- 172.21.15.31:0/1415668884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6930154b20 con 0x7f69300c9b40 2024-09-11T00:44:41.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.870+0000 7f6926000640 1 -- 172.21.15.31:0/1415668884 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6542 (secure 0 0 0) 0x7f6920088220 con 0x7f69300c9b40 2024-09-11T00:44:41.872 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:41.872 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":102,"btime":"2024-09-11T00:44:26:658257+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:17.423168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":2}]} 2024-09-11T00:44:41.872 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 102 2024-09-11T00:44:41.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.872+0000 7f6934e00640 1 -- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6910077f70 msgr2=0x7f691007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.872+0000 7f6934e00640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6910077f70 0x7f691007a430 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f69301f06a0 tx=0x7f691c009040 comp rx=0 tx=0).stop 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.872+0000 7f6934e00640 1 -- 172.21.15.31:0/1415668884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69300c9b40 msgr2=0x7f69301ec980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.872+0000 7f6934e00640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69300c9b40 0x7f69301ec980 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6920009980 tx=0x7f6920032b90 comp rx=0 tx=0).stop 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.872+0000 7f6934e00640 1 -- 172.21.15.31:0/1415668884 shutdown_connections 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.872+0000 7f6934e00640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6910077f70 0x7f691007a430 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.872+0000 7f6934e00640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6930159650 0x7f69300ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.872+0000 7f6934e00640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69301534f0 0x7f69301584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.872+0000 7f6934e00640 1 --2- 172.21.15.31:0/1415668884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69300c9b40 0x7f69301ec980 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.872+0000 7f6934e00640 1 -- 172.21.15.31:0/1415668884 >> 172.21.15.31:0/1415668884 conn(0x7f6930068f10 msgr2=0x7f69300cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.873+0000 7f6934e00640 1 -- 172.21.15.31:0/1415668884 shutdown_connections 2024-09-11T00:44:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:41.873+0000 7f6934e00640 1 -- 172.21.15.31:0/1415668884 wait complete. 2024-09-11T00:44:41.892 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:44:43.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:44.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.075+0000 7f89c3e00640 1 -- 172.21.15.31:0/3167658658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a8004800 msgr2=0x7f89a8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:44.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.075+0000 7f89c3e00640 1 --2- 172.21.15.31:0/3167658658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a8004800 0x7f89a8004c80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f89ac001830 tx=0x7f89a8005830 comp rx=0 tx=0).stop 2024-09-11T00:44:44.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.075+0000 7f89c3e00640 1 -- 172.21.15.31:0/3167658658 shutdown_connections 2024-09-11T00:44:44.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.075+0000 7f89c3e00640 1 --2- 172.21.15.31:0/3167658658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89a8004800 0x7f89a8004c80 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:44.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.075+0000 7f89c3e00640 1 -- 172.21.15.31:0/3167658658 >> 172.21.15.31:0/3167658658 conn(0x7f89c4068f10 msgr2=0x7f89c40cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:44.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.075+0000 7f89c3e00640 1 -- 172.21.15.31:0/3167658658 shutdown_connections 2024-09-11T00:44:44.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.075+0000 7f89c3e00640 1 -- 172.21.15.31:0/3167658658 wait complete. 2024-09-11T00:44:44.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.077+0000 7f89c3e00640 1 Processor -- start 2024-09-11T00:44:44.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.077+0000 7f89c3e00640 1 -- start start 2024-09-11T00:44:44.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89c3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89a8004800 0x7f89c41f01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89c3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89c40b0e70 0x7f89c414b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89c3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89c41ed610 0x7f89c4159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89c3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f89c4157550 con 0x7f89c40b0e70 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89c3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f89c41573d0 con 0x7f89c41ed610 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89c3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f89c4156960 con 0x7f89a8004800 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89bbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89c40b0e70 0x7f89c414b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89bbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89c40b0e70 0x7f89c414b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60822/0 (socket says 172.21.15.31:60822) 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89bbe00640 1 -- 172.21.15.31:0/3372232277 learned_addr learned my addr 172.21.15.31:0/3372232277 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89c2a00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89a8004800 0x7f89c41f01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.078+0000 7f89c3400640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89c41ed610 0x7f89c4159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.079+0000 7f89bbe00640 1 -- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89a8004800 msgr2=0x7f89c41f01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.079+0000 7f89bbe00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89a8004800 0x7f89c41f01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:44.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.079+0000 7f89bbe00640 1 -- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89c41ed610 msgr2=0x7f89c4159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:44.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.079+0000 7f89bbe00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89c41ed610 0x7f89c4159460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:44.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.079+0000 7f89bbe00640 1 -- 172.21.15.31:0/3372232277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c41ef710 con 0x7f89c40b0e70 2024-09-11T00:44:44.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.079+0000 7f89bbe00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89c40b0e70 0x7f89c414b1e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f89ac001950 tx=0x7f89ac0060e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:44.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.080+0000 7f89baa00640 1 -- 172.21.15.31:0/3372232277 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac043070 con 0x7f89c40b0e70 2024-09-11T00:44:44.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.080+0000 7f89baa00640 1 -- 172.21.15.31:0/3372232277 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89ac003af0 con 0x7f89c40b0e70 2024-09-11T00:44:44.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.080+0000 7f89baa00640 1 -- 172.21.15.31:0/3372232277 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0064c0 con 0x7f89c40b0e70 2024-09-11T00:44:44.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.080+0000 7f89c3e00640 1 -- 172.21.15.31:0/3372232277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c40aea40 con 0x7f89c40b0e70 2024-09-11T00:44:44.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.080+0000 7f89c3e00640 1 -- 172.21.15.31:0/3372232277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c41ef9b0 con 0x7f89c40b0e70 2024-09-11T00:44:44.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.081+0000 7f89c3e00640 1 -- 172.21.15.31:0/3372232277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c40aef00 con 0x7f89c40b0e70 2024-09-11T00:44:44.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.082+0000 7f89baa00640 1 -- 172.21.15.31:0/3372232277 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f89ac033480 con 0x7f89c40b0e70 2024-09-11T00:44:44.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.083+0000 7f89baa00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f89a0077fa0 0x7f89a007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:44.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.084+0000 7f89baa00640 1 -- 172.21.15.31:0/3372232277 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8214+0+0 (secure 0 0 0) 0x7f89ac0ba310 con 0x7f89c40b0e70 2024-09-11T00:44:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.084+0000 7f89c2a00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f89a0077fa0 0x7f89a007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:44.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.085+0000 7f89c2a00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f89a0077fa0 0x7f89a007a460 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f89c40668e0 tx=0x7f89b4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.087+0000 7f89baa00640 1 -- 172.21.15.31:0/3372232277 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f89c40aef00 con 0x7f89c40b0e70 2024-09-11T00:44:44.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.350+0000 7f89c3e00640 1 -- 172.21.15.31:0/3372232277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89c40b1540 con 0x7f89c40b0e70 2024-09-11T00:44:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.351+0000 7f89baa00640 1 -- 172.21.15.31:0/3372232277 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+6542 (secure 0 0 0) 0x7f89ac081e90 con 0x7f89c40b0e70 2024-09-11T00:44:44.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:44.353 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":102,"btime":"2024-09-11T00:44:26:658257+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5777,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:17.423168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"c","rank":0,"incarnation":98,"state":"up:active","state_seq":14,"addr":"172.21.15.31:6835/2323279595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2323279595},{"type":"v1","addr":"172.21.15.31:6835","nonce":2323279595}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":2}]} 2024-09-11T00:44:44.353 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 102 2024-09-11T00:44:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 -- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f89a0077fa0 msgr2=0x7f89a007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f89a0077fa0 0x7f89a007a460 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f89c40668e0 tx=0x7f89b4005f90 comp rx=0 tx=0).stop 2024-09-11T00:44:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 -- 172.21.15.31:0/3372232277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89c40b0e70 msgr2=0x7f89c414b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89c40b0e70 0x7f89c414b1e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f89ac001950 tx=0x7f89ac0060e0 comp rx=0 tx=0).stop 2024-09-11T00:44:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 -- 172.21.15.31:0/3372232277 shutdown_connections 2024-09-11T00:44:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f89a0077fa0 0x7f89a007a460 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89c41ed610 0x7f89c4159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:44.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89c40b0e70 0x7f89c414b1e0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:44.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 --2- 172.21.15.31:0/3372232277 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89a8004800 0x7f89c41f01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:44.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 -- 172.21.15.31:0/3372232277 >> 172.21.15.31:0/3372232277 conn(0x7f89c4068f10 msgr2=0x7f89c4155f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:44.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 -- 172.21.15.31:0/3372232277 shutdown_connections 2024-09-11T00:44:44.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:44.354+0000 7f89c3e00640 1 -- 172.21.15.31:0/3372232277 wait complete. 2024-09-11T00:44:44.374 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:44:46.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:46.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.551+0000 7fd145a00640 1 -- 172.21.15.31:0/1330096579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd124003c30 msgr2=0x7fd124004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:46.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.551+0000 7fd145a00640 1 --2- 172.21.15.31:0/1330096579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd124003c30 0x7fd124004030 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd130006f80 tx=0x7fd124005830 comp rx=0 tx=0).stop 2024-09-11T00:44:46.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.551+0000 7fd145a00640 1 -- 172.21.15.31:0/1330096579 shutdown_connections 2024-09-11T00:44:46.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.551+0000 7fd145a00640 1 --2- 172.21.15.31:0/1330096579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd124003c30 0x7fd124004030 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:46.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.551+0000 7fd145a00640 1 -- 172.21.15.31:0/1330096579 >> 172.21.15.31:0/1330096579 conn(0x7fd1400aa6f0 msgr2=0x7fd1400aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:46.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.552+0000 7fd145a00640 1 -- 172.21.15.31:0/1330096579 shutdown_connections 2024-09-11T00:44:46.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.552+0000 7fd145a00640 1 -- 172.21.15.31:0/1330096579 wait complete. 2024-09-11T00:44:46.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.553+0000 7fd145a00640 1 Processor -- start 2024-09-11T00:44:46.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.553+0000 7fd145a00640 1 -- start start 2024-09-11T00:44:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.554+0000 7fd145a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd124003c30 0x7fd1401ed790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.554+0000 7fd145a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1400ab7e0 0x7fd1401df180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.554+0000 7fd145a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd140149b00 0x7fd140149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.554+0000 7fd145a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd140151810 con 0x7fd140149b00 2024-09-11T00:44:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.554+0000 7fd145a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1400c85a0 con 0x7fd124003c30 2024-09-11T00:44:46.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.554+0000 7fd145a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd140151990 con 0x7fd1400ab7e0 2024-09-11T00:44:46.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.554+0000 7fd145000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd140149b00 0x7fd140149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:46.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.554+0000 7fd13fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd124003c30 0x7fd1401ed790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:46.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.554+0000 7fd145000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd140149b00 0x7fd140149f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60838/0 (socket says 172.21.15.31:60838) 2024-09-11T00:44:46.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.554+0000 7fd145000640 1 -- 172.21.15.31:0/1796518707 learned_addr learned my addr 172.21.15.31:0/1796518707 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:46.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.555+0000 7fd13f400640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1400ab7e0 0x7fd1401df180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:46.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.555+0000 7fd145000640 1 -- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1400ab7e0 msgr2=0x7fd1401df180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:46.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.555+0000 7fd145000640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1400ab7e0 0x7fd1401df180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:46.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.555+0000 7fd145000640 1 -- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd124003c30 msgr2=0x7fd1401ed790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:46.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.555+0000 7fd145000640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd124003c30 0x7fd1401ed790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:46.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.555+0000 7fd145000640 1 -- 172.21.15.31:0/1796518707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1400ad240 con 0x7fd140149b00 2024-09-11T00:44:46.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.555+0000 7fd145000640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd140149b00 0x7fd140149f80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fd138011bb0 tx=0x7fd13800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:46.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.555+0000 7fd137e00640 1 -- 172.21.15.31:0/1796518707 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd138003d80 con 0x7fd140149b00 2024-09-11T00:44:46.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.556+0000 7fd137e00640 1 -- 172.21.15.31:0/1796518707 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd138004b20 con 0x7fd140149b00 2024-09-11T00:44:46.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.556+0000 7fd137e00640 1 -- 172.21.15.31:0/1796518707 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd138004030 con 0x7fd140149b00 2024-09-11T00:44:46.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.556+0000 7fd145a00640 1 -- 172.21.15.31:0/1796518707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1401dfae0 con 0x7fd140149b00 2024-09-11T00:44:46.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.556+0000 7fd145a00640 1 -- 172.21.15.31:0/1796518707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1401df6c0 con 0x7fd140149b00 2024-09-11T00:44:46.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.557+0000 7fd145a00640 1 -- 172.21.15.31:0/1796518707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd14006caf0 con 0x7fd140149b00 2024-09-11T00:44:46.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.558+0000 7fd137e00640 1 -- 172.21.15.31:0/1796518707 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd138018030 con 0x7fd140149b00 2024-09-11T00:44:46.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.559+0000 7fd137e00640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd120077f70 0x7fd12007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:46.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.559+0000 7fd137e00640 1 -- 172.21.15.31:0/1796518707 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7fd13809b1e0 con 0x7fd140149b00 2024-09-11T00:44:46.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.559+0000 7fd13fe00640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd120077f70 0x7fd12007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:46.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.560+0000 7fd13fe00640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd120077f70 0x7fd12007a430 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd130033000 tx=0x7fd13002d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:46.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.565+0000 7fd137e00640 1 -- 172.21.15.31:0/1796518707 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd138062c50 con 0x7fd140149b00 2024-09-11T00:44:46.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.835+0000 7fd145a00640 1 -- 172.21.15.31:0/1796518707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1401e0000 con 0x7fd140149b00 2024-09-11T00:44:46.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.835+0000 7fd137e00640 1 -- 172.21.15.31:0/1796518707 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5709 (secure 0 0 0) 0x7fd138067680 con 0x7fd140149b00 2024-09-11T00:44:46.838 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:46.838 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":105,"btime":"2024-09-11T00:44:46:706944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:45.713002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:44:46.839 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 105 2024-09-11T00:44:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 -- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd120077f70 msgr2=0x7fd12007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd120077f70 0x7fd12007a430 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd130033000 tx=0x7fd13002d000 comp rx=0 tx=0).stop 2024-09-11T00:44:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 -- 172.21.15.31:0/1796518707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd140149b00 msgr2=0x7fd140149f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd140149b00 0x7fd140149f80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fd138011bb0 tx=0x7fd13800f790 comp rx=0 tx=0).stop 2024-09-11T00:44:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 -- 172.21.15.31:0/1796518707 shutdown_connections 2024-09-11T00:44:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd120077f70 0x7fd12007a430 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:46.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd140149b00 0x7fd140149f80 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd1400ab7e0 0x7fd1401df180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 --2- 172.21.15.31:0/1796518707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd124003c30 0x7fd1401ed790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 -- 172.21.15.31:0/1796518707 >> 172.21.15.31:0/1796518707 conn(0x7fd1400aa6f0 msgr2=0x7fd14006cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 -- 172.21.15.31:0/1796518707 shutdown_connections 2024-09-11T00:44:46.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:46.838+0000 7fd145a00640 1 -- 172.21.15.31:0/1796518707 wait complete. 2024-09-11T00:44:46.859 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-11T00:44:46.859 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:44:46.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.044+0000 7fe522400640 1 -- 172.21.15.31:0/1254076314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe51c0c5c60 msgr2=0x7fe51c06caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.044+0000 7fe522400640 1 --2- 172.21.15.31:0/1254076314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe51c0c5c60 0x7fe51c06caf0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe51c06db80 tx=0x7fe50402a4d0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.044+0000 7fe522400640 1 -- 172.21.15.31:0/1254076314 shutdown_connections 2024-09-11T00:44:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.044+0000 7fe522400640 1 --2- 172.21.15.31:0/1254076314 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe51c0c6090 0x7fe51c06d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.044+0000 7fe522400640 1 --2- 172.21.15.31:0/1254076314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe51c0c5c60 0x7fe51c06caf0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.044+0000 7fe522400640 1 --2- 172.21.15.31:0/1254076314 >> v2:172.21.15.204:3300/0 conn(0x7fe51c0c4ef0 0x7fe51c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.044+0000 7fe522400640 1 -- 172.21.15.31:0/1254076314 >> 172.21.15.31:0/1254076314 conn(0x7fe51c0c89d0 msgr2=0x7fe51c0c8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.044+0000 7fe522400640 1 -- 172.21.15.31:0/1254076314 shutdown_connections 2024-09-11T00:44:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.044+0000 7fe522400640 1 -- 172.21.15.31:0/1254076314 wait complete. 2024-09-11T00:44:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.046+0000 7fe522400640 1 Processor -- start 2024-09-11T00:44:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.046+0000 7fe522400640 1 -- start start 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe522400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe51c0c4ef0 0x7fe51c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe522400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe51c0c5c60 0x7fe51c155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe522400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe51c0c6090 0x7fe51c157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe522400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c0c8690 con 0x7fe51c0c5c60 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe522400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c0c6ce0 con 0x7fe51c0c4ef0 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe522400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c151930 con 0x7fe51c0c6090 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe513e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe51c0c5c60 0x7fe51c155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe51b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe51c0c4ef0 0x7fe51c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe513e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe51c0c5c60 0x7fe51c155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60848/0 (socket says 172.21.15.31:60848) 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe513e00640 1 -- 172.21.15.31:0/2116283105 learned_addr learned my addr 172.21.15.31:0/2116283105 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe51be00640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe51c0c6090 0x7fe51c157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe513e00640 1 -- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe51c0c6090 msgr2=0x7fe51c157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe513e00640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe51c0c6090 0x7fe51c157cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe513e00640 1 -- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe51c0c4ef0 msgr2=0x7fe51c0abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe513e00640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe51c0c4ef0 0x7fe51c0abfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe513e00640 1 -- 172.21.15.31:0/2116283105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c1ecd10 con 0x7fe51c0c5c60 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.047+0000 7fe51b400640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe51c0c4ef0 0x7fe51c0abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.048+0000 7fe513e00640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe51c0c5c60 0x7fe51c155dd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe5040099c0 tx=0x7fe5040039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.048+0000 7fe513400640 1 -- 172.21.15.31:0/2116283105 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50403f070 con 0x7fe51c0c5c60 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.048+0000 7fe513400640 1 -- 172.21.15.31:0/2116283105 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe504004890 con 0x7fe51c0c5c60 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.048+0000 7fe513400640 1 -- 172.21.15.31:0/2116283105 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe504035030 con 0x7fe51c0c5c60 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.048+0000 7fe522400640 1 -- 172.21.15.31:0/2116283105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51c1ecfb0 con 0x7fe51c0c5c60 2024-09-11T00:44:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.049+0000 7fe522400640 1 -- 172.21.15.31:0/2116283105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51c1e9da0 con 0x7fe51c0c5c60 2024-09-11T00:44:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.050+0000 7fe522400640 1 -- 172.21.15.31:0/2116283105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe51c0c5720 con 0x7fe51c0c5c60 2024-09-11T00:44:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.051+0000 7fe513400640 1 -- 172.21.15.31:0/2116283105 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe504004ce0 con 0x7fe51c0c5c60 2024-09-11T00:44:47.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.052+0000 7fe513400640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe4f8077fa0 0x7fe4f807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:47.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.052+0000 7fe513400640 1 -- 172.21.15.31:0/2116283105 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7fe5040bb850 con 0x7fe51c0c5c60 2024-09-11T00:44:47.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.053+0000 7fe51b400640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe4f8077fa0 0x7fe4f807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:47.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.054+0000 7fe51b400640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe4f8077fa0 0x7fe4f807a460 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005fd0 tx=0x7fe50c00b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:47.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.058+0000 7fe513400640 1 -- 172.21.15.31:0/2116283105 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe5040832c0 con 0x7fe51c0c5c60 2024-09-11T00:44:47.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.344+0000 7fe522400640 1 -- 172.21.15.31:0/2116283105 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe51c1e1bf0 con 0x7fe51c0c5c60 2024-09-11T00:44:47.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.344+0000 7fe513400640 1 -- 172.21.15.31:0/2116283105 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5709 (secure 0 0 0) 0x7fe504087cf0 con 0x7fe51c0c5c60 2024-09-11T00:44:47.347 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:47.347 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":105,"btime":"2024-09-11T00:44:46:706944+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:45.713002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:44:47.348 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 105 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 -- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe4f8077fa0 msgr2=0x7fe4f807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe4f8077fa0 0x7fe4f807a460 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005fd0 tx=0x7fe50c00b2e0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 -- 172.21.15.31:0/2116283105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe51c0c5c60 msgr2=0x7fe51c155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe51c0c5c60 0x7fe51c155dd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe5040099c0 tx=0x7fe5040039e0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 -- 172.21.15.31:0/2116283105 shutdown_connections 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe4f8077fa0 0x7fe4f807a460 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe51c0c6090 0x7fe51c157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe51c0c5c60 0x7fe51c155dd0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 --2- 172.21.15.31:0/2116283105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe51c0c4ef0 0x7fe51c0abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 -- 172.21.15.31:0/2116283105 >> 172.21.15.31:0/2116283105 conn(0x7fe51c0c89d0 msgr2=0x7fe51c153180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 -- 172.21.15.31:0/2116283105 shutdown_connections 2024-09-11T00:44:47.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:47.347+0000 7fe522400640 1 -- 172.21.15.31:0/2116283105 wait complete. 2024-09-11T00:44:47.367 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:44:46:706944+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 105, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5773": { "addr": "172.21.15.204:6837/3513641573", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 3513641573, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 3513641573, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5773, "incarnation": 88, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 77 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:43:22.726458+0000", "qdb_cluster": [ 5773 ], "qdb_leader": 5773, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5773 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 105, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5777": { "addr": "172.21.15.204:6835/1348312176", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1348312176, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 1348312176, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5777, "incarnation": 103, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:44:45.713002+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5777 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:44:47.584 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:44:47.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:44:49.368 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:49.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.549+0000 7f88eee00640 1 -- 172.21.15.31:0/162039065 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f88e80bf240 msgr2=0x7f88e80b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:49.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.549+0000 7f88eee00640 1 --2- 172.21.15.31:0/162039065 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f88e80bf240 0x7f88e80b0be0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f88d4009820 tx=0x7f88d402a480 comp rx=0 tx=0).stop 2024-09-11T00:44:49.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.549+0000 7f88eee00640 1 -- 172.21.15.31:0/162039065 shutdown_connections 2024-09-11T00:44:49.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.549+0000 7f88eee00640 1 --2- 172.21.15.31:0/162039065 >> v2:172.21.15.204:3300/0 conn(0x7f88e8156e40 0x7f88e80b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:49.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.549+0000 7f88eee00640 1 --2- 172.21.15.31:0/162039065 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f88e80bf240 0x7f88e80b0be0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:49.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.549+0000 7f88eee00640 1 -- 172.21.15.31:0/162039065 >> 172.21.15.31:0/162039065 conn(0x7f88e8068ac0 msgr2=0x7f88e8068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:49.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.549+0000 7f88eee00640 1 -- 172.21.15.31:0/162039065 shutdown_connections 2024-09-11T00:44:49.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.550+0000 7f88eee00640 1 -- 172.21.15.31:0/162039065 wait complete. 2024-09-11T00:44:49.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.551+0000 7f88eee00640 1 Processor -- start 2024-09-11T00:44:49.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.552+0000 7f88eee00640 1 -- start start 2024-09-11T00:44:49.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88eee00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f88e80bf240 0x7f88e80acef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:49.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88eee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e80c6730 0x7f88e81e0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:49.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88eee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f88e8156e40 0x7f88e81e1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88eee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f88e8159de0 con 0x7f88e80c6730 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88eee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f88e8156640 con 0x7f88e8156e40 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88eee00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f88e8159f60 con 0x7f88e80bf240 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88ed000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e80c6730 0x7f88e81e0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88eda00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f88e80bf240 0x7f88e80acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88ed000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e80c6730 0x7f88e81e0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45844/0 (socket says 172.21.15.31:45844) 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88ed000640 1 -- 172.21.15.31:0/3019998287 learned_addr learned my addr 172.21.15.31:0/3019998287 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.553+0000 7f88ee400640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f88e8156e40 0x7f88e81e1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88eda00640 1 -- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f88e8156e40 msgr2=0x7f88e81e1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88eda00640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f88e8156e40 0x7f88e81e1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88eda00640 1 -- 172.21.15.31:0/3019998287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e80c6730 msgr2=0x7f88e81e0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:49.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88eda00640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e80c6730 0x7f88e81e0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:49.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88eda00640 1 -- 172.21.15.31:0/3019998287 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e80ca2f0 con 0x7f88e80bf240 2024-09-11T00:44:49.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88eda00640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f88e80bf240 0x7f88e80acef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00c3c0 tx=0x7f88dc00c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:49.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88e5600640 1 -- 172.21.15.31:0/3019998287 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc011850 con 0x7f88e80bf240 2024-09-11T00:44:49.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88e5600640 1 -- 172.21.15.31:0/3019998287 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88dc017070 con 0x7f88e80bf240 2024-09-11T00:44:49.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88eee00640 1 -- 172.21.15.31:0/3019998287 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e81ece50 con 0x7f88e80bf240 2024-09-11T00:44:49.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88e5600640 1 -- 172.21.15.31:0/3019998287 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc011de0 con 0x7f88e80bf240 2024-09-11T00:44:49.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.554+0000 7f88eee00640 1 -- 172.21.15.31:0/3019998287 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e81e1e30 con 0x7f88e80bf240 2024-09-11T00:44:49.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.556+0000 7f88eee00640 1 -- 172.21.15.31:0/3019998287 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e806e850 con 0x7f88e80bf240 2024-09-11T00:44:49.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.557+0000 7f88e5600640 1 -- 172.21.15.31:0/3019998287 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f88dc01c460 con 0x7f88e80bf240 2024-09-11T00:44:49.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.558+0000 7f88e5600640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f88c8077f70 0x7f88c807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:49.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.558+0000 7f88e5600640 1 -- 172.21.15.31:0/3019998287 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7f88dc09a460 con 0x7f88e80bf240 2024-09-11T00:44:49.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.558+0000 7f88ed000640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f88c8077f70 0x7f88c807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:49.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.559+0000 7f88ed000640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f88c8077f70 0x7f88c807a430 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f88e80ca610 tx=0x7f88d4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:49.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.562+0000 7f88e5600640 1 -- 172.21.15.31:0/3019998287 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f88dc062520 con 0x7f88e80bf240 2024-09-11T00:44:49.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.842+0000 7f88eee00640 1 -- 172.21.15.31:0/3019998287 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88e80c2980 con 0x7f88e80bf240 2024-09-11T00:44:49.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.843+0000 7f88e5600640 1 -- 172.21.15.31:0/3019998287 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5716 (secure 0 0 0) 0x7f88dc061e20 con 0x7f88e80bf240 2024-09-11T00:44:49.846 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:49.846 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":106,"btime":"2024-09-11T00:44:47:708766+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:44:49.846 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 106 2024-09-11T00:44:49.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 -- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f88c8077f70 msgr2=0x7f88c807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:49.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f88c8077f70 0x7f88c807a430 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f88e80ca610 tx=0x7f88d4004730 comp rx=0 tx=0).stop 2024-09-11T00:44:49.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 -- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f88e80bf240 msgr2=0x7f88e80acef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:49.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f88e80bf240 0x7f88e80acef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00c3c0 tx=0x7f88dc00c7c0 comp rx=0 tx=0).stop 2024-09-11T00:44:49.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 -- 172.21.15.31:0/3019998287 shutdown_connections 2024-09-11T00:44:49.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f88c8077f70 0x7f88c807a430 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:49.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f88e8156e40 0x7f88e81e1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:49.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f88e80c6730 0x7f88e81e0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:49.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 --2- 172.21.15.31:0/3019998287 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f88e80bf240 0x7f88e80acef0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:49.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 -- 172.21.15.31:0/3019998287 >> 172.21.15.31:0/3019998287 conn(0x7f88e8068ac0 msgr2=0x7f88e80afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:49.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 -- 172.21.15.31:0/3019998287 shutdown_connections 2024-09-11T00:44:49.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:49.846+0000 7f88eee00640 1 -- 172.21.15.31:0/3019998287 wait complete. 2024-09-11T00:44:49.866 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=6402 2024-09-11T00:44:49.866 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.c 2024-09-11T00:44:49.867 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-11T00:44:49.867 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-11T00:44:49.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-11T00:44:49.871 INFO:tasks.ceph.mds.c:Started 2024-09-11T00:44:49.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:50.037 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-11T00:44:50.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.069+0000 7f0d93200640 1 -- 172.21.15.31:0/1323445871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d940bf240 msgr2=0x7f0d940b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:50.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.069+0000 7f0d93200640 1 --2- 172.21.15.31:0/1323445871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d940bf240 0x7f0d940b0be0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0d940b1c80 tx=0x7f0d8002a560 comp rx=0 tx=0).stop 2024-09-11T00:44:50.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.069+0000 7f0d93200640 1 -- 172.21.15.31:0/1323445871 shutdown_connections 2024-09-11T00:44:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.069+0000 7f0d93200640 1 --2- 172.21.15.31:0/1323445871 >> v2:172.21.15.204:3300/0 conn(0x7f0d94156e40 0x7f0d940b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.069+0000 7f0d93200640 1 --2- 172.21.15.31:0/1323445871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d940bf240 0x7f0d940b0be0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.069+0000 7f0d93200640 1 -- 172.21.15.31:0/1323445871 >> 172.21.15.31:0/1323445871 conn(0x7f0d94068ac0 msgr2=0x7f0d94068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.069+0000 7f0d93200640 1 -- 172.21.15.31:0/1323445871 shutdown_connections 2024-09-11T00:44:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.070+0000 7f0d93200640 1 -- 172.21.15.31:0/1323445871 wait complete. 2024-09-11T00:44:50.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.071+0000 7f0d93200640 1 Processor -- start 2024-09-11T00:44:50.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.072+0000 7f0d93200640 1 -- start start 2024-09-11T00:44:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d93200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d940bf240 0x7f0d9414eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d93200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0d940c6730 0x7f0d940c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d93200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94156e40 0x7f0d941ef6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d93200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0d94159de0 con 0x7f0d94156e40 2024-09-11T00:44:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d93200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0d94156640 con 0x7f0d940bf240 2024-09-11T00:44:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d93200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0d94159f60 con 0x7f0d940c6730 2024-09-11T00:44:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d92800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94156e40 0x7f0d941ef6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d91e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d940bf240 0x7f0d9414eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d92800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94156e40 0x7f0d941ef6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45886/0 (socket says 172.21.15.31:45886) 2024-09-11T00:44:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d91e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d940bf240 0x7f0d9414eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:51122/0 (socket says 172.21.15.31:51122) 2024-09-11T00:44:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d91400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0d940c6730 0x7f0d940c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.073+0000 7f0d92800640 1 -- 172.21.15.31:0/496089023 learned_addr learned my addr 172.21.15.31:0/496089023 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d91e00640 1 -- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0d940c6730 msgr2=0x7f0d940c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d91e00640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0d940c6730 0x7f0d940c9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d91e00640 1 -- 172.21.15.31:0/496089023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94156e40 msgr2=0x7f0d941ef6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d91e00640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94156e40 0x7f0d941ef6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d91e00640 1 -- 172.21.15.31:0/496089023 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d940ac6d0 con 0x7f0d940bf240 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d91e00640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d940bf240 0x7f0d9414eba0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c3c0 tx=0x7f0d8400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d8a000640 1 -- 172.21.15.31:0/496089023 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84017070 con 0x7f0d940bf240 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d8a000640 1 -- 172.21.15.31:0/496089023 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d840053c0 con 0x7f0d940bf240 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d8a000640 1 -- 172.21.15.31:0/496089023 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84003850 con 0x7f0d940bf240 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d93200640 1 -- 172.21.15.31:0/496089023 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9414f0e0 con 0x7f0d940bf240 2024-09-11T00:44:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.074+0000 7f0d93200640 1 -- 172.21.15.31:0/496089023 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d941e1250 con 0x7f0d940bf240 2024-09-11T00:44:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.076+0000 7f0d93200640 1 -- 172.21.15.31:0/496089023 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d9406e850 con 0x7f0d940bf240 2024-09-11T00:44:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.077+0000 7f0d8a000640 1 -- 172.21.15.31:0/496089023 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0d840039f0 con 0x7f0d940bf240 2024-09-11T00:44:50.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.078+0000 7f0d8a000640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d74077f70 0x7f0d7407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:50.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.078+0000 7f0d8a000640 1 -- 172.21.15.31:0/496089023 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7f0d84099d60 con 0x7f0d940bf240 2024-09-11T00:44:50.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.079+0000 7f0d91400640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d74077f70 0x7f0d7407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:50.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.079+0000 7f0d91400640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d74077f70 0x7f0d7407a430 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0d80034000 tx=0x7f0d80004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:50.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.082+0000 7f0d8a000640 1 -- 172.21.15.31:0/496089023 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0d840617d0 con 0x7f0d940bf240 2024-09-11T00:44:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.354+0000 7f0d93200640 1 -- 172.21.15.31:0/496089023 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d940c5f50 con 0x7f0d940bf240 2024-09-11T00:44:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.355+0000 7f0d8a000640 1 -- 172.21.15.31:0/496089023 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5716 (secure 0 0 0) 0x7f0d84066200 con 0x7f0d940bf240 2024-09-11T00:44:50.358 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:50.358 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":106,"btime":"2024-09-11T00:44:47:708766+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:44:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 106 2024-09-11T00:44:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 -- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d74077f70 msgr2=0x7f0d7407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d74077f70 0x7f0d7407a430 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0d80034000 tx=0x7f0d80004730 comp rx=0 tx=0).stop 2024-09-11T00:44:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 -- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d940bf240 msgr2=0x7f0d9414eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d940bf240 0x7f0d9414eba0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c3c0 tx=0x7f0d8400c880 comp rx=0 tx=0).stop 2024-09-11T00:44:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 -- 172.21.15.31:0/496089023 shutdown_connections 2024-09-11T00:44:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d74077f70 0x7f0d7407a430 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:50.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94156e40 0x7f0d941ef6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:50.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0d940c6730 0x7f0d940c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:50.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 --2- 172.21.15.31:0/496089023 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0d940bf240 0x7f0d9414eba0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:50.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 -- 172.21.15.31:0/496089023 >> 172.21.15.31:0/496089023 conn(0x7f0d94068ac0 msgr2=0x7f0d940af0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:50.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 -- 172.21.15.31:0/496089023 shutdown_connections 2024-09-11T00:44:50.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:50.358+0000 7f0d93200640 1 -- 172.21.15.31:0/496089023 wait complete. 2024-09-11T00:44:50.378 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-11T00:44:51.672 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:44:51.670+0000 7f4ca5c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:44:52.381 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:44:52.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.554+0000 7ff80b200640 1 -- 172.21.15.31:0/3786324330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c14c880 msgr2=0x7ff80c0c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:52.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.554+0000 7ff80b200640 1 --2- 172.21.15.31:0/3786324330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c14c880 0x7ff80c0c8ac0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc005fd0 tx=0x7ff7fc02a5d0 comp rx=0 tx=0).stop 2024-09-11T00:44:52.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.554+0000 7ff80b200640 1 -- 172.21.15.31:0/3786324330 shutdown_connections 2024-09-11T00:44:52.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.554+0000 7ff80b200640 1 --2- 172.21.15.31:0/3786324330 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff80c06cc10 0x7ff80c159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:52.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.554+0000 7ff80b200640 1 --2- 172.21.15.31:0/3786324330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c14c880 0x7ff80c0c8ac0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:52.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.554+0000 7ff80b200640 1 -- 172.21.15.31:0/3786324330 >> 172.21.15.31:0/3786324330 conn(0x7ff80c0adb00 msgr2=0x7ff80c0adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:52.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.554+0000 7ff80b200640 1 -- 172.21.15.31:0/3786324330 shutdown_connections 2024-09-11T00:44:52.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.554+0000 7ff80b200640 1 -- 172.21.15.31:0/3786324330 wait complete. 2024-09-11T00:44:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.556+0000 7ff80b200640 1 Processor -- start 2024-09-11T00:44:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.556+0000 7ff80b200640 1 -- start start 2024-09-11T00:44:52.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.557+0000 7ff80b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff80c06cc10 0x7ff80c154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:52.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.557+0000 7ff80b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0c9000 0x7ff80c1475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:52.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.557+0000 7ff80b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff80c14c880 0x7ff80c147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:52.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.557+0000 7ff80b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c14cc60 con 0x7ff80c0c9000 2024-09-11T00:44:52.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.557+0000 7ff80b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c156810 con 0x7ff80c14c880 2024-09-11T00:44:52.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.557+0000 7ff80b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff80c06edc0 con 0x7ff80c06cc10 2024-09-11T00:44:52.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.557+0000 7ff809400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0c9000 0x7ff80c1475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:52.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.557+0000 7ff809e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff80c06cc10 0x7ff80c154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:52.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.557+0000 7ff809400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0c9000 0x7ff80c1475e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45906/0 (socket says 172.21.15.31:45906) 2024-09-11T00:44:52.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.557+0000 7ff809400640 1 -- 172.21.15.31:0/2228502522 learned_addr learned my addr 172.21.15.31:0/2228502522 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:44:52.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.558+0000 7ff80a800640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff80c14c880 0x7ff80c147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:52.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.558+0000 7ff809e00640 1 -- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff80c14c880 msgr2=0x7ff80c147b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:52.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.558+0000 7ff809e00640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff80c14c880 0x7ff80c147b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:52.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.558+0000 7ff809e00640 1 -- 172.21.15.31:0/2228502522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0c9000 msgr2=0x7ff80c1475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:52.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.558+0000 7ff809e00640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0c9000 0x7ff80c1475e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:52.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.558+0000 7ff809e00640 1 -- 172.21.15.31:0/2228502522 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c159f40 con 0x7ff80c06cc10 2024-09-11T00:44:52.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.558+0000 7ff809400640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0c9000 0x7ff80c1475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:44:52.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.558+0000 7ff809e00640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff80c06cc10 0x7ff80c154f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc031000 tx=0x7ff7fc005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.559+0000 7ff802000640 1 -- 172.21.15.31:0/2228502522 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc03e030 con 0x7ff80c06cc10 2024-09-11T00:44:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.559+0000 7ff802000640 1 -- 172.21.15.31:0/2228502522 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7fc003a70 con 0x7ff80c06cc10 2024-09-11T00:44:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.559+0000 7ff80b200640 1 -- 172.21.15.31:0/2228502522 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80c157870 con 0x7ff80c06cc10 2024-09-11T00:44:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.559+0000 7ff802000640 1 -- 172.21.15.31:0/2228502522 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc035040 con 0x7ff80c06cc10 2024-09-11T00:44:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.559+0000 7ff80b200640 1 -- 172.21.15.31:0/2228502522 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80c1eff70 con 0x7ff80c06cc10 2024-09-11T00:44:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.560+0000 7ff80b200640 1 -- 172.21.15.31:0/2228502522 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff80c0c59a0 con 0x7ff80c06cc10 2024-09-11T00:44:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.562+0000 7ff802000640 1 -- 172.21.15.31:0/2228502522 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff7fc031400 con 0x7ff80c06cc10 2024-09-11T00:44:52.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.563+0000 7ff802000640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff7ec077f70 0x7ff7ec07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:44:52.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.563+0000 7ff802000640 1 -- 172.21.15.31:0/2228502522 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7ff7fc0bb320 con 0x7ff80c06cc10 2024-09-11T00:44:52.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.563+0000 7ff809400640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff7ec077f70 0x7ff7ec07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:44:52.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.564+0000 7ff809400640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff7ec077f70 0x7ff7ec07a430 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff80c06f210 tx=0x7ff7f8005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:44:52.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.566+0000 7ff802000640 1 -- 172.21.15.31:0/2228502522 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff7fc082e10 con 0x7ff80c06cc10 2024-09-11T00:44:52.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.814+0000 7ff80b200640 1 -- 172.21.15.31:0/2228502522 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff80c153300 con 0x7ff80c06cc10 2024-09-11T00:44:52.818 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:44:52.818 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"btime":"2024-09-11T00:44:50:427343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:44:52.818 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 107 2024-09-11T00:44:52.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.815+0000 7ff802000640 1 -- 172.21.15.31:0/2228502522 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6542 (secure 0 0 0) 0x7ff7fc087840 con 0x7ff80c06cc10 2024-09-11T00:44:52.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 -- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff7ec077f70 msgr2=0x7ff7ec07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:52.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff7ec077f70 0x7ff7ec07a430 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff80c06f210 tx=0x7ff7f8005f90 comp rx=0 tx=0).stop 2024-09-11T00:44:52.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 -- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff80c06cc10 msgr2=0x7ff80c154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:44:52.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff80c06cc10 0x7ff80c154f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc031000 tx=0x7ff7fc005750 comp rx=0 tx=0).stop 2024-09-11T00:44:52.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 -- 172.21.15.31:0/2228502522 shutdown_connections 2024-09-11T00:44:52.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff7ec077f70 0x7ff7ec07a430 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:52.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff80c14c880 0x7ff80c147b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:52.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff80c0c9000 0x7ff80c1475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:52.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 --2- 172.21.15.31:0/2228502522 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff80c06cc10 0x7ff80c154f60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:44:52.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 -- 172.21.15.31:0/2228502522 >> 172.21.15.31:0/2228502522 conn(0x7ff80c0adb00 msgr2=0x7ff80c0acf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:44:52.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.817+0000 7ff80b200640 1 -- 172.21.15.31:0/2228502522 shutdown_connections 2024-09-11T00:44:52.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:44:52.818+0000 7ff80b200640 1 -- 172.21.15.31:0/2228502522 wait complete. 2024-09-11T00:44:52.942 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-11T00:44:52.942 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before thrashing 2024-09-11T00:45:01.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.130+0000 7f963d200640 1 -- 172.21.15.31:0/2294560033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96380cc220 msgr2=0x7f963806e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.130+0000 7f963d200640 1 --2- 172.21.15.31:0/2294560033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96380cc220 0x7f963806e680 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9630005fd0 tx=0x7f963002aa10 comp rx=0 tx=0).stop 2024-09-11T00:45:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.131+0000 7f963d200640 1 -- 172.21.15.31:0/2294560033 shutdown_connections 2024-09-11T00:45:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.131+0000 7f963d200640 1 --2- 172.21.15.31:0/2294560033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96380cc220 0x7f963806e680 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.131+0000 7f963d200640 1 --2- 172.21.15.31:0/2294560033 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9638156960 0x7f963806e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.131+0000 7f963d200640 1 -- 172.21.15.31:0/2294560033 >> 172.21.15.31:0/2294560033 conn(0x7f9638068f10 msgr2=0x7f96380cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.131+0000 7f963d200640 1 -- 172.21.15.31:0/2294560033 shutdown_connections 2024-09-11T00:45:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.131+0000 7f963d200640 1 -- 172.21.15.31:0/2294560033 wait complete. 2024-09-11T00:45:02.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.132+0000 7f963d200640 1 Processor -- start 2024-09-11T00:45:02.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.132+0000 7f963d200640 1 -- start start 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f963d200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96380cc220 0x7f96380b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f963d200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638150020 0x7f96380c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f963d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9638156960 0x7f96380c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f963d200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96380c6ca0 con 0x7f9638156960 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f963d200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f96380c6b20 con 0x7f9638150020 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f963d200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f96381573d0 con 0x7f96380cc220 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f9637e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9638156960 0x7f96380c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f9637400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96380cc220 0x7f96380b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f9637e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9638156960 0x7f96380c45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57912/0 (socket says 172.21.15.31:57912) 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f9637e00640 1 -- 172.21.15.31:0/1370445447 learned_addr learned my addr 172.21.15.31:0/1370445447 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:02.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f9636a00640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638150020 0x7f96380c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f9637400640 1 -- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638150020 msgr2=0x7f96380c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f9637400640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638150020 0x7f96380c4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f9637400640 1 -- 172.21.15.31:0/1370445447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9638156960 msgr2=0x7f96380c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f9637400640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9638156960 0x7f96380c45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.134+0000 7f9637400640 1 -- 172.21.15.31:0/1370445447 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96380aef30 con 0x7f96380cc220 2024-09-11T00:45:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.135+0000 7f9637400640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96380cc220 0x7f96380b0210 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f962800c3f0 tx=0x7f962800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.135+0000 7f962f400640 1 -- 172.21.15.31:0/1370445447 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628017070 con 0x7f96380cc220 2024-09-11T00:45:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.135+0000 7f962f400640 1 -- 172.21.15.31:0/1370445447 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96280053c0 con 0x7f96380cc220 2024-09-11T00:45:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.135+0000 7f962f400640 1 -- 172.21.15.31:0/1370445447 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628003850 con 0x7f96380cc220 2024-09-11T00:45:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.135+0000 7f963d200640 1 -- 172.21.15.31:0/1370445447 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96380af210 con 0x7f96380cc220 2024-09-11T00:45:02.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.135+0000 7f963d200640 1 -- 172.21.15.31:0/1370445447 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96380c4080 con 0x7f96380cc220 2024-09-11T00:45:02.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.136+0000 7f963d200640 1 -- 172.21.15.31:0/1370445447 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96380af730 con 0x7f96380cc220 2024-09-11T00:45:02.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.138+0000 7f962f400640 1 -- 172.21.15.31:0/1370445447 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f96280039f0 con 0x7f96380cc220 2024-09-11T00:45:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.139+0000 7f962f400640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9618077f70 0x7f961807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.139+0000 7f962f400640 1 -- 172.21.15.31:0/1370445447 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7f9628099d30 con 0x7f96380cc220 2024-09-11T00:45:02.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.139+0000 7f9636a00640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9618077f70 0x7f961807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:02.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.140+0000 7f9636a00640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9618077f70 0x7f961807a430 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f96381e9d40 tx=0x7f9624005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:02.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.143+0000 7f962f400640 1 -- 172.21.15.31:0/1370445447 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f96280617a0 con 0x7f96380cc220 2024-09-11T00:45:02.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.397+0000 7f963d200640 1 -- 172.21.15.31:0/1370445447 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96380ab280 con 0x7f96380cc220 2024-09-11T00:45:02.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.398+0000 7f962f400640 1 -- 172.21.15.31:0/1370445447 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6542 (secure 0 0 0) 0x7f96380ab280 con 0x7f96380cc220 2024-09-11T00:45:02.400 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:02.401 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"btime":"2024-09-11T00:44:50:427343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:45:02.401 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 107 2024-09-11T00:45:02.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 -- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9618077f70 msgr2=0x7f961807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:02.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9618077f70 0x7f961807a430 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f96381e9d40 tx=0x7f9624005f90 comp rx=0 tx=0).stop 2024-09-11T00:45:02.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 -- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96380cc220 msgr2=0x7f96380b0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:02.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96380cc220 0x7f96380b0210 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f962800c3f0 tx=0x7f962800c8b0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 -- 172.21.15.31:0/1370445447 shutdown_connections 2024-09-11T00:45:02.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9618077f70 0x7f961807a430 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9638156960 0x7f96380c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638150020 0x7f96380c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 --2- 172.21.15.31:0/1370445447 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f96380cc220 0x7f96380b0210 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 -- 172.21.15.31:0/1370445447 >> 172.21.15.31:0/1370445447 conn(0x7f9638068f10 msgr2=0x7f9638153ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 -- 172.21.15.31:0/1370445447 shutdown_connections 2024-09-11T00:45:02.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.400+0000 7f963d200640 1 -- 172.21.15.31:0/1370445447 wait complete. 2024-09-11T00:45:02.420 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-11T00:45:02.420 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:45:02.420 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:45:02.438 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:45:02.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.631+0000 7f582b200640 1 -- 172.21.15.31:0/2525673721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f582c071f00 msgr2=0x7f582c072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.631+0000 7f582b200640 1 --2- 172.21.15.31:0/2525673721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f582c071f00 0x7f582c072390 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f582c0c7610 tx=0x7f582402aac0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.631+0000 7f582b200640 1 -- 172.21.15.31:0/2525673721 shutdown_connections 2024-09-11T00:45:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.631+0000 7f582b200640 1 --2- 172.21.15.31:0/2525673721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f582c071f00 0x7f582c072390 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.631+0000 7f582b200640 1 --2- 172.21.15.31:0/2525673721 >> v2:172.21.15.204:3300/0 conn(0x7f582c0c7bc0 0x7f582c0c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.631+0000 7f582b200640 1 -- 172.21.15.31:0/2525673721 >> 172.21.15.31:0/2525673721 conn(0x7f582c0c9a50 msgr2=0x7f582c0c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.631+0000 7f582b200640 1 -- 172.21.15.31:0/2525673721 shutdown_connections 2024-09-11T00:45:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.631+0000 7f582b200640 1 -- 172.21.15.31:0/2525673721 wait complete. 2024-09-11T00:45:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.633+0000 7f582b200640 1 Processor -- start 2024-09-11T00:45:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.633+0000 7f582b200640 1 -- start start 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f582b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f582c071f00 0x7f582c1ef070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f582b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f582c0c3cd0 0x7f582c14e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f582b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f582c0c7bc0 0x7f582c1ecd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f582b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f582c1546d0 con 0x7f582c071f00 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f582b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f582c0c8a80 con 0x7f582c0c3cd0 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f582b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f582c154850 con 0x7f582c0c7bc0 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f5829e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f582c071f00 0x7f582c1ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f5829e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f582c071f00 0x7f582c1ef070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57920/0 (socket says 172.21.15.31:57920) 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f5829e00640 1 -- 172.21.15.31:0/294761359 learned_addr learned my addr 172.21.15.31:0/294761359 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f5829400640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f582c0c3cd0 0x7f582c14e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:02.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f582a800640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f582c0c7bc0 0x7f582c1ecd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f5829e00640 1 -- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f582c0c7bc0 msgr2=0x7f582c1ecd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f5829e00640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f582c0c7bc0 0x7f582c1ecd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.634+0000 7f5829e00640 1 -- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f582c0c3cd0 msgr2=0x7f582c14e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.635+0000 7f5829e00640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f582c0c3cd0 0x7f582c14e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.635+0000 7f5829e00640 1 -- 172.21.15.31:0/294761359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f582c0a9bd0 con 0x7f582c071f00 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.635+0000 7f5829e00640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f582c071f00 0x7f582c1ef070 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f581c00e790 tx=0x7f581c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.635+0000 7f5822000640 1 -- 172.21.15.31:0/294761359 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581c00eea0 con 0x7f582c071f00 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.635+0000 7f582b200640 1 -- 172.21.15.31:0/294761359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f582c14eee0 con 0x7f582c071f00 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.635+0000 7f5822000640 1 -- 172.21.15.31:0/294761359 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f581c00cd50 con 0x7f582c071f00 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.635+0000 7f5822000640 1 -- 172.21.15.31:0/294761359 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581c010690 con 0x7f582c071f00 2024-09-11T00:45:02.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.635+0000 7f582b200640 1 -- 172.21.15.31:0/294761359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f582c152460 con 0x7f582c071f00 2024-09-11T00:45:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.636+0000 7f582b200640 1 -- 172.21.15.31:0/294761359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f582c0c7fa0 con 0x7f582c071f00 2024-09-11T00:45:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.637+0000 7f5822000640 1 -- 172.21.15.31:0/294761359 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f581c010830 con 0x7f582c071f00 2024-09-11T00:45:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.638+0000 7f5822000640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f580c077f70 0x7f580c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.638+0000 7f5822000640 1 -- 172.21.15.31:0/294761359 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7f581c09b0a0 con 0x7f582c071f00 2024-09-11T00:45:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.639+0000 7f5829400640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f580c077f70 0x7f580c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.640+0000 7f5829400640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f580c077f70 0x7f580c07a430 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f582c06f9d0 tx=0x7f5818009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:02.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.643+0000 7f5822000640 1 -- 172.21.15.31:0/294761359 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f581c062b10 con 0x7f582c071f00 2024-09-11T00:45:02.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.964+0000 7f582b200640 1 -- 172.21.15.31:0/294761359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f582c0c6840 con 0x7f582c071f00 2024-09-11T00:45:02.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.965+0000 7f5822000640 1 -- 172.21.15.31:0/294761359 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6542 (secure 0 0 0) 0x7f581c067540 con 0x7f582c071f00 2024-09-11T00:45:02.968 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:02.968 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"btime":"2024-09-11T00:44:50:427343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:45:02.968 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 107 2024-09-11T00:45:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 -- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f580c077f70 msgr2=0x7f580c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f580c077f70 0x7f580c07a430 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f582c06f9d0 tx=0x7f5818009040 comp rx=0 tx=0).stop 2024-09-11T00:45:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 -- 172.21.15.31:0/294761359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f582c071f00 msgr2=0x7f582c1ef070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f582c071f00 0x7f582c1ef070 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f581c00e790 tx=0x7f581c00ec50 comp rx=0 tx=0).stop 2024-09-11T00:45:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 -- 172.21.15.31:0/294761359 shutdown_connections 2024-09-11T00:45:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f580c077f70 0x7f580c07a430 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f582c0c7bc0 0x7f582c1ecd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f582c0c3cd0 0x7f582c14e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 --2- 172.21.15.31:0/294761359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f582c071f00 0x7f582c1ef070 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:02.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 -- 172.21.15.31:0/294761359 >> 172.21.15.31:0/294761359 conn(0x7f582c0c9a50 msgr2=0x7f582c0c2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:02.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 -- 172.21.15.31:0/294761359 shutdown_connections 2024-09-11T00:45:02.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:02.968+0000 7f582b200640 1 -- 172.21.15.31:0/294761359 wait complete. 2024-09-11T00:45:02.994 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:45:04.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.175+0000 7fd5dfa00640 1 -- 172.21.15.31:0/1340285241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5d8153f20 msgr2=0x7fd5d81516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.175+0000 7fd5dfa00640 1 --2- 172.21.15.31:0/1340285241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5d8153f20 0x7fd5d81516e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0005fd0 tx=0x7fd5d002a670 comp rx=0 tx=0).stop 2024-09-11T00:45:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.175+0000 7fd5dfa00640 1 -- 172.21.15.31:0/1340285241 shutdown_connections 2024-09-11T00:45:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.175+0000 7fd5dfa00640 1 --2- 172.21.15.31:0/1340285241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5d8153f20 0x7fd5d81516e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.175+0000 7fd5dfa00640 1 --2- 172.21.15.31:0/1340285241 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5d8072480 0x7fd5d81511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.175+0000 7fd5dfa00640 1 -- 172.21.15.31:0/1340285241 >> 172.21.15.31:0/1340285241 conn(0x7fd5d80a9750 msgr2=0x7fd5d80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.175+0000 7fd5dfa00640 1 -- 172.21.15.31:0/1340285241 shutdown_connections 2024-09-11T00:45:05.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.175+0000 7fd5dfa00640 1 -- 172.21.15.31:0/1340285241 wait complete. 2024-09-11T00:45:05.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.177+0000 7fd5dfa00640 1 Processor -- start 2024-09-11T00:45:05.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.177+0000 7fd5dfa00640 1 -- start start 2024-09-11T00:45:05.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dfa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5d8072480 0x7fd5d81f0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:05.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dfa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5d80c23a0 0x7fd5d806e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:05.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dfa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5d8153f20 0x7fd5d81466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:05.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dfa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d8074200 con 0x7fd5d8072480 2024-09-11T00:45:05.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dfa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d8074080 con 0x7fd5d8153f20 2024-09-11T00:45:05.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dfa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd5d8072860 con 0x7fd5d80c23a0 2024-09-11T00:45:05.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dd000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5d8072480 0x7fd5d81f0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dd000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5d8072480 0x7fd5d81f0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57938/0 (socket says 172.21.15.31:57938) 2024-09-11T00:45:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dd000640 1 -- 172.21.15.31:0/53628957 learned_addr learned my addr 172.21.15.31:0/53628957 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5d7e00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5d80c23a0 0x7fd5d806e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dda00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5d8153f20 0x7fd5d81466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dd000640 1 -- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5d80c23a0 msgr2=0x7fd5d806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dd000640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5d80c23a0 0x7fd5d806e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dd000640 1 -- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5d8153f20 msgr2=0x7fd5d81466f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dd000640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5d8153f20 0x7fd5d81466f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:05.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.178+0000 7fd5dd000640 1 -- 172.21.15.31:0/53628957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d8148930 con 0x7fd5d8072480 2024-09-11T00:45:05.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.179+0000 7fd5dd000640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5d8072480 0x7fd5d81f0080 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800db80 tx=0x7fd5c800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:05.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.179+0000 7fd5d4c00640 1 -- 172.21.15.31:0/53628957 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8017070 con 0x7fd5d8072480 2024-09-11T00:45:05.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.179+0000 7fd5d4c00640 1 -- 172.21.15.31:0/53628957 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5c8003c60 con 0x7fd5d8072480 2024-09-11T00:45:05.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.179+0000 7fd5d4c00640 1 -- 172.21.15.31:0/53628957 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c800be60 con 0x7fd5d8072480 2024-09-11T00:45:05.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.179+0000 7fd5dfa00640 1 -- 172.21.15.31:0/53628957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d80c44f0 con 0x7fd5d8072480 2024-09-11T00:45:05.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.179+0000 7fd5dfa00640 1 -- 172.21.15.31:0/53628957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d81ef5a0 con 0x7fd5d8072480 2024-09-11T00:45:05.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.180+0000 7fd5dfa00640 1 -- 172.21.15.31:0/53628957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5d81511a0 con 0x7fd5d8072480 2024-09-11T00:45:05.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.182+0000 7fd5d4c00640 1 -- 172.21.15.31:0/53628957 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd5c8004280 con 0x7fd5d8072480 2024-09-11T00:45:05.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.183+0000 7fd5d4c00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd5b4077fa0 0x7fd5b407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:05.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.184+0000 7fd5d4c00640 1 -- 172.21.15.31:0/53628957 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7fd5c809ad20 con 0x7fd5d8072480 2024-09-11T00:45:05.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.184+0000 7fd5d7e00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd5b4077fa0 0x7fd5b407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:05.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.185+0000 7fd5d7e00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd5b4077fa0 0x7fd5b407a460 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd5d814ca80 tx=0x7fd5c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:05.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.188+0000 7fd5d4c00640 1 -- 172.21.15.31:0/53628957 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd5c8020030 con 0x7fd5d8072480 2024-09-11T00:45:05.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.476+0000 7fd5dfa00640 1 -- 172.21.15.31:0/53628957 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5d814c4f0 con 0x7fd5d8072480 2024-09-11T00:45:05.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.477+0000 7fd5d4c00640 1 -- 172.21.15.31:0/53628957 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6542 (secure 0 0 0) 0x7fd5c8062790 con 0x7fd5d8072480 2024-09-11T00:45:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:05.479 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"btime":"2024-09-11T00:44:50:427343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:45:05.479 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 107 2024-09-11T00:45:05.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.479+0000 7fd5dfa00640 1 -- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd5b4077fa0 msgr2=0x7fd5b407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:05.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.479+0000 7fd5dfa00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd5b4077fa0 0x7fd5b407a460 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd5d814ca80 tx=0x7fd5c0009040 comp rx=0 tx=0).stop 2024-09-11T00:45:05.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.479+0000 7fd5dfa00640 1 -- 172.21.15.31:0/53628957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5d8072480 msgr2=0x7fd5d81f0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:05.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.479+0000 7fd5dfa00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5d8072480 0x7fd5d81f0080 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800db80 tx=0x7fd5c800b630 comp rx=0 tx=0).stop 2024-09-11T00:45:05.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.480+0000 7fd5dfa00640 1 -- 172.21.15.31:0/53628957 shutdown_connections 2024-09-11T00:45:05.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.480+0000 7fd5dfa00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd5b4077fa0 0x7fd5b407a460 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:05.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.480+0000 7fd5dfa00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd5d8153f20 0x7fd5d81466f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:05.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.480+0000 7fd5dfa00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd5d80c23a0 0x7fd5d806e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:05.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.480+0000 7fd5dfa00640 1 --2- 172.21.15.31:0/53628957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5d8072480 0x7fd5d81f0080 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:05.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.480+0000 7fd5dfa00640 1 -- 172.21.15.31:0/53628957 >> 172.21.15.31:0/53628957 conn(0x7fd5d80a9750 msgr2=0x7fd5d80a8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:05.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.480+0000 7fd5dfa00640 1 -- 172.21.15.31:0/53628957 shutdown_connections 2024-09-11T00:45:05.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:05.480+0000 7fd5dfa00640 1 -- 172.21.15.31:0/53628957 wait complete. 2024-09-11T00:45:05.498 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:45:07.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:07.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.679+0000 7fa199200640 1 -- 172.21.15.31:0/2148251072 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa194072480 msgr2=0x7fa1941511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:07.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.679+0000 7fa199200640 1 --2- 172.21.15.31:0/2148251072 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa194072480 0x7fa1941511a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa184009830 tx=0x7fa18402a5a0 comp rx=0 tx=0).stop 2024-09-11T00:45:07.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.679+0000 7fa199200640 1 -- 172.21.15.31:0/2148251072 shutdown_connections 2024-09-11T00:45:07.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.679+0000 7fa199200640 1 --2- 172.21.15.31:0/2148251072 >> v2:172.21.15.204:3300/0 conn(0x7fa194153f20 0x7fa1941516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:07.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.679+0000 7fa199200640 1 --2- 172.21.15.31:0/2148251072 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa194072480 0x7fa1941511a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:07.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.679+0000 7fa199200640 1 -- 172.21.15.31:0/2148251072 >> 172.21.15.31:0/2148251072 conn(0x7fa1940a9750 msgr2=0x7fa1940a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:07.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.679+0000 7fa199200640 1 -- 172.21.15.31:0/2148251072 shutdown_connections 2024-09-11T00:45:07.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.679+0000 7fa199200640 1 -- 172.21.15.31:0/2148251072 wait complete. 2024-09-11T00:45:07.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.681+0000 7fa199200640 1 Processor -- start 2024-09-11T00:45:07.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.681+0000 7fa199200640 1 -- start start 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.682+0000 7fa199200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa194072480 0x7fa1940c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.682+0000 7fa199200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1940c23a0 0x7fa1941ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.682+0000 7fa199200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa194153f20 0x7fa1941ecb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.682+0000 7fa199200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa194074200 con 0x7fa194072480 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.682+0000 7fa199200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa194074080 con 0x7fa1940c23a0 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.682+0000 7fa199200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa194072860 con 0x7fa194153f20 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.682+0000 7fa191e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa194072480 0x7fa1940c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.683+0000 7fa191e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa194072480 0x7fa1940c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57948/0 (socket says 172.21.15.31:57948) 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.683+0000 7fa191e00640 1 -- 172.21.15.31:0/3990738244 learned_addr learned my addr 172.21.15.31:0/3990738244 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.683+0000 7fa189400640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1940c23a0 0x7fa1941ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.683+0000 7fa192800640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa194153f20 0x7fa1941ecb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:07.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.683+0000 7fa191e00640 1 -- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa194153f20 msgr2=0x7fa1941ecb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:07.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.683+0000 7fa191e00640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa194153f20 0x7fa1941ecb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:07.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.683+0000 7fa191e00640 1 -- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1940c23a0 msgr2=0x7fa1941ec5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:07.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.683+0000 7fa191e00640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1940c23a0 0x7fa1941ec5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:07.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.683+0000 7fa191e00640 1 -- 172.21.15.31:0/3990738244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1941ec110 con 0x7fa194072480 2024-09-11T00:45:07.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.683+0000 7fa191e00640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa194072480 0x7fa1940c1040 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa184000c00 tx=0x7fa184032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:07.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.684+0000 7fa18aa00640 1 -- 172.21.15.31:0/3990738244 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184040070 con 0x7fa194072480 2024-09-11T00:45:07.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.684+0000 7fa18aa00640 1 -- 172.21.15.31:0/3990738244 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1840038e0 con 0x7fa194072480 2024-09-11T00:45:07.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.684+0000 7fa18aa00640 1 -- 172.21.15.31:0/3990738244 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184005c90 con 0x7fa194072480 2024-09-11T00:45:07.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.684+0000 7fa199200640 1 -- 172.21.15.31:0/3990738244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1941ed050 con 0x7fa194072480 2024-09-11T00:45:07.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.684+0000 7fa199200640 1 -- 172.21.15.31:0/3990738244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1941efee0 con 0x7fa194072480 2024-09-11T00:45:07.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.685+0000 7fa199200640 1 -- 172.21.15.31:0/3990738244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1941511a0 con 0x7fa194072480 2024-09-11T00:45:07.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.687+0000 7fa18aa00640 1 -- 172.21.15.31:0/3990738244 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa18404f480 con 0x7fa194072480 2024-09-11T00:45:07.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.688+0000 7fa18aa00640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa170077f70 0x7fa17007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:07.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.688+0000 7fa18aa00640 1 -- 172.21.15.31:0/3990738244 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7fa1840bc150 con 0x7fa194072480 2024-09-11T00:45:07.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.689+0000 7fa189400640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa170077f70 0x7fa17007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:07.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.690+0000 7fa189400640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa170077f70 0x7fa17007a430 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa17c007fd0 tx=0x7fa17c001410 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:07.697+0000 7fa18aa00640 1 -- 172.21.15.31:0/3990738244 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa1941ec110 con 0x7fa194072480 2024-09-11T00:45:08.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.018+0000 7fa199200640 1 -- 172.21.15.31:0/3990738244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa194151d00 con 0x7fa194072480 2024-09-11T00:45:08.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.018+0000 7fa18aa00640 1 -- 172.21.15.31:0/3990738244 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6542 (secure 0 0 0) 0x7fa184083c40 con 0x7fa194072480 2024-09-11T00:45:08.020 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:08.020 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"btime":"2024-09-11T00:44:50:427343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:45:08.021 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 107 2024-09-11T00:45:08.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.021+0000 7fa199200640 1 -- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa170077f70 msgr2=0x7fa17007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:08.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.021+0000 7fa199200640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa170077f70 0x7fa17007a430 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa17c007fd0 tx=0x7fa17c001410 comp rx=0 tx=0).stop 2024-09-11T00:45:08.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.021+0000 7fa199200640 1 -- 172.21.15.31:0/3990738244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa194072480 msgr2=0x7fa1940c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:08.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.021+0000 7fa199200640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa194072480 0x7fa1940c1040 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa184000c00 tx=0x7fa184032c10 comp rx=0 tx=0).stop 2024-09-11T00:45:08.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.021+0000 7fa199200640 1 -- 172.21.15.31:0/3990738244 shutdown_connections 2024-09-11T00:45:08.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.021+0000 7fa199200640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa170077f70 0x7fa17007a430 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:08.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.021+0000 7fa199200640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa194153f20 0x7fa1941ecb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:08.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.021+0000 7fa199200640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1940c23a0 0x7fa1941ec5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:08.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.021+0000 7fa199200640 1 --2- 172.21.15.31:0/3990738244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa194072480 0x7fa1940c1040 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:08.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.021+0000 7fa199200640 1 -- 172.21.15.31:0/3990738244 >> 172.21.15.31:0/3990738244 conn(0x7fa1940a9750 msgr2=0x7fa194146ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:08.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.022+0000 7fa199200640 1 -- 172.21.15.31:0/3990738244 shutdown_connections 2024-09-11T00:45:08.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:08.022+0000 7fa199200640 1 -- 172.21.15.31:0/3990738244 wait complete. 2024-09-11T00:45:08.040 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:45:10.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:10.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:10.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.223+0000 7f419b200640 1 -- 172.21.15.31:0/78831873 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f419c151600 msgr2=0x7f419c0c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.223+0000 7f419b200640 1 --2- 172.21.15.31:0/78831873 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f419c151600 0x7f419c0c61b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4194005fd0 tx=0x7f419402a780 comp rx=0 tx=0).stop 2024-09-11T00:45:10.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.223+0000 7f419b200640 1 -- 172.21.15.31:0/78831873 shutdown_connections 2024-09-11T00:45:10.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.223+0000 7f419b200640 1 --2- 172.21.15.31:0/78831873 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f419c151600 0x7f419c0c61b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.223+0000 7f419b200640 1 --2- 172.21.15.31:0/78831873 >> v2:172.21.15.204:3300/0 conn(0x7f419c0a6e20 0x7f419c1510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.223+0000 7f419b200640 1 -- 172.21.15.31:0/78831873 >> 172.21.15.31:0/78831873 conn(0x7f419c06caf0 msgr2=0x7f419c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:10.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.223+0000 7f419b200640 1 -- 172.21.15.31:0/78831873 shutdown_connections 2024-09-11T00:45:10.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.223+0000 7f419b200640 1 -- 172.21.15.31:0/78831873 wait complete. 2024-09-11T00:45:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.225+0000 7f419b200640 1 Processor -- start 2024-09-11T00:45:10.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.225+0000 7f419b200640 1 -- start start 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f419b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f419c0a6e20 0x7f419c14cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f419b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f419c0ac2c0 0x7f419c0728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f419b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f419c151600 0x7f419c1ef250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f419b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f419c0acf30 con 0x7f419c151600 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f419b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f419c0acdb0 con 0x7f419c0a6e20 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f419b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f419c0c66f0 con 0x7f419c0ac2c0 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f419a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f419c151600 0x7f419c1ef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f419a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f419c151600 0x7f419c1ef250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46698/0 (socket says 172.21.15.31:46698) 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f419a800640 1 -- 172.21.15.31:0/2092795311 learned_addr learned my addr 172.21.15.31:0/2092795311 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f4199e00640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f419c0a6e20 0x7f419c14cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.226+0000 7f4199400640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f419c0ac2c0 0x7f419c0728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.227+0000 7f419a800640 1 -- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f419c0ac2c0 msgr2=0x7f419c0728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.227+0000 7f419a800640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f419c0ac2c0 0x7f419c0728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.227+0000 7f419a800640 1 -- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f419c0a6e20 msgr2=0x7f419c14cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.227+0000 7f419a800640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f419c0a6e20 0x7f419c14cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.227+0000 7f419a800640 1 -- 172.21.15.31:0/2092795311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419c14c8a0 con 0x7f419c151600 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.227+0000 7f419a800640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f419c151600 0x7f419c1ef250 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4194034040 tx=0x7f4194003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.228+0000 7f4192000640 1 -- 172.21.15.31:0/2092795311 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4194040070 con 0x7f419c151600 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.228+0000 7f4192000640 1 -- 172.21.15.31:0/2092795311 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4194004330 con 0x7f419c151600 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.228+0000 7f419b200640 1 -- 172.21.15.31:0/2092795311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419c1efb20 con 0x7f419c151600 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.228+0000 7f4192000640 1 -- 172.21.15.31:0/2092795311 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4194004740 con 0x7f419c151600 2024-09-11T00:45:10.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.228+0000 7f419b200640 1 -- 172.21.15.31:0/2092795311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f419c0c2230 con 0x7f419c151600 2024-09-11T00:45:10.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.229+0000 7f419b200640 1 -- 172.21.15.31:0/2092795311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f419c0c6b50 con 0x7f419c151600 2024-09-11T00:45:10.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.231+0000 7f4192000640 1 -- 172.21.15.31:0/2092795311 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f419403c420 con 0x7f419c151600 2024-09-11T00:45:10.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.232+0000 7f4192000640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f417c077f70 0x7f417c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.232+0000 7f4192000640 1 -- 172.21.15.31:0/2092795311 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7f41940bbbf0 con 0x7f419c151600 2024-09-11T00:45:10.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.233+0000 7f4199e00640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f417c077f70 0x7f417c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.234+0000 7f4199e00640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f417c077f70 0x7f417c07a430 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f418c007fd0 tx=0x7f418c001410 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:10.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.237+0000 7f4192000640 1 -- 172.21.15.31:0/2092795311 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f41940835d0 con 0x7f419c151600 2024-09-11T00:45:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.395+0000 7f797ca00640 1 -- 172.21.15.31:0/584526267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79780c4cb0 msgr2=0x7f7978074230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.395+0000 7f797ca00640 1 --2- 172.21.15.31:0/584526267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79780c4cb0 0x7f7978074230 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f79780c2f70 tx=0x7f796802a540 comp rx=0 tx=0).stop 2024-09-11T00:45:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.395+0000 7f797ca00640 1 -- 172.21.15.31:0/584526267 shutdown_connections 2024-09-11T00:45:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.395+0000 7f797ca00640 1 --2- 172.21.15.31:0/584526267 >> v2:172.21.15.204:3300/0 conn(0x7f79780c5090 0x7f797806caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.395+0000 7f797ca00640 1 --2- 172.21.15.31:0/584526267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79780c4cb0 0x7f7978074230 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.395+0000 7f797ca00640 1 -- 172.21.15.31:0/584526267 >> 172.21.15.31:0/584526267 conn(0x7f79780aa760 msgr2=0x7f79780aab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:10.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.395+0000 7f797ca00640 1 -- 172.21.15.31:0/584526267 shutdown_connections 2024-09-11T00:45:10.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.395+0000 7f797ca00640 1 -- 172.21.15.31:0/584526267 wait complete. 2024-09-11T00:45:10.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.397+0000 7f797ca00640 1 Processor -- start 2024-09-11T00:45:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.397+0000 7f797ca00640 1 -- start start 2024-09-11T00:45:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.398+0000 7f797ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79780c4cb0 0x7f79781551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.398+0000 7f797ca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79780c84e0 0x7f797814ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.398+0000 7f797ca00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7978157bd0 0x7f7978156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.398+0000 7f797ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f797806d1b0 con 0x7f79780c4cb0 2024-09-11T00:45:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.398+0000 7f797ca00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f797806d030 con 0x7f79780c84e0 2024-09-11T00:45:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.398+0000 7f797ca00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f797806d330 con 0x7f7978157bd0 2024-09-11T00:45:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.398+0000 7f7977400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79780c4cb0 0x7f79781551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.398+0000 7f7977400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79780c4cb0 0x7f79781551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46712/0 (socket says 172.21.15.31:46712) 2024-09-11T00:45:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.398+0000 7f7977400640 1 -- 172.21.15.31:0/3762266370 learned_addr learned my addr 172.21.15.31:0/3762266370 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.399+0000 7f7976a00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79780c84e0 0x7f797814ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.399+0000 7f7977e00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7978157bd0 0x7f7978156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.399+0000 7f7977400640 1 -- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7978157bd0 msgr2=0x7f7978156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.399+0000 7f7977400640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7978157bd0 0x7f7978156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.399+0000 7f7977400640 1 -- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79780c84e0 msgr2=0x7f797814ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.399+0000 7f7977400640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79780c84e0 0x7f797814ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.399+0000 7f7977400640 1 -- 172.21.15.31:0/3762266370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797814b120 con 0x7f79780c4cb0 2024-09-11T00:45:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.399+0000 7f7977e00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7978157bd0 0x7f7978156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:45:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.399+0000 7f7977400640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79780c4cb0 0x7f79781551a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f79680098f0 tx=0x7f7968032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.400+0000 7f796f400640 1 -- 172.21.15.31:0/3762266370 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7968040070 con 0x7f79780c4cb0 2024-09-11T00:45:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.400+0000 7f796f400640 1 -- 172.21.15.31:0/3762266370 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7968005310 con 0x7f79780c4cb0 2024-09-11T00:45:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.400+0000 7f796f400640 1 -- 172.21.15.31:0/3762266370 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7968005c90 con 0x7f79780c4cb0 2024-09-11T00:45:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.400+0000 7f797ca00640 1 -- 172.21.15.31:0/3762266370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79781ddb30 con 0x7f79780c4cb0 2024-09-11T00:45:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.400+0000 7f797ca00640 1 -- 172.21.15.31:0/3762266370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79781dd850 con 0x7f79780c4cb0 2024-09-11T00:45:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.401+0000 7f797ca00640 1 -- 172.21.15.31:0/3762266370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f797806a170 con 0x7f79780c4cb0 2024-09-11T00:45:10.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.404+0000 7f796f400640 1 -- 172.21.15.31:0/3762266370 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f79680049b0 con 0x7f79780c4cb0 2024-09-11T00:45:10.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.405+0000 7f796f400640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7958077f70 0x7f795807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.405+0000 7f796f400640 1 -- 172.21.15.31:0/3762266370 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7f79680bc070 con 0x7f79780c4cb0 2024-09-11T00:45:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.405+0000 7f7976a00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7958077f70 0x7f795807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.406+0000 7f7976a00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7958077f70 0x7f795807a430 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7978151140 tx=0x7f7964005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.407+0000 7f796f400640 1 -- 172.21.15.31:0/3762266370 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f797806a170 con 0x7f79780c4cb0 2024-09-11T00:45:10.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.516+0000 7f419b200640 1 -- 172.21.15.31:0/2092795311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f419c0aca90 con 0x7f419c151600 2024-09-11T00:45:10.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.517+0000 7f4192000640 1 -- 172.21.15.31:0/2092795311 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6542 (secure 0 0 0) 0x7f4194088000 con 0x7f419c151600 2024-09-11T00:45:10.519 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:10.519 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"btime":"2024-09-11T00:44:50:427343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:45:10.519 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 107 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 -- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f417c077f70 msgr2=0x7f417c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f417c077f70 0x7f417c07a430 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f418c007fd0 tx=0x7f418c001410 comp rx=0 tx=0).stop 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 -- 172.21.15.31:0/2092795311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f419c151600 msgr2=0x7f419c1ef250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f419c151600 0x7f419c1ef250 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4194034040 tx=0x7f4194003ba0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 -- 172.21.15.31:0/2092795311 shutdown_connections 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f417c077f70 0x7f417c07a430 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f419c151600 0x7f419c1ef250 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f419c0ac2c0 0x7f419c0728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 --2- 172.21.15.31:0/2092795311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f419c0a6e20 0x7f419c14cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 -- 172.21.15.31:0/2092795311 >> 172.21.15.31:0/2092795311 conn(0x7f419c06caf0 msgr2=0x7f419c151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:10.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 -- 172.21.15.31:0/2092795311 shutdown_connections 2024-09-11T00:45:10.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.520+0000 7f419b200640 1 -- 172.21.15.31:0/2092795311 wait complete. 2024-09-11T00:45:10.540 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:45:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.715+0000 7f797ca00640 1 -- 172.21.15.31:0/3762266370 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7978074230 con 0x7f79780c4cb0 2024-09-11T00:45:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.716+0000 7f796f400640 1 -- 172.21.15.31:0/3762266370 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6542 (secure 0 0 0) 0x7f79680842d0 con 0x7f79780c4cb0 2024-09-11T00:45:10.718 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:10.718 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"btime":"2024-09-11T00:44:50:427343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:45:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 107 2024-09-11T00:45:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 -- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7958077f70 msgr2=0x7f795807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7958077f70 0x7f795807a430 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7978151140 tx=0x7f7964005f90 comp rx=0 tx=0).stop 2024-09-11T00:45:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 -- 172.21.15.31:0/3762266370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79780c4cb0 msgr2=0x7f79781551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79780c4cb0 0x7f79781551a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f79680098f0 tx=0x7f7968032c10 comp rx=0 tx=0).stop 2024-09-11T00:45:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 -- 172.21.15.31:0/3762266370 shutdown_connections 2024-09-11T00:45:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7958077f70 0x7f795807a430 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7978157bd0 0x7f7978156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79780c84e0 0x7f797814ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 --2- 172.21.15.31:0/3762266370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79780c4cb0 0x7f79781551a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 -- 172.21.15.31:0/3762266370 >> 172.21.15.31:0/3762266370 conn(0x7f79780aa760 msgr2=0x7f79780c71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 -- 172.21.15.31:0/3762266370 shutdown_connections 2024-09-11T00:45:10.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.718+0000 7f797ca00640 1 -- 172.21.15.31:0/3762266370 wait complete. 2024-09-11T00:45:10.736 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-09-11T00:45:10.737 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-11T00:45:10.737 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:45:10.755 INFO:tasks.ceph.mds.b:Stopped 2024-09-11T00:45:10.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.949+0000 7fb048a00640 1 -- 172.21.15.31:0/1443221639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0440c9eb0 msgr2=0x7fb04406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.949+0000 7fb048a00640 1 --2- 172.21.15.31:0/1443221639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0440c9eb0 0x7fb04406dd20 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb04406f360 tx=0x7fb02c02a5d0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.949+0000 7fb048a00640 1 -- 172.21.15.31:0/1443221639 shutdown_connections 2024-09-11T00:45:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.949+0000 7fb048a00640 1 --2- 172.21.15.31:0/1443221639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0440c9eb0 0x7fb04406dd20 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.949+0000 7fb048a00640 1 -- 172.21.15.31:0/1443221639 >> 172.21.15.31:0/1443221639 conn(0x7fb0440cbec0 msgr2=0x7fb0440cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:10.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.949+0000 7fb048a00640 1 -- 172.21.15.31:0/1443221639 shutdown_connections 2024-09-11T00:45:10.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.949+0000 7fb048a00640 1 -- 172.21.15.31:0/1443221639 wait complete. 2024-09-11T00:45:10.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.950+0000 7fb048a00640 1 Processor -- start 2024-09-11T00:45:10.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.951+0000 7fb048a00640 1 -- start start 2024-09-11T00:45:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.951+0000 7fb048a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0440bf1a0 0x7fb04414a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.951+0000 7fb048a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0440c6690 0x7fb0440c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.951+0000 7fb048a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb04414f300 0x7fb04414f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.951+0000 7fb048a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb0440c6a70 con 0x7fb0440c6690 2024-09-11T00:45:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.951+0000 7fb048a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb044155f90 con 0x7fb0440bf1a0 2024-09-11T00:45:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.951+0000 7fb048a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb0440c6bf0 con 0x7fb04414f300 2024-09-11T00:45:10.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb03be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0440c6690 0x7fb0440c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb03be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0440c6690 0x7fb0440c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46720/0 (socket says 172.21.15.31:46720) 2024-09-11T00:45:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb03be00640 1 -- 172.21.15.31:0/52029852 learned_addr learned my addr 172.21.15.31:0/52029852 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb042800640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0440bf1a0 0x7fb04414a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb043200640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb04414f300 0x7fb04414f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb03be00640 1 -- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb04414f300 msgr2=0x7fb04414f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb03be00640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb04414f300 0x7fb04414f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb03be00640 1 -- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0440bf1a0 msgr2=0x7fb04414a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb03be00640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0440bf1a0 0x7fb04414a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:10.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb03be00640 1 -- 172.21.15.31:0/52029852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04414a0e0 con 0x7fb0440c6690 2024-09-11T00:45:10.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.952+0000 7fb042800640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0440bf1a0 0x7fb04414a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:45:10.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.953+0000 7fb03be00640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0440c6690 0x7fb0440c5530 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0099c0 tx=0x7fb02c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:10.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.953+0000 7fb03aa00640 1 -- 172.21.15.31:0/52029852 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c03e030 con 0x7fb0440c6690 2024-09-11T00:45:10.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.953+0000 7fb03aa00640 1 -- 172.21.15.31:0/52029852 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb02c003db0 con 0x7fb0440c6690 2024-09-11T00:45:10.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.953+0000 7fb048a00640 1 -- 172.21.15.31:0/52029852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0440abf90 con 0x7fb0440c6690 2024-09-11T00:45:10.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.953+0000 7fb03aa00640 1 -- 172.21.15.31:0/52029852 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c035040 con 0x7fb0440c6690 2024-09-11T00:45:10.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.953+0000 7fb048a00640 1 -- 172.21.15.31:0/52029852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0441e2410 con 0x7fb0440c6690 2024-09-11T00:45:10.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.954+0000 7fb048a00640 1 -- 172.21.15.31:0/52029852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0440b1410 con 0x7fb0440c6690 2024-09-11T00:45:10.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.956+0000 7fb03aa00640 1 -- 172.21.15.31:0/52029852 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb02c004a00 con 0x7fb0440c6690 2024-09-11T00:45:10.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.957+0000 7fb03aa00640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb020077f70 0x7fb02007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:10.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.957+0000 7fb03aa00640 1 -- 172.21.15.31:0/52029852 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7fb02c0baff0 con 0x7fb0440c6690 2024-09-11T00:45:10.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.958+0000 7fb042800640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb020077f70 0x7fb02007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:10.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.958+0000 7fb042800640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb020077f70 0x7fb02007a430 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb034007fd0 tx=0x7fb034009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:10.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:10.962+0000 7fb03aa00640 1 -- 172.21.15.31:0/52029852 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb02c0830f0 con 0x7fb0440c6690 2024-09-11T00:45:11.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.242+0000 7fb048a00640 1 -- 172.21.15.31:0/52029852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb044075fe0 con 0x7fb0440c6690 2024-09-11T00:45:11.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.243+0000 7fb03aa00640 1 -- 172.21.15.31:0/52029852 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6542 (secure 0 0 0) 0x7fb02c0829f0 con 0x7fb0440c6690 2024-09-11T00:45:11.246 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:11.246 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"btime":"2024-09-11T00:44:50:427343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:45:11.246 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 107 2024-09-11T00:45:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 -- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb020077f70 msgr2=0x7fb02007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb020077f70 0x7fb02007a430 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb034007fd0 tx=0x7fb034009040 comp rx=0 tx=0).stop 2024-09-11T00:45:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 -- 172.21.15.31:0/52029852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0440c6690 msgr2=0x7fb0440c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0440c6690 0x7fb0440c5530 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0099c0 tx=0x7fb02c0038a0 comp rx=0 tx=0).stop 2024-09-11T00:45:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 -- 172.21.15.31:0/52029852 shutdown_connections 2024-09-11T00:45:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb020077f70 0x7fb02007a430 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb04414f300 0x7fb04414f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0440c6690 0x7fb0440c5530 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:11.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 --2- 172.21.15.31:0/52029852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0440bf1a0 0x7fb04414a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:11.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 -- 172.21.15.31:0/52029852 >> 172.21.15.31:0/52029852 conn(0x7fb0440cbec0 msgr2=0x7fb0440b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:11.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 -- 172.21.15.31:0/52029852 shutdown_connections 2024-09-11T00:45:11.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:11.246+0000 7fb048a00640 1 -- 172.21.15.31:0/52029852 wait complete. 2024-09-11T00:45:11.263 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:45:12.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:12.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.710+0000 7f3546200640 1 -- 172.21.15.31:0/2494996358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3540159670 msgr2=0x7f3540159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:12.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.710+0000 7f3546200640 1 --2- 172.21.15.31:0/2494996358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3540159670 0x7f3540159aa0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3538005fd0 tx=0x7f353802a940 comp rx=0 tx=0).stop 2024-09-11T00:45:12.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.710+0000 7f3546200640 1 -- 172.21.15.31:0/2494996358 shutdown_connections 2024-09-11T00:45:12.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.710+0000 7f3546200640 1 --2- 172.21.15.31:0/2494996358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3540159670 0x7f3540159aa0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:12.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.710+0000 7f3546200640 1 -- 172.21.15.31:0/2494996358 >> 172.21.15.31:0/2494996358 conn(0x7f35400a9750 msgr2=0x7f35400a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:12.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.710+0000 7f3546200640 1 -- 172.21.15.31:0/2494996358 shutdown_connections 2024-09-11T00:45:12.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.710+0000 7f3546200640 1 -- 172.21.15.31:0/2494996358 wait complete. 2024-09-11T00:45:12.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.712+0000 7f3546200640 1 Processor -- start 2024-09-11T00:45:12.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.712+0000 7f3546200640 1 -- start start 2024-09-11T00:45:12.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f3546200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35400c5b10 0x7f3540158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:12.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f3546200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3540156ad0 0x7f35401e34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:12.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f3546200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3540159670 0x7f3540149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:12.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f3546200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f35400c7320 con 0x7f35400c5b10 2024-09-11T00:45:12.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f3546200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f35400c3950 con 0x7f3540159670 2024-09-11T00:45:12.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f3546200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3540153b90 con 0x7f3540156ad0 2024-09-11T00:45:12.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f353f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35400c5b10 0x7f3540158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:12.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f353f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35400c5b10 0x7f3540158e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46732/0 (socket says 172.21.15.31:46732) 2024-09-11T00:45:12.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f353f400640 1 -- 172.21.15.31:0/871816625 learned_addr learned my addr 172.21.15.31:0/871816625 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:12.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f353fe00640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3540159670 0x7f3540149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.713+0000 7f353ea00640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3540156ad0 0x7f35401e34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f353f400640 1 -- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3540156ad0 msgr2=0x7f35401e34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f353f400640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3540156ad0 0x7f35401e34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f353f400640 1 -- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3540159670 msgr2=0x7f3540149a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f353f400640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3540159670 0x7f3540149a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f353f400640 1 -- 172.21.15.31:0/871816625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354014edd0 con 0x7f35400c5b10 2024-09-11T00:45:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f353ea00640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3540156ad0 0x7f35401e34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:45:12.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f353f400640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35400c5b10 0x7f3540158e60 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f353000c450 tx=0x7f353000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f3537400640 1 -- 172.21.15.31:0/871816625 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35300052d0 con 0x7f35400c5b10 2024-09-11T00:45:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f3537400640 1 -- 172.21.15.31:0/871816625 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3530005470 con 0x7f35400c5b10 2024-09-11T00:45:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f3537400640 1 -- 172.21.15.31:0/871816625 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530012870 con 0x7f35400c5b10 2024-09-11T00:45:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f3546200640 1 -- 172.21.15.31:0/871816625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354014f0a0 con 0x7f35400c5b10 2024-09-11T00:45:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.714+0000 7f3546200640 1 -- 172.21.15.31:0/871816625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f354014f660 con 0x7f35400c5b10 2024-09-11T00:45:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.715+0000 7f3546200640 1 -- 172.21.15.31:0/871816625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35400c3d90 con 0x7f35400c5b10 2024-09-11T00:45:12.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.717+0000 7f3537400640 1 -- 172.21.15.31:0/871816625 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f35300035f0 con 0x7f35400c5b10 2024-09-11T00:45:12.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.718+0000 7f3537400640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f351c077f70 0x7f351c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:12.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.718+0000 7f3537400640 1 -- 172.21.15.31:0/871816625 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7f353009a2c0 con 0x7f35400c5b10 2024-09-11T00:45:12.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.718+0000 7f353ea00640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f351c077f70 0x7f351c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:12.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.719+0000 7f353ea00640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f351c077f70 0x7f351c07a430 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f35400c6b60 tx=0x7f3528009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:12.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.722+0000 7f3537400640 1 -- 172.21.15.31:0/871816625 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f35400c3d90 con 0x7f35400c5b10 2024-09-11T00:45:12.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.957+0000 7f3546200640 1 -- 172.21.15.31:0/871816625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35400c40a0 con 0x7f35400c5b10 2024-09-11T00:45:12.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.957+0000 7f3537400640 1 -- 172.21.15.31:0/871816625 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6542 (secure 0 0 0) 0x7f35400c40a0 con 0x7f35400c5b10 2024-09-11T00:45:12.959 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:12.959 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"btime":"2024-09-11T00:44:50:427343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:45:12.960 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 107 2024-09-11T00:45:12.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 -- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f351c077f70 msgr2=0x7f351c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:12.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f351c077f70 0x7f351c07a430 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f35400c6b60 tx=0x7f3528009040 comp rx=0 tx=0).stop 2024-09-11T00:45:12.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 -- 172.21.15.31:0/871816625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35400c5b10 msgr2=0x7f3540158e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:12.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35400c5b10 0x7f3540158e60 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f353000c450 tx=0x7f353000c910 comp rx=0 tx=0).stop 2024-09-11T00:45:12.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 -- 172.21.15.31:0/871816625 shutdown_connections 2024-09-11T00:45:12.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f351c077f70 0x7f351c07a430 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:12.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3540159670 0x7f3540149a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:12.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3540156ad0 0x7f35401e34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:12.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 --2- 172.21.15.31:0/871816625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35400c5b10 0x7f3540158e60 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:12.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 -- 172.21.15.31:0/871816625 >> 172.21.15.31:0/871816625 conn(0x7f35400a9750 msgr2=0x7f354006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:12.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 -- 172.21.15.31:0/871816625 shutdown_connections 2024-09-11T00:45:12.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:12.960+0000 7f3546200640 1 -- 172.21.15.31:0/871816625 wait complete. 2024-09-11T00:45:12.977 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:45:13.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.425+0000 7f145ca00640 1 -- 172.21.15.31:0/1544825115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458072480 msgr2=0x7f14581511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.425+0000 7f145ca00640 1 --2- 172.21.15.31:0/1544825115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458072480 0x7f14581511a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1458075260 tx=0x7f144802a660 comp rx=0 tx=0).stop 2024-09-11T00:45:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.425+0000 7f145ca00640 1 -- 172.21.15.31:0/1544825115 shutdown_connections 2024-09-11T00:45:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.425+0000 7f145ca00640 1 --2- 172.21.15.31:0/1544825115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458072480 0x7f14581511a0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.425+0000 7f145ca00640 1 -- 172.21.15.31:0/1544825115 >> 172.21.15.31:0/1544825115 conn(0x7f14580a9750 msgr2=0x7f14580a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.425+0000 7f145ca00640 1 -- 172.21.15.31:0/1544825115 shutdown_connections 2024-09-11T00:45:13.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.425+0000 7f145ca00640 1 -- 172.21.15.31:0/1544825115 wait complete. 2024-09-11T00:45:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.426+0000 7f145ca00640 1 Processor -- start 2024-09-11T00:45:13.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.426+0000 7f145ca00640 1 -- start start 2024-09-11T00:45:13.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.427+0000 7f145ca00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1458072480 0x7f1458144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:13.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.427+0000 7f145ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14580c23a0 0x7f1458146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:13.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.427+0000 7f145ca00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1458153f20 0x7f14581472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:13.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.427+0000 7f145ca00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1458074200 con 0x7f14580c23a0 2024-09-11T00:45:13.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.427+0000 7f145ca00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1458074080 con 0x7f1458153f20 2024-09-11T00:45:13.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.427+0000 7f145ca00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1458072860 con 0x7f1458072480 2024-09-11T00:45:13.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14580c23a0 0x7f1458146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:13.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14580c23a0 0x7f1458146d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46760/0 (socket says 172.21.15.31:46760) 2024-09-11T00:45:13.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455400640 1 -- 172.21.15.31:0/2573761383 learned_addr learned my addr 172.21.15.31:0/2573761383 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:13.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455e00640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1458072480 0x7f1458144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:13.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1456800640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1458153f20 0x7f14581472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:13.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455400640 1 -- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1458072480 msgr2=0x7f1458144140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:13.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455400640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1458072480 0x7f1458144140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:13.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455400640 1 -- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1458153f20 msgr2=0x7f14581472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:13.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455400640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1458153f20 0x7f14581472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:13.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455400640 1 -- 172.21.15.31:0/2573761383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14581527f0 con 0x7f14580c23a0 2024-09-11T00:45:13.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455e00640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1458072480 0x7f1458144140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:45:13.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.428+0000 7f1455400640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14580c23a0 0x7f1458146d70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f144400c420 tx=0x7f144400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:13.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.429+0000 7f144e000640 1 -- 172.21.15.31:0/2573761383 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444017070 con 0x7f14580c23a0 2024-09-11T00:45:13.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.429+0000 7f144e000640 1 -- 172.21.15.31:0/2573761383 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14440053c0 con 0x7f14580c23a0 2024-09-11T00:45:13.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.429+0000 7f144e000640 1 -- 172.21.15.31:0/2573761383 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444003850 con 0x7f14580c23a0 2024-09-11T00:45:13.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.429+0000 7f145ca00640 1 -- 172.21.15.31:0/2573761383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14581ee500 con 0x7f14580c23a0 2024-09-11T00:45:13.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.429+0000 7f145ca00640 1 -- 172.21.15.31:0/2573761383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14581497d0 con 0x7f14580c23a0 2024-09-11T00:45:13.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.430+0000 7f145ca00640 1 -- 172.21.15.31:0/2573761383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14581511a0 con 0x7f14580c23a0 2024-09-11T00:45:13.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.431+0000 7f144e000640 1 -- 172.21.15.31:0/2573761383 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f14440039f0 con 0x7f14580c23a0 2024-09-11T00:45:13.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.432+0000 7f144e000640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1438077f70 0x7f143807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:13.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.432+0000 7f144e000640 1 -- 172.21.15.31:0/2573761383 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8300+0+0 (secure 0 0 0) 0x7f1444099ce0 con 0x7f14580c23a0 2024-09-11T00:45:13.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.433+0000 7f1455e00640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1438077f70 0x7f143807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:13.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.433+0000 7f1455e00640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1438077f70 0x7f143807a430 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1448033000 tx=0x7f1448004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:13.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.436+0000 7f144e000640 1 -- 172.21.15.31:0/2573761383 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f14440617b0 con 0x7f14580c23a0 2024-09-11T00:45:13.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.669+0000 7f145ca00640 1 -- 172.21.15.31:0/2573761383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1458147b10 con 0x7f14580c23a0 2024-09-11T00:45:13.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.669+0000 7f144e000640 1 -- 172.21.15.31:0/2573761383 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6542 (secure 0 0 0) 0x7f14440661e0 con 0x7f14580c23a0 2024-09-11T00:45:13.740 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:13.741 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":107,"btime":"2024-09-11T00:44:50:427343+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:44:47.708761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5777":{"gid":5777,"name":"d","rank":0,"incarnation":103,"state":"up:active","state_seq":8,"addr":"172.21.15.204:6835/1348312176","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1348312176},{"type":"v1","addr":"172.21.15.204:6835","nonce":1348312176}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5777,"qdb_cluster":[5777]},"id":2}]} 2024-09-11T00:45:13.742 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 107 2024-09-11T00:45:13.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 -- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1438077f70 msgr2=0x7f143807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:13.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1438077f70 0x7f143807a430 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1448033000 tx=0x7f1448004730 comp rx=0 tx=0).stop 2024-09-11T00:45:13.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 -- 172.21.15.31:0/2573761383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14580c23a0 msgr2=0x7f1458146d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:13.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14580c23a0 0x7f1458146d70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f144400c420 tx=0x7f144400c8e0 comp rx=0 tx=0).stop 2024-09-11T00:45:13.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 -- 172.21.15.31:0/2573761383 shutdown_connections 2024-09-11T00:45:13.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1438077f70 0x7f143807a430 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:13.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1458153f20 0x7f14581472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:13.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14580c23a0 0x7f1458146d70 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:13.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 --2- 172.21.15.31:0/2573761383 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1458072480 0x7f1458144140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:13.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 -- 172.21.15.31:0/2573761383 >> 172.21.15.31:0/2573761383 conn(0x7f14580a9750 msgr2=0x7f14580c30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:13.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 -- 172.21.15.31:0/2573761383 shutdown_connections 2024-09-11T00:45:13.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:13.672+0000 7f145ca00640 1 -- 172.21.15.31:0/2573761383 wait complete. 2024-09-11T00:45:13.744 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:45:14.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:15.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.138+0000 7f97e6600640 1 -- 172.21.15.31:0/2910756293 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97e01518b0 msgr2=0x7f97e00c37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.138+0000 7f97e6600640 1 --2- 172.21.15.31:0/2910756293 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97e01518b0 0x7f97e00c37d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009be0 tx=0x7f97d802a940 comp rx=0 tx=0).stop 2024-09-11T00:45:15.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.138+0000 7f97e6600640 1 -- 172.21.15.31:0/2910756293 shutdown_connections 2024-09-11T00:45:15.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.138+0000 7f97e6600640 1 --2- 172.21.15.31:0/2910756293 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97e01518b0 0x7f97e00c37d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.138+0000 7f97e6600640 1 -- 172.21.15.31:0/2910756293 >> 172.21.15.31:0/2910756293 conn(0x7f97e00a9750 msgr2=0x7f97e00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:15.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.138+0000 7f97e6600640 1 -- 172.21.15.31:0/2910756293 shutdown_connections 2024-09-11T00:45:15.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.139+0000 7f97e6600640 1 -- 172.21.15.31:0/2910756293 wait complete. 2024-09-11T00:45:15.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.140+0000 7f97e6600640 1 Processor -- start 2024-09-11T00:45:15.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.140+0000 7f97e6600640 1 -- start start 2024-09-11T00:45:15.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.141+0000 7f97e6600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072480 0x7f97e00c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.141+0000 7f97e6600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97e00c23a0 0x7f97e0153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.141+0000 7f97e6600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97e01518b0 0x7f97e01537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.141+0000 7f97e6600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0074080 con 0x7f97e0072480 2024-09-11T00:45:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.141+0000 7f97e6600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0074dd0 con 0x7f97e00c23a0 2024-09-11T00:45:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.141+0000 7f97e6600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f97e0074200 con 0x7f97e01518b0 2024-09-11T00:45:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.141+0000 7f97df400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072480 0x7f97e00c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.141+0000 7f97df400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072480 0x7f97e00c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46764/0 (socket says 172.21.15.31:46764) 2024-09-11T00:45:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.141+0000 7f97df400640 1 -- 172.21.15.31:0/674843057 learned_addr learned my addr 172.21.15.31:0/674843057 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:15.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97dfe00640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97e01518b0 0x7f97e01537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97df400640 1 -- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97e01518b0 msgr2=0x7f97e01537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97df400640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97e01518b0 0x7f97e01537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97df400640 1 -- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97e00c23a0 msgr2=0x7f97e0153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97dea00640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97e00c23a0 0x7f97e0153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97df400640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97e00c23a0 0x7f97e0153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97df400640 1 -- 172.21.15.31:0/674843057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e01e9df0 con 0x7f97e0072480 2024-09-11T00:45:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97dea00640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97e00c23a0 0x7f97e0153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:45:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97df400640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072480 0x7f97e00c1040 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c730 tx=0x7f97d000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:15.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97d7400640 1 -- 172.21.15.31:0/674843057 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0016020 con 0x7f97e0072480 2024-09-11T00:45:15.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97d7400640 1 -- 172.21.15.31:0/674843057 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97d000a930 con 0x7f97e0072480 2024-09-11T00:45:15.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97e6600640 1 -- 172.21.15.31:0/674843057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e01ed0a0 con 0x7f97e0072480 2024-09-11T00:45:15.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97d7400640 1 -- 172.21.15.31:0/674843057 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d000ad80 con 0x7f97e0072480 2024-09-11T00:45:15.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.142+0000 7f97e6600640 1 -- 172.21.15.31:0/674843057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e006fb40 con 0x7f97e0072480 2024-09-11T00:45:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.144+0000 7f97e6600640 1 -- 172.21.15.31:0/674843057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97e014d680 con 0x7f97e0072480 2024-09-11T00:45:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.145+0000 7f97d7400640 1 -- 172.21.15.31:0/674843057 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f97d001a3b0 con 0x7f97e0072480 2024-09-11T00:45:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.146+0000 7f97d7400640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97c0077f70 0x7f97c007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.146+0000 7f97d7400640 1 -- 172.21.15.31:0/674843057 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8386+0+0 (secure 0 0 0) 0x7f97d009a9f0 con 0x7f97e0072480 2024-09-11T00:45:15.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.146+0000 7f97dea00640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97c0077f70 0x7f97c007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.147+0000 7f97dea00640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97c0077f70 0x7f97c007a430 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f97e014e7a0 tx=0x7f97cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:15.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.150+0000 7f97d7400640 1 -- 172.21.15.31:0/674843057 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f97e014d680 con 0x7f97e0072480 2024-09-11T00:45:15.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.384+0000 7f97e6600640 1 -- 172.21.15.31:0/674843057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97e0153f20 con 0x7f97e0072480 2024-09-11T00:45:15.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.385+0000 7f97d7400640 1 -- 172.21.15.31:0/674843057 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+5706 (secure 0 0 0) 0x7f97d0020030 con 0x7f97e0072480 2024-09-11T00:45:15.387 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:15.387 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":108,"btime":"2024-09-11T00:45:14:418443+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:14.418438+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:45:15.387 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 108 2024-09-11T00:45:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 -- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97c0077f70 msgr2=0x7f97c007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97c0077f70 0x7f97c007a430 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f97e014e7a0 tx=0x7f97cc009040 comp rx=0 tx=0).stop 2024-09-11T00:45:15.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 -- 172.21.15.31:0/674843057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072480 msgr2=0x7f97e00c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072480 0x7f97e00c1040 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c730 tx=0x7f97d000cbf0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 -- 172.21.15.31:0/674843057 shutdown_connections 2024-09-11T00:45:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97c0077f70 0x7f97c007a430 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97e01518b0 0x7f97e01537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97e00c23a0 0x7f97e0153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 --2- 172.21.15.31:0/674843057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97e0072480 0x7f97e00c1040 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 -- 172.21.15.31:0/674843057 >> 172.21.15.31:0/674843057 conn(0x7f97e00a9750 msgr2=0x7f97e014b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 -- 172.21.15.31:0/674843057 shutdown_connections 2024-09-11T00:45:15.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.387+0000 7f97e6600640 1 -- 172.21.15.31:0/674843057 wait complete. 2024-09-11T00:45:15.405 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-11T00:45:15.405 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:45:15.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.565+0000 7fb743c00640 1 -- 172.21.15.31:0/2257523089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb73c072480 msgr2=0x7fb73c1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.565+0000 7fb743c00640 1 --2- 172.21.15.31:0/2257523089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb73c072480 0x7fb73c1511a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009830 tx=0x7fb72c02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.565+0000 7fb743c00640 1 -- 172.21.15.31:0/2257523089 shutdown_connections 2024-09-11T00:45:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.565+0000 7fb743c00640 1 --2- 172.21.15.31:0/2257523089 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb73c153f20 0x7fb73c1516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.565+0000 7fb743c00640 1 --2- 172.21.15.31:0/2257523089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb73c072480 0x7fb73c1511a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.565+0000 7fb743c00640 1 -- 172.21.15.31:0/2257523089 >> 172.21.15.31:0/2257523089 conn(0x7fb73c0a9750 msgr2=0x7fb73c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.565+0000 7fb743c00640 1 -- 172.21.15.31:0/2257523089 shutdown_connections 2024-09-11T00:45:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.565+0000 7fb743c00640 1 -- 172.21.15.31:0/2257523089 wait complete. 2024-09-11T00:45:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.567+0000 7fb743c00640 1 Processor -- start 2024-09-11T00:45:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.567+0000 7fb743c00640 1 -- start start 2024-09-11T00:45:15.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.567+0000 7fb743c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb73c072480 0x7fb73c06e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.567+0000 7fb743c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb73c0c23a0 0x7fb73c0c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.567+0000 7fb743c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb73c153f20 0x7fb73c0c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb743c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c074200 con 0x7fb73c072480 2024-09-11T00:45:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb743c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c074080 con 0x7fb73c153f20 2024-09-11T00:45:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb743c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb73c072860 con 0x7fb73c0c23a0 2024-09-11T00:45:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb741200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb73c072480 0x7fb73c06e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb741200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb73c072480 0x7fb73c06e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46774/0 (socket says 172.21.15.31:46774) 2024-09-11T00:45:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb741200640 1 -- 172.21.15.31:0/3254213036 learned_addr learned my addr 172.21.15.31:0/3254213036 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb741c00640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb73c153f20 0x7fb73c0c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb733e00640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb73c0c23a0 0x7fb73c0c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb741200640 1 -- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb73c0c23a0 msgr2=0x7fb73c0c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb741200640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb73c0c23a0 0x7fb73c0c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb741200640 1 -- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb73c153f20 msgr2=0x7fb73c0c33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb741200640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb73c153f20 0x7fb73c0c33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.568+0000 7fb741200640 1 -- 172.21.15.31:0/3254213036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73c06f440 con 0x7fb73c072480 2024-09-11T00:45:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.569+0000 7fb741200640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb73c072480 0x7fb73c06e820 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009950 tx=0x7fb72c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.569+0000 7fb739600640 1 -- 172.21.15.31:0/3254213036 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c040070 con 0x7fb73c072480 2024-09-11T00:45:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.569+0000 7fb739600640 1 -- 172.21.15.31:0/3254213036 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb72c0038e0 con 0x7fb73c072480 2024-09-11T00:45:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.569+0000 7fb739600640 1 -- 172.21.15.31:0/3254213036 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c005c90 con 0x7fb73c072480 2024-09-11T00:45:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.569+0000 7fb743c00640 1 -- 172.21.15.31:0/3254213036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb73c1df670 con 0x7fb73c072480 2024-09-11T00:45:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.569+0000 7fb743c00640 1 -- 172.21.15.31:0/3254213036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb73c06ed60 con 0x7fb73c072480 2024-09-11T00:45:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.570+0000 7fb743c00640 1 -- 172.21.15.31:0/3254213036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb73c14caf0 con 0x7fb73c072480 2024-09-11T00:45:15.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.571+0000 7fb739600640 1 -- 172.21.15.31:0/3254213036 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb72c04d030 con 0x7fb73c072480 2024-09-11T00:45:15.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.572+0000 7fb739600640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb718077fa0 0x7fb71807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.572+0000 7fb739600640 1 -- 172.21.15.31:0/3254213036 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8386+0+0 (secure 0 0 0) 0x7fb72c0bbd80 con 0x7fb73c072480 2024-09-11T00:45:15.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.573+0000 7fb733e00640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb718077fa0 0x7fb71807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.573+0000 7fb733e00640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb718077fa0 0x7fb71807a460 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0c7940 tx=0x7fb724007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:15.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.576+0000 7fb739600640 1 -- 172.21.15.31:0/3254213036 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb72c083de0 con 0x7fb73c072480 2024-09-11T00:45:15.745 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.815+0000 7fb743c00640 1 -- 172.21.15.31:0/3254213036 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb73c0c3150 con 0x7fb73c072480 2024-09-11T00:45:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.816+0000 7fb739600640 1 -- 172.21.15.31:0/3254213036 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+5709 (secure 0 0 0) 0x7fb72c0836e0 con 0x7fb73c072480 2024-09-11T00:45:15.818 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:15.818 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":109,"btime":"2024-09-11T00:45:15:709312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:14.740134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:45:15.818 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 109 2024-09-11T00:45:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 -- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb718077fa0 msgr2=0x7fb71807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb718077fa0 0x7fb71807a460 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0c7940 tx=0x7fb724007eb0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 -- 172.21.15.31:0/3254213036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb73c072480 msgr2=0x7fb73c06e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb73c072480 0x7fb73c06e820 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009950 tx=0x7fb72c032c10 comp rx=0 tx=0).stop 2024-09-11T00:45:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 -- 172.21.15.31:0/3254213036 shutdown_connections 2024-09-11T00:45:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb718077fa0 0x7fb71807a460 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb73c153f20 0x7fb73c0c33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb73c0c23a0 0x7fb73c0c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 --2- 172.21.15.31:0/3254213036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb73c072480 0x7fb73c06e820 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 -- 172.21.15.31:0/3254213036 >> 172.21.15.31:0/3254213036 conn(0x7fb73c0a9750 msgr2=0x7fb73c0a8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.818+0000 7fb743c00640 1 -- 172.21.15.31:0/3254213036 shutdown_connections 2024-09-11T00:45:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.819+0000 7fb743c00640 1 -- 172.21.15.31:0/3254213036 wait complete. 2024-09-11T00:45:15.837 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:45:15:709312+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 109, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5773": { "addr": "172.21.15.204:6837/3513641573", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 3513641573, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 3513641573, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5773, "incarnation": 88, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 77 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:43:22.726458+0000", "qdb_cluster": [ 5773 ], "qdb_leader": 5773, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5773 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 109, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5834": { "addr": "172.21.15.31:6835/1396366029", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 1396366029, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 1396366029, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5834, "incarnation": 108, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:45:14.740134+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5834 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:45:15.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.908+0000 7fb44ea00640 1 -- 172.21.15.31:0/3700077092 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb448072480 msgr2=0x7fb4481511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.908+0000 7fb44ea00640 1 --2- 172.21.15.31:0/3700077092 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb448072480 0x7fb4481511a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb438009830 tx=0x7fb43802a5a0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.908+0000 7fb44ea00640 1 -- 172.21.15.31:0/3700077092 shutdown_connections 2024-09-11T00:45:15.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.908+0000 7fb44ea00640 1 --2- 172.21.15.31:0/3700077092 >> v2:172.21.15.204:3300/0 conn(0x7fb448153f20 0x7fb4481516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.908+0000 7fb44ea00640 1 --2- 172.21.15.31:0/3700077092 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb448072480 0x7fb4481511a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.908+0000 7fb44ea00640 1 -- 172.21.15.31:0/3700077092 >> 172.21.15.31:0/3700077092 conn(0x7fb4480a9750 msgr2=0x7fb4480a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:15.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.908+0000 7fb44ea00640 1 -- 172.21.15.31:0/3700077092 shutdown_connections 2024-09-11T00:45:15.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.908+0000 7fb44ea00640 1 -- 172.21.15.31:0/3700077092 wait complete. 2024-09-11T00:45:15.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.910+0000 7fb44ea00640 1 Processor -- start 2024-09-11T00:45:15.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.910+0000 7fb44ea00640 1 -- start start 2024-09-11T00:45:15.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb44ea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb448072480 0x7fb448146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb44ea00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4480c23a0 0x7fb4481ed510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb44ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb448153f20 0x7fb4481df3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb44ea00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb448074200 con 0x7fb448153f20 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb44ea00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb448074080 con 0x7fb448072480 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb44ea00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb448072860 con 0x7fb4480c23a0 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb44ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb448153f20 0x7fb4481df3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb44ca00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb448153f20 0x7fb4481df3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46784/0 (socket says 172.21.15.31:46784) 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb44ca00640 1 -- 172.21.15.31:0/2581874546 learned_addr learned my addr 172.21.15.31:0/2581874546 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb447e00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb448072480 0x7fb448146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.911+0000 7fb447400640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4480c23a0 0x7fb4481ed510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.912+0000 7fb44ca00640 1 -- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4480c23a0 msgr2=0x7fb4481ed510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.912+0000 7fb44ca00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4480c23a0 0x7fb4481ed510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.912+0000 7fb44ca00640 1 -- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb448072480 msgr2=0x7fb448146b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:15.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.912+0000 7fb44ca00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb448072480 0x7fb448146b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:15.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.912+0000 7fb44ca00640 1 -- 172.21.15.31:0/2581874546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4481edfc0 con 0x7fb448153f20 2024-09-11T00:45:15.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.912+0000 7fb44ca00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb448153f20 0x7fb4481df3a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb44000ee70 tx=0x7fb44000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.912+0000 7fb43fe00640 1 -- 172.21.15.31:0/2581874546 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb440004d20 con 0x7fb448153f20 2024-09-11T00:45:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.912+0000 7fb43fe00640 1 -- 172.21.15.31:0/2581874546 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb440003970 con 0x7fb448153f20 2024-09-11T00:45:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.912+0000 7fb44ea00640 1 -- 172.21.15.31:0/2581874546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4481ee2a0 con 0x7fb448153f20 2024-09-11T00:45:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.912+0000 7fb43fe00640 1 -- 172.21.15.31:0/2581874546 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb440003dc0 con 0x7fb448153f20 2024-09-11T00:45:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.913+0000 7fb44ea00640 1 -- 172.21.15.31:0/2581874546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4481481b0 con 0x7fb448153f20 2024-09-11T00:45:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.914+0000 7fb44ea00640 1 -- 172.21.15.31:0/2581874546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4481511a0 con 0x7fb448153f20 2024-09-11T00:45:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.915+0000 7fb43fe00640 1 -- 172.21.15.31:0/2581874546 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb44000b040 con 0x7fb448153f20 2024-09-11T00:45:15.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.916+0000 7fb43fe00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb428077f70 0x7fb42807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:15.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.916+0000 7fb43fe00640 1 -- 172.21.15.31:0/2581874546 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8386+0+0 (secure 0 0 0) 0x7fb44009afc0 con 0x7fb448153f20 2024-09-11T00:45:15.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.916+0000 7fb447e00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb428077f70 0x7fb42807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:15.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.917+0000 7fb447e00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb428077f70 0x7fb42807a430 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb438034000 tx=0x7fb438004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:15.920+0000 7fb43fe00640 1 -- 172.21.15.31:0/2581874546 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb440062950 con 0x7fb448153f20 2024-09-11T00:45:16.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.151+0000 7fb44ea00640 1 -- 172.21.15.31:0/2581874546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb44806a170 con 0x7fb448153f20 2024-09-11T00:45:16.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.152+0000 7fb43fe00640 1 -- 172.21.15.31:0/2581874546 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+5709 (secure 0 0 0) 0x7fb440067380 con 0x7fb448153f20 2024-09-11T00:45:16.154 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:16.154 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":109,"btime":"2024-09-11T00:45:15:709312+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:14.740134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:45:16.154 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 109 2024-09-11T00:45:16.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 -- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb428077f70 msgr2=0x7fb42807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:16.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb428077f70 0x7fb42807a430 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb438034000 tx=0x7fb438004730 comp rx=0 tx=0).stop 2024-09-11T00:45:16.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 -- 172.21.15.31:0/2581874546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb448153f20 msgr2=0x7fb4481df3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:16.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb448153f20 0x7fb4481df3a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb44000ee70 tx=0x7fb44000c740 comp rx=0 tx=0).stop 2024-09-11T00:45:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 -- 172.21.15.31:0/2581874546 shutdown_connections 2024-09-11T00:45:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb428077f70 0x7fb42807a430 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb448153f20 0x7fb4481df3a0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb4480c23a0 0x7fb4481ed510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 --2- 172.21.15.31:0/2581874546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb448072480 0x7fb448146b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 -- 172.21.15.31:0/2581874546 >> 172.21.15.31:0/2581874546 conn(0x7fb4480a9750 msgr2=0x7fb44814b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 -- 172.21.15.31:0/2581874546 shutdown_connections 2024-09-11T00:45:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:16.154+0000 7fb44ea00640 1 -- 172.21.15.31:0/2581874546 wait complete. 2024-09-11T00:45:16.172 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:45:17.698 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:45:17.703 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:45:17.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:18.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.007+0000 7f5adec00640 1 -- 172.21.15.31:0/1758056773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac0003a50 msgr2=0x7f5ac0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:18.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.007+0000 7f5adec00640 1 --2- 172.21.15.31:0/1758056773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac0003a50 0x7f5ac0003e70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00c610 tx=0x7f5ac0005830 comp rx=0 tx=0).stop 2024-09-11T00:45:18.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.007+0000 7f5adec00640 1 -- 172.21.15.31:0/1758056773 shutdown_connections 2024-09-11T00:45:18.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.007+0000 7f5adec00640 1 --2- 172.21.15.31:0/1758056773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac0003a50 0x7f5ac0003e70 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.007+0000 7f5adec00640 1 -- 172.21.15.31:0/1758056773 >> 172.21.15.31:0/1758056773 conn(0x7f5ad80694f0 msgr2=0x7f5ad80c9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:18.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.007+0000 7f5adec00640 1 -- 172.21.15.31:0/1758056773 shutdown_connections 2024-09-11T00:45:18.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.007+0000 7f5adec00640 1 -- 172.21.15.31:0/1758056773 wait complete. 2024-09-11T00:45:18.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.008+0000 7f5adec00640 1 Processor -- start 2024-09-11T00:45:18.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.009+0000 7f5adec00640 1 -- start start 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.009+0000 7f5adec00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac0003a50 0x7f5ad80c3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adec00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad806f310 0x7f5ad8149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adec00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ad8149fc0 0x7f5ad806dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adec00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad80c92e0 con 0x7f5ad806f310 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adec00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad80c8760 con 0x7f5ac0003a50 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adec00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad80c9460 con 0x7f5ad8149fc0 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad806f310 0x7f5ad8149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad806f310 0x7f5ad8149a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35200/0 (socket says 172.21.15.31:35200) 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adce00640 1 -- 172.21.15.31:0/285635567 learned_addr learned my addr 172.21.15.31:0/285635567 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5add800640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac0003a50 0x7f5ad80c3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:18.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5ade200640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ad8149fc0 0x7f5ad806dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adce00640 1 -- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ad8149fc0 msgr2=0x7f5ad806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adce00640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ad8149fc0 0x7f5ad806dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adce00640 1 -- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac0003a50 msgr2=0x7f5ad80c3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adce00640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac0003a50 0x7f5ad80c3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5adce00640 1 -- 172.21.15.31:0/285635567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad80c46d0 con 0x7f5ad806f310 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5add800640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac0003a50 0x7f5ad80c3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.010+0000 7f5ade200640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ad8149fc0 0x7f5ad806dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.011+0000 7f5adce00640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad806f310 0x7f5ad8149a80 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4011a40 tx=0x7f5ac4011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.011+0000 7f5ad5600640 1 -- 172.21.15.31:0/285635567 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4004460 con 0x7f5ad806f310 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.011+0000 7f5ad5600640 1 -- 172.21.15.31:0/285635567 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ac400f9a0 con 0x7f5ad806f310 2024-09-11T00:45:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.011+0000 7f5ad5600640 1 -- 172.21.15.31:0/285635567 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac400fdf0 con 0x7f5ad806f310 2024-09-11T00:45:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.011+0000 7f5adec00640 1 -- 172.21.15.31:0/285635567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad81e7cb0 con 0x7f5ad806f310 2024-09-11T00:45:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.011+0000 7f5adec00640 1 -- 172.21.15.31:0/285635567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad81e8270 con 0x7f5ad806f310 2024-09-11T00:45:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.012+0000 7f5adec00640 1 -- 172.21.15.31:0/285635567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad80c8c80 con 0x7f5ad806f310 2024-09-11T00:45:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.013+0000 7f5ad5600640 1 -- 172.21.15.31:0/285635567 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5ac4005020 con 0x7f5ad806f310 2024-09-11T00:45:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.014+0000 7f5ad5600640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ab8077f70 0x7f5ab807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.014+0000 7f5ad5600640 1 -- 172.21.15.31:0/285635567 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8386+0+0 (secure 0 0 0) 0x7f5ac409b3e0 con 0x7f5ad806f310 2024-09-11T00:45:18.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.014+0000 7f5add800640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ab8077f70 0x7f5ab807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:18.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.015+0000 7f5add800640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ab8077f70 0x7f5ab807a430 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5acc033000 tx=0x7f5acc001bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.018+0000 7f5ad5600640 1 -- 172.21.15.31:0/285635567 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5ad80c46d0 con 0x7f5ad806f310 2024-09-11T00:45:18.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:18.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.251+0000 7f5adec00640 1 -- 172.21.15.31:0/285635567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ad80aed10 con 0x7f5ad806f310 2024-09-11T00:45:18.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.252+0000 7f5ad5600640 1 -- 172.21.15.31:0/285635567 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5714 (secure 0 0 0) 0x7f5ac4063440 con 0x7f5ad806f310 2024-09-11T00:45:18.254 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:18.254 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":111,"btime":"2024-09-11T00:45:17:724320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:18.254 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 111 2024-09-11T00:45:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.254+0000 7f5adec00640 1 -- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ab8077f70 msgr2=0x7f5ab807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.254+0000 7f5adec00640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ab8077f70 0x7f5ab807a430 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5acc033000 tx=0x7f5acc001bc0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.254+0000 7f5adec00640 1 -- 172.21.15.31:0/285635567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad806f310 msgr2=0x7f5ad8149a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.254+0000 7f5adec00640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad806f310 0x7f5ad8149a80 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4011a40 tx=0x7f5ac4011f00 comp rx=0 tx=0).stop 2024-09-11T00:45:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.255+0000 7f5adec00640 1 -- 172.21.15.31:0/285635567 shutdown_connections 2024-09-11T00:45:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.255+0000 7f5adec00640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5ab8077f70 0x7f5ab807a430 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.255+0000 7f5adec00640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ad8149fc0 0x7f5ad806dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.255+0000 7f5adec00640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ad806f310 0x7f5ad8149a80 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.255+0000 7f5adec00640 1 --2- 172.21.15.31:0/285635567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac0003a50 0x7f5ad80c3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.255+0000 7f5adec00640 1 -- 172.21.15.31:0/285635567 >> 172.21.15.31:0/285635567 conn(0x7f5ad80694f0 msgr2=0x7f5ad8156170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.255+0000 7f5adec00640 1 -- 172.21.15.31:0/285635567 shutdown_connections 2024-09-11T00:45:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.255+0000 7f5adec00640 1 -- 172.21.15.31:0/285635567 wait complete. 2024-09-11T00:45:18.273 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5777 2024-09-11T00:45:18.273 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.d 2024-09-11T00:45:18.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.332+0000 7f1daaa00640 1 -- 172.21.15.31:0/2911537558 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1da41511c0 msgr2=0x7f1da4151650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:18.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.332+0000 7f1daaa00640 1 --2- 172.21.15.31:0/2911537558 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1da41511c0 0x7f1da4151650 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009c10 tx=0x7f1d9c02a840 comp rx=0 tx=0).stop 2024-09-11T00:45:18.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.332+0000 7f1daaa00640 1 -- 172.21.15.31:0/2911537558 shutdown_connections 2024-09-11T00:45:18.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.332+0000 7f1daaa00640 1 --2- 172.21.15.31:0/2911537558 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1da41511c0 0x7f1da4151650 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.332+0000 7f1daaa00640 1 -- 172.21.15.31:0/2911537558 >> 172.21.15.31:0/2911537558 conn(0x7f1da40a9750 msgr2=0x7f1da40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.332+0000 7f1daaa00640 1 -- 172.21.15.31:0/2911537558 shutdown_connections 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.332+0000 7f1daaa00640 1 -- 172.21.15.31:0/2911537558 wait complete. 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.334+0000 7f1daaa00640 1 Processor -- start 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.334+0000 7f1daaa00640 1 -- start start 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1daaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da4072480 0x7f1da414ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1daaa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1da41511c0 0x7f1da41df930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1daaa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1da4154b60 0x7f1da414af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1daaa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1da40a7140 con 0x7f1da4072480 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1daaa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1da40c23a0 con 0x7f1da41511c0 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1daaa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1da40c36b0 con 0x7f1da4154b60 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1da3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da4072480 0x7f1da414ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:18.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1da3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da4072480 0x7f1da414ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35208/0 (socket says 172.21.15.31:35208) 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1d9be00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1da41511c0 0x7f1da41df930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1da3e00640 1 -- 172.21.15.31:0/705908545 learned_addr learned my addr 172.21.15.31:0/705908545 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1da8a00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1da4154b60 0x7f1da414af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1da3e00640 1 -- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1da4154b60 msgr2=0x7f1da414af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1da3e00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1da4154b60 0x7f1da414af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1da3e00640 1 -- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1da41511c0 msgr2=0x7f1da41df930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.335+0000 7f1da3e00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1da41511c0 0x7f1da41df930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.336+0000 7f1da3e00640 1 -- 172.21.15.31:0/705908545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da40c76e0 con 0x7f1da4072480 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.336+0000 7f1da3e00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da4072480 0x7f1da414ff10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400c500 tx=0x7f1d9400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.336+0000 7f1da0c00640 1 -- 172.21.15.31:0/705908545 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94012830 con 0x7f1da4072480 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.336+0000 7f1da0c00640 1 -- 172.21.15.31:0/705908545 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d94017070 con 0x7f1da4072480 2024-09-11T00:45:18.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.336+0000 7f1da0c00640 1 -- 172.21.15.31:0/705908545 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94012d60 con 0x7f1da4072480 2024-09-11T00:45:18.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.336+0000 7f1daaa00640 1 -- 172.21.15.31:0/705908545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da414c210 con 0x7f1da4072480 2024-09-11T00:45:18.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.336+0000 7f1daaa00640 1 -- 172.21.15.31:0/705908545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da41e9ca0 con 0x7f1da4072480 2024-09-11T00:45:18.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.337+0000 7f1daaa00640 1 -- 172.21.15.31:0/705908545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da4072860 con 0x7f1da4072480 2024-09-11T00:45:18.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.338+0000 7f1da0c00640 1 -- 172.21.15.31:0/705908545 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1d9401c460 con 0x7f1da4072480 2024-09-11T00:45:18.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.339+0000 7f1da0c00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d80077f70 0x7f1d8007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:18.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.339+0000 7f1da0c00640 1 -- 172.21.15.31:0/705908545 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8386+0+0 (secure 0 0 0) 0x7f1d9409a4e0 con 0x7f1da4072480 2024-09-11T00:45:18.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.340+0000 7f1d9be00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d80077f70 0x7f1d8007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:18.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.340+0000 7f1d9be00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d80077f70 0x7f1d8007a430 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c007fd0 tx=0x7f1d8c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:18.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.344+0000 7f1da0c00640 1 -- 172.21.15.31:0/705908545 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1d94061e70 con 0x7f1da4072480 2024-09-11T00:45:18.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.575+0000 7f1daaa00640 1 -- 172.21.15.31:0/705908545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1da40a6f20 con 0x7f1da4072480 2024-09-11T00:45:18.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.576+0000 7f1da0c00640 1 -- 172.21.15.31:0/705908545 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5714 (secure 0 0 0) 0x7f1d940668a0 con 0x7f1da4072480 2024-09-11T00:45:18.578 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:18.578 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":111,"btime":"2024-09-11T00:45:17:724320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 111 2024-09-11T00:45:18.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.578+0000 7f1daaa00640 1 -- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d80077f70 msgr2=0x7f1d8007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:18.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.578+0000 7f1daaa00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d80077f70 0x7f1d8007a430 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c007fd0 tx=0x7f1d8c009040 comp rx=0 tx=0).stop 2024-09-11T00:45:18.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.578+0000 7f1daaa00640 1 -- 172.21.15.31:0/705908545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da4072480 msgr2=0x7f1da414ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.578+0000 7f1daaa00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da4072480 0x7f1da414ff10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400c500 tx=0x7f1d9400c9c0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.578+0000 7f1daaa00640 1 -- 172.21.15.31:0/705908545 shutdown_connections 2024-09-11T00:45:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.578+0000 7f1daaa00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d80077f70 0x7f1d8007a430 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.578+0000 7f1daaa00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1da4154b60 0x7f1da414af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.578+0000 7f1daaa00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1da41511c0 0x7f1da41df930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.578+0000 7f1daaa00640 1 --2- 172.21.15.31:0/705908545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da4072480 0x7f1da414ff10 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.578+0000 7f1daaa00640 1 -- 172.21.15.31:0/705908545 >> 172.21.15.31:0/705908545 conn(0x7f1da40a9750 msgr2=0x7f1da41532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.579+0000 7f1daaa00640 1 -- 172.21.15.31:0/705908545 shutdown_connections 2024-09-11T00:45:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:18.579+0000 7f1daaa00640 1 -- 172.21.15.31:0/705908545 wait complete. 2024-09-11T00:45:18.596 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:45:20.443 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-11T00:45:20.442+0000 7f748be00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:45:20.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:20.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.768+0000 7fdd13e00640 1 -- 172.21.15.31:0/801888341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd140ac2c0 msgr2=0x7fdd140ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:20.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.768+0000 7fdd13e00640 1 --2- 172.21.15.31:0/801888341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd140ac2c0 0x7fdd140ac6a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdd04005fd0 tx=0x7fdd0402a4b0 comp rx=0 tx=0).stop 2024-09-11T00:45:20.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.768+0000 7fdd13e00640 1 -- 172.21.15.31:0/801888341 shutdown_connections 2024-09-11T00:45:20.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.768+0000 7fdd13e00640 1 --2- 172.21.15.31:0/801888341 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd140a6e20 0x7fdd141510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:20.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.768+0000 7fdd13e00640 1 --2- 172.21.15.31:0/801888341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd140ac2c0 0x7fdd140ac6a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:20.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.768+0000 7fdd13e00640 1 -- 172.21.15.31:0/801888341 >> 172.21.15.31:0/801888341 conn(0x7fdd1406caf0 msgr2=0x7fdd1406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:20.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.768+0000 7fdd13e00640 1 -- 172.21.15.31:0/801888341 shutdown_connections 2024-09-11T00:45:20.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.768+0000 7fdd13e00640 1 -- 172.21.15.31:0/801888341 wait complete. 2024-09-11T00:45:20.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.770+0000 7fdd13e00640 1 Processor -- start 2024-09-11T00:45:20.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.770+0000 7fdd13e00640 1 -- start start 2024-09-11T00:45:20.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd13e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd140a6e20 0x7fdd140c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:20.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd13e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd140ac2c0 0x7fdd141decb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:20.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd13e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd140acbe0 0x7fdd1406de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:20.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd13e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdd14151780 con 0x7fdd140a6e20 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd13e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdd14151600 con 0x7fdd140acbe0 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd13e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdd14151900 con 0x7fdd140ac2c0 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd12a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd140a6e20 0x7fdd140c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd12a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd140a6e20 0x7fdd140c4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35218/0 (socket says 172.21.15.31:35218) 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd12a00640 1 -- 172.21.15.31:0/1524052948 learned_addr learned my addr 172.21.15.31:0/1524052948 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd0be00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd140ac2c0 0x7fdd141decb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.771+0000 7fdd13400640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd140acbe0 0x7fdd1406de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd12a00640 1 -- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd140ac2c0 msgr2=0x7fdd141decb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd12a00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd140ac2c0 0x7fdd141decb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd12a00640 1 -- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd140acbe0 msgr2=0x7fdd1406de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd12a00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd140acbe0 0x7fdd1406de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd12a00640 1 -- 172.21.15.31:0/1524052948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd141eca80 con 0x7fdd140a6e20 2024-09-11T00:45:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd12a00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd140a6e20 0x7fdd140c4560 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009850 tx=0x7fdd040038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:20.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd0aa00640 1 -- 172.21.15.31:0/1524052948 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0403f070 con 0x7fdd140a6e20 2024-09-11T00:45:20.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd0aa00640 1 -- 172.21.15.31:0/1524052948 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd040047a0 con 0x7fdd140a6e20 2024-09-11T00:45:20.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd0aa00640 1 -- 172.21.15.31:0/1524052948 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd04035030 con 0x7fdd140a6e20 2024-09-11T00:45:20.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd13e00640 1 -- 172.21.15.31:0/1524052948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd14148ed0 con 0x7fdd140a6e20 2024-09-11T00:45:20.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.772+0000 7fdd13e00640 1 -- 172.21.15.31:0/1524052948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd141e96f0 con 0x7fdd140a6e20 2024-09-11T00:45:20.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.774+0000 7fdd13e00640 1 -- 172.21.15.31:0/1524052948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd141ec5c0 con 0x7fdd140a6e20 2024-09-11T00:45:20.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.775+0000 7fdd0aa00640 1 -- 172.21.15.31:0/1524052948 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdd04004b90 con 0x7fdd140a6e20 2024-09-11T00:45:20.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.775+0000 7fdd0aa00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcf0077f70 0x7fdcf007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:20.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.776+0000 7fdd0aa00640 1 -- 172.21.15.31:0/1524052948 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8386+0+0 (secure 0 0 0) 0x7fdd040bb7c0 con 0x7fdd140a6e20 2024-09-11T00:45:20.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.776+0000 7fdd0be00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcf0077f70 0x7fdcf007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:20.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.777+0000 7fdd0be00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcf0077f70 0x7fdcf007a430 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdd140ac950 tx=0x7fdcfc005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:20.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:20.781+0000 7fdd0aa00640 1 -- 172.21.15.31:0/1524052948 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdd141ec5c0 con 0x7fdd140a6e20 2024-09-11T00:45:21.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.017+0000 7fdd13e00640 1 -- 172.21.15.31:0/1524052948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd141510c0 con 0x7fdd140a6e20 2024-09-11T00:45:21.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.018+0000 7fdd0aa00640 1 -- 172.21.15.31:0/1524052948 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5714 (secure 0 0 0) 0x7fdd040831d0 con 0x7fdd140a6e20 2024-09-11T00:45:21.020 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:21.020 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":111,"btime":"2024-09-11T00:45:17:724320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:21.020 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 111 2024-09-11T00:45:21.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 -- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcf0077f70 msgr2=0x7fdcf007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:21.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcf0077f70 0x7fdcf007a430 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdd140ac950 tx=0x7fdcfc005f90 comp rx=0 tx=0).stop 2024-09-11T00:45:21.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 -- 172.21.15.31:0/1524052948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd140a6e20 msgr2=0x7fdd140c4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:21.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd140a6e20 0x7fdd140c4560 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009850 tx=0x7fdd040038a0 comp rx=0 tx=0).stop 2024-09-11T00:45:21.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 -- 172.21.15.31:0/1524052948 shutdown_connections 2024-09-11T00:45:21.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdcf0077f70 0x7fdcf007a430 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:21.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd140acbe0 0x7fdd1406de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:21.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdd140ac2c0 0x7fdd141decb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:21.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 --2- 172.21.15.31:0/1524052948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd140a6e20 0x7fdd140c4560 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:21.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 -- 172.21.15.31:0/1524052948 >> 172.21.15.31:0/1524052948 conn(0x7fdd1406caf0 msgr2=0x7fdd1414e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:21.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 -- 172.21.15.31:0/1524052948 shutdown_connections 2024-09-11T00:45:21.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:21.020+0000 7fdd13e00640 1 -- 172.21.15.31:0/1524052948 wait complete. 2024-09-11T00:45:21.040 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:45:23.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:23.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.215+0000 7f53d1800640 1 -- 172.21.15.31:0/3556553486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc1539c0 msgr2=0x7f53cc159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.215+0000 7f53d1800640 1 --2- 172.21.15.31:0/3556553486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc1539c0 0x7f53cc159670 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f53cc06e7f0 tx=0x7f53c402a8b0 comp rx=0 tx=0).stop 2024-09-11T00:45:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.216+0000 7f53d1800640 1 -- 172.21.15.31:0/3556553486 shutdown_connections 2024-09-11T00:45:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.216+0000 7f53d1800640 1 --2- 172.21.15.31:0/3556553486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc1539c0 0x7f53cc159670 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.216+0000 7f53d1800640 1 --2- 172.21.15.31:0/3556553486 >> v2:172.21.15.204:3300/0 conn(0x7f53cc156ad0 0x7f53cc153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.216+0000 7f53d1800640 1 -- 172.21.15.31:0/3556553486 >> 172.21.15.31:0/3556553486 conn(0x7f53cc0a9750 msgr2=0x7f53cc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.216+0000 7f53d1800640 1 -- 172.21.15.31:0/3556553486 shutdown_connections 2024-09-11T00:45:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.216+0000 7f53d1800640 1 -- 172.21.15.31:0/3556553486 wait complete. 2024-09-11T00:45:23.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.217+0000 7f53d1800640 1 Processor -- start 2024-09-11T00:45:23.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.217+0000 7f53d1800640 1 -- start start 2024-09-11T00:45:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.218+0000 7f53d1800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53cc0c5b10 0x7f53cc152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.218+0000 7f53d1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc1539c0 0x7f53cc1545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.218+0000 7f53d1800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53cc156ad0 0x7f53cc14e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.218+0000 7f53d1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc156f20 con 0x7f53cc1539c0 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.218+0000 7f53d1800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc0c7320 con 0x7f53cc156ad0 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.218+0000 7f53d1800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f53cc0c3950 con 0x7f53cc0c5b10 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.218+0000 7f53c2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc1539c0 0x7f53cc1545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53caa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53cc0c5b10 0x7f53cc152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53c2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc1539c0 0x7f53cc1545d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35230/0 (socket says 172.21.15.31:35230) 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53c2000640 1 -- 172.21.15.31:0/866856139 learned_addr learned my addr 172.21.15.31:0/866856139 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53cb400640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53cc156ad0 0x7f53cc14e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53c2000640 1 -- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53cc0c5b10 msgr2=0x7f53cc152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53c2000640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53cc0c5b10 0x7f53cc152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53c2000640 1 -- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53cc156ad0 msgr2=0x7f53cc14e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53c2000640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53cc156ad0 0x7f53cc14e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53c2000640 1 -- 172.21.15.31:0/866856139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53cc1effb0 con 0x7f53cc1539c0 2024-09-11T00:45:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53caa00640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53cc0c5b10 0x7f53cc152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:45:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.219+0000 7f53c2000640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc1539c0 0x7f53cc1545d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f53b400e9f0 tx=0x7f53b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.220+0000 7f53c3400640 1 -- 172.21.15.31:0/866856139 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b400cd80 con 0x7f53cc1539c0 2024-09-11T00:45:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.220+0000 7f53c3400640 1 -- 172.21.15.31:0/866856139 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f53b4010430 con 0x7f53cc1539c0 2024-09-11T00:45:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.220+0000 7f53c3400640 1 -- 172.21.15.31:0/866856139 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4010840 con 0x7f53cc1539c0 2024-09-11T00:45:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.220+0000 7f53d1800640 1 -- 172.21.15.31:0/866856139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53cc1f0290 con 0x7f53cc1539c0 2024-09-11T00:45:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.220+0000 7f53d1800640 1 -- 172.21.15.31:0/866856139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53cc155450 con 0x7f53cc1539c0 2024-09-11T00:45:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.221+0000 7f53d1800640 1 -- 172.21.15.31:0/866856139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53cc159670 con 0x7f53cc1539c0 2024-09-11T00:45:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.222+0000 7f53c3400640 1 -- 172.21.15.31:0/866856139 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f53b40109e0 con 0x7f53cc1539c0 2024-09-11T00:45:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.223+0000 7f53c3400640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53a8077f70 0x7f53a807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.223+0000 7f53c3400640 1 -- 172.21.15.31:0/866856139 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8386+0+0 (secure 0 0 0) 0x7f53b409b2c0 con 0x7f53cc1539c0 2024-09-11T00:45:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.223+0000 7f53caa00640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53a8077f70 0x7f53a807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.224+0000 7f53caa00640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53a8077f70 0x7f53a807a430 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f53cc156690 tx=0x7f53bc007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:23.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.227+0000 7f53c3400640 1 -- 172.21.15.31:0/866856139 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f53b4062d90 con 0x7f53cc1539c0 2024-09-11T00:45:23.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.460+0000 7f53d1800640 1 -- 172.21.15.31:0/866856139 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53cc0c1e70 con 0x7f53cc1539c0 2024-09-11T00:45:23.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.461+0000 7f53c3400640 1 -- 172.21.15.31:0/866856139 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5714 (secure 0 0 0) 0x7f53b40677c0 con 0x7f53cc1539c0 2024-09-11T00:45:23.463 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:23.463 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":111,"btime":"2024-09-11T00:45:17:724320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:23.463 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 111 2024-09-11T00:45:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 -- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53a8077f70 msgr2=0x7f53a807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53a8077f70 0x7f53a807a430 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f53cc156690 tx=0x7f53bc007e90 comp rx=0 tx=0).stop 2024-09-11T00:45:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 -- 172.21.15.31:0/866856139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc1539c0 msgr2=0x7f53cc1545d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc1539c0 0x7f53cc1545d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f53b400e9f0 tx=0x7f53b400eeb0 comp rx=0 tx=0).stop 2024-09-11T00:45:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 -- 172.21.15.31:0/866856139 shutdown_connections 2024-09-11T00:45:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53a8077f70 0x7f53a807a430 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53cc156ad0 0x7f53cc14e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53cc1539c0 0x7f53cc1545d0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 --2- 172.21.15.31:0/866856139 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53cc0c5b10 0x7f53cc152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 -- 172.21.15.31:0/866856139 >> 172.21.15.31:0/866856139 conn(0x7f53cc0a9750 msgr2=0x7f53cc0c0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 -- 172.21.15.31:0/866856139 shutdown_connections 2024-09-11T00:45:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:23.463+0000 7f53d1800640 1 -- 172.21.15.31:0/866856139 wait complete. 2024-09-11T00:45:23.481 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:45:25.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:25.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.652+0000 7fa4ba400640 1 -- 172.21.15.31:0/1696905402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b40c23a0 msgr2=0x7fa4b41516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:25.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.652+0000 7fa4ba400640 1 --2- 172.21.15.31:0/1696905402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b40c23a0 0x7fa4b41516e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40708c0 tx=0x7fa49c02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:45:25.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.652+0000 7fa4ba400640 1 -- 172.21.15.31:0/1696905402 shutdown_connections 2024-09-11T00:45:25.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.652+0000 7fa4ba400640 1 --2- 172.21.15.31:0/1696905402 >> v2:172.21.15.204:3300/0 conn(0x7fa4b4153f20 0x7fa4b40c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:25.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.652+0000 7fa4ba400640 1 --2- 172.21.15.31:0/1696905402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b40c23a0 0x7fa4b41516e0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:25.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.653+0000 7fa4ba400640 1 --2- 172.21.15.31:0/1696905402 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b4072480 0x7fa4b41511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:25.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.653+0000 7fa4ba400640 1 -- 172.21.15.31:0/1696905402 >> 172.21.15.31:0/1696905402 conn(0x7fa4b40a9750 msgr2=0x7fa4b40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:25.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.653+0000 7fa4ba400640 1 -- 172.21.15.31:0/1696905402 shutdown_connections 2024-09-11T00:45:25.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.653+0000 7fa4ba400640 1 -- 172.21.15.31:0/1696905402 wait complete. 2024-09-11T00:45:25.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.654+0000 7fa4ba400640 1 Processor -- start 2024-09-11T00:45:25.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.654+0000 7fa4ba400640 1 -- start start 2024-09-11T00:45:25.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.655+0000 7fa4ba400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b4072480 0x7fa4b406f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:25.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.655+0000 7fa4ba400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b40c23a0 0x7fa4b41ef470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:25.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.655+0000 7fa4ba400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b4153f20 0x7fa4b414aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:25.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.655+0000 7fa4ba400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b4074200 con 0x7fa4b4072480 2024-09-11T00:45:25.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.655+0000 7fa4ba400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b4074080 con 0x7fa4b40c23a0 2024-09-11T00:45:25.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.655+0000 7fa4ba400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b4072860 con 0x7fa4b4153f20 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.655+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b4072480 0x7fa4b406f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.655+0000 7fa4b3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b4072480 0x7fa4b406f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35242/0 (socket says 172.21.15.31:35242) 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.655+0000 7fa4b3400640 1 -- 172.21.15.31:0/3701113176 learned_addr learned my addr 172.21.15.31:0/3701113176 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4aaa00640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b40c23a0 0x7fa4b41ef470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4b3e00640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b4153f20 0x7fa4b414aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4b3400640 1 -- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b4153f20 msgr2=0x7fa4b414aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4b3400640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b4153f20 0x7fa4b414aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4b3400640 1 -- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b40c23a0 msgr2=0x7fa4b41ef470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4b3400640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b40c23a0 0x7fa4b41ef470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4b3400640 1 -- 172.21.15.31:0/3701113176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b41ece00 con 0x7fa4b4072480 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4b3e00640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b4153f20 0x7fa4b414aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4b3400640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b4072480 0x7fa4b406f4c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400ea20 tx=0x7fa4a400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4abe00640 1 -- 172.21.15.31:0/3701113176 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a400cd80 con 0x7fa4b4072480 2024-09-11T00:45:25.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4abe00640 1 -- 172.21.15.31:0/3701113176 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4a4005020 con 0x7fa4b4072480 2024-09-11T00:45:25.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4ba400640 1 -- 172.21.15.31:0/3701113176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b41ecb20 con 0x7fa4b4072480 2024-09-11T00:45:25.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.656+0000 7fa4abe00640 1 -- 172.21.15.31:0/3701113176 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4005450 con 0x7fa4b4072480 2024-09-11T00:45:25.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.657+0000 7fa4ba400640 1 -- 172.21.15.31:0/3701113176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b40c7430 con 0x7fa4b4072480 2024-09-11T00:45:25.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.658+0000 7fa4ba400640 1 -- 172.21.15.31:0/3701113176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b41511a0 con 0x7fa4b4072480 2024-09-11T00:45:25.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.659+0000 7fa4abe00640 1 -- 172.21.15.31:0/3701113176 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa4a40047a0 con 0x7fa4b4072480 2024-09-11T00:45:25.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.660+0000 7fa4abe00640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa490077f70 0x7fa49007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:25.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.660+0000 7fa4abe00640 1 -- 172.21.15.31:0/3701113176 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8386+0+0 (secure 0 0 0) 0x7fa4a409b260 con 0x7fa4b4072480 2024-09-11T00:45:25.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.660+0000 7fa4aaa00640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa490077f70 0x7fa49007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:25.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.661+0000 7fa4aaa00640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa490077f70 0x7fa49007a430 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa49c033000 tx=0x7fa49c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:25.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.664+0000 7fa4abe00640 1 -- 172.21.15.31:0/3701113176 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa4a40632c0 con 0x7fa4b4072480 2024-09-11T00:45:25.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.910+0000 7fa4ba400640 1 -- 172.21.15.31:0/3701113176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4b40a9df0 con 0x7fa4b4072480 2024-09-11T00:45:25.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.910+0000 7fa4abe00640 1 -- 172.21.15.31:0/3701113176 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5714 (secure 0 0 0) 0x7fa4a4062bc0 con 0x7fa4b4072480 2024-09-11T00:45:25.912 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:25.913 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":111,"btime":"2024-09-11T00:45:17:724320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 111 2024-09-11T00:45:25.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 -- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa490077f70 msgr2=0x7fa49007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:25.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa490077f70 0x7fa49007a430 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa49c033000 tx=0x7fa49c004730 comp rx=0 tx=0).stop 2024-09-11T00:45:25.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 -- 172.21.15.31:0/3701113176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b4072480 msgr2=0x7fa4b406f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:25.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b4072480 0x7fa4b406f4c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400ea20 tx=0x7fa4a400ee20 comp rx=0 tx=0).stop 2024-09-11T00:45:25.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 -- 172.21.15.31:0/3701113176 shutdown_connections 2024-09-11T00:45:25.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa490077f70 0x7fa49007a430 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b4153f20 0x7fa4b414aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b40c23a0 0x7fa4b41ef470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 --2- 172.21.15.31:0/3701113176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b4072480 0x7fa4b406f4c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 -- 172.21.15.31:0/3701113176 >> 172.21.15.31:0/3701113176 conn(0x7fa4b40a9750 msgr2=0x7fa4b40a8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 -- 172.21.15.31:0/3701113176 shutdown_connections 2024-09-11T00:45:25.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:25.913+0000 7fa4ba400640 1 -- 172.21.15.31:0/3701113176 wait complete. 2024-09-11T00:45:25.931 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:45:26.274 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-11T00:45:26.274 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:45:26.274 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:45:26.279 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:45:26.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:26.414 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:45:26.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.439+0000 7f2cce200640 1 -- 172.21.15.31:0/530806381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80c23a0 msgr2=0x7f2cc81516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:26.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.440+0000 7f2cce200640 1 --2- 172.21.15.31:0/530806381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80c23a0 0x7f2cc81516e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f2cc80708c0 tx=0x7f2cb402a5f0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.440+0000 7f2cce200640 1 -- 172.21.15.31:0/530806381 shutdown_connections 2024-09-11T00:45:26.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.440+0000 7f2cce200640 1 --2- 172.21.15.31:0/530806381 >> v2:172.21.15.204:3300/0 conn(0x7f2cc8153f20 0x7f2cc80c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.440+0000 7f2cce200640 1 --2- 172.21.15.31:0/530806381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80c23a0 0x7f2cc81516e0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.440+0000 7f2cce200640 1 --2- 172.21.15.31:0/530806381 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cc8072480 0x7f2cc81511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.440+0000 7f2cce200640 1 -- 172.21.15.31:0/530806381 >> 172.21.15.31:0/530806381 conn(0x7f2cc80a9750 msgr2=0x7f2cc80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:26.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.440+0000 7f2cce200640 1 -- 172.21.15.31:0/530806381 shutdown_connections 2024-09-11T00:45:26.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.440+0000 7f2cce200640 1 -- 172.21.15.31:0/530806381 wait complete. 2024-09-11T00:45:26.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.441+0000 7f2cce200640 1 Processor -- start 2024-09-11T00:45:26.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.441+0000 7f2cce200640 1 -- start start 2024-09-11T00:45:26.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.442+0000 7f2cce200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc8072480 0x7f2cc81dd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:26.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.442+0000 7f2cce200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cc80c23a0 0x7f2cc814fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:26.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.442+0000 7f2cce200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cc8153f20 0x7f2cc81edb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:26.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.442+0000 7f2cce200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc8074200 con 0x7f2cc8072480 2024-09-11T00:45:26.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.442+0000 7f2cce200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc8074080 con 0x7f2cc8153f20 2024-09-11T00:45:26.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.442+0000 7f2cce200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc8072860 con 0x7f2cc80c23a0 2024-09-11T00:45:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.442+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc8072480 0x7f2cc81dd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.442+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc8072480 0x7f2cc81dd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35266/0 (socket says 172.21.15.31:35266) 2024-09-11T00:45:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.442+0000 7f2cc7400640 1 -- 172.21.15.31:0/3030563750 learned_addr learned my addr 172.21.15.31:0/3030563750 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.443+0000 7f2cc6a00640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cc80c23a0 0x7f2cc814fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.443+0000 7f2cc7e00640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cc8153f20 0x7f2cc81edb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.443+0000 7f2cc7400640 1 -- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cc80c23a0 msgr2=0x7f2cc814fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.443+0000 7f2cc7400640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cc80c23a0 0x7f2cc814fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.443+0000 7f2cc7400640 1 -- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cc8153f20 msgr2=0x7f2cc81edb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.443+0000 7f2cc7400640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cc8153f20 0x7f2cc81edb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.443+0000 7f2cc7400640 1 -- 172.21.15.31:0/3030563750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc80c4660 con 0x7f2cc8072480 2024-09-11T00:45:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.443+0000 7f2cc7400640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc8072480 0x7f2cc81dd4f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800e9f0 tx=0x7f2cb800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.444+0000 7f2cbf400640 1 -- 172.21.15.31:0/3030563750 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb800cd20 con 0x7f2cc8072480 2024-09-11T00:45:26.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.444+0000 7f2cbf400640 1 -- 172.21.15.31:0/3030563750 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cb8005020 con 0x7f2cc8072480 2024-09-11T00:45:26.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.444+0000 7f2cbf400640 1 -- 172.21.15.31:0/3030563750 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8005450 con 0x7f2cc8072480 2024-09-11T00:45:26.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.444+0000 7f2cce200640 1 -- 172.21.15.31:0/3030563750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc80c4940 con 0x7f2cc8072480 2024-09-11T00:45:26.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.444+0000 7f2cce200640 1 -- 172.21.15.31:0/3030563750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc81ed630 con 0x7f2cc8072480 2024-09-11T00:45:26.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.445+0000 7f2cce200640 1 -- 172.21.15.31:0/3030563750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc81511a0 con 0x7f2cc8072480 2024-09-11T00:45:26.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.446+0000 7f2cbf400640 1 -- 172.21.15.31:0/3030563750 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2cb80047a0 con 0x7f2cc8072480 2024-09-11T00:45:26.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.447+0000 7f2cbf400640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ca8077f70 0x7f2ca807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:26.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.447+0000 7f2cbf400640 1 -- 172.21.15.31:0/3030563750 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8386+0+0 (secure 0 0 0) 0x7f2cb809b140 con 0x7f2cc8072480 2024-09-11T00:45:26.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.447+0000 7f2cc6a00640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ca8077f70 0x7f2ca807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:26.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.448+0000 7f2cc6a00640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ca8077f70 0x7f2ca807a430 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4033000 tx=0x7f2cb4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:26.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.451+0000 7f2cbf400640 1 -- 172.21.15.31:0/3030563750 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2cb8062c70 con 0x7f2cc8072480 2024-09-11T00:45:26.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.686+0000 7f2cce200640 1 -- 172.21.15.31:0/3030563750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cc8154490 con 0x7f2cc8072480 2024-09-11T00:45:26.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.687+0000 7f2cbf400640 1 -- 172.21.15.31:0/3030563750 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5714 (secure 0 0 0) 0x7f2cb80676a0 con 0x7f2cc8072480 2024-09-11T00:45:26.689 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:26.689 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":111,"btime":"2024-09-11T00:45:17:724320+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:43:22.726458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5773},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5773":{"gid":5773,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":77,"addr":"172.21.15.204:6837/3513641573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":3513641573},{"type":"v1","addr":"172.21.15.204:6837","nonce":3513641573}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5773,"qdb_cluster":[5773]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:26.689 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 111 2024-09-11T00:45:26.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 -- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ca8077f70 msgr2=0x7f2ca807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:26.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ca8077f70 0x7f2ca807a430 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4033000 tx=0x7f2cb4004730 comp rx=0 tx=0).stop 2024-09-11T00:45:26.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 -- 172.21.15.31:0/3030563750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc8072480 msgr2=0x7f2cc81dd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:26.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc8072480 0x7f2cc81dd4f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800e9f0 tx=0x7f2cb800edf0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 -- 172.21.15.31:0/3030563750 shutdown_connections 2024-09-11T00:45:26.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2ca8077f70 0x7f2ca807a430 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cc8153f20 0x7f2cc81edb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2cc80c23a0 0x7f2cc814fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 --2- 172.21.15.31:0/3030563750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc8072480 0x7f2cc81dd4f0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:26.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 -- 172.21.15.31:0/3030563750 >> 172.21.15.31:0/3030563750 conn(0x7f2cc80a9750 msgr2=0x7f2cc8154af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:26.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 -- 172.21.15.31:0/3030563750 shutdown_connections 2024-09-11T00:45:26.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:26.689+0000 7f2cce200640 1 -- 172.21.15.31:0/3030563750 wait complete. 2024-09-11T00:45:26.707 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:45:27.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.098+0000 7ff5d0600640 1 -- 172.21.15.31:0/637330489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5c81539c0 msgr2=0x7ff5c8159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.098+0000 7ff5d0600640 1 --2- 172.21.15.31:0/637330489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5c81539c0 0x7ff5c8159670 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff5c806e7f0 tx=0x7ff5c002aac0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.098+0000 7ff5d0600640 1 -- 172.21.15.31:0/637330489 shutdown_connections 2024-09-11T00:45:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.098+0000 7ff5d0600640 1 --2- 172.21.15.31:0/637330489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5c81539c0 0x7ff5c8159670 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.098+0000 7ff5d0600640 1 --2- 172.21.15.31:0/637330489 >> v2:172.21.15.204:3300/0 conn(0x7ff5c8156ad0 0x7ff5c8153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.098+0000 7ff5d0600640 1 -- 172.21.15.31:0/637330489 >> 172.21.15.31:0/637330489 conn(0x7ff5c80a9750 msgr2=0x7ff5c80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:28.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.098+0000 7ff5d0600640 1 -- 172.21.15.31:0/637330489 shutdown_connections 2024-09-11T00:45:28.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.098+0000 7ff5d0600640 1 -- 172.21.15.31:0/637330489 wait complete. 2024-09-11T00:45:28.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.099+0000 7ff5d0600640 1 Processor -- start 2024-09-11T00:45:28.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.099+0000 7ff5d0600640 1 -- start start 2024-09-11T00:45:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.100+0000 7ff5d0600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5c80c5b10 0x7ff5c80741b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.100+0000 7ff5d0600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff5c81539c0 0x7ff5c8155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.100+0000 7ff5d0600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5c8156ad0 0x7ff5c8151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.100+0000 7ff5d0600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c8156f20 con 0x7ff5c80c5b10 2024-09-11T00:45:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.100+0000 7ff5d0600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c80c7320 con 0x7ff5c8156ad0 2024-09-11T00:45:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.100+0000 7ff5d0600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff5c80c3950 con 0x7ff5c81539c0 2024-09-11T00:45:28.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5cdc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5c80c5b10 0x7ff5c80741b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5cdc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5c80c5b10 0x7ff5c80741b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39960/0 (socket says 172.21.15.31:39960) 2024-09-11T00:45:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5cdc00640 1 -- 172.21.15.31:0/2617239507 learned_addr learned my addr 172.21.15.31:0/2617239507 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5ce600640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5c8156ad0 0x7ff5c8151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5cd200640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff5c81539c0 0x7ff5c8155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5cdc00640 1 -- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff5c81539c0 msgr2=0x7ff5c8155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5cdc00640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff5c81539c0 0x7ff5c8155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5cdc00640 1 -- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5c8156ad0 msgr2=0x7ff5c8151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5cdc00640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5c8156ad0 0x7ff5c8151c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5cdc00640 1 -- 172.21.15.31:0/2617239507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c8074e10 con 0x7ff5c80c5b10 2024-09-11T00:45:28.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5ce600640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5c8156ad0 0x7ff5c8151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:45:28.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.101+0000 7ff5cdc00640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5c80c5b10 0x7ff5c80741b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00c480 tx=0x7ff5bc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:28.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.102+0000 7ff5c5600640 1 -- 172.21.15.31:0/2617239507 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0052d0 con 0x7ff5c80c5b10 2024-09-11T00:45:28.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.102+0000 7ff5c5600640 1 -- 172.21.15.31:0/2617239507 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5bc005470 con 0x7ff5c80c5b10 2024-09-11T00:45:28.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.102+0000 7ff5c5600640 1 -- 172.21.15.31:0/2617239507 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc012870 con 0x7ff5c80c5b10 2024-09-11T00:45:28.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.102+0000 7ff5d0600640 1 -- 172.21.15.31:0/2617239507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5c80750b0 con 0x7ff5c80c5b10 2024-09-11T00:45:28.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.102+0000 7ff5d0600640 1 -- 172.21.15.31:0/2617239507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5c80ad700 con 0x7ff5c80c5b10 2024-09-11T00:45:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.103+0000 7ff5d0600640 1 -- 172.21.15.31:0/2617239507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5c81507d0 con 0x7ff5c80c5b10 2024-09-11T00:45:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.104+0000 7ff5c5600640 1 -- 172.21.15.31:0/2617239507 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff5bc0035f0 con 0x7ff5c80c5b10 2024-09-11T00:45:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.105+0000 7ff5c5600640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff5a8077f70 0x7ff5a807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.105+0000 7ff5c5600640 1 -- 172.21.15.31:0/2617239507 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7ff5bc09a340 con 0x7ff5c80c5b10 2024-09-11T00:45:28.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.106+0000 7ff5cd200640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff5a8077f70 0x7ff5a807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.106+0000 7ff5cd200640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff5a8077f70 0x7ff5a807a430 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff5c814da40 tx=0x7ff5b4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:28.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.109+0000 7ff5c5600640 1 -- 172.21.15.31:0/2617239507 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff5c81507d0 con 0x7ff5c80c5b10 2024-09-11T00:45:28.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.350+0000 7ff5d0600640 1 -- 172.21.15.31:0/2617239507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5c8159670 con 0x7ff5c80c5b10 2024-09-11T00:45:28.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.350+0000 7ff5c5600640 1 -- 172.21.15.31:0/2617239507 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5711 (secure 0 0 0) 0x7ff5bc061d00 con 0x7ff5c80c5b10 2024-09-11T00:45:28.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:28.353 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":114,"btime":"2024-09-11T00:45:27:779211+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:27.184435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:28.353 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 114 2024-09-11T00:45:28.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 -- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff5a8077f70 msgr2=0x7ff5a807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff5a8077f70 0x7ff5a807a430 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff5c814da40 tx=0x7ff5b4005f90 comp rx=0 tx=0).stop 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 -- 172.21.15.31:0/2617239507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5c80c5b10 msgr2=0x7ff5c80741b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5c80c5b10 0x7ff5c80741b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00c480 tx=0x7ff5bc00c940 comp rx=0 tx=0).stop 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 -- 172.21.15.31:0/2617239507 shutdown_connections 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff5a8077f70 0x7ff5a807a430 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5c8156ad0 0x7ff5c8151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff5c81539c0 0x7ff5c8155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 --2- 172.21.15.31:0/2617239507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5c80c5b10 0x7ff5c80741b0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 -- 172.21.15.31:0/2617239507 >> 172.21.15.31:0/2617239507 conn(0x7ff5c80a9750 msgr2=0x7ff5c814fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.353+0000 7ff5d0600640 1 -- 172.21.15.31:0/2617239507 shutdown_connections 2024-09-11T00:45:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.354+0000 7ff5d0600640 1 -- 172.21.15.31:0/2617239507 wait complete. 2024-09-11T00:45:28.371 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-09-11T00:45:28.371 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:45:28.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:28.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.529+0000 7f45f0800640 1 -- 172.21.15.31:0/3786487672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45e8072480 msgr2=0x7f45e81511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.529+0000 7f45f0800640 1 --2- 172.21.15.31:0/3786487672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45e8072480 0x7f45e81511a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f45dc009860 tx=0x7f45dc02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.529+0000 7f45f0800640 1 -- 172.21.15.31:0/3786487672 shutdown_connections 2024-09-11T00:45:28.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.529+0000 7f45f0800640 1 --2- 172.21.15.31:0/3786487672 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45e8072480 0x7f45e81511a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.530+0000 7f45f0800640 1 -- 172.21.15.31:0/3786487672 >> 172.21.15.31:0/3786487672 conn(0x7f45e80a9750 msgr2=0x7f45e80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:28.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.530+0000 7f45f0800640 1 -- 172.21.15.31:0/3786487672 shutdown_connections 2024-09-11T00:45:28.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.530+0000 7f45f0800640 1 -- 172.21.15.31:0/3786487672 wait complete. 2024-09-11T00:45:28.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.531+0000 7f45f0800640 1 Processor -- start 2024-09-11T00:45:28.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.531+0000 7f45f0800640 1 -- start start 2024-09-11T00:45:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.532+0000 7f45f0800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45e8072480 0x7f45e80c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.532+0000 7f45f0800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45e80c23a0 0x7f45e81df310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.532+0000 7f45f0800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45e8153f20 0x7f45e81df850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.532+0000 7f45f0800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f45e8074200 con 0x7f45e8153f20 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.532+0000 7f45f0800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f45e8074080 con 0x7f45e80c23a0 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.532+0000 7f45f0800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f45e8072860 con 0x7f45e8072480 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ee800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45e8153f20 0x7f45e81df850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ede00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45e8072480 0x7f45e80c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ed400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45e80c23a0 0x7f45e81df310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ede00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45e8072480 0x7f45e80c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:59692/0 (socket says 172.21.15.31:59692) 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ede00640 1 -- 172.21.15.31:0/2265598388 learned_addr learned my addr 172.21.15.31:0/2265598388 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ee800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45e8153f20 0x7f45e81df850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39972/0 (socket says 172.21.15.31:39972) 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ee800640 1 -- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45e8072480 msgr2=0x7f45e80c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ee800640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45e8072480 0x7f45e80c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ee800640 1 -- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45e80c23a0 msgr2=0x7f45e81df310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ee800640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45e80c23a0 0x7f45e81df310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ee800640 1 -- 172.21.15.31:0/2265598388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e81ecd10 con 0x7f45e8153f20 2024-09-11T00:45:28.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.533+0000 7f45ee800640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45e8153f20 0x7f45e81df850 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f45e400ca70 tx=0x7f45e400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:28.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.534+0000 7f45e2000640 1 -- 172.21.15.31:0/2265598388 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4016020 con 0x7f45e8153f20 2024-09-11T00:45:28.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.534+0000 7f45e2000640 1 -- 172.21.15.31:0/2265598388 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45e400ac80 con 0x7f45e8153f20 2024-09-11T00:45:28.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.534+0000 7f45f0800640 1 -- 172.21.15.31:0/2265598388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e81ecff0 con 0x7f45e8153f20 2024-09-11T00:45:28.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.534+0000 7f45e2000640 1 -- 172.21.15.31:0/2265598388 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4012680 con 0x7f45e8153f20 2024-09-11T00:45:28.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.534+0000 7f45f0800640 1 -- 172.21.15.31:0/2265598388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e81dd7b0 con 0x7f45e8153f20 2024-09-11T00:45:28.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.535+0000 7f45f0800640 1 -- 172.21.15.31:0/2265598388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e8151770 con 0x7f45e8153f20 2024-09-11T00:45:28.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.536+0000 7f45e2000640 1 -- 172.21.15.31:0/2265598388 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f45e4012820 con 0x7f45e8153f20 2024-09-11T00:45:28.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.537+0000 7f45e2000640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45cc077f70 0x7f45cc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.537+0000 7f45e2000640 1 -- 172.21.15.31:0/2265598388 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7f45e409afe0 con 0x7f45e8153f20 2024-09-11T00:45:28.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.537+0000 7f45ede00640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45cc077f70 0x7f45cc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.538+0000 7f45ede00640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45cc077f70 0x7f45cc07a430 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f45dc035040 tx=0x7f45dc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:28.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.542+0000 7f45e2000640 1 -- 172.21.15.31:0/2265598388 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f45e8151770 con 0x7f45e8153f20 2024-09-11T00:45:28.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:28.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.773+0000 7f45f0800640 1 -- 172.21.15.31:0/2265598388 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45e81511a0 con 0x7f45e8153f20 2024-09-11T00:45:28.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.773+0000 7f45e2000640 1 -- 172.21.15.31:0/2265598388 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5711 (secure 0 0 0) 0x7f45e4020020 con 0x7f45e8153f20 2024-09-11T00:45:28.775 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:28.775 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":114,"btime":"2024-09-11T00:45:27:779211+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:27.184435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:28.775 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 114 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.775+0000 7f45f0800640 1 -- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45cc077f70 msgr2=0x7f45cc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.775+0000 7f45f0800640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45cc077f70 0x7f45cc07a430 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f45dc035040 tx=0x7f45dc004730 comp rx=0 tx=0).stop 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.775+0000 7f45f0800640 1 -- 172.21.15.31:0/2265598388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45e8153f20 msgr2=0x7f45e81df850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.775+0000 7f45f0800640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45e8153f20 0x7f45e81df850 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f45e400ca70 tx=0x7f45e400cf30 comp rx=0 tx=0).stop 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.775+0000 7f45f0800640 1 -- 172.21.15.31:0/2265598388 shutdown_connections 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.775+0000 7f45f0800640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f45cc077f70 0x7f45cc07a430 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.776+0000 7f45f0800640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45e8153f20 0x7f45e81df850 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.776+0000 7f45f0800640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f45e80c23a0 0x7f45e81df310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.776+0000 7f45f0800640 1 --2- 172.21.15.31:0/2265598388 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f45e8072480 0x7f45e80c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.776+0000 7f45f0800640 1 -- 172.21.15.31:0/2265598388 >> 172.21.15.31:0/2265598388 conn(0x7f45e80a9750 msgr2=0x7f45e80c30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.776+0000 7f45f0800640 1 -- 172.21.15.31:0/2265598388 shutdown_connections 2024-09-11T00:45:28.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.776+0000 7f45f0800640 1 -- 172.21.15.31:0/2265598388 wait complete. 2024-09-11T00:45:28.800 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:45:27:779211+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 114, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 114, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6601": { "addr": "172.21.15.204:6835/2113744259", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 2113744259, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 2113744259, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6601, "incarnation": 113, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:45:27.184435+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6601 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 111, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5834": { "addr": "172.21.15.31:6835/1396366029", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 1396366029, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 1396366029, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5834, "incarnation": 108, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:45:17.724314+0000", "qdb_cluster": [ 5834 ], "qdb_leader": 5834, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5834 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:45:28.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.872+0000 7fa4c0000640 1 -- 172.21.15.31:0/868879502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b8072480 msgr2=0x7fa4b81511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.872+0000 7fa4c0000640 1 --2- 172.21.15.31:0/868879502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b8072480 0x7fa4b81511a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80c4400 tx=0x7fa4ac02a520 comp rx=0 tx=0).stop 2024-09-11T00:45:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.872+0000 7fa4c0000640 1 -- 172.21.15.31:0/868879502 shutdown_connections 2024-09-11T00:45:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.872+0000 7fa4c0000640 1 --2- 172.21.15.31:0/868879502 >> v2:172.21.15.31:3300/0 conn(0x7fa4b8153f20 0x7fa4b80c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.872+0000 7fa4c0000640 1 --2- 172.21.15.31:0/868879502 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b80c23a0 0x7fa4b81516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.872+0000 7fa4c0000640 1 --2- 172.21.15.31:0/868879502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b8072480 0x7fa4b81511a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.872+0000 7fa4c0000640 1 -- 172.21.15.31:0/868879502 >> 172.21.15.31:0/868879502 conn(0x7fa4b80a9750 msgr2=0x7fa4b80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.872+0000 7fa4c0000640 1 -- 172.21.15.31:0/868879502 shutdown_connections 2024-09-11T00:45:28.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.872+0000 7fa4c0000640 1 -- 172.21.15.31:0/868879502 wait complete. 2024-09-11T00:45:28.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.874+0000 7fa4c0000640 1 Processor -- start 2024-09-11T00:45:28.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.874+0000 7fa4c0000640 1 -- start start 2024-09-11T00:45:28.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4c0000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b8072480 0x7fa4b81dee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4c0000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b80c23a0 0x7fa4b814bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4c0000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b8153f20 0x7fa4b814c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4c0000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b8074200 con 0x7fa4b8072480 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4c0000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b8074080 con 0x7fa4b80c23a0 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4c0000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b8072860 con 0x7fa4b8153f20 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4bd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b8072480 0x7fa4b81dee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4bd600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b8072480 0x7fa4b81dee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39986/0 (socket says 172.21.15.31:39986) 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4bd600640 1 -- 172.21.15.31:0/3272202625 learned_addr learned my addr 172.21.15.31:0/3272202625 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4bcc00640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b80c23a0 0x7fa4b814bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.875+0000 7fa4be000640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b8153f20 0x7fa4b814c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4bd600640 1 -- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b8153f20 msgr2=0x7fa4b814c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4bd600640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b8153f20 0x7fa4b814c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4bd600640 1 -- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b80c23a0 msgr2=0x7fa4b814bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:28.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4bd600640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b80c23a0 0x7fa4b814bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:28.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4bd600640 1 -- 172.21.15.31:0/3272202625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b80700b0 con 0x7fa4b8072480 2024-09-11T00:45:28.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4bcc00640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b80c23a0 0x7fa4b814bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:45:28.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4bd600640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b8072480 0x7fa4b81dee70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009780 tx=0x7fa4ac032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:28.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4b5600640 1 -- 172.21.15.31:0/3272202625 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac040070 con 0x7fa4b8072480 2024-09-11T00:45:28.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4b5600640 1 -- 172.21.15.31:0/3272202625 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4ac005310 con 0x7fa4b8072480 2024-09-11T00:45:28.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4b5600640 1 -- 172.21.15.31:0/3272202625 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac005c90 con 0x7fa4b8072480 2024-09-11T00:45:28.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.876+0000 7fa4c0000640 1 -- 172.21.15.31:0/3272202625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b81ed370 con 0x7fa4b8072480 2024-09-11T00:45:28.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.877+0000 7fa4c0000640 1 -- 172.21.15.31:0/3272202625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b80c3440 con 0x7fa4b8072480 2024-09-11T00:45:28.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.878+0000 7fa4c0000640 1 -- 172.21.15.31:0/3272202625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b81511a0 con 0x7fa4b8072480 2024-09-11T00:45:28.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.879+0000 7fa4b5600640 1 -- 172.21.15.31:0/3272202625 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa4ac0049b0 con 0x7fa4b8072480 2024-09-11T00:45:28.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.880+0000 7fa4b5600640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa498077f70 0x7fa49807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:28.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.880+0000 7fa4b5600640 1 -- 172.21.15.31:0/3272202625 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7fa4ac0bbbb0 con 0x7fa4b8072480 2024-09-11T00:45:28.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.880+0000 7fa4bcc00640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa498077f70 0x7fa49807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:28.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.881+0000 7fa4bcc00640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa498077f70 0x7fa49807a430 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa4b81587b0 tx=0x7fa4a4009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:28.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:28.884+0000 7fa4b5600640 1 -- 172.21.15.31:0/3272202625 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa4ac083bc0 con 0x7fa4b8072480 2024-09-11T00:45:29.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.121+0000 7fa4c0000640 1 -- 172.21.15.31:0/3272202625 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4b8154790 con 0x7fa4b8072480 2024-09-11T00:45:29.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.121+0000 7fa4b5600640 1 -- 172.21.15.31:0/3272202625 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+5708 (secure 0 0 0) 0x7fa4ac0834c0 con 0x7fa4b8072480 2024-09-11T00:45:29.123 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:29.123 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":115,"btime":"2024-09-11T00:45:28:781538+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:27.783835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:29.124 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 115 2024-09-11T00:45:29.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 -- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa498077f70 msgr2=0x7fa49807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:29.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa498077f70 0x7fa49807a430 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa4b81587b0 tx=0x7fa4a4009260 comp rx=0 tx=0).stop 2024-09-11T00:45:29.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 -- 172.21.15.31:0/3272202625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b8072480 msgr2=0x7fa4b81dee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:29.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b8072480 0x7fa4b81dee70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009780 tx=0x7fa4ac032c10 comp rx=0 tx=0).stop 2024-09-11T00:45:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 -- 172.21.15.31:0/3272202625 shutdown_connections 2024-09-11T00:45:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa498077f70 0x7fa49807a430 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa4b8153f20 0x7fa4b814c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4b80c23a0 0x7fa4b814bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 --2- 172.21.15.31:0/3272202625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4b8072480 0x7fa4b81dee70 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 -- 172.21.15.31:0/3272202625 >> 172.21.15.31:0/3272202625 conn(0x7fa4b80a9750 msgr2=0x7fa4b8153050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 -- 172.21.15.31:0/3272202625 shutdown_connections 2024-09-11T00:45:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:29.124+0000 7fa4c0000640 1 -- 172.21.15.31:0/3272202625 wait complete. 2024-09-11T00:45:29.141 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:45:30.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:30.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.983+0000 7f9607000640 1 -- 172.21.15.31:0/789495417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9600073f00 msgr2=0x7f9600070b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:30.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.983+0000 7f9607000640 1 --2- 172.21.15.31:0/789495417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9600073f00 0x7f9600070b90 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009860 tx=0x7f95ec02a420 comp rx=0 tx=0).stop 2024-09-11T00:45:30.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.983+0000 7f9607000640 1 -- 172.21.15.31:0/789495417 shutdown_connections 2024-09-11T00:45:30.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.983+0000 7f9607000640 1 --2- 172.21.15.31:0/789495417 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9600154230 0x7f96001546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:30.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.983+0000 7f9607000640 1 --2- 172.21.15.31:0/789495417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9600073f00 0x7f9600070b90 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:30.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.984+0000 7f9607000640 1 -- 172.21.15.31:0/789495417 >> 172.21.15.31:0/789495417 conn(0x7f96000c48c0 msgr2=0x7f96000c4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:30.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.984+0000 7f9607000640 1 -- 172.21.15.31:0/789495417 shutdown_connections 2024-09-11T00:45:30.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.984+0000 7f9607000640 1 -- 172.21.15.31:0/789495417 wait complete. 2024-09-11T00:45:30.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.985+0000 7f9607000640 1 Processor -- start 2024-09-11T00:45:30.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.985+0000 7f9607000640 1 -- start start 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.986+0000 7f9607000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9600073f00 0x7f960014d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9607000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f960014e140 0x7f96001ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9607000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9600154230 0x7f9600143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9607000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96000748d0 con 0x7f9600073f00 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9607000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f96000745f0 con 0x7f960014e140 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9607000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f96000c3ee0 con 0x7f9600154230 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9605c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9600073f00 0x7f960014d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9606600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9600154230 0x7f9600143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9605c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9600073f00 0x7f960014d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40006/0 (socket says 172.21.15.31:40006) 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9605c00640 1 -- 172.21.15.31:0/3465164562 learned_addr learned my addr 172.21.15.31:0/3465164562 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:30.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9605200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f960014e140 0x7f96001ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:30.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9605c00640 1 -- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9600154230 msgr2=0x7f9600143850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:30.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9605c00640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9600154230 0x7f9600143850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:30.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9605c00640 1 -- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f960014e140 msgr2=0x7f96001ebb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.987+0000 7f9605c00640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f960014e140 0x7f96001ebb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.988+0000 7f9605c00640 1 -- 172.21.15.31:0/3465164562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96000c5ea0 con 0x7f9600073f00 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.988+0000 7f9606600640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9600154230 0x7f9600143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.988+0000 7f9605200640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f960014e140 0x7f96001ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.988+0000 7f9605c00640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9600073f00 0x7f960014d010 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f95f400cbc0 tx=0x7f95f400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.988+0000 7f95fd600640 1 -- 172.21.15.31:0/3465164562 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4016020 con 0x7f9600073f00 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.988+0000 7f95fd600640 1 -- 172.21.15.31:0/3465164562 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95f400ae30 con 0x7f9600073f00 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.988+0000 7f9607000640 1 -- 172.21.15.31:0/3465164562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96000c6110 con 0x7f9600073f00 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.988+0000 7f95fd600640 1 -- 172.21.15.31:0/3465164562 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4012660 con 0x7f9600073f00 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.988+0000 7f9607000640 1 -- 172.21.15.31:0/3465164562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f960014d550 con 0x7f9600073f00 2024-09-11T00:45:30.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.990+0000 7f9607000640 1 -- 172.21.15.31:0/3465164562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96000c59b0 con 0x7f9600073f00 2024-09-11T00:45:30.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.991+0000 7f95fd600640 1 -- 172.21.15.31:0/3465164562 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f95f401b400 con 0x7f9600073f00 2024-09-11T00:45:30.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.992+0000 7f95fd600640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f95e0077f70 0x7f95e007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:30.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.992+0000 7f95fd600640 1 -- 172.21.15.31:0/3465164562 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7f95f409b0a0 con 0x7f9600073f00 2024-09-11T00:45:30.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.992+0000 7f9605200640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f95e0077f70 0x7f95e007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:30.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.993+0000 7f9605200640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f95e0077f70 0x7f95e007a430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f95ec034000 tx=0x7f95ec004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:30.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:30.996+0000 7f95fd600640 1 -- 172.21.15.31:0/3465164562 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f95f4020020 con 0x7f9600073f00 2024-09-11T00:45:31.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:31.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.310+0000 7f9607000640 1 -- 172.21.15.31:0/3465164562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96000c4320 con 0x7f9600073f00 2024-09-11T00:45:31.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.311+0000 7f95fd600640 1 -- 172.21.15.31:0/3465164562 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5715 (secure 0 0 0) 0x7f95f4062a60 con 0x7f9600073f00 2024-09-11T00:45:31.313 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:31.313 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":116,"btime":"2024-09-11T00:45:29:788371+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:31.313 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 116 2024-09-11T00:45:31.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.314+0000 7f9607000640 1 -- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f95e0077f70 msgr2=0x7f95e007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:31.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.314+0000 7f9607000640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f95e0077f70 0x7f95e007a430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f95ec034000 tx=0x7f95ec004730 comp rx=0 tx=0).stop 2024-09-11T00:45:31.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.314+0000 7f9607000640 1 -- 172.21.15.31:0/3465164562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9600073f00 msgr2=0x7f960014d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:31.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.314+0000 7f9607000640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9600073f00 0x7f960014d010 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f95f400cbc0 tx=0x7f95f400a430 comp rx=0 tx=0).stop 2024-09-11T00:45:31.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.314+0000 7f9607000640 1 -- 172.21.15.31:0/3465164562 shutdown_connections 2024-09-11T00:45:31.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.314+0000 7f9607000640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f95e0077f70 0x7f95e007a430 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.314+0000 7f9607000640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9600154230 0x7f9600143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.314+0000 7f9607000640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f960014e140 0x7f96001ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.314+0000 7f9607000640 1 --2- 172.21.15.31:0/3465164562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9600073f00 0x7f960014d010 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.314+0000 7f9607000640 1 -- 172.21.15.31:0/3465164562 >> 172.21.15.31:0/3465164562 conn(0x7f96000c48c0 msgr2=0x7f96001523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:31.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.315+0000 7f9607000640 1 -- 172.21.15.31:0/3465164562 shutdown_connections 2024-09-11T00:45:31.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.315+0000 7f9607000640 1 -- 172.21.15.31:0/3465164562 wait complete. 2024-09-11T00:45:31.335 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=5773 2024-09-11T00:45:31.335 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.b 2024-09-11T00:45:31.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.339+0000 7fb9f6a00640 1 -- 172.21.15.31:0/1700272412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f00c5b10 msgr2=0x7fb9f0156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:31.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.339+0000 7fb9f6a00640 1 --2- 172.21.15.31:0/1700272412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f00c5b10 0x7fb9f0156ad0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00c6d20 tx=0x7fb9e002a5f0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.339+0000 7fb9f6a00640 1 -- 172.21.15.31:0/1700272412 shutdown_connections 2024-09-11T00:45:31.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.339+0000 7fb9f6a00640 1 --2- 172.21.15.31:0/1700272412 >> v2:172.21.15.204:3300/0 conn(0x7fb9f00c6340 0x7fb9f0159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.339+0000 7fb9f6a00640 1 --2- 172.21.15.31:0/1700272412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f00c5b10 0x7fb9f0156ad0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.339+0000 7fb9f6a00640 1 -- 172.21.15.31:0/1700272412 >> 172.21.15.31:0/1700272412 conn(0x7fb9f00a9750 msgr2=0x7fb9f00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:31.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.339+0000 7fb9f6a00640 1 -- 172.21.15.31:0/1700272412 shutdown_connections 2024-09-11T00:45:31.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.340+0000 7fb9f6a00640 1 -- 172.21.15.31:0/1700272412 wait complete. 2024-09-11T00:45:31.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.341+0000 7fb9f6a00640 1 Processor -- start 2024-09-11T00:45:31.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.342+0000 7fb9f6a00640 1 -- start start 2024-09-11T00:45:31.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9f6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f00c5b10 0x7fb9f0074d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:31.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9f6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9f00c5ef0 0x7fb9f0151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:31.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9f6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f00c6340 0x7fb9f014cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:31.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9f6a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f0153b90 con 0x7fb9f00c5b10 2024-09-11T00:45:31.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9f6a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f00c3950 con 0x7fb9f00c6340 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9f6a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f0153d10 con 0x7fb9f00c5ef0 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f00c5b10 0x7fb9f0074d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9efe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f00c5b10 0x7fb9f0074d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40012/0 (socket says 172.21.15.31:40012) 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9efe00640 1 -- 172.21.15.31:0/3053228501 learned_addr learned my addr 172.21.15.31:0/3053228501 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9f4a00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f00c6340 0x7fb9f014cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.343+0000 7fb9ef400640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9f00c5ef0 0x7fb9f0151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.344+0000 7fb9efe00640 1 -- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9f00c5ef0 msgr2=0x7fb9f0151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.344+0000 7fb9efe00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9f00c5ef0 0x7fb9f0151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.344+0000 7fb9efe00640 1 -- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f00c6340 msgr2=0x7fb9f014cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.344+0000 7fb9efe00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f00c6340 0x7fb9f014cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.344+0000 7fb9efe00640 1 -- 172.21.15.31:0/3053228501 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f00ac040 con 0x7fb9f00c5b10 2024-09-11T00:45:31.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.344+0000 7fb9efe00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f00c5b10 0x7fb9f0074d90 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb9e002abe0 tx=0x7fb9e00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:31.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.344+0000 7fb9e7e00640 1 -- 172.21.15.31:0/3053228501 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e003f070 con 0x7fb9f00c5b10 2024-09-11T00:45:31.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.344+0000 7fb9e7e00640 1 -- 172.21.15.31:0/3053228501 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9e00047a0 con 0x7fb9f00c5b10 2024-09-11T00:45:31.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.344+0000 7fb9f6a00640 1 -- 172.21.15.31:0/3053228501 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f00ac320 con 0x7fb9f00c5b10 2024-09-11T00:45:31.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.344+0000 7fb9e7e00640 1 -- 172.21.15.31:0/3053228501 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0035030 con 0x7fb9f00c5b10 2024-09-11T00:45:31.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.345+0000 7fb9f6a00640 1 -- 172.21.15.31:0/3053228501 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f00acc60 con 0x7fb9f00c5b10 2024-09-11T00:45:31.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.346+0000 7fb9f6a00640 1 -- 172.21.15.31:0/3053228501 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9f0156ad0 con 0x7fb9f00c5b10 2024-09-11T00:45:31.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.347+0000 7fb9e7e00640 1 -- 172.21.15.31:0/3053228501 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb9e0004bf0 con 0x7fb9f00c5b10 2024-09-11T00:45:31.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.348+0000 7fb9e7e00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 0x7fb9d007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:31.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.348+0000 7fb9e7e00640 1 -- 172.21.15.31:0/3053228501 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7fb9e00bb860 con 0x7fb9f00c5b10 2024-09-11T00:45:31.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.348+0000 7fb9ef400640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 0x7fb9d007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:31.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.349+0000 7fb9ef400640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 0x7fb9d007a460 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0152ed0 tx=0x7fb9dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:31.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.353+0000 7fb9e7e00640 1 -- 172.21.15.31:0/3053228501 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb9e0083220 con 0x7fb9f00c5b10 2024-09-11T00:45:31.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.629+0000 7fb9f6a00640 1 -- 172.21.15.31:0/3053228501 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9f00aa1c0 con 0x7fb9f00c5b10 2024-09-11T00:45:31.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.629+0000 7fb9e7e00640 1 -- 172.21.15.31:0/3053228501 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5715 (secure 0 0 0) 0x7fb9e0087c50 con 0x7fb9f00c5b10 2024-09-11T00:45:31.631 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:31.631 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":116,"btime":"2024-09-11T00:45:29:788371+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:31.631 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 116 2024-09-11T00:45:31.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.631+0000 7fb9f6a00640 1 -- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 msgr2=0x7fb9d007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:31.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 0x7fb9d007a460 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0152ed0 tx=0x7fb9dc009040 comp rx=0 tx=0).stop 2024-09-11T00:45:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 -- 172.21.15.31:0/3053228501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f00c5b10 msgr2=0x7fb9f0074d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f00c5b10 0x7fb9f0074d90 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb9e002abe0 tx=0x7fb9e00038a0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 -- 172.21.15.31:0/3053228501 shutdown_connections 2024-09-11T00:45:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 0x7fb9d007a460 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f00c6340 0x7fb9f014cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9f00c5ef0 0x7fb9f0151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 --2- 172.21.15.31:0/3053228501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f00c5b10 0x7fb9f0074d90 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 -- 172.21.15.31:0/3053228501 >> 172.21.15.31:0/3053228501 conn(0x7fb9f00a9750 msgr2=0x7fb9f0151fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 -- 172.21.15.31:0/3053228501 shutdown_connections 2024-09-11T00:45:31.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:31.632+0000 7fb9f6a00640 1 -- 172.21.15.31:0/3053228501 wait complete. 2024-09-11T00:45:31.652 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:active state 2024-09-11T00:45:31.652 INFO:tasks.mds_thrash.fs.[b]:waiting for 10 secs before thrashing 2024-09-11T00:45:32.784 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:45:32.782+0000 7f623ca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:45:33.336 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-09-11T00:45:33.336 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-11T00:45:33.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-11T00:45:33.340 INFO:tasks.ceph.mds.b:Started 2024-09-11T00:45:33.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:33.481 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:45:33.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.524+0000 7fa57de00640 1 -- 172.21.15.31:0/2229405603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5780a8ed0 msgr2=0x7fa5780c7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:33.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.524+0000 7fa57de00640 1 --2- 172.21.15.31:0/2229405603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5780a8ed0 0x7fa5780c7d40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa564005fd0 tx=0x7fa56402a4b0 comp rx=0 tx=0).stop 2024-09-11T00:45:33.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.524+0000 7fa57de00640 1 -- 172.21.15.31:0/2229405603 shutdown_connections 2024-09-11T00:45:33.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.524+0000 7fa57de00640 1 --2- 172.21.15.31:0/2229405603 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5780a9420 0x7fa5780c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:33.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.524+0000 7fa57de00640 1 --2- 172.21.15.31:0/2229405603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5780a8ed0 0x7fa5780c7d40 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:33.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.524+0000 7fa57de00640 1 -- 172.21.15.31:0/2229405603 >> 172.21.15.31:0/2229405603 conn(0x7fa5780aba80 msgr2=0x7fa5780abe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:33.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.524+0000 7fa57de00640 1 -- 172.21.15.31:0/2229405603 shutdown_connections 2024-09-11T00:45:33.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.524+0000 7fa57de00640 1 -- 172.21.15.31:0/2229405603 wait complete. 2024-09-11T00:45:33.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.526+0000 7fa57de00640 1 Processor -- start 2024-09-11T00:45:33.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.526+0000 7fa57de00640 1 -- start start 2024-09-11T00:45:33.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.527+0000 7fa57de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5780a8ed0 0x7fa57814d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.527+0000 7fa57de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5780a9420 0x7fa57806e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.527+0000 7fa57de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa578151720 0x7fa578145030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.527+0000 7fa57de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa57806f590 con 0x7fa578151720 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.527+0000 7fa57de00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa57806f410 con 0x7fa5780a9420 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.527+0000 7fa57de00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa57806f7f0 con 0x7fa5780a8ed0 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa577400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa578151720 0x7fa578145030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa576000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5780a9420 0x7fa57806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa576a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5780a8ed0 0x7fa57814d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa576000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5780a9420 0x7fa57806e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:57434/0 (socket says 172.21.15.31:57434) 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa577400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa578151720 0x7fa578145030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40032/0 (socket says 172.21.15.31:40032) 2024-09-11T00:45:33.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa576000640 1 -- 172.21.15.31:0/3487830217 learned_addr learned my addr 172.21.15.31:0/3487830217 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:33.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa576a00640 1 -- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5780a9420 msgr2=0x7fa57806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:33.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa576a00640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5780a9420 0x7fa57806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:33.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa576a00640 1 -- 172.21.15.31:0/3487830217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa578151720 msgr2=0x7fa578145030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:33.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa576a00640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa578151720 0x7fa578145030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:33.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.528+0000 7fa576a00640 1 -- 172.21.15.31:0/3487830217 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57806ccd0 con 0x7fa5780a8ed0 2024-09-11T00:45:33.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.529+0000 7fa576a00640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5780a8ed0 0x7fa57814d2f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa56800c3f0 tx=0x7fa56800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:33.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.529+0000 7fa56f400640 1 -- 172.21.15.31:0/3487830217 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568017070 con 0x7fa5780a8ed0 2024-09-11T00:45:33.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.529+0000 7fa56f400640 1 -- 172.21.15.31:0/3487830217 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5680053c0 con 0x7fa5780a8ed0 2024-09-11T00:45:33.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.529+0000 7fa57de00640 1 -- 172.21.15.31:0/3487830217 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa57806d290 con 0x7fa5780a8ed0 2024-09-11T00:45:33.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.529+0000 7fa56f400640 1 -- 172.21.15.31:0/3487830217 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568003850 con 0x7fa5780a8ed0 2024-09-11T00:45:33.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.529+0000 7fa57de00640 1 -- 172.21.15.31:0/3487830217 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5780c33b0 con 0x7fa5780a8ed0 2024-09-11T00:45:33.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.531+0000 7fa57de00640 1 -- 172.21.15.31:0/3487830217 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa57806ed00 con 0x7fa5780a8ed0 2024-09-11T00:45:33.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.532+0000 7fa56f400640 1 -- 172.21.15.31:0/3487830217 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa5680039f0 con 0x7fa5780a8ed0 2024-09-11T00:45:33.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.533+0000 7fa56f400640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa558077fa0 0x7fa55807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:33.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.533+0000 7fa56f400640 1 -- 172.21.15.31:0/3487830217 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7fa568099d90 con 0x7fa5780a8ed0 2024-09-11T00:45:33.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.533+0000 7fa576000640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa558077fa0 0x7fa55807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:33.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.534+0000 7fa576000640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa558077fa0 0x7fa55807a460 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa57814eff0 tx=0x7fa564004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:33.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.537+0000 7fa56f400640 1 -- 172.21.15.31:0/3487830217 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa5680616c0 con 0x7fa5780a8ed0 2024-09-11T00:45:33.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.841+0000 7fa57de00640 1 -- 172.21.15.31:0/3487830217 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5780c3e40 con 0x7fa5780a8ed0 2024-09-11T00:45:33.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.842+0000 7fa56f400640 1 -- 172.21.15.31:0/3487830217 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6544 (secure 0 0 0) 0x7fa56806a430 con 0x7fa5780a8ed0 2024-09-11T00:45:33.844 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:33.844 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":117,"btime":"2024-09-11T00:45:33:802740+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:33.844 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 117 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.844+0000 7fa57de00640 1 -- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa558077fa0 msgr2=0x7fa55807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.844+0000 7fa57de00640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa558077fa0 0x7fa55807a460 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa57814eff0 tx=0x7fa564004730 comp rx=0 tx=0).stop 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.844+0000 7fa57de00640 1 -- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5780a8ed0 msgr2=0x7fa57814d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.844+0000 7fa57de00640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5780a8ed0 0x7fa57814d2f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa56800c3f0 tx=0x7fa56800c8b0 comp rx=0 tx=0).stop 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.845+0000 7fa57de00640 1 -- 172.21.15.31:0/3487830217 shutdown_connections 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.845+0000 7fa57de00640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa558077fa0 0x7fa55807a460 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.845+0000 7fa57de00640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa578151720 0x7fa578145030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.845+0000 7fa57de00640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5780a9420 0x7fa57806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.845+0000 7fa57de00640 1 --2- 172.21.15.31:0/3487830217 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5780a8ed0 0x7fa57814d2f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.845+0000 7fa57de00640 1 -- 172.21.15.31:0/3487830217 >> 172.21.15.31:0/3487830217 conn(0x7fa5780aba80 msgr2=0x7fa578151c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.845+0000 7fa57de00640 1 -- 172.21.15.31:0/3487830217 shutdown_connections 2024-09-11T00:45:33.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:33.845+0000 7fa57de00640 1 -- 172.21.15.31:0/3487830217 wait complete. 2024-09-11T00:45:33.867 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-09-11T00:45:33.868 INFO:tasks.mds_thrash.fs.[a]:waiting for 88 secs before thrashing 2024-09-11T00:45:41.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:41.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.838+0000 7f94b8c00640 1 -- 172.21.15.31:0/864054092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b4150020 msgr2=0x7f94b40ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:41.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.838+0000 7f94b8c00640 1 --2- 172.21.15.31:0/864054092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b4150020 0x7f94b40ad4e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f94b408d7a0 tx=0x7f94a402a4d0 comp rx=0 tx=0).stop 2024-09-11T00:45:41.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.838+0000 7f94b8c00640 1 -- 172.21.15.31:0/864054092 shutdown_connections 2024-09-11T00:45:41.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.838+0000 7f94b8c00640 1 --2- 172.21.15.31:0/864054092 >> v2:172.21.15.31:3300/0 conn(0x7f94b4156960 0x7f94b406e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:41.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.838+0000 7f94b8c00640 1 --2- 172.21.15.31:0/864054092 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b40ada20 0x7f94b406e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:41.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.838+0000 7f94b8c00640 1 --2- 172.21.15.31:0/864054092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b4150020 0x7f94b40ad4e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:41.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.838+0000 7f94b8c00640 1 -- 172.21.15.31:0/864054092 >> 172.21.15.31:0/864054092 conn(0x7f94b4068f10 msgr2=0x7f94b40cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:41.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.838+0000 7f94b8c00640 1 -- 172.21.15.31:0/864054092 shutdown_connections 2024-09-11T00:45:41.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.838+0000 7f94b8c00640 1 -- 172.21.15.31:0/864054092 wait complete. 2024-09-11T00:45:41.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.840+0000 7f94b8c00640 1 Processor -- start 2024-09-11T00:45:41.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.840+0000 7f94b8c00640 1 -- start start 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b8c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b40ada20 0x7f94b40c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b8c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4150020 0x7f94b414ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b8c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b4156960 0x7f94b414bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b8c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f94b4157550 con 0x7f94b4150020 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b8c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f94b41573d0 con 0x7f94b4156960 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b8c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f94b40cc220 con 0x7f94b40ada20 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4150020 0x7f94b414ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b3400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b40ada20 0x7f94b40c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4150020 0x7f94b414ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57476/0 (socket says 172.21.15.31:57476) 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b2a00640 1 -- 172.21.15.31:0/2410280 learned_addr learned my addr 172.21.15.31:0/2410280 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:41.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.841+0000 7f94b3e00640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b4156960 0x7f94b414bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94b3400640 1 -- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b4156960 msgr2=0x7f94b414bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94b3400640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b4156960 0x7f94b414bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94b3400640 1 -- 172.21.15.31:0/2410280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4150020 msgr2=0x7f94b414ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94b3400640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4150020 0x7f94b414ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94b3400640 1 -- 172.21.15.31:0/2410280 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b41ec1e0 con 0x7f94b40ada20 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94b2a00640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4150020 0x7f94b414ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94b3400640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b40ada20 0x7f94b40c4f90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f94b40c66e0 tx=0x7f94a40038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94ab400640 1 -- 172.21.15.31:0/2410280 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a403f070 con 0x7f94b40ada20 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94ab400640 1 -- 172.21.15.31:0/2410280 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94a40047a0 con 0x7f94b40ada20 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94ab400640 1 -- 172.21.15.31:0/2410280 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a4035030 con 0x7f94b40ada20 2024-09-11T00:45:41.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.842+0000 7f94b8c00640 1 -- 172.21.15.31:0/2410280 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b40c2c40 con 0x7f94b40ada20 2024-09-11T00:45:41.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.843+0000 7f94b8c00640 1 -- 172.21.15.31:0/2410280 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b40c54d0 con 0x7f94b40ada20 2024-09-11T00:45:41.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.844+0000 7f94b8c00640 1 -- 172.21.15.31:0/2410280 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b406ef00 con 0x7f94b40ada20 2024-09-11T00:45:41.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.845+0000 7f94ab400640 1 -- 172.21.15.31:0/2410280 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f94a4004bb0 con 0x7f94b40ada20 2024-09-11T00:45:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.846+0000 7f94ab400640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9494077f70 0x7f949407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.846+0000 7f94ab400640 1 -- 172.21.15.31:0/2410280 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7f94a40bbae0 con 0x7f94b40ada20 2024-09-11T00:45:41.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.847+0000 7f94b2a00640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9494077f70 0x7f949407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:41.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.847+0000 7f94b2a00640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9494077f70 0x7f949407a430 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f94b41e9c30 tx=0x7f94a0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:41.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:41.852+0000 7f94ab400640 1 -- 172.21.15.31:0/2410280 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f94b406ef00 con 0x7f94b40ada20 2024-09-11T00:45:42.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.148+0000 7f94b8c00640 1 -- 172.21.15.31:0/2410280 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94b40c9630 con 0x7f94b40ada20 2024-09-11T00:45:42.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.149+0000 7f94ab400640 1 -- 172.21.15.31:0/2410280 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6544 (secure 0 0 0) 0x7f94b40c9630 con 0x7f94b40ada20 2024-09-11T00:45:42.151 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:42.151 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":117,"btime":"2024-09-11T00:45:33:802740+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:42.151 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 117 2024-09-11T00:45:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 -- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9494077f70 msgr2=0x7f949407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9494077f70 0x7f949407a430 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f94b41e9c30 tx=0x7f94a0005f90 comp rx=0 tx=0).stop 2024-09-11T00:45:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 -- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b40ada20 msgr2=0x7f94b40c4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b40ada20 0x7f94b40c4f90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f94b40c66e0 tx=0x7f94a40038a0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 -- 172.21.15.31:0/2410280 shutdown_connections 2024-09-11T00:45:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9494077f70 0x7f949407a430 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f94b4156960 0x7f94b414bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94b4150020 0x7f94b414ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 --2- 172.21.15.31:0/2410280 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f94b40ada20 0x7f94b40c4f90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 -- 172.21.15.31:0/2410280 >> 172.21.15.31:0/2410280 conn(0x7f94b4068f10 msgr2=0x7f94b4154a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.152+0000 7f94b8c00640 1 -- 172.21.15.31:0/2410280 shutdown_connections 2024-09-11T00:45:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.153+0000 7f94b8c00640 1 -- 172.21.15.31:0/2410280 wait complete. 2024-09-11T00:45:42.172 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-11T00:45:42.172 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-11T00:45:42.172 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:45:42.192 INFO:tasks.ceph.mds.c:Stopped 2024-09-11T00:45:42.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:42.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.416+0000 7f7303e00640 1 -- 172.21.15.31:0/3308504891 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73040ac2c0 msgr2=0x7f73040ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:42.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.416+0000 7f7303e00640 1 --2- 172.21.15.31:0/3308504891 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73040ac2c0 0x7f73040ac6a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009830 tx=0x7f72f402a3f0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.416+0000 7f7303e00640 1 -- 172.21.15.31:0/3308504891 shutdown_connections 2024-09-11T00:45:42.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.416+0000 7f7303e00640 1 --2- 172.21.15.31:0/3308504891 >> v2:172.21.15.204:3300/0 conn(0x7f73040a6e20 0x7f73041510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.416+0000 7f7303e00640 1 --2- 172.21.15.31:0/3308504891 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73040ac2c0 0x7f73040ac6a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.416+0000 7f7303e00640 1 -- 172.21.15.31:0/3308504891 >> 172.21.15.31:0/3308504891 conn(0x7f730406caf0 msgr2=0x7f730406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:42.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.416+0000 7f7303e00640 1 -- 172.21.15.31:0/3308504891 shutdown_connections 2024-09-11T00:45:42.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.416+0000 7f7303e00640 1 -- 172.21.15.31:0/3308504891 wait complete. 2024-09-11T00:45:42.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.418+0000 7f7303e00640 1 Processor -- start 2024-09-11T00:45:42.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.418+0000 7f7303e00640 1 -- start start 2024-09-11T00:45:42.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.419+0000 7f7303e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73040a6e20 0x7f73041eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:42.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.419+0000 7f7303e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73040ac2c0 0x7f73041e8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.419+0000 7f7303e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73040acbe0 0x7f730406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.419+0000 7f7303e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7304151780 con 0x7f73040acbe0 2024-09-11T00:45:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.419+0000 7f7303e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7304151600 con 0x7f73040a6e20 2024-09-11T00:45:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.419+0000 7f7303e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7304151900 con 0x7f73040ac2c0 2024-09-11T00:45:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f72fbe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73040ac2c0 0x7f73041e8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f7302a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73040a6e20 0x7f73041eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f72fbe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73040ac2c0 0x7f73041e8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:43974/0 (socket says 172.21.15.31:43974) 2024-09-11T00:45:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f72fbe00640 1 -- 172.21.15.31:0/4042166471 learned_addr learned my addr 172.21.15.31:0/4042166471 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:42.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f7303400640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73040acbe0 0x7f730406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:42.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f7302a00640 1 -- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73040ac2c0 msgr2=0x7f73041e8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:42.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f7302a00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73040ac2c0 0x7f73041e8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f7302a00640 1 -- 172.21.15.31:0/4042166471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73040acbe0 msgr2=0x7f730406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f7302a00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73040acbe0 0x7f730406def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f7302a00640 1 -- 172.21.15.31:0/4042166471 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73040c4030 con 0x7f73040a6e20 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f7303400640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73040acbe0 0x7f730406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.420+0000 7f72fbe00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73040ac2c0 0x7f73041e8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.421+0000 7f7302a00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73040a6e20 0x7f73041eb5c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f72f4031000 tx=0x7f72f4032c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.421+0000 7f72faa00640 1 -- 172.21.15.31:0/4042166471 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4040070 con 0x7f73040a6e20 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.421+0000 7f72faa00640 1 -- 172.21.15.31:0/4042166471 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72f4005330 con 0x7f73040a6e20 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.421+0000 7f7303e00640 1 -- 172.21.15.31:0/4042166471 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7304072070 con 0x7f73040a6e20 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.421+0000 7f72faa00640 1 -- 172.21.15.31:0/4042166471 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f4005c90 con 0x7f73040a6e20 2024-09-11T00:45:42.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.421+0000 7f7303e00640 1 -- 172.21.15.31:0/4042166471 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73040745f0 con 0x7f73040a6e20 2024-09-11T00:45:42.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.423+0000 7f7303e00640 1 -- 172.21.15.31:0/4042166471 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73041510c0 con 0x7f73040a6e20 2024-09-11T00:45:42.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.424+0000 7f72faa00640 1 -- 172.21.15.31:0/4042166471 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f72f404c030 con 0x7f73040a6e20 2024-09-11T00:45:42.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.425+0000 7f72faa00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f72e0077fa0 0x7f72e007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:42.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.425+0000 7f72faa00640 1 -- 172.21.15.31:0/4042166471 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7f72f40bb9e0 con 0x7f73040a6e20 2024-09-11T00:45:42.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.425+0000 7f72fbe00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f72e0077fa0 0x7f72e007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:42.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.426+0000 7f72fbe00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f72e0077fa0 0x7f72e007a460 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0095f0 tx=0x7f72ec005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:42.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.430+0000 7f72faa00640 1 -- 172.21.15.31:0/4042166471 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f72f4083310 con 0x7f73040a6e20 2024-09-11T00:45:42.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.687+0000 7f7303e00640 1 -- 172.21.15.31:0/4042166471 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73041e91e0 con 0x7f73040a6e20 2024-09-11T00:45:42.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.688+0000 7f72faa00640 1 -- 172.21.15.31:0/4042166471 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6544 (secure 0 0 0) 0x7f72f4087d40 con 0x7f73040a6e20 2024-09-11T00:45:42.689 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:42.690 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":117,"btime":"2024-09-11T00:45:33:802740+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:42.690 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 117 2024-09-11T00:45:42.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.690+0000 7f7303e00640 1 -- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f72e0077fa0 msgr2=0x7f72e007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:42.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.690+0000 7f7303e00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f72e0077fa0 0x7f72e007a460 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0095f0 tx=0x7f72ec005f90 comp rx=0 tx=0).stop 2024-09-11T00:45:42.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.690+0000 7f7303e00640 1 -- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73040a6e20 msgr2=0x7f73041eb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.690+0000 7f7303e00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73040a6e20 0x7f73041eb5c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f72f4031000 tx=0x7f72f4032c10 comp rx=0 tx=0).stop 2024-09-11T00:45:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.690+0000 7f7303e00640 1 -- 172.21.15.31:0/4042166471 shutdown_connections 2024-09-11T00:45:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.690+0000 7f7303e00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f72e0077fa0 0x7f72e007a460 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.690+0000 7f7303e00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73040acbe0 0x7f730406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.690+0000 7f7303e00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f73040ac2c0 0x7f73041e8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.690+0000 7f7303e00640 1 --2- 172.21.15.31:0/4042166471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73040a6e20 0x7f73041eb5c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.690+0000 7f7303e00640 1 -- 172.21.15.31:0/4042166471 >> 172.21.15.31:0/4042166471 conn(0x7f730406caf0 msgr2=0x7f730414fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.691+0000 7f7303e00640 1 -- 172.21.15.31:0/4042166471 shutdown_connections 2024-09-11T00:45:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:42.691+0000 7f7303e00640 1 -- 172.21.15.31:0/4042166471 wait complete. 2024-09-11T00:45:42.710 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:45:44.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:44.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.889+0000 7f834e600640 1 -- 172.21.15.31:0/1251067469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8348153850 msgr2=0x7f8348156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:44.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.889+0000 7f834e600640 1 --2- 172.21.15.31:0/1251067469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8348153850 0x7f8348156350 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f8340005fd0 tx=0x7f834002a8a0 comp rx=0 tx=0).stop 2024-09-11T00:45:44.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.889+0000 7f834e600640 1 -- 172.21.15.31:0/1251067469 shutdown_connections 2024-09-11T00:45:44.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.889+0000 7f834e600640 1 --2- 172.21.15.31:0/1251067469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8348153850 0x7f8348156350 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.889+0000 7f834e600640 1 --2- 172.21.15.31:0/1251067469 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83480c6420 0x7f8348153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.889+0000 7f834e600640 1 -- 172.21.15.31:0/1251067469 >> 172.21.15.31:0/1251067469 conn(0x7f834806caf0 msgr2=0x7f834806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.889+0000 7f834e600640 1 -- 172.21.15.31:0/1251067469 shutdown_connections 2024-09-11T00:45:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.889+0000 7f834e600640 1 -- 172.21.15.31:0/1251067469 wait complete. 2024-09-11T00:45:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.891+0000 7f834e600640 1 Processor -- start 2024-09-11T00:45:44.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.891+0000 7f834e600640 1 -- start start 2024-09-11T00:45:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.892+0000 7f834e600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83480c5aa0 0x7f83481ec320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.892+0000 7f834e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83480c6420 0x7f8348074460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.892+0000 7f834e600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8348153850 0x7f83481595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.892+0000 7f834e600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f83480c5e80 con 0x7f83480c6420 2024-09-11T00:45:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.892+0000 7f834e600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f83480c2e80 con 0x7f8348153850 2024-09-11T00:45:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.892+0000 7f834e600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f83480c6000 con 0x7f83480c5aa0 2024-09-11T00:45:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.892+0000 7f8347e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83480c6420 0x7f8348074460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.892+0000 7f8347e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83480c6420 0x7f8348074460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57512/0 (socket says 172.21.15.31:57512) 2024-09-11T00:45:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.892+0000 7f8347e00640 1 -- 172.21.15.31:0/2035153690 learned_addr learned my addr 172.21.15.31:0/2035153690 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.892+0000 7f834d200640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83480c5aa0 0x7f83481ec320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.893+0000 7f834dc00640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8348153850 0x7f83481595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.893+0000 7f8347e00640 1 -- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83480c5aa0 msgr2=0x7f83481ec320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.893+0000 7f8347e00640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83480c5aa0 0x7f83481ec320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.893+0000 7f8347e00640 1 -- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8348153850 msgr2=0x7f83481595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.893+0000 7f8347e00640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8348153850 0x7f83481595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.893+0000 7f8347e00640 1 -- 172.21.15.31:0/2035153690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8348157d00 con 0x7f83480c6420 2024-09-11T00:45:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.893+0000 7f8347e00640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83480c6420 0x7f8348074460 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f833400e9f0 tx=0x7f833400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.894+0000 7f8344c00640 1 -- 172.21.15.31:0/2035153690 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833400cd80 con 0x7f83480c6420 2024-09-11T00:45:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.894+0000 7f8344c00640 1 -- 172.21.15.31:0/2035153690 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8334004020 con 0x7f83480c6420 2024-09-11T00:45:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.894+0000 7f8344c00640 1 -- 172.21.15.31:0/2035153690 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334004430 con 0x7f83480c6420 2024-09-11T00:45:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.894+0000 7f834e600640 1 -- 172.21.15.31:0/2035153690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f834814edd0 con 0x7f83480c6420 2024-09-11T00:45:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.894+0000 7f834e600640 1 -- 172.21.15.31:0/2035153690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83480722c0 con 0x7f83480c6420 2024-09-11T00:45:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.895+0000 7f834e600640 1 -- 172.21.15.31:0/2035153690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8348072050 con 0x7f83480c6420 2024-09-11T00:45:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.896+0000 7f8344c00640 1 -- 172.21.15.31:0/2035153690 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f83340035f0 con 0x7f83480c6420 2024-09-11T00:45:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.898+0000 7f8344c00640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8328077f70 0x7f832807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.898+0000 7f8344c00640 1 -- 172.21.15.31:0/2035153690 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7f833409b190 con 0x7f83480c6420 2024-09-11T00:45:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.898+0000 7f834d200640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8328077f70 0x7f832807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.899+0000 7f834d200640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8328077f70 0x7f832807a430 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f833c007fd0 tx=0x7f833c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:44.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:44.904+0000 7f8344c00640 1 -- 172.21.15.31:0/2035153690 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8334062be0 con 0x7f83480c6420 2024-09-11T00:45:45.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.191+0000 7f834e600640 1 -- 172.21.15.31:0/2035153690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83480c5520 con 0x7f83480c6420 2024-09-11T00:45:45.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.192+0000 7f8344c00640 1 -- 172.21.15.31:0/2035153690 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6544 (secure 0 0 0) 0x7f83480c5520 con 0x7f83480c6420 2024-09-11T00:45:45.194 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:45.194 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":117,"btime":"2024-09-11T00:45:33:802740+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:45.194 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 117 2024-09-11T00:45:45.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.194+0000 7f834e600640 1 -- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8328077f70 msgr2=0x7f832807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:45.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.194+0000 7f834e600640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8328077f70 0x7f832807a430 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f833c007fd0 tx=0x7f833c009040 comp rx=0 tx=0).stop 2024-09-11T00:45:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.194+0000 7f834e600640 1 -- 172.21.15.31:0/2035153690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83480c6420 msgr2=0x7f8348074460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.194+0000 7f834e600640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83480c6420 0x7f8348074460 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f833400e9f0 tx=0x7f833400eeb0 comp rx=0 tx=0).stop 2024-09-11T00:45:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.194+0000 7f834e600640 1 -- 172.21.15.31:0/2035153690 shutdown_connections 2024-09-11T00:45:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.194+0000 7f834e600640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8328077f70 0x7f832807a430 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.194+0000 7f834e600640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8348153850 0x7f83481595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.194+0000 7f834e600640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83480c6420 0x7f8348074460 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.194+0000 7f834e600640 1 --2- 172.21.15.31:0/2035153690 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f83480c5aa0 0x7f83481ec320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.194+0000 7f834e600640 1 -- 172.21.15.31:0/2035153690 >> 172.21.15.31:0/2035153690 conn(0x7f834806caf0 msgr2=0x7f83481562c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.195+0000 7f834e600640 1 -- 172.21.15.31:0/2035153690 shutdown_connections 2024-09-11T00:45:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:45.195+0000 7f834e600640 1 -- 172.21.15.31:0/2035153690 wait complete. 2024-09-11T00:45:45.216 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:45:47.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.403+0000 7f39a2800640 1 -- 172.21.15.31:0/3364699416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f399c0ada20 msgr2=0x7f399c06e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.403+0000 7f39a2800640 1 --2- 172.21.15.31:0/3364699416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f399c0ada20 0x7f399c06e140 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3988009830 tx=0x7f398802a510 comp rx=0 tx=0).stop 2024-09-11T00:45:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.404+0000 7f39a2800640 1 -- 172.21.15.31:0/3364699416 shutdown_connections 2024-09-11T00:45:47.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.404+0000 7f39a2800640 1 --2- 172.21.15.31:0/3364699416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f399c0ada20 0x7f399c06e140 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.404+0000 7f39a2800640 1 --2- 172.21.15.31:0/3364699416 >> v2:172.21.15.204:3300/0 conn(0x7f399c150020 0x7f399c0ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.404+0000 7f39a2800640 1 -- 172.21.15.31:0/3364699416 >> 172.21.15.31:0/3364699416 conn(0x7f399c068f10 msgr2=0x7f399c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.404+0000 7f39a2800640 1 -- 172.21.15.31:0/3364699416 shutdown_connections 2024-09-11T00:45:47.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.404+0000 7f39a2800640 1 -- 172.21.15.31:0/3364699416 wait complete. 2024-09-11T00:45:47.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.405+0000 7f39a2800640 1 Processor -- start 2024-09-11T00:45:47.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.405+0000 7f39a2800640 1 -- start start 2024-09-11T00:45:47.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.406+0000 7f39a2800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f399c0ada20 0x7f399c0ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:47.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.406+0000 7f39a2800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f399c150020 0x7f399c14ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:47.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.406+0000 7f39a2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f399c156960 0x7f399c14b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:47.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.406+0000 7f39a2800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f399c157550 con 0x7f399c156960 2024-09-11T00:45:47.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.406+0000 7f39a2800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f399c1573d0 con 0x7f399c150020 2024-09-11T00:45:47.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.406+0000 7f39a2800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f399c0cc220 con 0x7f399c0ada20 2024-09-11T00:45:47.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f399c156960 0x7f399c14b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:47.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f399c156960 0x7f399c14b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57526/0 (socket says 172.21.15.31:57526) 2024-09-11T00:45:47.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a1e00640 1 -- 172.21.15.31:0/2205214207 learned_addr learned my addr 172.21.15.31:0/2205214207 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:47.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a1400640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f399c0ada20 0x7f399c0ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:47.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a0a00640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f399c150020 0x7f399c14ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:47.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a1e00640 1 -- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f399c0ada20 msgr2=0x7f399c0ac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:47.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a1e00640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f399c0ada20 0x7f399c0ac620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:47.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a1e00640 1 -- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f399c150020 msgr2=0x7f399c14ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:47.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a1e00640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f399c150020 0x7f399c14ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:47.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a1e00640 1 -- 172.21.15.31:0/2205214207 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399c14f670 con 0x7f399c156960 2024-09-11T00:45:47.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.407+0000 7f39a1400640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f399c0ada20 0x7f399c0ac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:45:47.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.408+0000 7f39a1e00640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f399c156960 0x7f399c14b1a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f399400cb10 tx=0x7f399400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:47.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.408+0000 7f3999600640 1 -- 172.21.15.31:0/2205214207 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994016020 con 0x7f399c156960 2024-09-11T00:45:47.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.408+0000 7f3999600640 1 -- 172.21.15.31:0/2205214207 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f399400ad80 con 0x7f399c156960 2024-09-11T00:45:47.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.408+0000 7f3999600640 1 -- 172.21.15.31:0/2205214207 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994012680 con 0x7f399c156960 2024-09-11T00:45:47.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.408+0000 7f39a2800640 1 -- 172.21.15.31:0/2205214207 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399c0be990 con 0x7f399c156960 2024-09-11T00:45:47.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.408+0000 7f39a2800640 1 -- 172.21.15.31:0/2205214207 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f399c0bec30 con 0x7f399c156960 2024-09-11T00:45:47.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.409+0000 7f39a2800640 1 -- 172.21.15.31:0/2205214207 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f399c0c9630 con 0x7f399c156960 2024-09-11T00:45:47.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.411+0000 7f3999600640 1 -- 172.21.15.31:0/2205214207 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3994012820 con 0x7f399c156960 2024-09-11T00:45:47.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.412+0000 7f3999600640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f397c077f70 0x7f397c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:47.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.412+0000 7f3999600640 1 -- 172.21.15.31:0/2205214207 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7f399409afc0 con 0x7f399c156960 2024-09-11T00:45:47.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.412+0000 7f39a1400640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f397c077f70 0x7f397c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:47.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.413+0000 7f39a1400640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f397c077f70 0x7f397c07a430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f39900095f0 tx=0x7f3990005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:47.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.416+0000 7f3999600640 1 -- 172.21.15.31:0/2205214207 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f399c0c9630 con 0x7f399c156960 2024-09-11T00:45:47.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.670+0000 7f39a2800640 1 -- 172.21.15.31:0/2205214207 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f399c0ad4e0 con 0x7f399c156960 2024-09-11T00:45:47.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.670+0000 7f3999600640 1 -- 172.21.15.31:0/2205214207 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6544 (secure 0 0 0) 0x7f3994020020 con 0x7f399c156960 2024-09-11T00:45:47.672 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:47.672 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":117,"btime":"2024-09-11T00:45:33:802740+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 117 2024-09-11T00:45:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 -- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f397c077f70 msgr2=0x7f397c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f397c077f70 0x7f397c07a430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f39900095f0 tx=0x7f3990005f90 comp rx=0 tx=0).stop 2024-09-11T00:45:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 -- 172.21.15.31:0/2205214207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f399c156960 msgr2=0x7f399c14b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f399c156960 0x7f399c14b1a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f399400cb10 tx=0x7f399400a060 comp rx=0 tx=0).stop 2024-09-11T00:45:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 -- 172.21.15.31:0/2205214207 shutdown_connections 2024-09-11T00:45:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f397c077f70 0x7f397c07a430 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f399c156960 0x7f399c14b1a0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:47.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f399c150020 0x7f399c14ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:47.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 --2- 172.21.15.31:0/2205214207 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f399c0ada20 0x7f399c0ac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:47.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 -- 172.21.15.31:0/2205214207 >> 172.21.15.31:0/2205214207 conn(0x7f399c068f10 msgr2=0x7f399c1576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:47.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 -- 172.21.15.31:0/2205214207 shutdown_connections 2024-09-11T00:45:47.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:47.673+0000 7f39a2800640 1 -- 172.21.15.31:0/2205214207 wait complete. 2024-09-11T00:45:47.691 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:45:48.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:45:48.349 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:45:49.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.862+0000 7fb310800640 1 -- 172.21.15.31:0/17702481 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb308072480 msgr2=0x7fb308074a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.862+0000 7fb310800640 1 --2- 172.21.15.31:0/17702481 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb308072480 0x7fb308074a90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009830 tx=0x7fb2fc02a4d0 comp rx=0 tx=0).stop 2024-09-11T00:45:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.862+0000 7fb310800640 1 -- 172.21.15.31:0/17702481 shutdown_connections 2024-09-11T00:45:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.862+0000 7fb310800640 1 --2- 172.21.15.31:0/17702481 >> v2:172.21.15.204:3300/0 conn(0x7fb3081511c0 0x7fb308151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.862+0000 7fb310800640 1 --2- 172.21.15.31:0/17702481 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb308072480 0x7fb308074a90 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:49.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.862+0000 7fb310800640 1 -- 172.21.15.31:0/17702481 >> 172.21.15.31:0/17702481 conn(0x7fb3080a9750 msgr2=0x7fb3080a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:49.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.862+0000 7fb310800640 1 -- 172.21.15.31:0/17702481 shutdown_connections 2024-09-11T00:45:49.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.862+0000 7fb310800640 1 -- 172.21.15.31:0/17702481 wait complete. 2024-09-11T00:45:49.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.864+0000 7fb310800640 1 Processor -- start 2024-09-11T00:45:49.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.864+0000 7fb310800640 1 -- start start 2024-09-11T00:45:49.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb310800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb308072480 0x7fb3080c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:49.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb310800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3081511c0 0x7fb3081504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:49.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb310800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb308154b60 0x7fb30814f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:49.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb310800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb3080c23a0 con 0x7fb308154b60 2024-09-11T00:45:49.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb310800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb308154640 con 0x7fb3081511c0 2024-09-11T00:45:49.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb310800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb3080a7140 con 0x7fb308072480 2024-09-11T00:45:49.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb30e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb308154b60 0x7fb30814f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:49.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb30de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb308072480 0x7fb3080c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:49.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb30e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb308154b60 0x7fb30814f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50096/0 (socket says 172.21.15.31:50096) 2024-09-11T00:45:49.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb30e800640 1 -- 172.21.15.31:0/79000061 learned_addr learned my addr 172.21.15.31:0/79000061 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:49.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.865+0000 7fb30d400640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3081511c0 0x7fb3081504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.866+0000 7fb30e800640 1 -- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb308072480 msgr2=0x7fb3080c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.866+0000 7fb30e800640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb308072480 0x7fb3080c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.866+0000 7fb30e800640 1 -- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3081511c0 msgr2=0x7fb3081504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.866+0000 7fb30e800640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3081511c0 0x7fb3081504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.866+0000 7fb30e800640 1 -- 172.21.15.31:0/79000061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb308144940 con 0x7fb308154b60 2024-09-11T00:45:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.866+0000 7fb30e800640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb308154b60 0x7fb30814f7f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb30400eaa0 tx=0x7fb30400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.866+0000 7fb302000640 1 -- 172.21.15.31:0/79000061 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30400cdb0 con 0x7fb308154b60 2024-09-11T00:45:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.866+0000 7fb302000640 1 -- 172.21.15.31:0/79000061 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb304004c10 con 0x7fb308154b60 2024-09-11T00:45:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.866+0000 7fb302000640 1 -- 172.21.15.31:0/79000061 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304005020 con 0x7fb308154b60 2024-09-11T00:45:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.866+0000 7fb310800640 1 -- 172.21.15.31:0/79000061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb308147f50 con 0x7fb308154b60 2024-09-11T00:45:49.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.867+0000 7fb310800640 1 -- 172.21.15.31:0/79000061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3080c6350 con 0x7fb308154b60 2024-09-11T00:45:49.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.868+0000 7fb310800640 1 -- 172.21.15.31:0/79000061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3080c6550 con 0x7fb308154b60 2024-09-11T00:45:49.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.869+0000 7fb302000640 1 -- 172.21.15.31:0/79000061 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb30400f040 con 0x7fb308154b60 2024-09-11T00:45:49.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.870+0000 7fb302000640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2ec077f70 0x7fb2ec07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:49.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.870+0000 7fb302000640 1 -- 172.21.15.31:0/79000061 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7fb30409aef0 con 0x7fb308154b60 2024-09-11T00:45:49.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.871+0000 7fb30de00640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2ec077f70 0x7fb2ec07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:49.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.872+0000 7fb30de00640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2ec077f70 0x7fb2ec07a430 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc034000 tx=0x7fb2fc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:49.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:49.876+0000 7fb302000640 1 -- 172.21.15.31:0/79000061 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb3080c6550 con 0x7fb308154b60 2024-09-11T00:45:50.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.176+0000 7fb310800640 1 -- 172.21.15.31:0/79000061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb308072860 con 0x7fb308154b60 2024-09-11T00:45:50.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.177+0000 7fb302000640 1 -- 172.21.15.31:0/79000061 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6544 (secure 0 0 0) 0x7fb308147f50 con 0x7fb308154b60 2024-09-11T00:45:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:50.182 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":117,"btime":"2024-09-11T00:45:33:802740+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:50.184 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 117 2024-09-11T00:45:50.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 -- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2ec077f70 msgr2=0x7fb2ec07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:50.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2ec077f70 0x7fb2ec07a430 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc034000 tx=0x7fb2fc004730 comp rx=0 tx=0).stop 2024-09-11T00:45:50.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 -- 172.21.15.31:0/79000061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb308154b60 msgr2=0x7fb30814f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:50.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb308154b60 0x7fb30814f7f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb30400eaa0 tx=0x7fb30400ef60 comp rx=0 tx=0).stop 2024-09-11T00:45:50.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 -- 172.21.15.31:0/79000061 shutdown_connections 2024-09-11T00:45:50.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb2ec077f70 0x7fb2ec07a430 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:50.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb308154b60 0x7fb30814f7f0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:50.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb3081511c0 0x7fb3081504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:50.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 --2- 172.21.15.31:0/79000061 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb308072480 0x7fb3080c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:50.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 -- 172.21.15.31:0/79000061 >> 172.21.15.31:0/79000061 conn(0x7fb3080a9750 msgr2=0x7fb308152a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:50.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 -- 172.21.15.31:0/79000061 shutdown_connections 2024-09-11T00:45:50.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:50.180+0000 7fb310800640 1 -- 172.21.15.31:0/79000061 wait complete. 2024-09-11T00:45:50.200 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:45:52.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:52.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.372+0000 7f06f3e00640 1 -- 172.21.15.31:0/2192083624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06d8005390 msgr2=0x7f06d800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:52.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.372+0000 7f06f3e00640 1 --2- 172.21.15.31:0/2192083624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06d8005390 0x7f06d800b9c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f06ec001f30 tx=0x7f06d8005830 comp rx=0 tx=0).stop 2024-09-11T00:45:52.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.372+0000 7f06f3e00640 1 -- 172.21.15.31:0/2192083624 shutdown_connections 2024-09-11T00:45:52.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.372+0000 7f06f3e00640 1 --2- 172.21.15.31:0/2192083624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06d8005390 0x7f06d800b9c0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:52.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.372+0000 7f06f3e00640 1 -- 172.21.15.31:0/2192083624 >> 172.21.15.31:0/2192083624 conn(0x7f06f406caf0 msgr2=0x7f06f406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:52.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.372+0000 7f06f3e00640 1 -- 172.21.15.31:0/2192083624 shutdown_connections 2024-09-11T00:45:52.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.372+0000 7f06f3e00640 1 -- 172.21.15.31:0/2192083624 wait complete. 2024-09-11T00:45:52.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.374+0000 7f06f3e00640 1 Processor -- start 2024-09-11T00:45:52.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.374+0000 7f06f3e00640 1 -- start start 2024-09-11T00:45:52.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.375+0000 7f06f3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06d8005390 0x7f06f414ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:52.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.375+0000 7f06f3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06f41568a0 0x7f06f4156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:52.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.375+0000 7f06f3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06f4155930 0x7f06f4155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:52.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.375+0000 7f06f3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f06f4151450 con 0x7f06d8005390 2024-09-11T00:45:52.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.375+0000 7f06f3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41512d0 con 0x7f06f41568a0 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.375+0000 7f06f3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f06f40c5d20 con 0x7f06f4155930 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.375+0000 7f06f2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06d8005390 0x7f06f414ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.375+0000 7f06f2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06d8005390 0x7f06f414ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50114/0 (socket says 172.21.15.31:50114) 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.375+0000 7f06f2a00640 1 -- 172.21.15.31:0/1030288416 learned_addr learned my addr 172.21.15.31:0/1030288416 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06f3400640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06f4155930 0x7f06f4155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06ebe00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06f41568a0 0x7f06f4156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06f2a00640 1 -- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06f4155930 msgr2=0x7f06f4155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06f2a00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06f4155930 0x7f06f4155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06f2a00640 1 -- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06f41568a0 msgr2=0x7f06f4156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06f2a00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06f41568a0 0x7f06f4156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:52.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06f2a00640 1 -- 172.21.15.31:0/1030288416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f41582e0 con 0x7f06d8005390 2024-09-11T00:45:52.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06f2a00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06d8005390 0x7f06f414ad80 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f06e4006630 tx=0x7f06e40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:52.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06eaa00640 1 -- 172.21.15.31:0/1030288416 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4015070 con 0x7f06d8005390 2024-09-11T00:45:52.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06f3e00640 1 -- 172.21.15.31:0/1030288416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06f41ecbd0 con 0x7f06d8005390 2024-09-11T00:45:52.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06eaa00640 1 -- 172.21.15.31:0/1030288416 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06e4011840 con 0x7f06d8005390 2024-09-11T00:45:52.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.376+0000 7f06eaa00640 1 -- 172.21.15.31:0/1030288416 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4011c70 con 0x7f06d8005390 2024-09-11T00:45:52.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.377+0000 7f06f3e00640 1 -- 172.21.15.31:0/1030288416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f4074850 con 0x7f06d8005390 2024-09-11T00:45:52.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.378+0000 7f06f3e00640 1 -- 172.21.15.31:0/1030288416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06f414f310 con 0x7f06d8005390 2024-09-11T00:45:52.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.379+0000 7f06eaa00640 1 -- 172.21.15.31:0/1030288416 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f06e40103c0 con 0x7f06d8005390 2024-09-11T00:45:52.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.380+0000 7f06eaa00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f06d0077fa0 0x7f06d007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:52.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.380+0000 7f06eaa00640 1 -- 172.21.15.31:0/1030288416 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7f06e409b820 con 0x7f06d8005390 2024-09-11T00:45:52.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.380+0000 7f06ebe00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f06d0077fa0 0x7f06d007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:52.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.381+0000 7f06ebe00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f06d0077fa0 0x7f06d007a460 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f06f406d270 tx=0x7f06dc005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:52.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.385+0000 7f06eaa00640 1 -- 172.21.15.31:0/1030288416 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f06f414f310 con 0x7f06d8005390 2024-09-11T00:45:52.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.701+0000 7f06f3e00640 1 -- 172.21.15.31:0/1030288416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06f4074360 con 0x7f06d8005390 2024-09-11T00:45:52.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.702+0000 7f06eaa00640 1 -- 172.21.15.31:0/1030288416 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6544 (secure 0 0 0) 0x7f06e4063870 con 0x7f06d8005390 2024-09-11T00:45:52.705 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:52.705 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":117,"btime":"2024-09-11T00:45:33:802740+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:52.705 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 117 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 -- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f06d0077fa0 msgr2=0x7f06d007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f06d0077fa0 0x7f06d007a460 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f06f406d270 tx=0x7f06dc005f90 comp rx=0 tx=0).stop 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 -- 172.21.15.31:0/1030288416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06d8005390 msgr2=0x7f06f414ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06d8005390 0x7f06f414ad80 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f06e4006630 tx=0x7f06e40039c0 comp rx=0 tx=0).stop 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 -- 172.21.15.31:0/1030288416 shutdown_connections 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f06d0077fa0 0x7f06d007a460 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06f4155930 0x7f06f4155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06f41568a0 0x7f06f4156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 --2- 172.21.15.31:0/1030288416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06d8005390 0x7f06f414ad80 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 -- 172.21.15.31:0/1030288416 >> 172.21.15.31:0/1030288416 conn(0x7f06f406caf0 msgr2=0x7f06f4153410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:52.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 -- 172.21.15.31:0/1030288416 shutdown_connections 2024-09-11T00:45:52.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:52.705+0000 7f06f3e00640 1 -- 172.21.15.31:0/1030288416 wait complete. 2024-09-11T00:45:52.728 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:45:54.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:54.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.899+0000 7f8de2c00640 1 -- 172.21.15.31:0/4200332775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ddc150020 msgr2=0x7f8ddc0ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:54.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.899+0000 7f8de2c00640 1 --2- 172.21.15.31:0/4200332775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ddc150020 0x7f8ddc0ad4e0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc153d20 tx=0x7f8dd002a510 comp rx=0 tx=0).stop 2024-09-11T00:45:54.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.899+0000 7f8de2c00640 1 -- 172.21.15.31:0/4200332775 shutdown_connections 2024-09-11T00:45:54.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.899+0000 7f8de2c00640 1 --2- 172.21.15.31:0/4200332775 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ddc06e850 0x7f8ddc160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:54.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.899+0000 7f8de2c00640 1 --2- 172.21.15.31:0/4200332775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ddc150020 0x7f8ddc0ad4e0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:54.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.899+0000 7f8de2c00640 1 -- 172.21.15.31:0/4200332775 >> 172.21.15.31:0/4200332775 conn(0x7f8ddc068f10 msgr2=0x7f8ddc0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:54.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.899+0000 7f8de2c00640 1 -- 172.21.15.31:0/4200332775 shutdown_connections 2024-09-11T00:45:54.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.899+0000 7f8de2c00640 1 -- 172.21.15.31:0/4200332775 wait complete. 2024-09-11T00:45:54.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.901+0000 7f8de2c00640 1 Processor -- start 2024-09-11T00:45:54.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.901+0000 7f8de2c00640 1 -- start start 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de2c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ddc06e850 0x7f8ddc1f0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de2c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ddc0ada20 0x7f8ddc0bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de2c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ddc150020 0x7f8ddc0c0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de2c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc157550 con 0x7f8ddc150020 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de2c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc1573d0 con 0x7f8ddc06e850 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de2c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8ddc156960 con 0x7f8ddc0ada20 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de2200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ddc150020 0x7f8ddc0c0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de2200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ddc150020 0x7f8ddc0c0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50120/0 (socket says 172.21.15.31:50120) 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de2200640 1 -- 172.21.15.31:0/3630423877 learned_addr learned my addr 172.21.15.31:0/3630423877 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de0e00640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ddc0ada20 0x7f8ddc0bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:54.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.902+0000 7f8de1800640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ddc06e850 0x7f8ddc1f0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:54.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8de2200640 1 -- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ddc0ada20 msgr2=0x7f8ddc0bf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8de2200640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ddc0ada20 0x7f8ddc0bf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8de2200640 1 -- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ddc06e850 msgr2=0x7f8ddc1f0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8de2200640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ddc06e850 0x7f8ddc1f0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8de2200640 1 -- 172.21.15.31:0/3630423877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ddc1f0c30 con 0x7f8ddc150020 2024-09-11T00:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8de2200640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ddc150020 0x7f8ddc0c0ea0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400e990 tx=0x7f8dd400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8dd9600640 1 -- 172.21.15.31:0/3630423877 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4004d20 con 0x7f8ddc150020 2024-09-11T00:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8dd9600640 1 -- 172.21.15.31:0/3630423877 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8dd400cb00 con 0x7f8ddc150020 2024-09-11T00:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8de2c00640 1 -- 172.21.15.31:0/3630423877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ddc1f0950 con 0x7f8ddc150020 2024-09-11T00:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8dd9600640 1 -- 172.21.15.31:0/3630423877 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4005020 con 0x7f8ddc150020 2024-09-11T00:45:54.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.903+0000 7f8de2c00640 1 -- 172.21.15.31:0/3630423877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ddc0c0680 con 0x7f8ddc150020 2024-09-11T00:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.905+0000 7f8de2c00640 1 -- 172.21.15.31:0/3630423877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ddc0cc220 con 0x7f8ddc150020 2024-09-11T00:45:54.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.906+0000 7f8dd9600640 1 -- 172.21.15.31:0/3630423877 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8dd400f040 con 0x7f8ddc150020 2024-09-11T00:45:54.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.907+0000 7f8dd9600640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8dbc077f70 0x7f8dbc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:54.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.907+0000 7f8dd9600640 1 -- 172.21.15.31:0/3630423877 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7f8dd409ae20 con 0x7f8ddc150020 2024-09-11T00:45:54.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.907+0000 7f8de1800640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8dbc077f70 0x7f8dbc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:54.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.908+0000 7f8de1800640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8dbc077f70 0x7f8dbc07a430 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0034000 tx=0x7f8dd0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:54.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:54.913+0000 7f8dd9600640 1 -- 172.21.15.31:0/3630423877 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8dd4016030 con 0x7f8ddc150020 2024-09-11T00:45:55.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.163+0000 7f8de2c00640 1 -- 172.21.15.31:0/3630423877 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ddc0c5e00 con 0x7f8ddc150020 2024-09-11T00:45:55.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.164+0000 7f8dd9600640 1 -- 172.21.15.31:0/3630423877 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6544 (secure 0 0 0) 0x7f8dd4062750 con 0x7f8ddc150020 2024-09-11T00:45:55.167 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:55.167 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":117,"btime":"2024-09-11T00:45:33:802740+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:55.167 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 117 2024-09-11T00:45:55.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.166+0000 7f8de2c00640 1 -- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8dbc077f70 msgr2=0x7f8dbc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.166+0000 7f8de2c00640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8dbc077f70 0x7f8dbc07a430 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0034000 tx=0x7f8dd0004730 comp rx=0 tx=0).stop 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.166+0000 7f8de2c00640 1 -- 172.21.15.31:0/3630423877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ddc150020 msgr2=0x7f8ddc0c0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.166+0000 7f8de2c00640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ddc150020 0x7f8ddc0c0ea0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400e990 tx=0x7f8dd400ee50 comp rx=0 tx=0).stop 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.166+0000 7f8de2c00640 1 -- 172.21.15.31:0/3630423877 shutdown_connections 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.166+0000 7f8de2c00640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8dbc077f70 0x7f8dbc07a430 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.166+0000 7f8de2c00640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ddc150020 0x7f8ddc0c0ea0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.166+0000 7f8de2c00640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ddc0ada20 0x7f8ddc0bf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.166+0000 7f8de2c00640 1 --2- 172.21.15.31:0/3630423877 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ddc06e850 0x7f8ddc1f0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.167+0000 7f8de2c00640 1 -- 172.21.15.31:0/3630423877 >> 172.21.15.31:0/3630423877 conn(0x7f8ddc068f10 msgr2=0x7f8ddc14ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.167+0000 7f8de2c00640 1 -- 172.21.15.31:0/3630423877 shutdown_connections 2024-09-11T00:45:55.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:55.167+0000 7f8de2c00640 1 -- 172.21.15.31:0/3630423877 wait complete. 2024-09-11T00:45:55.185 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:45:57.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:57.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.371+0000 7efeba000640 1 -- 172.21.15.31:0/269401257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efeb40c6420 msgr2=0x7efeb4153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:57.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.371+0000 7efeba000640 1 --2- 172.21.15.31:0/269401257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efeb40c6420 0x7efeb4153310 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efea0005fd0 tx=0x7efea002a5d0 comp rx=0 tx=0).stop 2024-09-11T00:45:57.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.372+0000 7efeba000640 1 -- 172.21.15.31:0/269401257 shutdown_connections 2024-09-11T00:45:57.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.372+0000 7efeba000640 1 --2- 172.21.15.31:0/269401257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efeb40c6420 0x7efeb4153310 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:57.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.372+0000 7efeba000640 1 --2- 172.21.15.31:0/269401257 >> v2:172.21.15.204:3300/0 conn(0x7efeb40c5aa0 0x7efeb40c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:57.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.372+0000 7efeba000640 1 -- 172.21.15.31:0/269401257 >> 172.21.15.31:0/269401257 conn(0x7efeb406caf0 msgr2=0x7efeb406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:57.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.372+0000 7efeba000640 1 -- 172.21.15.31:0/269401257 shutdown_connections 2024-09-11T00:45:57.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.372+0000 7efeba000640 1 -- 172.21.15.31:0/269401257 wait complete. 2024-09-11T00:45:57.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.373+0000 7efeba000640 1 Processor -- start 2024-09-11T00:45:57.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.374+0000 7efeba000640 1 -- start start 2024-09-11T00:45:57.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeba000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efeb40c5aa0 0x7efeb4151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:57.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeba000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efeb40c6420 0x7efeb40aa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:57.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeba000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeb4153850 0x7efeb40aade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:57.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeba000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efeb40c2e80 con 0x7efeb4153850 2024-09-11T00:45:57.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeba000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efeb40c2ba0 con 0x7efeb40c6420 2024-09-11T00:45:57.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeba000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7efeb40c5e80 con 0x7efeb40c5aa0 2024-09-11T00:45:57.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeb9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeb4153850 0x7efeb40aade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:57.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeb3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efeb40c6420 0x7efeb40aa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:57.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeb9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeb4153850 0x7efeb40aade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50132/0 (socket says 172.21.15.31:50132) 2024-09-11T00:45:57.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeb9600640 1 -- 172.21.15.31:0/210057779 learned_addr learned my addr 172.21.15.31:0/210057779 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:57.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.375+0000 7efeb8c00640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efeb40c5aa0 0x7efeb4151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:57.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeb9600640 1 -- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efeb40c5aa0 msgr2=0x7efeb4151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:57.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeb9600640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efeb40c5aa0 0x7efeb4151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:57.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeb9600640 1 -- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efeb40c6420 msgr2=0x7efeb40aa8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:57.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeb9600640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efeb40c6420 0x7efeb40aa8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:57.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeb9600640 1 -- 172.21.15.31:0/210057779 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb41ece30 con 0x7efeb4153850 2024-09-11T00:45:57.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeb9600640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeb4153850 0x7efeb40aade0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7efeac00c700 tx=0x7efeac00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:57.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeb0c00640 1 -- 172.21.15.31:0/210057779 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac003b70 con 0x7efeb4153850 2024-09-11T00:45:57.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeb0c00640 1 -- 172.21.15.31:0/210057779 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efeac003d10 con 0x7efeb4153850 2024-09-11T00:45:57.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeb0c00640 1 -- 172.21.15.31:0/210057779 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac00aab0 con 0x7efeb4153850 2024-09-11T00:45:57.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeba000640 1 -- 172.21.15.31:0/210057779 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb41ecb50 con 0x7efeb4153850 2024-09-11T00:45:57.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.376+0000 7efeba000640 1 -- 172.21.15.31:0/210057779 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb4151750 con 0x7efeb4153850 2024-09-11T00:45:57.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.378+0000 7efeba000640 1 -- 172.21.15.31:0/210057779 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb40c3f80 con 0x7efeb4153850 2024-09-11T00:45:57.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.379+0000 7efeb0c00640 1 -- 172.21.15.31:0/210057779 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7efeac00ac50 con 0x7efeb4153850 2024-09-11T00:45:57.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.380+0000 7efeb0c00640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efe94077fa0 0x7efe9407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:57.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.381+0000 7efeb0c00640 1 -- 172.21.15.31:0/210057779 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8472+0+0 (secure 0 0 0) 0x7efeac09a490 con 0x7efeb4153850 2024-09-11T00:45:57.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.381+0000 7efeb8c00640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efe94077fa0 0x7efe9407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:57.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.382+0000 7efeb8c00640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efe94077fa0 0x7efe9407a460 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efeb4152080 tx=0x7efea8005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:57.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.385+0000 7efeb0c00640 1 -- 172.21.15.31:0/210057779 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efeb40c3f80 con 0x7efeb4153850 2024-09-11T00:45:57.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.669+0000 7efeba000640 1 -- 172.21.15.31:0/210057779 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efeb40c7150 con 0x7efeb4153850 2024-09-11T00:45:57.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.670+0000 7efeb0c00640 1 -- 172.21.15.31:0/210057779 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6544 (secure 0 0 0) 0x7efeb40c7690 con 0x7efeb4153850 2024-09-11T00:45:57.673 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:45:57.673 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":117,"btime":"2024-09-11T00:45:33:802740+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:17.724314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5834":{"gid":5834,"name":"c","rank":0,"incarnation":108,"state":"up:active","state_seq":10,"addr":"172.21.15.31:6835/1396366029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1396366029},{"type":"v1","addr":"172.21.15.31:6835","nonce":1396366029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5834,"qdb_cluster":[5834]},"id":2}]} 2024-09-11T00:45:57.673 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 117 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 -- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efe94077fa0 msgr2=0x7efe9407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efe94077fa0 0x7efe9407a460 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efeb4152080 tx=0x7efea8005f90 comp rx=0 tx=0).stop 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 -- 172.21.15.31:0/210057779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeb4153850 msgr2=0x7efeb40aade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeb4153850 0x7efeb40aade0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7efeac00c700 tx=0x7efeac00cbc0 comp rx=0 tx=0).stop 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 -- 172.21.15.31:0/210057779 shutdown_connections 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efe94077fa0 0x7efe9407a460 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeb4153850 0x7efeb40aade0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efeb40c6420 0x7efeb40aa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 --2- 172.21.15.31:0/210057779 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7efeb40c5aa0 0x7efeb4151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 -- 172.21.15.31:0/210057779 >> 172.21.15.31:0/210057779 conn(0x7efeb406caf0 msgr2=0x7efeb40a81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:57.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 -- 172.21.15.31:0/210057779 shutdown_connections 2024-09-11T00:45:57.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:57.673+0000 7efeba000640 1 -- 172.21.15.31:0/210057779 wait complete. 2024-09-11T00:45:57.692 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:45:59.693 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:45:59.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.869+0000 7fad6aa00640 1 -- 172.21.15.31:0/821501745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad64150020 msgr2=0x7fad640ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:59.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.869+0000 7fad6aa00640 1 --2- 172.21.15.31:0/821501745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad64150020 0x7fad640ad4e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fad64153d20 tx=0x7fad5802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:45:59.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.869+0000 7fad6aa00640 1 -- 172.21.15.31:0/821501745 shutdown_connections 2024-09-11T00:45:59.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.869+0000 7fad6aa00640 1 --2- 172.21.15.31:0/821501745 >> v2:172.21.15.204:3300/0 conn(0x7fad640ada20 0x7fad6406e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:59.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.869+0000 7fad6aa00640 1 --2- 172.21.15.31:0/821501745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad64150020 0x7fad640ad4e0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:59.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.869+0000 7fad6aa00640 1 -- 172.21.15.31:0/821501745 >> 172.21.15.31:0/821501745 conn(0x7fad64068f10 msgr2=0x7fad640cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:45:59.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.870+0000 7fad6aa00640 1 -- 172.21.15.31:0/821501745 shutdown_connections 2024-09-11T00:45:59.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.870+0000 7fad6aa00640 1 -- 172.21.15.31:0/821501745 wait complete. 2024-09-11T00:45:59.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.871+0000 7fad6aa00640 1 Processor -- start 2024-09-11T00:45:59.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.872+0000 7fad6aa00640 1 -- start start 2024-09-11T00:45:59.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad6aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad640ada20 0x7fad640b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:59.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad6aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad64150020 0x7fad64154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:59.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad6aa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad64156960 0x7fad6414efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:59.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad6aa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fad64157550 con 0x7fad64150020 2024-09-11T00:45:59.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad6aa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fad641573d0 con 0x7fad64156960 2024-09-11T00:45:59.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad6aa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fad640cc220 con 0x7fad640ada20 2024-09-11T00:45:59.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad68c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad64150020 0x7fad64154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:59.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad69600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad640ada20 0x7fad640b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:59.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad69600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad640ada20 0x7fad640b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:54052/0 (socket says 172.21.15.31:54052) 2024-09-11T00:45:59.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad69600640 1 -- 172.21.15.31:0/388426310 learned_addr learned my addr 172.21.15.31:0/388426310 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:45:59.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.873+0000 7fad6a000640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad64156960 0x7fad6414efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:59.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.874+0000 7fad68c00640 1 -- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad640ada20 msgr2=0x7fad640b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:59.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.874+0000 7fad68c00640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad640ada20 0x7fad640b10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:59.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.874+0000 7fad68c00640 1 -- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad64156960 msgr2=0x7fad6414efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:45:59.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.874+0000 7fad68c00640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad64156960 0x7fad6414efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:45:59.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.874+0000 7fad68c00640 1 -- 172.21.15.31:0/388426310 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad640af840 con 0x7fad64150020 2024-09-11T00:45:59.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.874+0000 7fad68c00640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad64150020 0x7fad64154c40 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fad5000e700 tx=0x7fad5000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:59.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.874+0000 7fad61600640 1 -- 172.21.15.31:0/388426310 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5000cab0 con 0x7fad64150020 2024-09-11T00:45:59.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.874+0000 7fad61600640 1 -- 172.21.15.31:0/388426310 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad5000cd40 con 0x7fad64150020 2024-09-11T00:45:59.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.875+0000 7fad6aa00640 1 -- 172.21.15.31:0/388426310 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad640b0830 con 0x7fad64150020 2024-09-11T00:45:59.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.875+0000 7fad61600640 1 -- 172.21.15.31:0/388426310 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50010690 con 0x7fad64150020 2024-09-11T00:45:59.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.875+0000 7fad6aa00640 1 -- 172.21.15.31:0/388426310 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad64155180 con 0x7fad64150020 2024-09-11T00:45:59.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.876+0000 7fad6aa00640 1 -- 172.21.15.31:0/388426310 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad640be620 con 0x7fad64150020 2024-09-11T00:45:59.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.878+0000 7fad61600640 1 -- 172.21.15.31:0/388426310 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fad50010830 con 0x7fad64150020 2024-09-11T00:45:59.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.879+0000 7fad61600640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad44077f70 0x7fad4407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:45:59.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.879+0000 7fad61600640 1 -- 172.21.15.31:0/388426310 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7fad5009b160 con 0x7fad64150020 2024-09-11T00:45:59.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.879+0000 7fad69600640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad44077f70 0x7fad4407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:45:59.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.880+0000 7fad69600640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad44077f70 0x7fad4407a430 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fad64153730 tx=0x7fad58004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:45:59.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:45:59.885+0000 7fad61600640 1 -- 172.21.15.31:0/388426310 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fad50062ad0 con 0x7fad64150020 2024-09-11T00:46:00.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.123+0000 7fad6aa00640 1 -- 172.21.15.31:0/388426310 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad640ad4e0 con 0x7fad64150020 2024-09-11T00:46:00.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.123+0000 7fad61600640 1 -- 172.21.15.31:0/388426310 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+5710 (secure 0 0 0) 0x7fad50067500 con 0x7fad64150020 2024-09-11T00:46:00.126 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:46:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":118,"btime":"2024-09-11T00:45:59:426600+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:59.426594+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:46:00.127 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 118 2024-09-11T00:46:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 -- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad44077f70 msgr2=0x7fad4407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad44077f70 0x7fad4407a430 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fad64153730 tx=0x7fad58004730 comp rx=0 tx=0).stop 2024-09-11T00:46:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 -- 172.21.15.31:0/388426310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad64150020 msgr2=0x7fad64154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad64150020 0x7fad64154c40 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fad5000e700 tx=0x7fad5000ebc0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 -- 172.21.15.31:0/388426310 shutdown_connections 2024-09-11T00:46:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fad44077f70 0x7fad4407a430 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fad64156960 0x7fad6414efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad64150020 0x7fad64154c40 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 --2- 172.21.15.31:0/388426310 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fad640ada20 0x7fad640b10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 -- 172.21.15.31:0/388426310 >> 172.21.15.31:0/388426310 conn(0x7fad64068f10 msgr2=0x7fad64153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:46:00.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 -- 172.21.15.31:0/388426310 shutdown_connections 2024-09-11T00:46:00.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.126+0000 7fad6aa00640 1 -- 172.21.15.31:0/388426310 wait complete. 2024-09-11T00:46:00.151 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-11T00:46:00.151 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:46:00.151 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:46:00.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.309+0000 7f8959000640 1 -- 172.21.15.31:0/9849257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89540c5b10 msgr2=0x7f8954156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:00.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.309+0000 7f8959000640 1 --2- 172.21.15.31:0/9849257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89540c5b10 0x7f8954156ad0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8944009830 tx=0x7f894402a510 comp rx=0 tx=0).stop 2024-09-11T00:46:00.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.309+0000 7f8959000640 1 -- 172.21.15.31:0/9849257 shutdown_connections 2024-09-11T00:46:00.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.309+0000 7f8959000640 1 --2- 172.21.15.31:0/9849257 >> v2:172.21.15.204:3300/0 conn(0x7f89540c6310 0x7f8954159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.309+0000 7f8959000640 1 --2- 172.21.15.31:0/9849257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89540c5b10 0x7f8954156ad0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.309+0000 7f8959000640 1 -- 172.21.15.31:0/9849257 >> 172.21.15.31:0/9849257 conn(0x7f89540a9750 msgr2=0x7f89540a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:46:00.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.310+0000 7f8959000640 1 -- 172.21.15.31:0/9849257 shutdown_connections 2024-09-11T00:46:00.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.310+0000 7f8959000640 1 -- 172.21.15.31:0/9849257 wait complete. 2024-09-11T00:46:00.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.311+0000 7f8959000640 1 Processor -- start 2024-09-11T00:46:00.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.311+0000 7f8959000640 1 -- start start 2024-09-11T00:46:00.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8959000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89540c5b10 0x7f8954074740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:00.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8959000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89540c5ef0 0x7f8954072380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:00.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8959000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89540c6310 0x7f89541ebff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:00.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8959000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8954153b90 con 0x7f89540c5ef0 2024-09-11T00:46:00.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8959000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f89540c3950 con 0x7f89540c6310 2024-09-11T00:46:00.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8959000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8954153d10 con 0x7f89540c5b10 2024-09-11T00:46:00.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8949400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89540c5ef0 0x7f8954072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:00.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8949400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89540c5ef0 0x7f8954072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59164/0 (socket says 172.21.15.31:59164) 2024-09-11T00:46:00.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8951e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89540c5b10 0x7f8954074740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:00.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8949400640 1 -- 172.21.15.31:0/972409511 learned_addr learned my addr 172.21.15.31:0/972409511 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:46:00.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.312+0000 7f8952800640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89540c6310 0x7f89541ebff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:00.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f8951e00640 1 -- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89540c6310 msgr2=0x7f89541ebff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:00.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f8951e00640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89540c6310 0x7f89541ebff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f8951e00640 1 -- 172.21.15.31:0/972409511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89540c5ef0 msgr2=0x7f8954072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:00.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f8951e00640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89540c5ef0 0x7f8954072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f8951e00640 1 -- 172.21.15.31:0/972409511 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89541532e0 con 0x7f89540c5b10 2024-09-11T00:46:00.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f8949400640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89540c5ef0 0x7f8954072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:46:00.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f8951e00640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89540c5b10 0x7f8954074740 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8944031000 tx=0x7f8944032b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:46:00.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f894aa00640 1 -- 172.21.15.31:0/972409511 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944040070 con 0x7f89540c5b10 2024-09-11T00:46:00.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f894aa00640 1 -- 172.21.15.31:0/972409511 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8944003900 con 0x7f89540c5b10 2024-09-11T00:46:00.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f8959000640 1 -- 172.21.15.31:0/972409511 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895414c620 con 0x7f89540c5b10 2024-09-11T00:46:00.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.313+0000 7f894aa00640 1 -- 172.21.15.31:0/972409511 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944003d50 con 0x7f89540c5b10 2024-09-11T00:46:00.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.314+0000 7f8959000640 1 -- 172.21.15.31:0/972409511 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8954074240 con 0x7f89540c5b10 2024-09-11T00:46:00.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.315+0000 7f8959000640 1 -- 172.21.15.31:0/972409511 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8954156ad0 con 0x7f89540c5b10 2024-09-11T00:46:00.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.316+0000 7f894aa00640 1 -- 172.21.15.31:0/972409511 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f894404d030 con 0x7f89540c5b10 2024-09-11T00:46:00.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.317+0000 7f894aa00640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8930077f70 0x7f893007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:00.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.317+0000 7f894aa00640 1 -- 172.21.15.31:0/972409511 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7f89440bbc30 con 0x7f89540c5b10 2024-09-11T00:46:00.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.317+0000 7f8949400640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8930077f70 0x7f893007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:00.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.318+0000 7f8949400640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8930077f70 0x7f893007a430 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8954155a20 tx=0x7f893c0018e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:46:00.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.321+0000 7f894aa00640 1 -- 172.21.15.31:0/972409511 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f895414c620 con 0x7f89540c5b10 2024-09-11T00:46:00.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.557+0000 7f8959000640 1 -- 172.21.15.31:0/972409511 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89540c1e70 con 0x7f89540c5b10 2024-09-11T00:46:00.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.558+0000 7f894aa00640 1 -- 172.21.15.31:0/972409511 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5713 (secure 0 0 0) 0x7f8944083510 con 0x7f89540c5b10 2024-09-11T00:46:00.560 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:46:00.561 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":119,"btime":"2024-09-11T00:46:00:428386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:45:59.748514+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:46:00.561 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 119 2024-09-11T00:46:00.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 -- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8930077f70 msgr2=0x7f893007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:00.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8930077f70 0x7f893007a430 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8954155a20 tx=0x7f893c0018e0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 -- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89540c5b10 msgr2=0x7f8954074740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:00.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89540c5b10 0x7f8954074740 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8944031000 tx=0x7f8944032b90 comp rx=0 tx=0).stop 2024-09-11T00:46:00.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 -- 172.21.15.31:0/972409511 shutdown_connections 2024-09-11T00:46:00.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8930077f70 0x7f893007a430 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89540c6310 0x7f89541ebff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89540c5ef0 0x7f8954072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 --2- 172.21.15.31:0/972409511 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f89540c5b10 0x7f8954074740 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:00.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 -- 172.21.15.31:0/972409511 >> 172.21.15.31:0/972409511 conn(0x7f89540a9750 msgr2=0x7f89540a76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:46:00.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 -- 172.21.15.31:0/972409511 shutdown_connections 2024-09-11T00:46:00.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:00.560+0000 7f8959000640 1 -- 172.21.15.31:0/972409511 wait complete. 2024-09-11T00:46:00.580 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:46:00:428386+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 119, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 116, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6601": { "addr": "172.21.15.204:6835/2113744259", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 2113744259, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 2113744259, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6601, "incarnation": 113, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:45:29.788358+0000", "qdb_cluster": [ 6601 ], "qdb_leader": 6601, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6601 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 119, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5957": { "addr": "172.21.15.204:6837/1082799504", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 1082799504, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 1082799504, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5957, "incarnation": 118, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:45:59.748514+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5957 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:46:02.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:46:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.776+0000 7fdfee200640 1 -- 172.21.15.31:0/500226001 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfe8156960 msgr2=0x7fdfe806e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.776+0000 7fdfee200640 1 --2- 172.21.15.31:0/500226001 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfe8156960 0x7fdfe806e140 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4009830 tx=0x7fdfd402a480 comp rx=0 tx=0).stop 2024-09-11T00:46:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.777+0000 7fdfee200640 1 -- 172.21.15.31:0/500226001 shutdown_connections 2024-09-11T00:46:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.777+0000 7fdfee200640 1 --2- 172.21.15.31:0/500226001 >> v2:172.21.15.204:3300/0 conn(0x7fdfe80cc220 0x7fdfe806e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.777+0000 7fdfee200640 1 --2- 172.21.15.31:0/500226001 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfe8156960 0x7fdfe806e140 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.777+0000 7fdfee200640 1 -- 172.21.15.31:0/500226001 >> 172.21.15.31:0/500226001 conn(0x7fdfe8068f10 msgr2=0x7fdfe80cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:46:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.777+0000 7fdfee200640 1 -- 172.21.15.31:0/500226001 shutdown_connections 2024-09-11T00:46:02.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.777+0000 7fdfee200640 1 -- 172.21.15.31:0/500226001 wait complete. 2024-09-11T00:46:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.778+0000 7fdfee200640 1 Processor -- start 2024-09-11T00:46:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.779+0000 7fdfee200640 1 -- start start 2024-09-11T00:46:02.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.779+0000 7fdfee200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfe80cc220 0x7fdfe80c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfee200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8150020 0x7fdfe814e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfee200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8156960 0x7fdfe814e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfee200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe80c6ca0 con 0x7fdfe8150020 2024-09-11T00:46:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfee200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe80c6b20 con 0x7fdfe8156960 2024-09-11T00:46:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfee200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdfe81573d0 con 0x7fdfe80cc220 2024-09-11T00:46:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfe7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8150020 0x7fdfe814e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfe7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8150020 0x7fdfe814e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59166/0 (socket says 172.21.15.31:59166) 2024-09-11T00:46:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfe7e00640 1 -- 172.21.15.31:0/2458283313 learned_addr learned my addr 172.21.15.31:0/2458283313 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:46:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfece00640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfe80cc220 0x7fdfe80c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfed800640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8156960 0x7fdfe814e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:02.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfe7e00640 1 -- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfe80cc220 msgr2=0x7fdfe80c5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:02.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfe7e00640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfe80cc220 0x7fdfe80c5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfe7e00640 1 -- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8156960 msgr2=0x7fdfe814e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfe7e00640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8156960 0x7fdfe814e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.780+0000 7fdfe7e00640 1 -- 172.21.15.31:0/2458283313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe80ae680 con 0x7fdfe8150020 2024-09-11T00:46:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.781+0000 7fdfe7e00640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8150020 0x7fdfe814e270 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4000c00 tx=0x7fdfd4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:46:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.781+0000 7fdfe4c00640 1 -- 172.21.15.31:0/2458283313 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4005c90 con 0x7fdfe8150020 2024-09-11T00:46:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.781+0000 7fdfe4c00640 1 -- 172.21.15.31:0/2458283313 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdfd4031e50 con 0x7fdfe8150020 2024-09-11T00:46:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.781+0000 7fdfee200640 1 -- 172.21.15.31:0/2458283313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe80c1e40 con 0x7fdfe8150020 2024-09-11T00:46:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.781+0000 7fdfe4c00640 1 -- 172.21.15.31:0/2458283313 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4033290 con 0x7fdfe8150020 2024-09-11T00:46:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.781+0000 7fdfee200640 1 -- 172.21.15.31:0/2458283313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe80c2400 con 0x7fdfe8150020 2024-09-11T00:46:02.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.783+0000 7fdfee200640 1 -- 172.21.15.31:0/2458283313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfe80c7120 con 0x7fdfe8150020 2024-09-11T00:46:02.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.784+0000 7fdfe4c00640 1 -- 172.21.15.31:0/2458283313 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdfd4033430 con 0x7fdfe8150020 2024-09-11T00:46:02.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.785+0000 7fdfe4c00640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfc8077f70 0x7fdfc807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:02.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.785+0000 7fdfe4c00640 1 -- 172.21.15.31:0/2458283313 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7fdfd40bb930 con 0x7fdfe8150020 2024-09-11T00:46:02.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.785+0000 7fdfece00640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfc8077f70 0x7fdfc807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:02.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.786+0000 7fdfece00640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfc8077f70 0x7fdfc807a430 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc007fd0 tx=0x7fdfdc001430 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:46:02.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:02.789+0000 7fdfe4c00640 1 -- 172.21.15.31:0/2458283313 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdfe80c7120 con 0x7fdfe8150020 2024-09-11T00:46:03.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.047+0000 7fdfee200640 1 -- 172.21.15.31:0/2458283313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfe80ad4e0 con 0x7fdfe8150020 2024-09-11T00:46:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.047+0000 7fdfe4c00640 1 -- 172.21.15.31:0/2458283313 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+5718 (secure 0 0 0) 0x7fdfd403f030 con 0x7fdfe8150020 2024-09-11T00:46:03.050 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:46:03.050 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":121,"btime":"2024-09-11T00:46:02:442429+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:46:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 121 2024-09-11T00:46:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 -- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfc8077f70 msgr2=0x7fdfc807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfc8077f70 0x7fdfc807a430 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc007fd0 tx=0x7fdfdc001430 comp rx=0 tx=0).stop 2024-09-11T00:46:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 -- 172.21.15.31:0/2458283313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8150020 msgr2=0x7fdfe814e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8150020 0x7fdfe814e270 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4000c00 tx=0x7fdfd4033c20 comp rx=0 tx=0).stop 2024-09-11T00:46:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 -- 172.21.15.31:0/2458283313 shutdown_connections 2024-09-11T00:46:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdfc8077f70 0x7fdfc807a430 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8156960 0x7fdfe814e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdfe8150020 0x7fdfe814e270 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 --2- 172.21.15.31:0/2458283313 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdfe80cc220 0x7fdfe80c5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 -- 172.21.15.31:0/2458283313 >> 172.21.15.31:0/2458283313 conn(0x7fdfe8068f10 msgr2=0x7fdfe814f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:46:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 -- 172.21.15.31:0/2458283313 shutdown_connections 2024-09-11T00:46:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:03.050+0000 7fdfee200640 1 -- 172.21.15.31:0/2458283313 wait complete. 2024-09-11T00:46:03.070 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=5834 2024-09-11T00:46:03.071 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.c 2024-09-11T00:46:03.819 INFO:tasks.ceph.mds.b.smithi204.stderr:2024-09-11T00:46:03.817+0000 7f63cc400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:46:04.072 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-11T00:46:04.072 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-11T00:46:04.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-11T00:46:04.077 INFO:tasks.ceph.mds.c:Started 2024-09-11T00:46:04.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:46:04.225 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-11T00:46:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.273+0000 7f52a3200640 1 -- 172.21.15.31:0/2823176430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a4156960 msgr2=0x7f52a406e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.273+0000 7f52a3200640 1 --2- 172.21.15.31:0/2823176430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a4156960 0x7f52a406e140 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5290005fd0 tx=0x7f529002a540 comp rx=0 tx=0).stop 2024-09-11T00:46:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.274+0000 7f52a3200640 1 -- 172.21.15.31:0/2823176430 shutdown_connections 2024-09-11T00:46:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.274+0000 7f52a3200640 1 --2- 172.21.15.31:0/2823176430 >> v2:172.21.15.204:3300/0 conn(0x7f52a40cc220 0x7f52a406e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.274+0000 7f52a3200640 1 --2- 172.21.15.31:0/2823176430 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a4156960 0x7f52a406e140 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.274+0000 7f52a3200640 1 -- 172.21.15.31:0/2823176430 >> 172.21.15.31:0/2823176430 conn(0x7f52a4068f10 msgr2=0x7f52a40cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:46:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.274+0000 7f52a3200640 1 -- 172.21.15.31:0/2823176430 shutdown_connections 2024-09-11T00:46:04.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.274+0000 7f52a3200640 1 -- 172.21.15.31:0/2823176430 wait complete. 2024-09-11T00:46:04.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.276+0000 7f52a3200640 1 Processor -- start 2024-09-11T00:46:04.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.276+0000 7f52a3200640 1 -- start start 2024-09-11T00:46:04.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a40cc220 0x7f52a414acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:04.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a4150020 0x7f52a4158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:04.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a3200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a4156960 0x7f52a41ef9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:04.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f52a40c6ca0 con 0x7f52a40cc220 2024-09-11T00:46:04.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f52a40c6b20 con 0x7f52a4150020 2024-09-11T00:46:04.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a3200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f52a41573d0 con 0x7f52a4156960 2024-09-11T00:46:04.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a1400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a4150020 0x7f52a4158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:04.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a40cc220 0x7f52a414acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:04.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a1400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a4150020 0x7f52a4158160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:43442/0 (socket says 172.21.15.31:43442) 2024-09-11T00:46:04.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a2800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a4156960 0x7f52a41ef9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:04.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.277+0000 7f52a1400640 1 -- 172.21.15.31:0/3156675853 learned_addr learned my addr 172.21.15.31:0/3156675853 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:46:04.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.278+0000 7f52a1400640 1 -- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a4156960 msgr2=0x7f52a41ef9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:04.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.278+0000 7f52a1400640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a4156960 0x7f52a41ef9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:04.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.278+0000 7f52a1400640 1 -- 172.21.15.31:0/3156675853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a40cc220 msgr2=0x7f52a414acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:04.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.278+0000 7f52a1400640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a40cc220 0x7f52a414acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:04.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.278+0000 7f52a1400640 1 -- 172.21.15.31:0/3156675853 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a414b3f0 con 0x7f52a4150020 2024-09-11T00:46:04.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.278+0000 7f52a1400640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a4150020 0x7f52a4158160 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5290006010 tx=0x7f52900038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:46:04.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.279+0000 7f529a000640 1 -- 172.21.15.31:0/3156675853 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529003e030 con 0x7f52a4150020 2024-09-11T00:46:04.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.279+0000 7f529a000640 1 -- 172.21.15.31:0/3156675853 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5290003d90 con 0x7f52a4150020 2024-09-11T00:46:04.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.279+0000 7f529a000640 1 -- 172.21.15.31:0/3156675853 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5290035040 con 0x7f52a4150020 2024-09-11T00:46:04.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.279+0000 7f52a3200640 1 -- 172.21.15.31:0/3156675853 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a4150610 con 0x7f52a4150020 2024-09-11T00:46:04.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.279+0000 7f52a3200640 1 -- 172.21.15.31:0/3156675853 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a414f4b0 con 0x7f52a4150020 2024-09-11T00:46:04.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.280+0000 7f52a3200640 1 -- 172.21.15.31:0/3156675853 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a4150400 con 0x7f52a4150020 2024-09-11T00:46:04.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.282+0000 7f529a000640 1 -- 172.21.15.31:0/3156675853 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f52900049e0 con 0x7f52a4150020 2024-09-11T00:46:04.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.283+0000 7f529a000640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5284077fc0 0x7f528407a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:46:04.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.283+0000 7f529a000640 1 -- 172.21.15.31:0/3156675853 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7f52900bafb0 con 0x7f52a4150020 2024-09-11T00:46:04.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.283+0000 7f52a1e00640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5284077fc0 0x7f528407a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:46:04.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.284+0000 7f52a1e00640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5284077fc0 0x7f528407a480 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f52a40bf940 tx=0x7f5294009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:46:04.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.287+0000 7f529a000640 1 -- 172.21.15.31:0/3156675853 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5290082a00 con 0x7f52a4150020 2024-09-11T00:46:04.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.571+0000 7f52a3200640 1 -- 172.21.15.31:0/3156675853 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52a40c22f0 con 0x7f52a4150020 2024-09-11T00:46:04.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.572+0000 7f529a000640 1 -- 172.21.15.31:0/3156675853 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6544 (secure 0 0 0) 0x7f5290087430 con 0x7f52a4150020 2024-09-11T00:46:04.575 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:46:04.575 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":122,"btime":"2024-09-11T00:46:04:450048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6804,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:46:04.575 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 122 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 -- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5284077fc0 msgr2=0x7f528407a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5284077fc0 0x7f528407a480 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f52a40bf940 tx=0x7f5294009040 comp rx=0 tx=0).stop 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 -- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a4150020 msgr2=0x7f52a4158160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a4150020 0x7f52a4158160 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5290006010 tx=0x7f52900038a0 comp rx=0 tx=0).stop 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 -- 172.21.15.31:0/3156675853 shutdown_connections 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5284077fc0 0x7f528407a480 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a4156960 0x7f52a41ef9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a4150020 0x7f52a4158160 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 --2- 172.21.15.31:0/3156675853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a40cc220 0x7f52a414acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 -- 172.21.15.31:0/3156675853 >> 172.21.15.31:0/3156675853 conn(0x7f52a4068f10 msgr2=0x7f52a4154a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 -- 172.21.15.31:0/3156675853 shutdown_connections 2024-09-11T00:46:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:46:04.575+0000 7f52a3200640 1 -- 172.21.15.31:0/3156675853 wait complete. 2024-09-11T00:46:04.595 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-11T00:46:04.595 INFO:tasks.mds_thrash.fs.[b]:waiting for 98 secs before thrashing 2024-09-11T00:46:18.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:46:18.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:46:48.553 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:46:48.557 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:47:01.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.060+0000 7fe291c00640 1 -- 172.21.15.31:0/1258495742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe28c0b1160 msgr2=0x7fe28c15a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.060+0000 7fe291c00640 1 --2- 172.21.15.31:0/1258495742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe28c0b1160 0x7fe28c15a1a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009830 tx=0x7fe27c02a530 comp rx=0 tx=0).stop 2024-09-11T00:47:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.060+0000 7fe291c00640 1 -- 172.21.15.31:0/1258495742 shutdown_connections 2024-09-11T00:47:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.060+0000 7fe291c00640 1 --2- 172.21.15.31:0/1258495742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe28c0b1160 0x7fe28c15a1a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.060+0000 7fe291c00640 1 -- 172.21.15.31:0/1258495742 >> 172.21.15.31:0/1258495742 conn(0x7fe28c0cc260 msgr2=0x7fe28c0ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.060+0000 7fe291c00640 1 -- 172.21.15.31:0/1258495742 shutdown_connections 2024-09-11T00:47:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.060+0000 7fe291c00640 1 -- 172.21.15.31:0/1258495742 wait complete. 2024-09-11T00:47:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.062+0000 7fe291c00640 1 Processor -- start 2024-09-11T00:47:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.062+0000 7fe291c00640 1 -- start start 2024-09-11T00:47:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe291c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe28c0b1160 0x7fe28c1efc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe291c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe28c0bf2c0 0x7fe28c1df6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe291c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe28c0c9b30 0x7fe28c1e15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe291c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c0b1540 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe291c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c06e5c0 con 0x7fe28c0b1160 2024-09-11T00:47:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe291c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe28c0ae540 con 0x7fe28c0c9b30 2024-09-11T00:47:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe28a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe28c0bf2c0 0x7fe28c1df6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe28a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe28c0bf2c0 0x7fe28c1df6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44940/0 (socket says 172.21.15.31:44940) 2024-09-11T00:47:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe28a000640 1 -- 172.21.15.31:0/651393792 learned_addr learned my addr 172.21.15.31:0/651393792 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe28aa00640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe28c0b1160 0x7fe28c1efc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.063+0000 7fe28b400640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe28c0c9b30 0x7fe28c1e15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.064+0000 7fe28a000640 1 -- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe28c0c9b30 msgr2=0x7fe28c1e15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.064+0000 7fe28a000640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe28c0c9b30 0x7fe28c1e15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.064+0000 7fe28a000640 1 -- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe28c0b1160 msgr2=0x7fe28c1efc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.064+0000 7fe28a000640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe28c0b1160 0x7fe28c1efc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.064+0000 7fe28a000640 1 -- 172.21.15.31:0/651393792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c1f01b0 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.064+0000 7fe28a000640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe28c0bf2c0 0x7fe28c1df6d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fe27800c480 tx=0x7fe27800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.065+0000 7fe282a00640 1 -- 172.21.15.31:0/651393792 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278016020 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.065+0000 7fe282a00640 1 -- 172.21.15.31:0/651393792 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe278012920 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.065+0000 7fe282a00640 1 -- 172.21.15.31:0/651393792 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278012d70 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.065+0000 7fe291c00640 1 -- 172.21.15.31:0/651393792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c156a20 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.065+0000 7fe291c00640 1 -- 172.21.15.31:0/651393792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c1ef470 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.066+0000 7fe291c00640 1 -- 172.21.15.31:0/651393792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe28c0c7680 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.067+0000 7fe282a00640 1 -- 172.21.15.31:0/651393792 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe27801b400 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.068+0000 7fe282a00640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe26c077f70 0x7fe26c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.069+0000 7fe282a00640 1 -- 172.21.15.31:0/651393792 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7fe27809a720 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.069+0000 7fe28aa00640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe26c077f70 0x7fe26c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.070+0000 7fe28aa00640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe26c077f70 0x7fe26c07a430 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe27c034000 tx=0x7fe27c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.075+0000 7fe282a00640 1 -- 172.21.15.31:0/651393792 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe28c0c7680 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.352+0000 7fe291c00640 1 -- 172.21.15.31:0/651393792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe28c1dfc10 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.352+0000 7fe282a00640 1 -- 172.21.15.31:0/651393792 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6544 (secure 0 0 0) 0x7fe278020030 con 0x7fe28c0bf2c0 2024-09-11T00:47:02.355 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:02.355 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":122,"btime":"2024-09-11T00:46:04:450048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6804,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:02.355 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 122 2024-09-11T00:47:02.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 -- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe26c077f70 msgr2=0x7fe26c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:02.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe26c077f70 0x7fe26c07a430 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe27c034000 tx=0x7fe27c004730 comp rx=0 tx=0).stop 2024-09-11T00:47:02.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 -- 172.21.15.31:0/651393792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe28c0bf2c0 msgr2=0x7fe28c1df6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:02.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe28c0bf2c0 0x7fe28c1df6d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fe27800c480 tx=0x7fe27800c940 comp rx=0 tx=0).stop 2024-09-11T00:47:02.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 -- 172.21.15.31:0/651393792 shutdown_connections 2024-09-11T00:47:02.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe26c077f70 0x7fe26c07a430 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe28c0c9b30 0x7fe28c1e15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe28c0bf2c0 0x7fe28c1df6d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 --2- 172.21.15.31:0/651393792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe28c0b1160 0x7fe28c1efc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 -- 172.21.15.31:0/651393792 >> 172.21.15.31:0/651393792 conn(0x7fe28c0cc260 msgr2=0x7fe28c0c9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:02.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 -- 172.21.15.31:0/651393792 shutdown_connections 2024-09-11T00:47:02.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.355+0000 7fe291c00640 1 -- 172.21.15.31:0/651393792 wait complete. 2024-09-11T00:47:02.375 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-11T00:47:02.376 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:47:02.376 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:47:02.392 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:47:02.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.592+0000 7f54e5e00640 1 -- 172.21.15.31:0/4082902212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e00c6760 msgr2=0x7f54e00c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.592+0000 7f54e5e00640 1 --2- 172.21.15.31:0/4082902212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e00c6760 0x7f54e00c8c80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009830 tx=0x7f54d402a5c0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.592+0000 7f54e5e00640 1 -- 172.21.15.31:0/4082902212 shutdown_connections 2024-09-11T00:47:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.592+0000 7f54e5e00640 1 --2- 172.21.15.31:0/4082902212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e00c6760 0x7f54e00c8c80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.592+0000 7f54e5e00640 1 -- 172.21.15.31:0/4082902212 >> 172.21.15.31:0/4082902212 conn(0x7f54e00694f0 msgr2=0x7f54e00c9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:02.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.592+0000 7f54e5e00640 1 -- 172.21.15.31:0/4082902212 shutdown_connections 2024-09-11T00:47:02.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.592+0000 7f54e5e00640 1 -- 172.21.15.31:0/4082902212 wait complete. 2024-09-11T00:47:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.594+0000 7f54e5e00640 1 Processor -- start 2024-09-11T00:47:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.594+0000 7f54e5e00640 1 -- start start 2024-09-11T00:47:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.595+0000 7f54e5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e00c5a20 0x7f54e01e0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.595+0000 7f54e5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e00c6760 0x7f54e01535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.595+0000 7f54e5e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f54e00c6b40 0x7f54e01ed430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:02.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.595+0000 7f54e5e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54e00c92e0 con 0x7f54e00c6760 2024-09-11T00:47:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.595+0000 7f54e5e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f54e00c8760 con 0x7f54e00c5a20 2024-09-11T00:47:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.595+0000 7f54e5e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f54e00c9460 con 0x7f54e00c6b40 2024-09-11T00:47:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54dfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e00c6760 0x7f54e01535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54e4a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e00c5a20 0x7f54e01e0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54dfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e00c6760 0x7f54e01535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44954/0 (socket says 172.21.15.31:44954) 2024-09-11T00:47:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54dfe00640 1 -- 172.21.15.31:0/257196196 learned_addr learned my addr 172.21.15.31:0/257196196 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:02.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54e5400640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f54e00c6b40 0x7f54e01ed430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54dfe00640 1 -- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f54e00c6b40 msgr2=0x7f54e01ed430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54dfe00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f54e00c6b40 0x7f54e01ed430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54dfe00640 1 -- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e00c5a20 msgr2=0x7f54e01e0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54dfe00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e00c5a20 0x7f54e01e0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54dfe00640 1 -- 172.21.15.31:0/257196196 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e01ddc40 con 0x7f54e00c6760 2024-09-11T00:47:02.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.596+0000 7f54e4a00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e00c5a20 0x7f54e01e0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.597+0000 7f54dfe00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e00c6760 0x7f54e01535a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00c400 tx=0x7f54cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.597+0000 7f54dcc00640 1 -- 172.21.15.31:0/257196196 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc017070 con 0x7f54e00c6760 2024-09-11T00:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.597+0000 7f54e5e00640 1 -- 172.21.15.31:0/257196196 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e01ddf20 con 0x7f54e00c6760 2024-09-11T00:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.597+0000 7f54dcc00640 1 -- 172.21.15.31:0/257196196 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54cc0035f0 con 0x7f54e00c6760 2024-09-11T00:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.597+0000 7f54dcc00640 1 -- 172.21.15.31:0/257196196 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc003a40 con 0x7f54e00c6760 2024-09-11T00:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.597+0000 7f54e5e00640 1 -- 172.21.15.31:0/257196196 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e01e0740 con 0x7f54e00c6760 2024-09-11T00:47:02.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.598+0000 7f54e5e00640 1 -- 172.21.15.31:0/257196196 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54e00c7370 con 0x7f54e00c6760 2024-09-11T00:47:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.599+0000 7f54dcc00640 1 -- 172.21.15.31:0/257196196 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f54cc003be0 con 0x7f54e00c6760 2024-09-11T00:47:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.600+0000 7f54dcc00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f54c0077fa0 0x7f54c007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.600+0000 7f54dcc00640 1 -- 172.21.15.31:0/257196196 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7f54cc099e50 con 0x7f54e00c6760 2024-09-11T00:47:02.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.601+0000 7f54e4a00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f54c0077fa0 0x7f54c007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:02.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.601+0000 7f54e4a00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f54c0077fa0 0x7f54c007a460 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f54d4034000 tx=0x7f54d4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:02.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.606+0000 7f54dcc00640 1 -- 172.21.15.31:0/257196196 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f54e00c7370 con 0x7f54e00c6760 2024-09-11T00:47:02.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.891+0000 7f54e5e00640 1 -- 172.21.15.31:0/257196196 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54e00c5e00 con 0x7f54e00c6760 2024-09-11T00:47:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.891+0000 7f54dcc00640 1 -- 172.21.15.31:0/257196196 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6544 (secure 0 0 0) 0x7f54e00c5e00 con 0x7f54e00c6760 2024-09-11T00:47:02.894 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:02.894 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":122,"btime":"2024-09-11T00:46:04:450048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6804,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 122 2024-09-11T00:47:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 -- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f54c0077fa0 msgr2=0x7f54c007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f54c0077fa0 0x7f54c007a460 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f54d4034000 tx=0x7f54d4004730 comp rx=0 tx=0).stop 2024-09-11T00:47:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 -- 172.21.15.31:0/257196196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e00c6760 msgr2=0x7f54e01535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e00c6760 0x7f54e01535a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00c400 tx=0x7f54cc00c8c0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 -- 172.21.15.31:0/257196196 shutdown_connections 2024-09-11T00:47:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f54c0077fa0 0x7f54c007a460 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f54e00c6b40 0x7f54e01ed430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54e00c6760 0x7f54e01535a0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 --2- 172.21.15.31:0/257196196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e00c5a20 0x7f54e01e0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:02.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 -- 172.21.15.31:0/257196196 >> 172.21.15.31:0/257196196 conn(0x7f54e00694f0 msgr2=0x7f54e00c3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:02.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 -- 172.21.15.31:0/257196196 shutdown_connections 2024-09-11T00:47:02.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:02.894+0000 7f54e5e00640 1 -- 172.21.15.31:0/257196196 wait complete. 2024-09-11T00:47:02.914 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:47:04.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:05.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.085+0000 7ff839200640 1 -- 172.21.15.31:0/477192082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8340c23a0 msgr2=0x7ff8340c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.085+0000 7ff839200640 1 --2- 172.21.15.31:0/477192082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8340c23a0 0x7ff8340c30c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff8340c5500 tx=0x7ff81c02a680 comp rx=0 tx=0).stop 2024-09-11T00:47:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.085+0000 7ff839200640 1 -- 172.21.15.31:0/477192082 shutdown_connections 2024-09-11T00:47:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.085+0000 7ff839200640 1 --2- 172.21.15.31:0/477192082 >> v2:172.21.15.204:3300/0 conn(0x7ff8341518b0 0x7ff8340c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.085+0000 7ff839200640 1 --2- 172.21.15.31:0/477192082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8340c23a0 0x7ff8340c30c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.085+0000 7ff839200640 1 -- 172.21.15.31:0/477192082 >> 172.21.15.31:0/477192082 conn(0x7ff8340a9750 msgr2=0x7ff8340a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.085+0000 7ff839200640 1 -- 172.21.15.31:0/477192082 shutdown_connections 2024-09-11T00:47:05.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.085+0000 7ff839200640 1 -- 172.21.15.31:0/477192082 wait complete. 2024-09-11T00:47:05.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.087+0000 7ff839200640 1 Processor -- start 2024-09-11T00:47:05.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.087+0000 7ff839200640 1 -- start start 2024-09-11T00:47:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff839200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff834072480 0x7ff8341f01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff839200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8340c23a0 0x7ff834147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff839200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff8341518b0 0x7ff83406e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff839200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff834074200 con 0x7ff834072480 2024-09-11T00:47:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff839200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff834074080 con 0x7ff8340c23a0 2024-09-11T00:47:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff839200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff834072860 con 0x7ff8341518b0 2024-09-11T00:47:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff831e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff834072480 0x7ff8341f01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff82be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8340c23a0 0x7ff834147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff831e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff834072480 0x7ff8341f01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44976/0 (socket says 172.21.15.31:44976) 2024-09-11T00:47:05.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff831e00640 1 -- 172.21.15.31:0/2245536737 learned_addr learned my addr 172.21.15.31:0/2245536737 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:05.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.088+0000 7ff832800640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff8341518b0 0x7ff83406e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff831e00640 1 -- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff8341518b0 msgr2=0x7ff83406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff831e00640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff8341518b0 0x7ff83406e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff831e00640 1 -- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8340c23a0 msgr2=0x7ff834147170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff831e00640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8340c23a0 0x7ff834147170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff831e00640 1 -- 172.21.15.31:0/2245536737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8340c3130 con 0x7ff834072480 2024-09-11T00:47:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff831e00640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff834072480 0x7ff8341f01f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff82400c480 tx=0x7ff82400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff82a000640 1 -- 172.21.15.31:0/2245536737 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8240052d0 con 0x7ff834072480 2024-09-11T00:47:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff82a000640 1 -- 172.21.15.31:0/2245536737 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff824005470 con 0x7ff834072480 2024-09-11T00:47:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff82a000640 1 -- 172.21.15.31:0/2245536737 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824012870 con 0x7ff834072480 2024-09-11T00:47:05.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff839200640 1 -- 172.21.15.31:0/2245536737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8341dd620 con 0x7ff834072480 2024-09-11T00:47:05.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.089+0000 7ff839200640 1 -- 172.21.15.31:0/2245536737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8341476b0 con 0x7ff834072480 2024-09-11T00:47:05.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.091+0000 7ff839200640 1 -- 172.21.15.31:0/2245536737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff834153f20 con 0x7ff834072480 2024-09-11T00:47:05.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.091+0000 7ff82a000640 1 -- 172.21.15.31:0/2245536737 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff8240035f0 con 0x7ff834072480 2024-09-11T00:47:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.092+0000 7ff82a000640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff810077fa0 0x7ff81007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.093+0000 7ff82a000640 1 -- 172.21.15.31:0/2245536737 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7ff82409a270 con 0x7ff834072480 2024-09-11T00:47:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.093+0000 7ff82be00640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff810077fa0 0x7ff81007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:05.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.093+0000 7ff82be00640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff810077fa0 0x7ff81007a460 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff81c033000 tx=0x7ff81c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:05.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.097+0000 7ff82a000640 1 -- 172.21.15.31:0/2245536737 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff824061c40 con 0x7ff834072480 2024-09-11T00:47:05.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.330+0000 7ff839200640 1 -- 172.21.15.31:0/2245536737 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8340a9530 con 0x7ff834072480 2024-09-11T00:47:05.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.330+0000 7ff82a000640 1 -- 172.21.15.31:0/2245536737 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6544 (secure 0 0 0) 0x7ff824066670 con 0x7ff834072480 2024-09-11T00:47:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:05.333 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":122,"btime":"2024-09-11T00:46:04:450048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6804,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:05.334 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 122 2024-09-11T00:47:05.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 -- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff810077fa0 msgr2=0x7ff81007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:05.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff810077fa0 0x7ff81007a460 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff81c033000 tx=0x7ff81c004730 comp rx=0 tx=0).stop 2024-09-11T00:47:05.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 -- 172.21.15.31:0/2245536737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff834072480 msgr2=0x7ff8341f01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:05.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff834072480 0x7ff8341f01f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff82400c480 tx=0x7ff82400c940 comp rx=0 tx=0).stop 2024-09-11T00:47:05.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 -- 172.21.15.31:0/2245536737 shutdown_connections 2024-09-11T00:47:05.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff810077fa0 0x7ff81007a460 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:05.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff8341518b0 0x7ff83406e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:05.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8340c23a0 0x7ff834147170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:05.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 --2- 172.21.15.31:0/2245536737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff834072480 0x7ff8341f01f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:05.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 -- 172.21.15.31:0/2245536737 >> 172.21.15.31:0/2245536737 conn(0x7ff8340a9750 msgr2=0x7ff83414e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:05.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 -- 172.21.15.31:0/2245536737 shutdown_connections 2024-09-11T00:47:05.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:05.333+0000 7ff839200640 1 -- 172.21.15.31:0/2245536737 wait complete. 2024-09-11T00:47:05.351 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:47:07.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:07.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.520+0000 7fbb2cc00640 1 -- 172.21.15.31:0/1649805695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb280a6e20 msgr2=0x7fbb281510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:07.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.520+0000 7fbb2cc00640 1 --2- 172.21.15.31:0/1649805695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb280a6e20 0x7fbb281510c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009830 tx=0x7fbb1002a4a0 comp rx=0 tx=0).stop 2024-09-11T00:47:07.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.520+0000 7fbb2cc00640 1 -- 172.21.15.31:0/1649805695 shutdown_connections 2024-09-11T00:47:07.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.520+0000 7fbb2cc00640 1 --2- 172.21.15.31:0/1649805695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb280a6e20 0x7fbb281510c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:07.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.520+0000 7fbb2cc00640 1 -- 172.21.15.31:0/1649805695 >> 172.21.15.31:0/1649805695 conn(0x7fbb2806caf0 msgr2=0x7fbb2806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:07.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.520+0000 7fbb2cc00640 1 -- 172.21.15.31:0/1649805695 shutdown_connections 2024-09-11T00:47:07.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.520+0000 7fbb2cc00640 1 -- 172.21.15.31:0/1649805695 wait complete. 2024-09-11T00:47:07.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.522+0000 7fbb2cc00640 1 Processor -- start 2024-09-11T00:47:07.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.522+0000 7fbb2cc00640 1 -- start start 2024-09-11T00:47:07.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb2cc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb280a6e20 0x7fbb28149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:07.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb2cc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb280ac2c0 0x7fbb280c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:07.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb2cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb28151600 0x7fbb280c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:07.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb2cc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbb280c68c0 con 0x7fbb28151600 2024-09-11T00:47:07.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb2cc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbb280c5d20 con 0x7fbb280a6e20 2024-09-11T00:47:07.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb2cc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbb280c6a40 con 0x7fbb280ac2c0 2024-09-11T00:47:07.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb27e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb28151600 0x7fbb280c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:07.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb27400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb280a6e20 0x7fbb28149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:07.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb27e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb28151600 0x7fbb280c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44986/0 (socket says 172.21.15.31:44986) 2024-09-11T00:47:07.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb27e00640 1 -- 172.21.15.31:0/2568484486 learned_addr learned my addr 172.21.15.31:0/2568484486 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:07.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.523+0000 7fbb1ea00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb280ac2c0 0x7fbb280c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.524+0000 7fbb27400640 1 -- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb280ac2c0 msgr2=0x7fbb280c0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.524+0000 7fbb27400640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb280ac2c0 0x7fbb280c0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.524+0000 7fbb27400640 1 -- 172.21.15.31:0/2568484486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb28151600 msgr2=0x7fbb280c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.524+0000 7fbb27400640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb28151600 0x7fbb280c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.524+0000 7fbb27400640 1 -- 172.21.15.31:0/2568484486 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2806fe70 con 0x7fbb280a6e20 2024-09-11T00:47:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.524+0000 7fbb27e00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb28151600 0x7fbb280c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:47:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.524+0000 7fbb27400640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb280a6e20 0x7fbb28149000 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800c430 tx=0x7fbb1800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:07.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.524+0000 7fbb1fe00640 1 -- 172.21.15.31:0/2568484486 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18017070 con 0x7fbb280a6e20 2024-09-11T00:47:07.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.525+0000 7fbb1fe00640 1 -- 172.21.15.31:0/2568484486 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb180053c0 con 0x7fbb280a6e20 2024-09-11T00:47:07.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.525+0000 7fbb1fe00640 1 -- 172.21.15.31:0/2568484486 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb18003850 con 0x7fbb280a6e20 2024-09-11T00:47:07.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.525+0000 7fbb2cc00640 1 -- 172.21.15.31:0/2568484486 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb2806f8b0 con 0x7fbb280a6e20 2024-09-11T00:47:07.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.525+0000 7fbb2cc00640 1 -- 172.21.15.31:0/2568484486 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb28072180 con 0x7fbb280a6e20 2024-09-11T00:47:07.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.526+0000 7fbb2cc00640 1 -- 172.21.15.31:0/2568484486 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb280c6e30 con 0x7fbb280a6e20 2024-09-11T00:47:07.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.527+0000 7fbb1fe00640 1 -- 172.21.15.31:0/2568484486 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbb180039f0 con 0x7fbb280a6e20 2024-09-11T00:47:07.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.528+0000 7fbb1fe00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb04077fa0 0x7fbb0407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:07.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.528+0000 7fbb1fe00640 1 -- 172.21.15.31:0/2568484486 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7fbb18099ed0 con 0x7fbb280a6e20 2024-09-11T00:47:07.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.528+0000 7fbb1ea00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb04077fa0 0x7fbb0407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:07.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.529+0000 7fbb1ea00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb04077fa0 0x7fbb0407a460 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbb10034000 tx=0x7fbb10004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:07.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.533+0000 7fbb1fe00640 1 -- 172.21.15.31:0/2568484486 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbb180617c0 con 0x7fbb280a6e20 2024-09-11T00:47:07.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.768+0000 7fbb2cc00640 1 -- 172.21.15.31:0/2568484486 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb281581c0 con 0x7fbb280a6e20 2024-09-11T00:47:07.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.769+0000 7fbb1fe00640 1 -- 172.21.15.31:0/2568484486 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6544 (secure 0 0 0) 0x7fbb180661f0 con 0x7fbb280a6e20 2024-09-11T00:47:07.771 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:07.772 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":122,"btime":"2024-09-11T00:46:04:450048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6804,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:07.772 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 122 2024-09-11T00:47:07.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 -- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb04077fa0 msgr2=0x7fbb0407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:07.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb04077fa0 0x7fbb0407a460 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbb10034000 tx=0x7fbb10004730 comp rx=0 tx=0).stop 2024-09-11T00:47:07.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 -- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb280a6e20 msgr2=0x7fbb28149000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:07.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb280a6e20 0x7fbb28149000 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800c430 tx=0x7fbb1800c8f0 comp rx=0 tx=0).stop 2024-09-11T00:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 -- 172.21.15.31:0/2568484486 shutdown_connections 2024-09-11T00:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb04077fa0 0x7fbb0407a460 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb28151600 0x7fbb280c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb280ac2c0 0x7fbb280c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 --2- 172.21.15.31:0/2568484486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb280a6e20 0x7fbb28149000 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 -- 172.21.15.31:0/2568484486 >> 172.21.15.31:0/2568484486 conn(0x7fbb2806caf0 msgr2=0x7fbb28153da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 -- 172.21.15.31:0/2568484486 shutdown_connections 2024-09-11T00:47:07.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:07.771+0000 7fbb2cc00640 1 -- 172.21.15.31:0/2568484486 wait complete. 2024-09-11T00:47:07.789 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:47:09.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:09.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.958+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3918341526 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1b01534f0 msgr2=0x7ff1b0150020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.958+0000 7ff1b5a00640 1 --2- 172.21.15.31:0/3918341526 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1b01534f0 0x7ff1b0150020 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8005fd0 tx=0x7ff1a802a780 comp rx=0 tx=0).stop 2024-09-11T00:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.958+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3918341526 shutdown_connections 2024-09-11T00:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.958+0000 7ff1b5a00640 1 --2- 172.21.15.31:0/3918341526 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1b01534f0 0x7ff1b0150020 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.958+0000 7ff1b5a00640 1 --2- 172.21.15.31:0/3918341526 >> v2:172.21.15.204:3300/0 conn(0x7ff1b00c6bb0 0x7ff1b006dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.958+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3918341526 >> 172.21.15.31:0/3918341526 conn(0x7ff1b006caf0 msgr2=0x7ff1b006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.958+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3918341526 shutdown_connections 2024-09-11T00:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.959+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3918341526 wait complete. 2024-09-11T00:47:09.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.960+0000 7ff1b5a00640 1 Processor -- start 2024-09-11T00:47:09.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.960+0000 7ff1b5a00640 1 -- start start 2024-09-11T00:47:09.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1b5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1b00c2700 0x7ff1b00a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:09.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1b5a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1b00c6bb0 0x7ff1b0154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:09.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1b5a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1b01534f0 0x7ff1b0154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1b5a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b00c7110 con 0x7ff1b00c2700 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1b5a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b00c6f90 con 0x7ff1b00c6bb0 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1b5a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b0153940 con 0x7ff1b01534f0 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1afe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1b00c2700 0x7ff1b00a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1afe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1b00c2700 0x7ff1b00a8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35532/0 (socket says 172.21.15.31:35532) 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1afe00640 1 -- 172.21.15.31:0/3105479812 learned_addr learned my addr 172.21.15.31:0/3105479812 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1af400640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1b00c6bb0 0x7ff1b0154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.961+0000 7ff1b5000640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1b01534f0 0x7ff1b0154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.962+0000 7ff1afe00640 1 -- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1b01534f0 msgr2=0x7ff1b0154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.962+0000 7ff1afe00640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1b01534f0 0x7ff1b0154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.962+0000 7ff1afe00640 1 -- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1b00c6bb0 msgr2=0x7ff1b0154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:09.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.962+0000 7ff1afe00640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1b00c6bb0 0x7ff1b0154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:09.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.962+0000 7ff1afe00640 1 -- 172.21.15.31:0/3105479812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b01ef900 con 0x7ff1b00c2700 2024-09-11T00:47:09.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.962+0000 7ff1afe00640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1b00c2700 0x7ff1b00a8460 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000ea60 tx=0x7ff1a000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:09.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.962+0000 7ff1a7e00640 1 -- 172.21.15.31:0/3105479812 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a000cde0 con 0x7ff1b00c2700 2024-09-11T00:47:09.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.962+0000 7ff1a7e00640 1 -- 172.21.15.31:0/3105479812 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1a0004020 con 0x7ff1b00c2700 2024-09-11T00:47:09.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.962+0000 7ff1a7e00640 1 -- 172.21.15.31:0/3105479812 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0004450 con 0x7ff1b00c2700 2024-09-11T00:47:09.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.962+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3105479812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b00a9fe0 con 0x7ff1b00c2700 2024-09-11T00:47:09.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.963+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3105479812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b00a9990 con 0x7ff1b00c2700 2024-09-11T00:47:09.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.964+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3105479812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1b006dc60 con 0x7ff1b00c2700 2024-09-11T00:47:09.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.965+0000 7ff1a7e00640 1 -- 172.21.15.31:0/3105479812 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff1a00035f0 con 0x7ff1b00c2700 2024-09-11T00:47:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.966+0000 7ff1a7e00640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff190077fa0 0x7ff19007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.966+0000 7ff1a7e00640 1 -- 172.21.15.31:0/3105479812 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7ff1a009b0f0 con 0x7ff1b00c2700 2024-09-11T00:47:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.967+0000 7ff1af400640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff190077fa0 0x7ff19007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:09.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.967+0000 7ff1af400640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff190077fa0 0x7ff19007a460 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff19c007fd0 tx=0x7ff19c001370 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:09.971+0000 7ff1a7e00640 1 -- 172.21.15.31:0/3105479812 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff1a0062b60 con 0x7ff1b00c2700 2024-09-11T00:47:10.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.203+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3105479812 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1b006a250 con 0x7ff1b00c2700 2024-09-11T00:47:10.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.204+0000 7ff1a7e00640 1 -- 172.21.15.31:0/3105479812 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6544 (secure 0 0 0) 0x7ff1a0067590 con 0x7ff1b00c2700 2024-09-11T00:47:10.207 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:10.207 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":122,"btime":"2024-09-11T00:46:04:450048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6804,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:10.207 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 122 2024-09-11T00:47:10.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff190077fa0 msgr2=0x7ff19007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:10.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff190077fa0 0x7ff19007a460 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff19c007fd0 tx=0x7ff19c001370 comp rx=0 tx=0).stop 2024-09-11T00:47:10.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3105479812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1b00c2700 msgr2=0x7ff1b00a8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:10.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1b00c2700 0x7ff1b00a8460 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000ea60 tx=0x7ff1a000ef20 comp rx=0 tx=0).stop 2024-09-11T00:47:10.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3105479812 shutdown_connections 2024-09-11T00:47:10.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff190077fa0 0x7ff19007a460 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:10.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1b01534f0 0x7ff1b0154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1b00c6bb0 0x7ff1b0154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 --2- 172.21.15.31:0/3105479812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1b00c2700 0x7ff1b00a8460 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3105479812 >> 172.21.15.31:0/3105479812 conn(0x7ff1b006caf0 msgr2=0x7ff1b006ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3105479812 shutdown_connections 2024-09-11T00:47:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:10.206+0000 7ff1b5a00640 1 -- 172.21.15.31:0/3105479812 wait complete. 2024-09-11T00:47:10.225 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:47:12.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:12.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.393+0000 7f14c5c00640 1 -- 172.21.15.31:0/4053993077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14c0156300 msgr2=0x7f14c00c5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.393+0000 7f14c5c00640 1 --2- 172.21.15.31:0/4053993077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14c0156300 0x7f14c00c5b40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009860 tx=0x7f14b002a510 comp rx=0 tx=0).stop 2024-09-11T00:47:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.393+0000 7f14c5c00640 1 -- 172.21.15.31:0/4053993077 shutdown_connections 2024-09-11T00:47:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.393+0000 7f14c5c00640 1 --2- 172.21.15.31:0/4053993077 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14c0156300 0x7f14c00c5b40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.393+0000 7f14c5c00640 1 -- 172.21.15.31:0/4053993077 >> 172.21.15.31:0/4053993077 conn(0x7f14c00a9750 msgr2=0x7f14c00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.393+0000 7f14c5c00640 1 -- 172.21.15.31:0/4053993077 shutdown_connections 2024-09-11T00:47:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.393+0000 7f14c5c00640 1 -- 172.21.15.31:0/4053993077 wait complete. 2024-09-11T00:47:12.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.395+0000 7f14c5c00640 1 Processor -- start 2024-09-11T00:47:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.395+0000 7f14c5c00640 1 -- start start 2024-09-11T00:47:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14c5c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14c006fa70 0x7f14c00747c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14c5c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14c00c23a0 0x7f14c014e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14c5c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14c0156300 0x7f14c014e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14c5c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f14c0156df0 con 0x7f14c00c23a0 2024-09-11T00:47:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14c5c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f14c006f5d0 con 0x7f14c006fa70 2024-09-11T00:47:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14c5c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f14c0156f70 con 0x7f14c0156300 2024-09-11T00:47:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14b6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14c00c23a0 0x7f14c014e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:12.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14bea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14c006fa70 0x7f14c00747c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:12.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14bea00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14c006fa70 0x7f14c00747c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:47070/0 (socket says 172.21.15.31:47070) 2024-09-11T00:47:12.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14bea00640 1 -- 172.21.15.31:0/119733559 learned_addr learned my addr 172.21.15.31:0/119733559 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:12.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.396+0000 7f14bf400640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14c0156300 0x7f14c014e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:12.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14b6000640 1 -- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14c0156300 msgr2=0x7f14c014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:12.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14b6000640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14c0156300 0x7f14c014e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:12.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14b6000640 1 -- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14c006fa70 msgr2=0x7f14c00747c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:12.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14b6000640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14c006fa70 0x7f14c00747c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:12.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14b6000640 1 -- 172.21.15.31:0/119733559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c00ad0b0 con 0x7f14c00c23a0 2024-09-11T00:47:12.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14bea00640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14c006fa70 0x7f14c00747c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:47:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14b6000640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14c00c23a0 0x7f14c014e4a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f14c0153e80 tx=0x7f14a800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14b7400640 1 -- 172.21.15.31:0/119733559 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8011790 con 0x7f14c00c23a0 2024-09-11T00:47:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14b7400640 1 -- 172.21.15.31:0/119733559 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14a8017070 con 0x7f14c00c23a0 2024-09-11T00:47:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14b7400640 1 -- 172.21.15.31:0/119733559 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8011d00 con 0x7f14c00c23a0 2024-09-11T00:47:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.397+0000 7f14c5c00640 1 -- 172.21.15.31:0/119733559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c00ad390 con 0x7f14c00c23a0 2024-09-11T00:47:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.398+0000 7f14c5c00640 1 -- 172.21.15.31:0/119733559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c0074d00 con 0x7f14c00c23a0 2024-09-11T00:47:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.399+0000 7f14c5c00640 1 -- 172.21.15.31:0/119733559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c00c5b40 con 0x7f14c00c23a0 2024-09-11T00:47:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.400+0000 7f14b7400640 1 -- 172.21.15.31:0/119733559 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f14a801c460 con 0x7f14c00c23a0 2024-09-11T00:47:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.401+0000 7f14b7400640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f149c077f70 0x7f149c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.401+0000 7f14b7400640 1 -- 172.21.15.31:0/119733559 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7f14a809a410 con 0x7f14c00c23a0 2024-09-11T00:47:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.401+0000 7f14bea00640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f149c077f70 0x7f149c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:12.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.402+0000 7f14bea00640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f149c077f70 0x7f149c07a430 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009860 tx=0x7f14b0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:12.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.405+0000 7f14b7400640 1 -- 172.21.15.31:0/119733559 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f14a8061d80 con 0x7f14c00c23a0 2024-09-11T00:47:12.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.644+0000 7f14c5c00640 1 -- 172.21.15.31:0/119733559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14c014bfe0 con 0x7f14c00c23a0 2024-09-11T00:47:12.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.644+0000 7f14b7400640 1 -- 172.21.15.31:0/119733559 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6544 (secure 0 0 0) 0x7f14a80667b0 con 0x7f14c00c23a0 2024-09-11T00:47:12.646 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:12.647 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":122,"btime":"2024-09-11T00:46:04:450048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6804,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:12.647 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 122 2024-09-11T00:47:12.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.646+0000 7f14c5c00640 1 -- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f149c077f70 msgr2=0x7f149c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:12.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.646+0000 7f14c5c00640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f149c077f70 0x7f149c07a430 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009860 tx=0x7f14b0004730 comp rx=0 tx=0).stop 2024-09-11T00:47:12.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.647+0000 7f14c5c00640 1 -- 172.21.15.31:0/119733559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14c00c23a0 msgr2=0x7f14c014e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:12.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.647+0000 7f14c5c00640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14c00c23a0 0x7f14c014e4a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f14c0153e80 tx=0x7f14a800c800 comp rx=0 tx=0).stop 2024-09-11T00:47:12.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.647+0000 7f14c5c00640 1 -- 172.21.15.31:0/119733559 shutdown_connections 2024-09-11T00:47:12.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.647+0000 7f14c5c00640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f149c077f70 0x7f149c07a430 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:12.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.647+0000 7f14c5c00640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f14c0156300 0x7f14c014e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:12.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.647+0000 7f14c5c00640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14c00c23a0 0x7f14c014e4a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:12.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.647+0000 7f14c5c00640 1 --2- 172.21.15.31:0/119733559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f14c006fa70 0x7f14c00747c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:12.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.647+0000 7f14c5c00640 1 -- 172.21.15.31:0/119733559 >> 172.21.15.31:0/119733559 conn(0x7f14c00a9750 msgr2=0x7f14c006e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:12.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.647+0000 7f14c5c00640 1 -- 172.21.15.31:0/119733559 shutdown_connections 2024-09-11T00:47:12.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:12.647+0000 7f14c5c00640 1 -- 172.21.15.31:0/119733559 wait complete. 2024-09-11T00:47:12.663 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:47:14.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:14.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.829+0000 7fb583200640 1 -- 172.21.15.31:0/2906284420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb57c072480 msgr2=0x7fb57c1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:14.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.829+0000 7fb583200640 1 --2- 172.21.15.31:0/2906284420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb57c072480 0x7fb57c1511a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb57c075260 tx=0x7fb56c02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:47:14.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.829+0000 7fb583200640 1 -- 172.21.15.31:0/2906284420 shutdown_connections 2024-09-11T00:47:14.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.829+0000 7fb583200640 1 --2- 172.21.15.31:0/2906284420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb57c072480 0x7fb57c1511a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:14.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.829+0000 7fb583200640 1 -- 172.21.15.31:0/2906284420 >> 172.21.15.31:0/2906284420 conn(0x7fb57c0a9750 msgr2=0x7fb57c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:14.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.829+0000 7fb583200640 1 -- 172.21.15.31:0/2906284420 shutdown_connections 2024-09-11T00:47:14.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.829+0000 7fb583200640 1 -- 172.21.15.31:0/2906284420 wait complete. 2024-09-11T00:47:14.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.831+0000 7fb583200640 1 Processor -- start 2024-09-11T00:47:14.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.831+0000 7fb583200640 1 -- start start 2024-09-11T00:47:14.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.831+0000 7fb583200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb57c072480 0x7fb57c0ad050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:14.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb583200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb57c0c23a0 0x7fb57c149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb583200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb57c153f20 0x7fb57c1e9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb583200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c074200 con 0x7fb57c153f20 2024-09-11T00:47:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb583200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c074080 con 0x7fb57c0c23a0 2024-09-11T00:47:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb583200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb57c072860 con 0x7fb57c072480 2024-09-11T00:47:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb581200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb57c153f20 0x7fb57c1e9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb57be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb57c072480 0x7fb57c0ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb573e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb57c0c23a0 0x7fb57c149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb581200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb57c153f20 0x7fb57c1e9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35538/0 (socket says 172.21.15.31:35538) 2024-09-11T00:47:14.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb581200640 1 -- 172.21.15.31:0/2117484945 learned_addr learned my addr 172.21.15.31:0/2117484945 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb581200640 1 -- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb57c072480 msgr2=0x7fb57c0ad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb581200640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb57c072480 0x7fb57c0ad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb581200640 1 -- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb57c0c23a0 msgr2=0x7fb57c149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb581200640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb57c0c23a0 0x7fb57c149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:14.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.832+0000 7fb581200640 1 -- 172.21.15.31:0/2117484945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57c14a9f0 con 0x7fb57c153f20 2024-09-11T00:47:14.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.833+0000 7fb581200640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb57c153f20 0x7fb57c1e9bb0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb57400c700 tx=0x7fb57400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:14.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.833+0000 7fb578c00640 1 -- 172.21.15.31:0/2117484945 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574003b70 con 0x7fb57c153f20 2024-09-11T00:47:14.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.833+0000 7fb578c00640 1 -- 172.21.15.31:0/2117484945 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb574003d10 con 0x7fb57c153f20 2024-09-11T00:47:14.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.833+0000 7fb583200640 1 -- 172.21.15.31:0/2117484945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb57c148490 con 0x7fb57c153f20 2024-09-11T00:47:14.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.833+0000 7fb578c00640 1 -- 172.21.15.31:0/2117484945 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57400aab0 con 0x7fb57c153f20 2024-09-11T00:47:14.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.833+0000 7fb583200640 1 -- 172.21.15.31:0/2117484945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb57c0ac850 con 0x7fb57c153f20 2024-09-11T00:47:14.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.834+0000 7fb583200640 1 -- 172.21.15.31:0/2117484945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb57c077240 con 0x7fb57c153f20 2024-09-11T00:47:14.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.835+0000 7fb578c00640 1 -- 172.21.15.31:0/2117484945 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb57400ac50 con 0x7fb57c153f20 2024-09-11T00:47:14.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.837+0000 7fb578c00640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb558077fa0 0x7fb55807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:14.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.837+0000 7fb578c00640 1 -- 172.21.15.31:0/2117484945 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7fb57409a5c0 con 0x7fb57c153f20 2024-09-11T00:47:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.837+0000 7fb57be00640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb558077fa0 0x7fb55807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.838+0000 7fb57be00640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb558077fa0 0x7fb55807a460 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb56c034000 tx=0x7fb56c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:14.841+0000 7fb578c00640 1 -- 172.21.15.31:0/2117484945 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb57c077240 con 0x7fb57c153f20 2024-09-11T00:47:15.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.073+0000 7fb583200640 1 -- 172.21.15.31:0/2117484945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb57c1511a0 con 0x7fb57c153f20 2024-09-11T00:47:15.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.074+0000 7fb578c00640 1 -- 172.21.15.31:0/2117484945 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6544 (secure 0 0 0) 0x7fb574061eb0 con 0x7fb57c153f20 2024-09-11T00:47:15.075 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:15.075 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":122,"btime":"2024-09-11T00:46:04:450048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6804,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:15.076 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 122 2024-09-11T00:47:15.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 -- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb558077fa0 msgr2=0x7fb55807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:15.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb558077fa0 0x7fb55807a460 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb56c034000 tx=0x7fb56c004730 comp rx=0 tx=0).stop 2024-09-11T00:47:15.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 -- 172.21.15.31:0/2117484945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb57c153f20 msgr2=0x7fb57c1e9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:15.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb57c153f20 0x7fb57c1e9bb0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb57400c700 tx=0x7fb57400cbc0 comp rx=0 tx=0).stop 2024-09-11T00:47:15.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 -- 172.21.15.31:0/2117484945 shutdown_connections 2024-09-11T00:47:15.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb558077fa0 0x7fb55807a460 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:15.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb57c153f20 0x7fb57c1e9bb0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:15.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb57c0c23a0 0x7fb57c149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:15.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 --2- 172.21.15.31:0/2117484945 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb57c072480 0x7fb57c0ad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:15.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 -- 172.21.15.31:0/2117484945 >> 172.21.15.31:0/2117484945 conn(0x7fb57c0a9750 msgr2=0x7fb57c14f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:15.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 -- 172.21.15.31:0/2117484945 shutdown_connections 2024-09-11T00:47:15.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:15.076+0000 7fb583200640 1 -- 172.21.15.31:0/2117484945 wait complete. 2024-09-11T00:47:15.094 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:47:17.094 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:17.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.266+0000 7f8ebe000640 1 -- 172.21.15.31:0/1068226534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8eb80c23a0 msgr2=0x7f8eb80c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:17.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.266+0000 7f8ebe000640 1 --2- 172.21.15.31:0/1068226534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8eb80c23a0 0x7f8eb80c30c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0005fd0 tx=0x7f8ea002a670 comp rx=0 tx=0).stop 2024-09-11T00:47:17.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.266+0000 7f8ebe000640 1 -- 172.21.15.31:0/1068226534 shutdown_connections 2024-09-11T00:47:17.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.266+0000 7f8ebe000640 1 --2- 172.21.15.31:0/1068226534 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8eb81518b0 0x7f8eb80c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:17.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.266+0000 7f8ebe000640 1 --2- 172.21.15.31:0/1068226534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8eb80c23a0 0x7f8eb80c30c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:17.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.266+0000 7f8ebe000640 1 -- 172.21.15.31:0/1068226534 >> 172.21.15.31:0/1068226534 conn(0x7f8eb80a9750 msgr2=0x7f8eb80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:17.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.266+0000 7f8ebe000640 1 -- 172.21.15.31:0/1068226534 shutdown_connections 2024-09-11T00:47:17.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.266+0000 7f8ebe000640 1 -- 172.21.15.31:0/1068226534 wait complete. 2024-09-11T00:47:17.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.268+0000 7f8ebe000640 1 Processor -- start 2024-09-11T00:47:17.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.268+0000 7f8ebe000640 1 -- start start 2024-09-11T00:47:17.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8ebe000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8eb8072480 0x7f8eb80c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:17.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8ebe000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8eb80c23a0 0x7f8eb814a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8ebe000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8eb81518b0 0x7f8eb814a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8ebe000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb8074200 con 0x7f8eb8072480 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8ebe000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb8074080 con 0x7f8eb80c23a0 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8ebe000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8eb8072860 con 0x7f8eb81518b0 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8eb7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8eb8072480 0x7f8eb80c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8eb7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8eb81518b0 0x7f8eb814a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8eb7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8eb8072480 0x7f8eb80c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35546/0 (socket says 172.21.15.31:35546) 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8eb7400640 1 -- 172.21.15.31:0/3435771379 learned_addr learned my addr 172.21.15.31:0/3435771379 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.269+0000 7f8eaea00640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8eb80c23a0 0x7f8eb814a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8eb7400640 1 -- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8eb81518b0 msgr2=0x7f8eb814a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8eb7400640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8eb81518b0 0x7f8eb814a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8eb7400640 1 -- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8eb80c23a0 msgr2=0x7f8eb814a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:17.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8eb7400640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8eb80c23a0 0x7f8eb814a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:17.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8eb7400640 1 -- 172.21.15.31:0/3435771379 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb814ad90 con 0x7f8eb8072480 2024-09-11T00:47:17.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8eb7400640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8eb8072480 0x7f8eb80c1040 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800c480 tx=0x7f8ea800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:17.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8eafe00640 1 -- 172.21.15.31:0/3435771379 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80052d0 con 0x7f8eb8072480 2024-09-11T00:47:17.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8eafe00640 1 -- 172.21.15.31:0/3435771379 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ea8005470 con 0x7f8eb8072480 2024-09-11T00:47:17.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8eafe00640 1 -- 172.21.15.31:0/3435771379 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea8012870 con 0x7f8eb8072480 2024-09-11T00:47:17.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8ebe000640 1 -- 172.21.15.31:0/3435771379 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb806fba0 con 0x7f8eb8072480 2024-09-11T00:47:17.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.270+0000 7f8ebe000640 1 -- 172.21.15.31:0/3435771379 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb80c1580 con 0x7f8eb8072480 2024-09-11T00:47:17.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.272+0000 7f8ebe000640 1 -- 172.21.15.31:0/3435771379 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eb80c3b90 con 0x7f8eb8072480 2024-09-11T00:47:17.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.273+0000 7f8eafe00640 1 -- 172.21.15.31:0/3435771379 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8ea80035f0 con 0x7f8eb8072480 2024-09-11T00:47:17.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.274+0000 7f8eafe00640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e94077fa0 0x7f8e9407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:17.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.274+0000 7f8eafe00640 1 -- 172.21.15.31:0/3435771379 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8558+0+0 (secure 0 0 0) 0x7f8ea809a370 con 0x7f8eb8072480 2024-09-11T00:47:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.274+0000 7f8eaea00640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e94077fa0 0x7f8e9407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.275+0000 7f8eaea00640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e94077fa0 0x7f8e9407a460 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0033000 tx=0x7f8ea0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:17.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.278+0000 7f8eafe00640 1 -- 172.21.15.31:0/3435771379 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8ea8061ce0 con 0x7f8eb8072480 2024-09-11T00:47:17.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.509+0000 7f8ebe000640 1 -- 172.21.15.31:0/3435771379 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8eb81efca0 con 0x7f8eb8072480 2024-09-11T00:47:17.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.509+0000 7f8eafe00640 1 -- 172.21.15.31:0/3435771379 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6544 (secure 0 0 0) 0x7f8ea8066710 con 0x7f8eb8072480 2024-09-11T00:47:17.511 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:17.511 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":122,"btime":"2024-09-11T00:46:04:450048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6804,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:45:29.788358+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6601":{"gid":6601,"name":"d","rank":0,"incarnation":113,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6835/2113744259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2113744259},{"type":"v1","addr":"172.21.15.204:6835","nonce":2113744259}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6601,"qdb_cluster":[6601]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:17.511 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 122 2024-09-11T00:47:17.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 -- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e94077fa0 msgr2=0x7f8e9407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e94077fa0 0x7f8e9407a460 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0033000 tx=0x7f8ea0004730 comp rx=0 tx=0).stop 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 -- 172.21.15.31:0/3435771379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8eb8072480 msgr2=0x7f8eb80c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8eb8072480 0x7f8eb80c1040 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800c480 tx=0x7f8ea800c940 comp rx=0 tx=0).stop 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 -- 172.21.15.31:0/3435771379 shutdown_connections 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e94077fa0 0x7f8e9407a460 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8eb81518b0 0x7f8eb814a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8eb80c23a0 0x7f8eb814a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 --2- 172.21.15.31:0/3435771379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8eb8072480 0x7f8eb80c1040 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 -- 172.21.15.31:0/3435771379 >> 172.21.15.31:0/3435771379 conn(0x7f8eb80a9750 msgr2=0x7f8eb8151180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 -- 172.21.15.31:0/3435771379 shutdown_connections 2024-09-11T00:47:17.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:17.512+0000 7f8ebe000640 1 -- 172.21.15.31:0/3435771379 wait complete. 2024-09-11T00:47:17.529 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:47:18.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:47:18.659 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:47:19.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:19.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.698+0000 7f6ee2200640 1 -- 172.21.15.31:0/2070285848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6edc074a30 msgr2=0x7f6edc06f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:19.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.698+0000 7f6ee2200640 1 --2- 172.21.15.31:0/2070285848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6edc074a30 0x7f6edc06f930 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8005fd0 tx=0x7f6ec802a540 comp rx=0 tx=0).stop 2024-09-11T00:47:19.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.698+0000 7f6ee2200640 1 -- 172.21.15.31:0/2070285848 shutdown_connections 2024-09-11T00:47:19.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.698+0000 7f6ee2200640 1 --2- 172.21.15.31:0/2070285848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6edc074a30 0x7f6edc06f930 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:19.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.698+0000 7f6ee2200640 1 --2- 172.21.15.31:0/2070285848 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6edc06dd70 0x7f6edc154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:19.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.698+0000 7f6ee2200640 1 -- 172.21.15.31:0/2070285848 >> 172.21.15.31:0/2070285848 conn(0x7f6edc0c4d20 msgr2=0x7f6edc0c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:19.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.698+0000 7f6ee2200640 1 -- 172.21.15.31:0/2070285848 shutdown_connections 2024-09-11T00:47:19.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.698+0000 7f6ee2200640 1 -- 172.21.15.31:0/2070285848 wait complete. 2024-09-11T00:47:19.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.700+0000 7f6ee2200640 1 Processor -- start 2024-09-11T00:47:19.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.700+0000 7f6ee2200640 1 -- start start 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6ee2200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6edc06dd70 0x7f6edc1f0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6ee2200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6edc074a30 0x7f6edc0be9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6ee2200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6edc150d60 0x7f6edc1eea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6ee2200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc071260 con 0x7f6edc06dd70 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6ee2200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc070c10 con 0x7f6edc150d60 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6ee2200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6edc06f470 con 0x7f6edc074a30 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6ee0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6edc06dd70 0x7f6edc1f0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6ee0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6edc06dd70 0x7f6edc1f0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58758/0 (socket says 172.21.15.31:58758) 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6ee0e00640 1 -- 172.21.15.31:0/1412624121 learned_addr learned my addr 172.21.15.31:0/1412624121 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6edbe00640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6edc074a30 0x7f6edc0be9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:19.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.701+0000 7f6ee1800640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6edc150d60 0x7f6edc1eea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:19.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ee0e00640 1 -- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6edc074a30 msgr2=0x7f6edc0be9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ee0e00640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6edc074a30 0x7f6edc0be9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ee0e00640 1 -- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6edc150d60 msgr2=0x7f6edc1eea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ee0e00640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6edc150d60 0x7f6edc1eea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ee0e00640 1 -- 172.21.15.31:0/1412624121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc152c00 con 0x7f6edc06dd70 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ee1800640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6edc150d60 0x7f6edc1eea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ee0e00640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6edc06dd70 0x7f6edc1f0040 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000e9f0 tx=0x7f6ed000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ed8c00640 1 -- 172.21.15.31:0/1412624121 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed000cd80 con 0x7f6edc06dd70 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ed8c00640 1 -- 172.21.15.31:0/1412624121 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ed0010430 con 0x7f6edc06dd70 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ed8c00640 1 -- 172.21.15.31:0/1412624121 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0010840 con 0x7f6edc06dd70 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ee2200640 1 -- 172.21.15.31:0/1412624121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6edc1ec130 con 0x7f6edc06dd70 2024-09-11T00:47:19.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.702+0000 7f6ee2200640 1 -- 172.21.15.31:0/1412624121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6edc153180 con 0x7f6edc06dd70 2024-09-11T00:47:19.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.704+0000 7f6ee2200640 1 -- 172.21.15.31:0/1412624121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6edc06e150 con 0x7f6edc06dd70 2024-09-11T00:47:19.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.705+0000 7f6ed8c00640 1 -- 172.21.15.31:0/1412624121 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6ed00109e0 con 0x7f6edc06dd70 2024-09-11T00:47:19.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.706+0000 7f6ed8c00640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ebc077f70 0x7f6ebc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:19.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.706+0000 7f6ed8c00640 1 -- 172.21.15.31:0/1412624121 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f6ed009b470 con 0x7f6edc06dd70 2024-09-11T00:47:19.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.707+0000 7f6edbe00640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ebc077f70 0x7f6ebc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:19.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.707+0000 7f6edbe00640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ebc077f70 0x7f6ebc07a430 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8033000 tx=0x7f6ec8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:19.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.710+0000 7f6ed8c00640 1 -- 172.21.15.31:0/1412624121 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6ed0062e10 con 0x7f6edc06dd70 2024-09-11T00:47:19.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.948+0000 7f6ee2200640 1 -- 172.21.15.31:0/1412624121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6edc0ae5b0 con 0x7f6edc06dd70 2024-09-11T00:47:19.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.948+0000 7f6ed8c00640 1 -- 172.21.15.31:0/1412624121 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5708 (secure 0 0 0) 0x7f6ed0067840 con 0x7f6edc06dd70 2024-09-11T00:47:19.950 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:19.950 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":123,"btime":"2024-09-11T00:47:19:441384+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:19.441381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:19.950 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 123 2024-09-11T00:47:19.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 -- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ebc077f70 msgr2=0x7f6ebc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ebc077f70 0x7f6ebc07a430 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8033000 tx=0x7f6ec8004730 comp rx=0 tx=0).stop 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 -- 172.21.15.31:0/1412624121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6edc06dd70 msgr2=0x7f6edc1f0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6edc06dd70 0x7f6edc1f0040 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000e9f0 tx=0x7f6ed000eeb0 comp rx=0 tx=0).stop 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 -- 172.21.15.31:0/1412624121 shutdown_connections 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ebc077f70 0x7f6ebc07a430 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6edc150d60 0x7f6edc1eea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6edc074a30 0x7f6edc0be9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 --2- 172.21.15.31:0/1412624121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6edc06dd70 0x7f6edc1f0040 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 -- 172.21.15.31:0/1412624121 >> 172.21.15.31:0/1412624121 conn(0x7f6edc0c4d20 msgr2=0x7f6edc0c6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 -- 172.21.15.31:0/1412624121 shutdown_connections 2024-09-11T00:47:19.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:19.951+0000 7f6ee2200640 1 -- 172.21.15.31:0/1412624121 wait complete. 2024-09-11T00:47:19.968 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-11T00:47:19.968 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:47:19.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.127+0000 7f7823200640 1 -- 172.21.15.31:0/246273833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f781c072480 msgr2=0x7f781c1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.127+0000 7f7823200640 1 --2- 172.21.15.31:0/246273833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f781c072480 0x7f781c1511a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f781c0c5500 tx=0x7f780c02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:47:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.127+0000 7f7823200640 1 -- 172.21.15.31:0/246273833 shutdown_connections 2024-09-11T00:47:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.127+0000 7f7823200640 1 --2- 172.21.15.31:0/246273833 >> v2:172.21.15.204:3300/0 conn(0x7f781c153f20 0x7f781c1516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:20.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.127+0000 7f7823200640 1 --2- 172.21.15.31:0/246273833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f781c072480 0x7f781c1511a0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:20.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.127+0000 7f7823200640 1 -- 172.21.15.31:0/246273833 >> 172.21.15.31:0/246273833 conn(0x7f781c0a9750 msgr2=0x7f781c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:20.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.127+0000 7f7823200640 1 -- 172.21.15.31:0/246273833 shutdown_connections 2024-09-11T00:47:20.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.127+0000 7f7823200640 1 -- 172.21.15.31:0/246273833 wait complete. 2024-09-11T00:47:20.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.128+0000 7f7823200640 1 Processor -- start 2024-09-11T00:47:20.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.129+0000 7f7823200640 1 -- start start 2024-09-11T00:47:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.129+0000 7f7823200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f781c072480 0x7f781c1e1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.129+0000 7f7823200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f781c0c23a0 0x7f781c0c3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7823200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f781c153f20 0x7f781c0c3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7823200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f781c074200 con 0x7f781c0c23a0 2024-09-11T00:47:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7823200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f781c074080 con 0x7f781c072480 2024-09-11T00:47:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7823200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f781c072860 con 0x7f781c153f20 2024-09-11T00:47:20.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7813400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f781c0c23a0 0x7f781c0c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f781be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f781c072480 0x7f781c1e1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f781be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f781c072480 0x7f781c1e1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:34240/0 (socket says 172.21.15.31:34240) 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7813400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f781c0c23a0 0x7f781c0c3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58782/0 (socket says 172.21.15.31:58782) 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f781be00640 1 -- 172.21.15.31:0/2430356437 learned_addr learned my addr 172.21.15.31:0/2430356437 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7821200640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f781c153f20 0x7f781c0c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7813400640 1 -- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f781c153f20 msgr2=0x7f781c0c3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7813400640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f781c153f20 0x7f781c0c3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7813400640 1 -- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f781c072480 msgr2=0x7f781c1e1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7813400640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f781c072480 0x7f781c1e1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.130+0000 7f7813400640 1 -- 172.21.15.31:0/2430356437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781c14a3c0 con 0x7f781c0c23a0 2024-09-11T00:47:20.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.131+0000 7f7813400640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f781c0c23a0 0x7f781c0c3230 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f780400c3e0 tx=0x7f780400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:20.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.131+0000 7f7818c00640 1 -- 172.21.15.31:0/2430356437 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7804011790 con 0x7f781c0c23a0 2024-09-11T00:47:20.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.131+0000 7f7818c00640 1 -- 172.21.15.31:0/2430356437 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7804017070 con 0x7f781c0c23a0 2024-09-11T00:47:20.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.131+0000 7f7823200640 1 -- 172.21.15.31:0/2430356437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f781c14a700 con 0x7f781c0c23a0 2024-09-11T00:47:20.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.131+0000 7f7818c00640 1 -- 172.21.15.31:0/2430356437 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7804011cc0 con 0x7f781c0c23a0 2024-09-11T00:47:20.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.131+0000 7f7823200640 1 -- 172.21.15.31:0/2430356437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f781c14af40 con 0x7f781c0c23a0 2024-09-11T00:47:20.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.132+0000 7f7823200640 1 -- 172.21.15.31:0/2430356437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f781c1511a0 con 0x7f781c0c23a0 2024-09-11T00:47:20.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.133+0000 7f7818c00640 1 -- 172.21.15.31:0/2430356437 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f780401c460 con 0x7f781c0c23a0 2024-09-11T00:47:20.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.134+0000 7f7818c00640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f77f8077f70 0x7f77f807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:20.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.134+0000 7f7818c00640 1 -- 172.21.15.31:0/2430356437 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f780409a610 con 0x7f781c0c23a0 2024-09-11T00:47:20.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.134+0000 7f781be00640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f77f8077f70 0x7f77f807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:20.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.135+0000 7f781be00640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f77f8077f70 0x7f77f807a430 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f780c0096c0 tx=0x7f780c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:20.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.138+0000 7f7818c00640 1 -- 172.21.15.31:0/2430356437 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f781c14a700 con 0x7f781c0c23a0 2024-09-11T00:47:20.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.371+0000 7f7823200640 1 -- 172.21.15.31:0/2430356437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f781c14b110 con 0x7f781c0c23a0 2024-09-11T00:47:20.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.372+0000 7f7818c00640 1 -- 172.21.15.31:0/2430356437 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5708 (secure 0 0 0) 0x7f7804061f20 con 0x7f781c0c23a0 2024-09-11T00:47:20.374 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:20.374 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":123,"btime":"2024-09-11T00:47:19:441384+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:19.441381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:20.374 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 123 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 -- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f77f8077f70 msgr2=0x7f77f807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f77f8077f70 0x7f77f807a430 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f780c0096c0 tx=0x7f780c004730 comp rx=0 tx=0).stop 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 -- 172.21.15.31:0/2430356437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f781c0c23a0 msgr2=0x7f781c0c3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f781c0c23a0 0x7f781c0c3230 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f780400c3e0 tx=0x7f780400c8a0 comp rx=0 tx=0).stop 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 -- 172.21.15.31:0/2430356437 shutdown_connections 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f77f8077f70 0x7f77f807a430 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f781c153f20 0x7f781c0c3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f781c0c23a0 0x7f781c0c3230 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 --2- 172.21.15.31:0/2430356437 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f781c072480 0x7f781c1e1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 -- 172.21.15.31:0/2430356437 >> 172.21.15.31:0/2430356437 conn(0x7f781c0a9750 msgr2=0x7f781c0c0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 -- 172.21.15.31:0/2430356437 shutdown_connections 2024-09-11T00:47:20.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:20.375+0000 7f7823200640 1 -- 172.21.15.31:0/2430356437 wait complete. 2024-09-11T00:47:20.393 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:47:19:441384+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 123, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 123, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6804": { "addr": "172.21.15.31:6835/2163558126", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 2163558126, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 2163558126, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6804, "incarnation": 123, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:47:19.441381+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6804 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 121, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5957": { "addr": "172.21.15.204:6837/1082799504", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 1082799504, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 1082799504, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5957, "incarnation": 118, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:46:02.442423+0000", "qdb_cluster": [ 5957 ], "qdb_leader": 5957, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5957 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:47:22.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.563+0000 7f5be3e00640 1 -- 172.21.15.31:0/3198625240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be40ac2c0 msgr2=0x7f5be40ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.563+0000 7f5be3e00640 1 --2- 172.21.15.31:0/3198625240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be40ac2c0 0x7f5be40ac6a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009860 tx=0x7f5bd402a480 comp rx=0 tx=0).stop 2024-09-11T00:47:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.563+0000 7f5be3e00640 1 -- 172.21.15.31:0/3198625240 shutdown_connections 2024-09-11T00:47:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.563+0000 7f5be3e00640 1 --2- 172.21.15.31:0/3198625240 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be40ac2c0 0x7f5be40ac6a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.563+0000 7f5be3e00640 1 -- 172.21.15.31:0/3198625240 >> 172.21.15.31:0/3198625240 conn(0x7f5be406caf0 msgr2=0x7f5be406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.563+0000 7f5be3e00640 1 -- 172.21.15.31:0/3198625240 shutdown_connections 2024-09-11T00:47:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.563+0000 7f5be3e00640 1 -- 172.21.15.31:0/3198625240 wait complete. 2024-09-11T00:47:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.565+0000 7f5be3e00640 1 Processor -- start 2024-09-11T00:47:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.565+0000 7f5be3e00640 1 -- start start 2024-09-11T00:47:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be40a6e20 0x7f5be41525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5be40ac2c0 0x7f5be414b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be40acbe0 0x7f5be414b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5be40c5d20 con 0x7f5be40a6e20 2024-09-11T00:47:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5be41519e0 con 0x7f5be40ac2c0 2024-09-11T00:47:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5be40c68c0 con 0x7f5be40acbe0 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be40a6e20 0x7f5be41525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be40a6e20 0x7f5be41525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58796/0 (socket says 172.21.15.31:58796) 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be2a00640 1 -- 172.21.15.31:0/1636657909 learned_addr learned my addr 172.21.15.31:0/1636657909 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5bda000640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5be40ac2c0 0x7f5be414b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be3400640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be40acbe0 0x7f5be414b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be2a00640 1 -- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be40acbe0 msgr2=0x7f5be414b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be2a00640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be40acbe0 0x7f5be414b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be2a00640 1 -- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5be40ac2c0 msgr2=0x7f5be414b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be2a00640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5be40ac2c0 0x7f5be414b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be2a00640 1 -- 172.21.15.31:0/1636657909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be4157e10 con 0x7f5be40a6e20 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.566+0000 7f5be3400640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be40acbe0 0x7f5be414b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:47:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.567+0000 7f5be2a00640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be40a6e20 0x7f5be41525d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009980 tx=0x7f5bd4032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:22.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.567+0000 7f5bdb400640 1 -- 172.21.15.31:0/1636657909 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4040070 con 0x7f5be40a6e20 2024-09-11T00:47:22.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.567+0000 7f5bdb400640 1 -- 172.21.15.31:0/1636657909 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bd4005310 con 0x7f5be40a6e20 2024-09-11T00:47:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.567+0000 7f5be3e00640 1 -- 172.21.15.31:0/1636657909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be4148d40 con 0x7f5be40a6e20 2024-09-11T00:47:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.567+0000 7f5bdb400640 1 -- 172.21.15.31:0/1636657909 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd40047a0 con 0x7f5be40a6e20 2024-09-11T00:47:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.567+0000 7f5be3e00640 1 -- 172.21.15.31:0/1636657909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be4148fe0 con 0x7f5be40a6e20 2024-09-11T00:47:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.568+0000 7f5be3e00640 1 -- 172.21.15.31:0/1636657909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5be40c6c60 con 0x7f5be40a6e20 2024-09-11T00:47:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.569+0000 7f5bdb400640 1 -- 172.21.15.31:0/1636657909 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5bd404c030 con 0x7f5be40a6e20 2024-09-11T00:47:22.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.570+0000 7f5bdb400640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bc0077f70 0x7f5bc007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:22.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.571+0000 7f5bdb400640 1 -- 172.21.15.31:0/1636657909 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f5bd40bb910 con 0x7f5be40a6e20 2024-09-11T00:47:22.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.571+0000 7f5bda000640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bc0077f70 0x7f5bc007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.571+0000 7f5bda000640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bc0077f70 0x7f5bc007a430 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc007fd0 tx=0x7f5bcc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:22.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.575+0000 7f5bdb400640 1 -- 172.21.15.31:0/1636657909 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5bd4083870 con 0x7f5be40a6e20 2024-09-11T00:47:22.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.820+0000 7f5be3e00640 1 -- 172.21.15.31:0/1636657909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5be41514b0 con 0x7f5be40a6e20 2024-09-11T00:47:22.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.820+0000 7f5bdb400640 1 -- 172.21.15.31:0/1636657909 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+5716 (secure 0 0 0) 0x7f5bd4083170 con 0x7f5be40a6e20 2024-09-11T00:47:22.822 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:22.822 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":126,"btime":"2024-09-11T00:47:22:793067+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:22.823 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 126 2024-09-11T00:47:22.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 -- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bc0077f70 msgr2=0x7f5bc007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:22.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bc0077f70 0x7f5bc007a430 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc007fd0 tx=0x7f5bcc009040 comp rx=0 tx=0).stop 2024-09-11T00:47:22.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 -- 172.21.15.31:0/1636657909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be40a6e20 msgr2=0x7f5be41525d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:22.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be40a6e20 0x7f5be41525d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009980 tx=0x7f5bd4032b90 comp rx=0 tx=0).stop 2024-09-11T00:47:22.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 -- 172.21.15.31:0/1636657909 shutdown_connections 2024-09-11T00:47:22.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5bc0077f70 0x7f5bc007a430 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:22.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5be40acbe0 0x7f5be414b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:22.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5be40ac2c0 0x7f5be414b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:22.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 --2- 172.21.15.31:0/1636657909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5be40a6e20 0x7f5be41525d0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:22.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 -- 172.21.15.31:0/1636657909 >> 172.21.15.31:0/1636657909 conn(0x7f5be406caf0 msgr2=0x7f5be40c46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:22.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 -- 172.21.15.31:0/1636657909 shutdown_connections 2024-09-11T00:47:22.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:22.823+0000 7f5be3e00640 1 -- 172.21.15.31:0/1636657909 wait complete. 2024-09-11T00:47:22.846 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6601 2024-09-11T00:47:22.846 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.d 2024-09-11T00:47:24.467 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-11T00:47:24.465+0000 7fe6dd400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:47:29.847 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-11T00:47:29.847 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:47:29.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:47:29.852 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:47:29.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:29.994 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:47:30.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.025+0000 7f529fe00640 1 -- 172.21.15.31:0/3599751257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a014ccc0 msgr2=0x7f52a0156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.025+0000 7f529fe00640 1 --2- 172.21.15.31:0/3599751257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a014ccc0 0x7f52a0156b00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5288009830 tx=0x7f528802a510 comp rx=0 tx=0).stop 2024-09-11T00:47:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.026+0000 7f529fe00640 1 -- 172.21.15.31:0/3599751257 shutdown_connections 2024-09-11T00:47:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.026+0000 7f529fe00640 1 --2- 172.21.15.31:0/3599751257 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a00c9a10 0x7f52a0159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.026+0000 7f529fe00640 1 --2- 172.21.15.31:0/3599751257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a014ccc0 0x7f52a0156b00 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.026+0000 7f529fe00640 1 -- 172.21.15.31:0/3599751257 >> 172.21.15.31:0/3599751257 conn(0x7f52a0068ee0 msgr2=0x7f52a00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.026+0000 7f529fe00640 1 -- 172.21.15.31:0/3599751257 shutdown_connections 2024-09-11T00:47:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.026+0000 7f529fe00640 1 -- 172.21.15.31:0/3599751257 wait complete. 2024-09-11T00:47:30.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.027+0000 7f529fe00640 1 Processor -- start 2024-09-11T00:47:30.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.027+0000 7f529fe00640 1 -- start start 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.028+0000 7f529fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a00c9a10 0x7f52a01ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.028+0000 7f529fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a014c8e0 0x7f52a0151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.028+0000 7f529fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a014ccc0 0x7f52a00c18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.028+0000 7f529fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f52a00c6490 con 0x7f52a00c9a10 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.028+0000 7f529fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f52a00c6800 con 0x7f52a014ccc0 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.028+0000 7f529fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f52a00c6c90 con 0x7f52a014c8e0 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f529ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a00c9a10 0x7f52a01ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f529ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a00c9a10 0x7f52a01ec620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33856/0 (socket says 172.21.15.31:33856) 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f529ea00640 1 -- 172.21.15.31:0/2219168856 learned_addr learned my addr 172.21.15.31:0/2219168856 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f5297e00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a014c8e0 0x7f52a0151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f529f400640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a014ccc0 0x7f52a00c18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f529ea00640 1 -- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a014c8e0 msgr2=0x7f52a0151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f529ea00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a014c8e0 0x7f52a0151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f529ea00640 1 -- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a014ccc0 msgr2=0x7f52a00c18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f529ea00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a014ccc0 0x7f52a00c18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f529ea00640 1 -- 172.21.15.31:0/2219168856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a01535a0 con 0x7f52a00c9a10 2024-09-11T00:47:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.029+0000 7f529ea00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a00c9a10 0x7f52a01ec620 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f529000c490 tx=0x7f529000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.030+0000 7f5296a00640 1 -- 172.21.15.31:0/2219168856 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5290011790 con 0x7f52a00c9a10 2024-09-11T00:47:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.030+0000 7f5296a00640 1 -- 172.21.15.31:0/2219168856 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5290017070 con 0x7f52a00c9a10 2024-09-11T00:47:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.030+0000 7f529fe00640 1 -- 172.21.15.31:0/2219168856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a0150160 con 0x7f52a00c9a10 2024-09-11T00:47:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.030+0000 7f5296a00640 1 -- 172.21.15.31:0/2219168856 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5290011ce0 con 0x7f52a00c9a10 2024-09-11T00:47:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.030+0000 7f529fe00640 1 -- 172.21.15.31:0/2219168856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a014c530 con 0x7f52a00c9a10 2024-09-11T00:47:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.031+0000 7f529fe00640 1 -- 172.21.15.31:0/2219168856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a01ebf10 con 0x7f52a00c9a10 2024-09-11T00:47:30.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.033+0000 7f5296a00640 1 -- 172.21.15.31:0/2219168856 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f529001c460 con 0x7f52a00c9a10 2024-09-11T00:47:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.033+0000 7f5296a00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f527c077fa0 0x7f527c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.034+0000 7f5296a00640 1 -- 172.21.15.31:0/2219168856 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f529009a510 con 0x7f52a00c9a10 2024-09-11T00:47:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.034+0000 7f5297e00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f527c077fa0 0x7f527c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.035+0000 7f5297e00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f527c077fa0 0x7f527c07a460 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5288034000 tx=0x7f5288004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:30.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.038+0000 7f5296a00640 1 -- 172.21.15.31:0/2219168856 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5290061e80 con 0x7f52a00c9a10 2024-09-11T00:47:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.351+0000 7f529fe00640 1 -- 172.21.15.31:0/2219168856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52a014ff50 con 0x7f52a00c9a10 2024-09-11T00:47:30.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.351+0000 7f5296a00640 1 -- 172.21.15.31:0/2219168856 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+5716 (secure 0 0 0) 0x7f52900668b0 con 0x7f52a00c9a10 2024-09-11T00:47:30.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:30.353 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":126,"btime":"2024-09-11T00:47:22:793067+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:30.354 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 126 2024-09-11T00:47:30.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.354+0000 7f529fe00640 1 -- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f527c077fa0 msgr2=0x7f527c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:30.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.354+0000 7f529fe00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f527c077fa0 0x7f527c07a460 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5288034000 tx=0x7f5288004730 comp rx=0 tx=0).stop 2024-09-11T00:47:30.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.354+0000 7f529fe00640 1 -- 172.21.15.31:0/2219168856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a00c9a10 msgr2=0x7f52a01ec620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.354+0000 7f529fe00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a00c9a10 0x7f52a01ec620 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f529000c490 tx=0x7f529000c950 comp rx=0 tx=0).stop 2024-09-11T00:47:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.354+0000 7f529fe00640 1 -- 172.21.15.31:0/2219168856 shutdown_connections 2024-09-11T00:47:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.355+0000 7f529fe00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f527c077fa0 0x7f527c07a460 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.355+0000 7f529fe00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52a014ccc0 0x7f52a00c18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.355+0000 7f529fe00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f52a014c8e0 0x7f52a0151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.355+0000 7f529fe00640 1 --2- 172.21.15.31:0/2219168856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52a00c9a10 0x7f52a01ec620 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.355+0000 7f529fe00640 1 -- 172.21.15.31:0/2219168856 >> 172.21.15.31:0/2219168856 conn(0x7f52a0068ee0 msgr2=0x7f52a00ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.355+0000 7f529fe00640 1 -- 172.21.15.31:0/2219168856 shutdown_connections 2024-09-11T00:47:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:30.355+0000 7f529fe00640 1 -- 172.21.15.31:0/2219168856 wait complete. 2024-09-11T00:47:30.375 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:47:32.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:32.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.551+0000 7f27f5200640 1 -- 172.21.15.31:0/734760520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f00c68b0 msgr2=0x7f27f01535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:32.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.551+0000 7f27f5200640 1 --2- 172.21.15.31:0/734760520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f00c68b0 0x7f27f01535a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009830 tx=0x7f27d802a510 comp rx=0 tx=0).stop 2024-09-11T00:47:32.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.551+0000 7f27f5200640 1 -- 172.21.15.31:0/734760520 shutdown_connections 2024-09-11T00:47:32.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.551+0000 7f27f5200640 1 --2- 172.21.15.31:0/734760520 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27f0153ae0 0x7f27f0156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:32.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.551+0000 7f27f5200640 1 --2- 172.21.15.31:0/734760520 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f00c68b0 0x7f27f01535a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:32.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.551+0000 7f27f5200640 1 -- 172.21.15.31:0/734760520 >> 172.21.15.31:0/734760520 conn(0x7f27f00aa760 msgr2=0x7f27f00aab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:32.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.551+0000 7f27f5200640 1 -- 172.21.15.31:0/734760520 shutdown_connections 2024-09-11T00:47:32.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.551+0000 7f27f5200640 1 -- 172.21.15.31:0/734760520 wait complete. 2024-09-11T00:47:32.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.553+0000 7f27f5200640 1 Processor -- start 2024-09-11T00:47:32.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.553+0000 7f27f5200640 1 -- start start 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27f5200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27f006cd60 0x7f27f014b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27f5200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f00c68b0 0x7f27f00741b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27f5200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27f0153ae0 0x7f27f006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27f5200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f27f006d2c0 con 0x7f27f006cd60 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27f5200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f27f006d140 con 0x7f27f00c68b0 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27f5200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f27f0156690 con 0x7f27f0153ae0 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27ef400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27f006cd60 0x7f27f014b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27ef400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27f006cd60 0x7f27f014b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33858/0 (socket says 172.21.15.31:33858) 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27ef400640 1 -- 172.21.15.31:0/2017731645 learned_addr learned my addr 172.21.15.31:0/2017731645 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27efe00640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27f0153ae0 0x7f27f006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:32.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.554+0000 7f27e6a00640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f00c68b0 0x7f27f00741b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:32.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.555+0000 7f27ef400640 1 -- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27f0153ae0 msgr2=0x7f27f006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:32.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.555+0000 7f27ef400640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27f0153ae0 0x7f27f006ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:32.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.555+0000 7f27ef400640 1 -- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f00c68b0 msgr2=0x7f27f00741b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:32.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.555+0000 7f27ef400640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f00c68b0 0x7f27f00741b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:32.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.555+0000 7f27ef400640 1 -- 172.21.15.31:0/2017731645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f00702a0 con 0x7f27f006cd60 2024-09-11T00:47:32.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.555+0000 7f27ef400640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27f006cd60 0x7f27f014b740 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f27e000c4c0 tx=0x7f27e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.555+0000 7f27e7e00640 1 -- 172.21.15.31:0/2017731645 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0011790 con 0x7f27f006cd60 2024-09-11T00:47:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.555+0000 7f27e7e00640 1 -- 172.21.15.31:0/2017731645 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27e0017070 con 0x7f27f006cd60 2024-09-11T00:47:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.556+0000 7f27e7e00640 1 -- 172.21.15.31:0/2017731645 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0011ce0 con 0x7f27f006cd60 2024-09-11T00:47:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.556+0000 7f27f5200640 1 -- 172.21.15.31:0/2017731645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f0070580 con 0x7f27f006cd60 2024-09-11T00:47:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.556+0000 7f27f5200640 1 -- 172.21.15.31:0/2017731645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f0155890 con 0x7f27f006cd60 2024-09-11T00:47:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.557+0000 7f27f5200640 1 -- 172.21.15.31:0/2017731645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f00ae710 con 0x7f27f006cd60 2024-09-11T00:47:32.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.558+0000 7f27e7e00640 1 -- 172.21.15.31:0/2017731645 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f27e001c460 con 0x7f27f006cd60 2024-09-11T00:47:32.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.559+0000 7f27e7e00640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f27cc077fa0 0x7f27cc07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:32.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.559+0000 7f27e7e00640 1 -- 172.21.15.31:0/2017731645 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f27e009a5c0 con 0x7f27f006cd60 2024-09-11T00:47:32.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.560+0000 7f27e6a00640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f27cc077fa0 0x7f27cc07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:32.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.561+0000 7f27e6a00640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f27cc077fa0 0x7f27cc07a460 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f27d8034000 tx=0x7f27d8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:32.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.565+0000 7f27e7e00640 1 -- 172.21.15.31:0/2017731645 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f27e0062520 con 0x7f27f006cd60 2024-09-11T00:47:32.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.848+0000 7f27f5200640 1 -- 172.21.15.31:0/2017731645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27f0159810 con 0x7f27f006cd60 2024-09-11T00:47:32.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.849+0000 7f27e7e00640 1 -- 172.21.15.31:0/2017731645 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6542 (secure 0 0 0) 0x7f27e0061e20 con 0x7f27f006cd60 2024-09-11T00:47:32.850 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:32.851 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"btime":"2024-09-11T00:47:30:455013+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:32.851 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 127 2024-09-11T00:47:32.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.851+0000 7f27f5200640 1 -- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f27cc077fa0 msgr2=0x7f27cc07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:32.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.851+0000 7f27f5200640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f27cc077fa0 0x7f27cc07a460 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f27d8034000 tx=0x7f27d8004730 comp rx=0 tx=0).stop 2024-09-11T00:47:32.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.851+0000 7f27f5200640 1 -- 172.21.15.31:0/2017731645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27f006cd60 msgr2=0x7f27f014b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:32.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.851+0000 7f27f5200640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27f006cd60 0x7f27f014b740 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f27e000c4c0 tx=0x7f27e000c8c0 comp rx=0 tx=0).stop 2024-09-11T00:47:32.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.851+0000 7f27f5200640 1 -- 172.21.15.31:0/2017731645 shutdown_connections 2024-09-11T00:47:32.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.851+0000 7f27f5200640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f27cc077fa0 0x7f27cc07a460 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:32.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.851+0000 7f27f5200640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f27f0153ae0 0x7f27f006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:32.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.851+0000 7f27f5200640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27f00c68b0 0x7f27f00741b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:32.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.851+0000 7f27f5200640 1 --2- 172.21.15.31:0/2017731645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27f006cd60 0x7f27f014b740 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:32.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.851+0000 7f27f5200640 1 -- 172.21.15.31:0/2017731645 >> 172.21.15.31:0/2017731645 conn(0x7f27f00aa760 msgr2=0x7f27f00a8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:32.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.852+0000 7f27f5200640 1 -- 172.21.15.31:0/2017731645 shutdown_connections 2024-09-11T00:47:32.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:32.852+0000 7f27f5200640 1 -- 172.21.15.31:0/2017731645 wait complete. 2024-09-11T00:47:32.869 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-11T00:47:32.870 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2024-09-11T00:47:42.596 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.783+0000 7f0864800640 1 -- 172.21.15.31:0/116171199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f085c072480 msgr2=0x7f085c1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.783+0000 7f0864800640 1 --2- 172.21.15.31:0/116171199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f085c072480 0x7f085c1511a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0850005fd0 tx=0x7f085002a670 comp rx=0 tx=0).stop 2024-09-11T00:47:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.784+0000 7f0864800640 1 -- 172.21.15.31:0/116171199 shutdown_connections 2024-09-11T00:47:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.784+0000 7f0864800640 1 --2- 172.21.15.31:0/116171199 >> v2:172.21.15.204:3300/0 conn(0x7f085c0c23a0 0x7f085c1516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.784+0000 7f0864800640 1 --2- 172.21.15.31:0/116171199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f085c072480 0x7f085c1511a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.784+0000 7f0864800640 1 -- 172.21.15.31:0/116171199 >> 172.21.15.31:0/116171199 conn(0x7f085c0a9750 msgr2=0x7f085c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.784+0000 7f0864800640 1 -- 172.21.15.31:0/116171199 shutdown_connections 2024-09-11T00:47:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.784+0000 7f0864800640 1 -- 172.21.15.31:0/116171199 wait complete. 2024-09-11T00:47:42.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.785+0000 7f0864800640 1 Processor -- start 2024-09-11T00:47:42.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.786+0000 7f0864800640 1 -- start start 2024-09-11T00:47:42.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.786+0000 7f0864800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f085c072480 0x7f085c0c37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:42.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.786+0000 7f0864800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f085c0c23a0 0x7f085c14c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.786+0000 7f0864800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f085c153f20 0x7f085c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.786+0000 7f0864800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f085c074200 con 0x7f085c0c23a0 2024-09-11T00:47:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.786+0000 7f0864800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f085c074080 con 0x7f085c072480 2024-09-11T00:47:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.786+0000 7f0864800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f085c072860 con 0x7f085c153f20 2024-09-11T00:47:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0861400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f085c0c23a0 0x7f085c14c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0861e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f085c072480 0x7f085c0c37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0861400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f085c0c23a0 0x7f085c14c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57460/0 (socket says 172.21.15.31:57460) 2024-09-11T00:47:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0861400640 1 -- 172.21.15.31:0/3686255302 learned_addr learned my addr 172.21.15.31:0/3686255302 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0862800640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f085c153f20 0x7f085c14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0861400640 1 -- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f085c153f20 msgr2=0x7f085c14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0861400640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f085c153f20 0x7f085c14c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0861400640 1 -- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f085c072480 msgr2=0x7f085c0c37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0861400640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f085c072480 0x7f085c0c37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0861400640 1 -- 172.21.15.31:0/3686255302 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085c06f7a0 con 0x7f085c0c23a0 2024-09-11T00:47:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.787+0000 7f0862800640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f085c153f20 0x7f085c14c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:47:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.788+0000 7f0861400640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f085c0c23a0 0x7f085c14c310 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f084c00ea20 tx=0x7f084c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.788+0000 7f0856000640 1 -- 172.21.15.31:0/3686255302 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084c00cd80 con 0x7f085c0c23a0 2024-09-11T00:47:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.788+0000 7f0856000640 1 -- 172.21.15.31:0/3686255302 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f084c004030 con 0x7f085c0c23a0 2024-09-11T00:47:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.788+0000 7f0864800640 1 -- 172.21.15.31:0/3686255302 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085c06fa80 con 0x7f085c0c23a0 2024-09-11T00:47:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.788+0000 7f0856000640 1 -- 172.21.15.31:0/3686255302 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084c004480 con 0x7f085c0c23a0 2024-09-11T00:47:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.788+0000 7f0864800640 1 -- 172.21.15.31:0/3686255302 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f085c070040 con 0x7f085c0c23a0 2024-09-11T00:47:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.789+0000 7f0864800640 1 -- 172.21.15.31:0/3686255302 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f085c073630 con 0x7f085c0c23a0 2024-09-11T00:47:42.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.791+0000 7f0856000640 1 -- 172.21.15.31:0/3686255302 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f084c0035f0 con 0x7f085c0c23a0 2024-09-11T00:47:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.792+0000 7f0856000640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0840077f70 0x7f084007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.792+0000 7f0856000640 1 -- 172.21.15.31:0/3686255302 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f084c09b2f0 con 0x7f085c0c23a0 2024-09-11T00:47:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.792+0000 7f0861e00640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0840077f70 0x7f084007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.793+0000 7f0861e00640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0840077f70 0x7f084007a430 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0850033000 tx=0x7f0850004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:42.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:42.797+0000 7f0856000640 1 -- 172.21.15.31:0/3686255302 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f085c073630 con 0x7f085c0c23a0 2024-09-11T00:47:43.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.087+0000 7f0864800640 1 -- 172.21.15.31:0/3686255302 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f085c1511a0 con 0x7f085c0c23a0 2024-09-11T00:47:43.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.088+0000 7f0856000640 1 -- 172.21.15.31:0/3686255302 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6542 (secure 0 0 0) 0x7f084c063250 con 0x7f085c0c23a0 2024-09-11T00:47:43.090 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:43.090 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"btime":"2024-09-11T00:47:30:455013+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:43.090 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 127 2024-09-11T00:47:43.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.090+0000 7f0864800640 1 -- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0840077f70 msgr2=0x7f084007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:43.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.090+0000 7f0864800640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0840077f70 0x7f084007a430 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0850033000 tx=0x7f0850004730 comp rx=0 tx=0).stop 2024-09-11T00:47:43.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.090+0000 7f0864800640 1 -- 172.21.15.31:0/3686255302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f085c0c23a0 msgr2=0x7f085c14c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:43.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.090+0000 7f0864800640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f085c0c23a0 0x7f085c14c310 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f084c00ea20 tx=0x7f084c00eee0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.090+0000 7f0864800640 1 -- 172.21.15.31:0/3686255302 shutdown_connections 2024-09-11T00:47:43.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.090+0000 7f0864800640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0840077f70 0x7f084007a430 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.090+0000 7f0864800640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f085c153f20 0x7f085c14c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.090+0000 7f0864800640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f085c0c23a0 0x7f085c14c310 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.090+0000 7f0864800640 1 --2- 172.21.15.31:0/3686255302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f085c072480 0x7f085c0c37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.090+0000 7f0864800640 1 -- 172.21.15.31:0/3686255302 >> 172.21.15.31:0/3686255302 conn(0x7f085c0a9750 msgr2=0x7f085c1532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:43.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.091+0000 7f0864800640 1 -- 172.21.15.31:0/3686255302 shutdown_connections 2024-09-11T00:47:43.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.091+0000 7f0864800640 1 -- 172.21.15.31:0/3686255302 wait complete. 2024-09-11T00:47:43.110 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-11T00:47:43.110 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-11T00:47:43.110 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:47:43.128 INFO:tasks.ceph.mds.b:Stopped 2024-09-11T00:47:43.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:43.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.297+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3229547710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bac0ac2c0 msgr2=0x7f2bac0ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:43.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.297+0000 7f2bb2a00640 1 --2- 172.21.15.31:0/3229547710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bac0ac2c0 0x7f2bac0ac6a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009830 tx=0x7f2ba002a4a0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.297+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3229547710 shutdown_connections 2024-09-11T00:47:43.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.297+0000 7f2bb2a00640 1 --2- 172.21.15.31:0/3229547710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bac0ac2c0 0x7f2bac0ac6a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.297+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3229547710 >> 172.21.15.31:0/3229547710 conn(0x7f2bac06caf0 msgr2=0x7f2bac06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:43.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.298+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3229547710 shutdown_connections 2024-09-11T00:47:43.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.298+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3229547710 wait complete. 2024-09-11T00:47:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.299+0000 7f2bb2a00640 1 Processor -- start 2024-09-11T00:47:43.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.300+0000 7f2bb2a00640 1 -- start start 2024-09-11T00:47:43.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.301+0000 7f2bb2a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bac0a6e20 0x7f2bac0c0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.301+0000 7f2bb2a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bac0ac2c0 0x7f2bac1575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bac0acbe0 0x7f2bac0745c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb2a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2bac0c5d20 con 0x7f2bac0ac2c0 2024-09-11T00:47:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb2a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2bac1519e0 con 0x7f2bac0a6e20 2024-09-11T00:47:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb2a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2bac0c68c0 con 0x7f2bac0acbe0 2024-09-11T00:47:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb1600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bac0a6e20 0x7f2bac0c0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb1600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bac0a6e20 0x7f2bac0c0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33290/0 (socket says 172.21.15.31:33290) 2024-09-11T00:47:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb1600640 1 -- 172.21.15.31:0/3341460667 learned_addr learned my addr 172.21.15.31:0/3341460667 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb0c00640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bac0ac2c0 0x7f2bac1575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:43.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb2000640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bac0acbe0 0x7f2bac0745c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb1600640 1 -- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bac0acbe0 msgr2=0x7f2bac0745c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.302+0000 7f2bb1600640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bac0acbe0 0x7f2bac0745c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.303+0000 7f2bb1600640 1 -- 172.21.15.31:0/3341460667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bac0ac2c0 msgr2=0x7f2bac1575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.303+0000 7f2bb1600640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bac0ac2c0 0x7f2bac1575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.303+0000 7f2bb1600640 1 -- 172.21.15.31:0/3341460667 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bac1ec480 con 0x7f2bac0a6e20 2024-09-11T00:47:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.303+0000 7f2bb0c00640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bac0ac2c0 0x7f2bac1575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:47:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.303+0000 7f2bb1600640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bac0a6e20 0x7f2bac0c0fd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009950 tx=0x7f2ba0032c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.303+0000 7f2ba9600640 1 -- 172.21.15.31:0/3341460667 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0040070 con 0x7f2bac0a6e20 2024-09-11T00:47:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.303+0000 7f2ba9600640 1 -- 172.21.15.31:0/3341460667 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ba0005310 con 0x7f2bac0a6e20 2024-09-11T00:47:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.303+0000 7f2ba9600640 1 -- 172.21.15.31:0/3341460667 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00047a0 con 0x7f2bac0a6e20 2024-09-11T00:47:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.303+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3341460667 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bac1ec760 con 0x7f2bac0a6e20 2024-09-11T00:47:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.304+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3341460667 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bac1567c0 con 0x7f2bac0a6e20 2024-09-11T00:47:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.305+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3341460667 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bac0c6c90 con 0x7f2bac0a6e20 2024-09-11T00:47:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.307+0000 7f2ba9600640 1 -- 172.21.15.31:0/3341460667 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2ba0004940 con 0x7f2bac0a6e20 2024-09-11T00:47:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.308+0000 7f2ba9600640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b8c077fa0 0x7f2b8c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.308+0000 7f2ba9600640 1 -- 172.21.15.31:0/3341460667 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f2ba00bba20 con 0x7f2bac0a6e20 2024-09-11T00:47:43.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.308+0000 7f2bb0c00640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b8c077fa0 0x7f2b8c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:43.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.309+0000 7f2bb0c00640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b8c077fa0 0x7f2b8c07a460 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0c5160 tx=0x7f2b98009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:43.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.312+0000 7f2ba9600640 1 -- 172.21.15.31:0/3341460667 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2ba0083390 con 0x7f2bac0a6e20 2024-09-11T00:47:43.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.572+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3341460667 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2bac0694f0 con 0x7f2bac0a6e20 2024-09-11T00:47:43.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.573+0000 7f2ba9600640 1 -- 172.21.15.31:0/3341460667 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6542 (secure 0 0 0) 0x7f2ba0087dc0 con 0x7f2bac0a6e20 2024-09-11T00:47:43.575 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:43.575 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"btime":"2024-09-11T00:47:30:455013+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 127 2024-09-11T00:47:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b8c077fa0 msgr2=0x7f2b8c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b8c077fa0 0x7f2b8c07a460 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0c5160 tx=0x7f2b98009040 comp rx=0 tx=0).stop 2024-09-11T00:47:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bac0a6e20 msgr2=0x7f2bac0c0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bac0a6e20 0x7f2bac0c0fd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009950 tx=0x7f2ba0032c00 comp rx=0 tx=0).stop 2024-09-11T00:47:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3341460667 shutdown_connections 2024-09-11T00:47:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b8c077fa0 0x7f2b8c07a460 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bac0acbe0 0x7f2bac0745c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bac0ac2c0 0x7f2bac1575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 --2- 172.21.15.31:0/3341460667 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bac0a6e20 0x7f2bac0c0fd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3341460667 >> 172.21.15.31:0/3341460667 conn(0x7f2bac06caf0 msgr2=0x7f2bac0c6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3341460667 shutdown_connections 2024-09-11T00:47:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:43.575+0000 7f2bb2a00640 1 -- 172.21.15.31:0/3341460667 wait complete. 2024-09-11T00:47:43.594 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:47:45.596 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:45.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.780+0000 7f0eb4000640 1 -- 172.21.15.31:0/591807926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0eac072480 msgr2=0x7f0eac1511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:45.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.780+0000 7f0eb4000640 1 --2- 172.21.15.31:0/591807926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0eac072480 0x7f0eac1511c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009830 tx=0x7f0ea002a5a0 comp rx=0 tx=0).stop 2024-09-11T00:47:45.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.780+0000 7f0eb4000640 1 -- 172.21.15.31:0/591807926 shutdown_connections 2024-09-11T00:47:45.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.780+0000 7f0eb4000640 1 --2- 172.21.15.31:0/591807926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eac0c2410 0x7f0eac151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:45.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.780+0000 7f0eb4000640 1 --2- 172.21.15.31:0/591807926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0eac072480 0x7f0eac1511c0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:45.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.780+0000 7f0eb4000640 1 -- 172.21.15.31:0/591807926 >> 172.21.15.31:0/591807926 conn(0x7f0eac0a9750 msgr2=0x7f0eac0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:45.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.780+0000 7f0eb4000640 1 -- 172.21.15.31:0/591807926 shutdown_connections 2024-09-11T00:47:45.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.780+0000 7f0eb4000640 1 -- 172.21.15.31:0/591807926 wait complete. 2024-09-11T00:47:45.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.782+0000 7f0eb4000640 1 Processor -- start 2024-09-11T00:47:45.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.782+0000 7f0eb4000640 1 -- start start 2024-09-11T00:47:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.783+0000 7f0eb4000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0eac072480 0x7f0eac1527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.783+0000 7f0eb4000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eac074080 0x7f0eac0c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.783+0000 7f0eb4000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eac0c2410 0x7f0eac14e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.783+0000 7f0eb4000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac0729e0 con 0x7f0eac072480 2024-09-11T00:47:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.783+0000 7f0eb4000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac072860 con 0x7f0eac0c2410 2024-09-11T00:47:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.783+0000 7f0eb4000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0eac153f20 con 0x7f0eac074080 2024-09-11T00:47:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.783+0000 7f0eb1600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0eac072480 0x7f0eac1527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.783+0000 7f0eb1600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0eac072480 0x7f0eac1527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57486/0 (socket says 172.21.15.31:57486) 2024-09-11T00:47:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.783+0000 7f0eb1600640 1 -- 172.21.15.31:0/52248013 learned_addr learned my addr 172.21.15.31:0/52248013 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.784+0000 7f0eb0c00640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eac074080 0x7f0eac0c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.784+0000 7f0eb2000640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eac0c2410 0x7f0eac14e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.784+0000 7f0eb1600640 1 -- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eac074080 msgr2=0x7f0eac0c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.784+0000 7f0eb1600640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eac074080 0x7f0eac0c6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.784+0000 7f0eb1600640 1 -- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eac0c2410 msgr2=0x7f0eac14e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.784+0000 7f0eb1600640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eac0c2410 0x7f0eac14e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.784+0000 7f0eb1600640 1 -- 172.21.15.31:0/52248013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac1506e0 con 0x7f0eac072480 2024-09-11T00:47:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.784+0000 7f0eb1600640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0eac072480 0x7f0eac1527e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009950 tx=0x7f0ea0032c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.785+0000 7f0ea9600640 1 -- 172.21.15.31:0/52248013 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0040070 con 0x7f0eac072480 2024-09-11T00:47:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.785+0000 7f0ea9600640 1 -- 172.21.15.31:0/52248013 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ea0032e70 con 0x7f0eac072480 2024-09-11T00:47:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.785+0000 7f0eb4000640 1 -- 172.21.15.31:0/52248013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac1efd60 con 0x7f0eac072480 2024-09-11T00:47:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.785+0000 7f0ea9600640 1 -- 172.21.15.31:0/52248013 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0003d90 con 0x7f0eac072480 2024-09-11T00:47:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.785+0000 7f0eb4000640 1 -- 172.21.15.31:0/52248013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eac1f0040 con 0x7f0eac072480 2024-09-11T00:47:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.786+0000 7f0eb4000640 1 -- 172.21.15.31:0/52248013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eac1511c0 con 0x7f0eac072480 2024-09-11T00:47:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.788+0000 7f0ea9600640 1 -- 172.21.15.31:0/52248013 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0ea004d030 con 0x7f0eac072480 2024-09-11T00:47:45.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.789+0000 7f0ea9600640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0e8c077fa0 0x7f0e8c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.789+0000 7f0ea9600640 1 -- 172.21.15.31:0/52248013 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f0ea00bbe30 con 0x7f0eac072480 2024-09-11T00:47:45.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.789+0000 7f0eb0c00640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0e8c077fa0 0x7f0e8c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:45.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.790+0000 7f0eb0c00640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0e8c077fa0 0x7f0e8c07a460 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0e98005fd0 tx=0x7f0e98005c50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:45.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:45.794+0000 7f0ea9600640 1 -- 172.21.15.31:0/52248013 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0ea0083df0 con 0x7f0eac072480 2024-09-11T00:47:46.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.072+0000 7f0eb4000640 1 -- 172.21.15.31:0/52248013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0eac154d40 con 0x7f0eac072480 2024-09-11T00:47:46.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.072+0000 7f0ea9600640 1 -- 172.21.15.31:0/52248013 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6542 (secure 0 0 0) 0x7f0ea00836f0 con 0x7f0eac072480 2024-09-11T00:47:46.074 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:46.074 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"btime":"2024-09-11T00:47:30:455013+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:46.075 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 127 2024-09-11T00:47:46.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 -- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0e8c077fa0 msgr2=0x7f0e8c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:46.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0e8c077fa0 0x7f0e8c07a460 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0e98005fd0 tx=0x7f0e98005c50 comp rx=0 tx=0).stop 2024-09-11T00:47:46.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 -- 172.21.15.31:0/52248013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0eac072480 msgr2=0x7f0eac1527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:46.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0eac072480 0x7f0eac1527e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009950 tx=0x7f0ea0032c80 comp rx=0 tx=0).stop 2024-09-11T00:47:46.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 -- 172.21.15.31:0/52248013 shutdown_connections 2024-09-11T00:47:46.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0e8c077fa0 0x7f0e8c07a460 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:46.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0eac0c2410 0x7f0eac14e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:46.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0eac074080 0x7f0eac0c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:46.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 --2- 172.21.15.31:0/52248013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0eac072480 0x7f0eac1527e0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:46.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 -- 172.21.15.31:0/52248013 >> 172.21.15.31:0/52248013 conn(0x7f0eac0a9750 msgr2=0x7f0eac14c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:46.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 -- 172.21.15.31:0/52248013 shutdown_connections 2024-09-11T00:47:46.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:46.075+0000 7f0eb4000640 1 -- 172.21.15.31:0/52248013 wait complete. 2024-09-11T00:47:46.095 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:47:48.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:48.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.264+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2738192235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7b00c23a0 msgr2=0x7fc7b00c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:48.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.264+0000 7fc7b7e00640 1 --2- 172.21.15.31:0/2738192235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7b00c23a0 0x7fc7b00c30c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc7b00c5500 tx=0x7fc79c02a5d0 comp rx=0 tx=0).stop 2024-09-11T00:47:48.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.264+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2738192235 shutdown_connections 2024-09-11T00:47:48.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.264+0000 7fc7b7e00640 1 --2- 172.21.15.31:0/2738192235 >> v2:172.21.15.204:3300/0 conn(0x7fc7b01518b0 0x7fc7b00c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:48.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.264+0000 7fc7b7e00640 1 --2- 172.21.15.31:0/2738192235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7b00c23a0 0x7fc7b00c30c0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:48.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.264+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2738192235 >> 172.21.15.31:0/2738192235 conn(0x7fc7b00a9750 msgr2=0x7fc7b00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:48.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.264+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2738192235 shutdown_connections 2024-09-11T00:47:48.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.264+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2738192235 wait complete. 2024-09-11T00:47:48.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.266+0000 7fc7b7e00640 1 Processor -- start 2024-09-11T00:47:48.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.266+0000 7fc7b7e00640 1 -- start start 2024-09-11T00:47:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7b0072480 0x7fc7b01f01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc7b00c23a0 0x7fc7b0147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b01518b0 0x7fc7b006e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0074200 con 0x7fc7b0072480 2024-09-11T00:47:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b7e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0074080 con 0x7fc7b01518b0 2024-09-11T00:47:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b7e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b0072860 con 0x7fc7b00c23a0 2024-09-11T00:47:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7b0072480 0x7fc7b01f01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7b0072480 0x7fc7b01f01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40210/0 (socket says 172.21.15.31:40210) 2024-09-11T00:47:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b5400640 1 -- 172.21.15.31:0/2612909675 learned_addr learned my addr 172.21.15.31:0/2612909675 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:48.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b4a00640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc7b00c23a0 0x7fc7b0147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:48.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.267+0000 7fc7b5e00640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b01518b0 0x7fc7b006e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.268+0000 7fc7b5400640 1 -- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc7b00c23a0 msgr2=0x7fc7b0147170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.268+0000 7fc7b5400640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc7b00c23a0 0x7fc7b0147170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.268+0000 7fc7b5400640 1 -- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b01518b0 msgr2=0x7fc7b006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.268+0000 7fc7b5400640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b01518b0 0x7fc7b006e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.268+0000 7fc7b5400640 1 -- 172.21.15.31:0/2612909675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b00c3130 con 0x7fc7b0072480 2024-09-11T00:47:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.268+0000 7fc7b5400640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7b0072480 0x7fc7b01f01f0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400cbd0 tx=0x7fc7a400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.268+0000 7fc7ad600640 1 -- 172.21.15.31:0/2612909675 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4016020 con 0x7fc7b0072480 2024-09-11T00:47:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.268+0000 7fc7ad600640 1 -- 172.21.15.31:0/2612909675 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7a400ae30 con 0x7fc7b0072480 2024-09-11T00:47:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.268+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2612909675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b01dd620 con 0x7fc7b0072480 2024-09-11T00:47:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.268+0000 7fc7ad600640 1 -- 172.21.15.31:0/2612909675 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4012660 con 0x7fc7b0072480 2024-09-11T00:47:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.269+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2612909675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b01476b0 con 0x7fc7b0072480 2024-09-11T00:47:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.270+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2612909675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b0153f20 con 0x7fc7b0072480 2024-09-11T00:47:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.271+0000 7fc7ad600640 1 -- 172.21.15.31:0/2612909675 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc7a401b400 con 0x7fc7b0072480 2024-09-11T00:47:48.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.272+0000 7fc7ad600640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc790077fa0 0x7fc79007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:48.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.272+0000 7fc7ad600640 1 -- 172.21.15.31:0/2612909675 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7fc7a409b380 con 0x7fc7b0072480 2024-09-11T00:47:48.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.273+0000 7fc7b4a00640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc790077fa0 0x7fc79007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.274+0000 7fc7b4a00640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc790077fa0 0x7fc79007a460 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc79c034000 tx=0x7fc79c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.277+0000 7fc7ad600640 1 -- 172.21.15.31:0/2612909675 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc7a4020020 con 0x7fc7b0072480 2024-09-11T00:47:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.560+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2612909675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7b00a9530 con 0x7fc7b0072480 2024-09-11T00:47:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.560+0000 7fc7ad600640 1 -- 172.21.15.31:0/2612909675 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6542 (secure 0 0 0) 0x7fc7a4062c90 con 0x7fc7b0072480 2024-09-11T00:47:48.562 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:48.562 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"btime":"2024-09-11T00:47:30:455013+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 127 2024-09-11T00:47:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc790077fa0 msgr2=0x7fc79007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc790077fa0 0x7fc79007a460 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc79c034000 tx=0x7fc79c004730 comp rx=0 tx=0).stop 2024-09-11T00:47:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2612909675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7b0072480 msgr2=0x7fc7b01f01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7b0072480 0x7fc7b01f01f0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400cbd0 tx=0x7fc7a400a430 comp rx=0 tx=0).stop 2024-09-11T00:47:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2612909675 shutdown_connections 2024-09-11T00:47:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc790077fa0 0x7fc79007a460 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc7b01518b0 0x7fc7b006e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc7b00c23a0 0x7fc7b0147170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 --2- 172.21.15.31:0/2612909675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7b0072480 0x7fc7b01f01f0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2612909675 >> 172.21.15.31:0/2612909675 conn(0x7fc7b00a9750 msgr2=0x7fc7b014e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2612909675 shutdown_connections 2024-09-11T00:47:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:48.563+0000 7fc7b7e00640 1 -- 172.21.15.31:0/2612909675 wait complete. 2024-09-11T00:47:48.581 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:47:48.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:47:48.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:47:50.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:50.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.761+0000 7f7a9e000640 1 -- 172.21.15.31:0/2516653855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a980bf240 msgr2=0x7f7a980b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:50.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.761+0000 7f7a9e000640 1 --2- 172.21.15.31:0/2516653855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a980bf240 0x7f7a980b0be0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7a840098e0 tx=0x7f7a8402a540 comp rx=0 tx=0).stop 2024-09-11T00:47:50.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.761+0000 7f7a9e000640 1 -- 172.21.15.31:0/2516653855 shutdown_connections 2024-09-11T00:47:50.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.761+0000 7f7a9e000640 1 --2- 172.21.15.31:0/2516653855 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a98156e40 0x7f7a980b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:50.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.761+0000 7f7a9e000640 1 --2- 172.21.15.31:0/2516653855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a980bf240 0x7f7a980b0be0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:50.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.761+0000 7f7a9e000640 1 -- 172.21.15.31:0/2516653855 >> 172.21.15.31:0/2516653855 conn(0x7f7a98068ac0 msgr2=0x7f7a98068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:50.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.761+0000 7f7a9e000640 1 -- 172.21.15.31:0/2516653855 shutdown_connections 2024-09-11T00:47:50.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.761+0000 7f7a9e000640 1 -- 172.21.15.31:0/2516653855 wait complete. 2024-09-11T00:47:50.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.763+0000 7f7a9e000640 1 Processor -- start 2024-09-11T00:47:50.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.763+0000 7f7a9e000640 1 -- start start 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.764+0000 7f7a9e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a980bf240 0x7f7a98148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.764+0000 7f7a9e000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a980c6730 0x7f7a981e0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.764+0000 7f7a9e000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a98156e40 0x7f7a980adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.764+0000 7f7a9e000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7a98159de0 con 0x7f7a98156e40 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.764+0000 7f7a9e000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7a981563c0 con 0x7f7a980bf240 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.764+0000 7f7a9e000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7a98159f60 con 0x7f7a980c6730 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.765+0000 7f7a9d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a98156e40 0x7f7a980adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.765+0000 7f7a9d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a98156e40 0x7f7a980adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40216/0 (socket says 172.21.15.31:40216) 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.765+0000 7f7a9d600640 1 -- 172.21.15.31:0/1345543685 learned_addr learned my addr 172.21.15.31:0/1345543685 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.765+0000 7f7a9cc00640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a980bf240 0x7f7a98148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:50.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.765+0000 7f7a97e00640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a980c6730 0x7f7a981e0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:50.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.765+0000 7f7a9d600640 1 -- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a980c6730 msgr2=0x7f7a981e0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:50.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.765+0000 7f7a9d600640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a980c6730 0x7f7a981e0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:50.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.765+0000 7f7a9d600640 1 -- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a980bf240 msgr2=0x7f7a98148f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:50.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.765+0000 7f7a9d600640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a980bf240 0x7f7a98148f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:50.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.765+0000 7f7a9d600640 1 -- 172.21.15.31:0/1345543685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a981eca80 con 0x7f7a98156e40 2024-09-11T00:47:50.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.766+0000 7f7a9d600640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a98156e40 0x7f7a980adb20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000e990 tx=0x7f7a9000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:50.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.766+0000 7f7a94c00640 1 -- 172.21.15.31:0/1345543685 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a90004d20 con 0x7f7a98156e40 2024-09-11T00:47:50.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.766+0000 7f7a94c00640 1 -- 172.21.15.31:0/1345543685 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a9000cb00 con 0x7f7a98156e40 2024-09-11T00:47:50.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.766+0000 7f7a94c00640 1 -- 172.21.15.31:0/1345543685 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a90005020 con 0x7f7a98156e40 2024-09-11T00:47:50.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.766+0000 7f7a9e000640 1 -- 172.21.15.31:0/1345543685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a981ec520 con 0x7f7a98156e40 2024-09-11T00:47:50.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.766+0000 7f7a9e000640 1 -- 172.21.15.31:0/1345543685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a981ec7a0 con 0x7f7a98156e40 2024-09-11T00:47:50.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.767+0000 7f7a9e000640 1 -- 172.21.15.31:0/1345543685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a9806e850 con 0x7f7a98156e40 2024-09-11T00:47:50.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.769+0000 7f7a94c00640 1 -- 172.21.15.31:0/1345543685 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7a9000f040 con 0x7f7a98156e40 2024-09-11T00:47:50.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.770+0000 7f7a94c00640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7a78077f70 0x7f7a7807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:50.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.770+0000 7f7a94c00640 1 -- 172.21.15.31:0/1345543685 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f7a9009aec0 con 0x7f7a98156e40 2024-09-11T00:47:50.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.771+0000 7f7a9cc00640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7a78077f70 0x7f7a7807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:50.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.772+0000 7f7a9cc00640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7a78077f70 0x7f7a7807a430 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c007fd0 tx=0x7f7a8c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:50.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:50.775+0000 7f7a94c00640 1 -- 172.21.15.31:0/1345543685 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7a981ec520 con 0x7f7a98156e40 2024-09-11T00:47:51.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.058+0000 7f7a9e000640 1 -- 172.21.15.31:0/1345543685 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a980b0310 con 0x7f7a98156e40 2024-09-11T00:47:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.058+0000 7f7a94c00640 1 -- 172.21.15.31:0/1345543685 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6542 (secure 0 0 0) 0x7f7a90016030 con 0x7f7a98156e40 2024-09-11T00:47:51.060 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:51.060 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"btime":"2024-09-11T00:47:30:455013+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:51.060 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 127 2024-09-11T00:47:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 -- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7a78077f70 msgr2=0x7f7a7807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7a78077f70 0x7f7a7807a430 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c007fd0 tx=0x7f7a8c009040 comp rx=0 tx=0).stop 2024-09-11T00:47:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 -- 172.21.15.31:0/1345543685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a98156e40 msgr2=0x7f7a980adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a98156e40 0x7f7a980adb20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000e990 tx=0x7f7a9000ee50 comp rx=0 tx=0).stop 2024-09-11T00:47:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 -- 172.21.15.31:0/1345543685 shutdown_connections 2024-09-11T00:47:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7a78077f70 0x7f7a7807a430 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a98156e40 0x7f7a980adb20 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7a980c6730 0x7f7a981e0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 --2- 172.21.15.31:0/1345543685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a980bf240 0x7f7a98148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:51.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 -- 172.21.15.31:0/1345543685 >> 172.21.15.31:0/1345543685 conn(0x7f7a98068ac0 msgr2=0x7f7a98152950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:51.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 -- 172.21.15.31:0/1345543685 shutdown_connections 2024-09-11T00:47:51.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:51.061+0000 7f7a9e000640 1 -- 172.21.15.31:0/1345543685 wait complete. 2024-09-11T00:47:51.082 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:47:53.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:53.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.256+0000 7f2381200640 1 -- 172.21.15.31:0/2551069557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f237c14c8b0 msgr2=0x7f237c0c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:53.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.256+0000 7f2381200640 1 --2- 172.21.15.31:0/2551069557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f237c14c8b0 0x7f237c0c6460 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f236c009830 tx=0x7f236c02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:47:53.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.256+0000 7f2381200640 1 -- 172.21.15.31:0/2551069557 shutdown_connections 2024-09-11T00:47:53.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.256+0000 7f2381200640 1 --2- 172.21.15.31:0/2551069557 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f237c06f5c0 0x7f237c06e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:53.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.256+0000 7f2381200640 1 --2- 172.21.15.31:0/2551069557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f237c14c8b0 0x7f237c0c6460 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:53.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.256+0000 7f2381200640 1 -- 172.21.15.31:0/2551069557 >> 172.21.15.31:0/2551069557 conn(0x7f237c0cbf00 msgr2=0x7f237c0cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:53.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.256+0000 7f2381200640 1 -- 172.21.15.31:0/2551069557 shutdown_connections 2024-09-11T00:47:53.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.256+0000 7f2381200640 1 -- 172.21.15.31:0/2551069557 wait complete. 2024-09-11T00:47:53.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.258+0000 7f2381200640 1 Processor -- start 2024-09-11T00:47:53.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.258+0000 7f2381200640 1 -- start start 2024-09-11T00:47:53.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f2381200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f237c06f5c0 0x7f237c153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:53.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f2381200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f237c0c99e0 0x7f237c1efb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:53.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f2381200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f237c14c8b0 0x7f237c1f0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:53.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f2381200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f237c14ce10 con 0x7f237c06f5c0 2024-09-11T00:47:53.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f2381200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f237c14cc90 con 0x7f237c0c99e0 2024-09-11T00:47:53.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f2381200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f237c1569a0 con 0x7f237c14c8b0 2024-09-11T00:47:53.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f237b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f237c06f5c0 0x7f237c153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:53.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f237b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f237c06f5c0 0x7f237c153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40226/0 (socket says 172.21.15.31:40226) 2024-09-11T00:47:53.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f237b400640 1 -- 172.21.15.31:0/482852786 learned_addr learned my addr 172.21.15.31:0/482852786 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:53.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f237aa00640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f237c0c99e0 0x7f237c1efb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:53.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.259+0000 7f237be00640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f237c14c8b0 0x7f237c1f0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.260+0000 7f237b400640 1 -- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f237c14c8b0 msgr2=0x7f237c1f0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.260+0000 7f237b400640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f237c14c8b0 0x7f237c1f0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.260+0000 7f237b400640 1 -- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f237c0c99e0 msgr2=0x7f237c1efb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.260+0000 7f237b400640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f237c0c99e0 0x7f237c1efb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.260+0000 7f237b400640 1 -- 172.21.15.31:0/482852786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237c159d50 con 0x7f237c06f5c0 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.260+0000 7f237aa00640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f237c0c99e0 0x7f237c1efb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.260+0000 7f237b400640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f237c06f5c0 0x7f237c153260 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f236c009950 tx=0x7f236c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.260+0000 7f2373400640 1 -- 172.21.15.31:0/482852786 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c005c90 con 0x7f237c06f5c0 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.260+0000 7f2373400640 1 -- 172.21.15.31:0/482852786 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f236c031e50 con 0x7f237c06f5c0 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.260+0000 7f2373400640 1 -- 172.21.15.31:0/482852786 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c005260 con 0x7f237c06f5c0 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.261+0000 7f2381200640 1 -- 172.21.15.31:0/482852786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f237c14f850 con 0x7f237c06f5c0 2024-09-11T00:47:53.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.261+0000 7f2381200640 1 -- 172.21.15.31:0/482852786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f237c14fe10 con 0x7f237c06f5c0 2024-09-11T00:47:53.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.262+0000 7f2381200640 1 -- 172.21.15.31:0/482852786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f237c0c6460 con 0x7f237c06f5c0 2024-09-11T00:47:53.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.263+0000 7f2373400640 1 -- 172.21.15.31:0/482852786 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f236c005400 con 0x7f237c06f5c0 2024-09-11T00:47:53.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.264+0000 7f2373400640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f235c077f70 0x7f235c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:53.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.264+0000 7f2373400640 1 -- 172.21.15.31:0/482852786 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f236c0bb8e0 con 0x7f237c06f5c0 2024-09-11T00:47:53.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.265+0000 7f237aa00640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f235c077f70 0x7f235c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:53.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.265+0000 7f237aa00640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f235c077f70 0x7f235c07a430 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2368007f20 tx=0x7f2368007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:53.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.272+0000 7f2373400640 1 -- 172.21.15.31:0/482852786 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f236c03f030 con 0x7f237c06f5c0 2024-09-11T00:47:53.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.576+0000 7f2381200640 1 -- 172.21.15.31:0/482852786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f237c06f3a0 con 0x7f237c06f5c0 2024-09-11T00:47:53.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.577+0000 7f2373400640 1 -- 172.21.15.31:0/482852786 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6542 (secure 0 0 0) 0x7f236c083160 con 0x7f237c06f5c0 2024-09-11T00:47:53.578 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:53.579 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"btime":"2024-09-11T00:47:30:455013+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:53.579 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 127 2024-09-11T00:47:53.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.579+0000 7f2381200640 1 -- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f235c077f70 msgr2=0x7f235c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:53.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.579+0000 7f2381200640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f235c077f70 0x7f235c07a430 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2368007f20 tx=0x7f2368007eb0 comp rx=0 tx=0).stop 2024-09-11T00:47:53.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.579+0000 7f2381200640 1 -- 172.21.15.31:0/482852786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f237c06f5c0 msgr2=0x7f237c153260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:53.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.579+0000 7f2381200640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f237c06f5c0 0x7f237c153260 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f236c009950 tx=0x7f236c033c20 comp rx=0 tx=0).stop 2024-09-11T00:47:53.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.579+0000 7f2381200640 1 -- 172.21.15.31:0/482852786 shutdown_connections 2024-09-11T00:47:53.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.579+0000 7f2381200640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f235c077f70 0x7f235c07a430 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:53.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.579+0000 7f2381200640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f237c14c8b0 0x7f237c1f0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:53.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.579+0000 7f2381200640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f237c0c99e0 0x7f237c1efb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:53.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.579+0000 7f2381200640 1 --2- 172.21.15.31:0/482852786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f237c06f5c0 0x7f237c153260 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:53.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.579+0000 7f2381200640 1 -- 172.21.15.31:0/482852786 >> 172.21.15.31:0/482852786 conn(0x7f237c0cbf00 msgr2=0x7f237c1551e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:53.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.580+0000 7f2381200640 1 -- 172.21.15.31:0/482852786 shutdown_connections 2024-09-11T00:47:53.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:53.580+0000 7f2381200640 1 -- 172.21.15.31:0/482852786 wait complete. 2024-09-11T00:47:53.598 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:47:55.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:55.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.787+0000 7fedad800640 1 -- 172.21.15.31:0/2090967874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feda80bf240 msgr2=0x7feda80b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:55.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.787+0000 7fedad800640 1 --2- 172.21.15.31:0/2090967874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feda80bf240 0x7feda80b0be0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fed900098e0 tx=0x7fed9002a540 comp rx=0 tx=0).stop 2024-09-11T00:47:55.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.787+0000 7fedad800640 1 -- 172.21.15.31:0/2090967874 shutdown_connections 2024-09-11T00:47:55.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.787+0000 7fedad800640 1 --2- 172.21.15.31:0/2090967874 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feda8156e40 0x7feda80b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:55.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.787+0000 7fedad800640 1 --2- 172.21.15.31:0/2090967874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feda80bf240 0x7feda80b0be0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:55.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.787+0000 7fedad800640 1 -- 172.21.15.31:0/2090967874 >> 172.21.15.31:0/2090967874 conn(0x7feda8068ac0 msgr2=0x7feda8068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:55.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.787+0000 7fedad800640 1 -- 172.21.15.31:0/2090967874 shutdown_connections 2024-09-11T00:47:55.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.787+0000 7fedad800640 1 -- 172.21.15.31:0/2090967874 wait complete. 2024-09-11T00:47:55.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.789+0000 7fedad800640 1 Processor -- start 2024-09-11T00:47:55.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.789+0000 7fedad800640 1 -- start start 2024-09-11T00:47:55.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.790+0000 7fedad800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feda80bf240 0x7feda80ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:55.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.790+0000 7fedad800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda80c6730 0x7feda8147130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:55.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.790+0000 7fedad800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feda8156e40 0x7feda81574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.790+0000 7fedad800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feda8159de0 con 0x7feda80bf240 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.790+0000 7fedad800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feda8156640 con 0x7feda80c6730 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.790+0000 7fedad800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7feda8159f60 con 0x7feda8156e40 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.790+0000 7feda7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feda80bf240 0x7feda80ad6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.790+0000 7feda7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feda80bf240 0x7feda80ad6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40244/0 (socket says 172.21.15.31:40244) 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.790+0000 7feda7e00640 1 -- 172.21.15.31:0/3099851286 learned_addr learned my addr 172.21.15.31:0/3099851286 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7fedace00640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feda8156e40 0x7feda81574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7fed9fe00640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda80c6730 0x7feda8147130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7feda7e00640 1 -- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feda8156e40 msgr2=0x7feda81574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7feda7e00640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feda8156e40 0x7feda81574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7feda7e00640 1 -- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda80c6730 msgr2=0x7feda8147130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:55.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7feda7e00640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda80c6730 0x7feda8147130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:55.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7feda7e00640 1 -- 172.21.15.31:0/3099851286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda81ece30 con 0x7feda80bf240 2024-09-11T00:47:55.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7fed9fe00640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda80c6730 0x7feda8147130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:47:55.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7feda7e00640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feda80bf240 0x7feda80ad6d0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fed9800c450 tx=0x7fed9800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:55.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7feda4c00640 1 -- 172.21.15.31:0/3099851286 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed98017070 con 0x7feda80bf240 2024-09-11T00:47:55.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7feda4c00640 1 -- 172.21.15.31:0/3099851286 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed980053c0 con 0x7feda80bf240 2024-09-11T00:47:55.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.791+0000 7feda4c00640 1 -- 172.21.15.31:0/3099851286 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed98003850 con 0x7feda80bf240 2024-09-11T00:47:55.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.792+0000 7fedad800640 1 -- 172.21.15.31:0/3099851286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda81df870 con 0x7feda80bf240 2024-09-11T00:47:55.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.792+0000 7fedad800640 1 -- 172.21.15.31:0/3099851286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda8147670 con 0x7feda80bf240 2024-09-11T00:47:55.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.793+0000 7fedad800640 1 -- 172.21.15.31:0/3099851286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feda806e850 con 0x7feda80bf240 2024-09-11T00:47:55.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.795+0000 7feda4c00640 1 -- 172.21.15.31:0/3099851286 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fed980039f0 con 0x7feda80bf240 2024-09-11T00:47:55.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.796+0000 7feda4c00640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed84077f70 0x7fed8407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:55.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.796+0000 7feda4c00640 1 -- 172.21.15.31:0/3099851286 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7fed98099eb0 con 0x7feda80bf240 2024-09-11T00:47:55.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.796+0000 7fed9fe00640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed84077f70 0x7fed8407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:55.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.797+0000 7fed9fe00640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed84077f70 0x7fed8407a430 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fed90034000 tx=0x7fed90004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:55.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:55.802+0000 7feda4c00640 1 -- 172.21.15.31:0/3099851286 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fed98061820 con 0x7feda80bf240 2024-09-11T00:47:56.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.070+0000 7fedad800640 1 -- 172.21.15.31:0/3099851286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feda80af7f0 con 0x7feda80bf240 2024-09-11T00:47:56.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.071+0000 7feda4c00640 1 -- 172.21.15.31:0/3099851286 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6542 (secure 0 0 0) 0x7fed98066250 con 0x7feda80bf240 2024-09-11T00:47:56.074 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:56.074 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"btime":"2024-09-11T00:47:30:455013+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:56.075 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 127 2024-09-11T00:47:56.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 -- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed84077f70 msgr2=0x7fed8407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:56.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed84077f70 0x7fed8407a430 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fed90034000 tx=0x7fed90004730 comp rx=0 tx=0).stop 2024-09-11T00:47:56.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 -- 172.21.15.31:0/3099851286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feda80bf240 msgr2=0x7feda80ad6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:56.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feda80bf240 0x7feda80ad6d0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fed9800c450 tx=0x7fed9800c910 comp rx=0 tx=0).stop 2024-09-11T00:47:56.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 -- 172.21.15.31:0/3099851286 shutdown_connections 2024-09-11T00:47:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed84077f70 0x7fed8407a430 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feda8156e40 0x7feda81574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda80c6730 0x7feda8147130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 --2- 172.21.15.31:0/3099851286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feda80bf240 0x7feda80ad6d0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 -- 172.21.15.31:0/3099851286 >> 172.21.15.31:0/3099851286 conn(0x7feda8068ac0 msgr2=0x7feda81528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 -- 172.21.15.31:0/3099851286 shutdown_connections 2024-09-11T00:47:56.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:56.074+0000 7fedad800640 1 -- 172.21.15.31:0/3099851286 wait complete. 2024-09-11T00:47:56.094 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:47:58.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:47:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.264+0000 7f25c7200640 1 -- 172.21.15.31:0/1176664166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25c8150020 msgr2=0x7f25c80ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.264+0000 7f25c7200640 1 --2- 172.21.15.31:0/1176664166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25c8150020 0x7f25c80ad4e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f25c806ed70 tx=0x7f25b802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:47:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.264+0000 7f25c7200640 1 -- 172.21.15.31:0/1176664166 shutdown_connections 2024-09-11T00:47:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.264+0000 7f25c7200640 1 --2- 172.21.15.31:0/1176664166 >> v2:172.21.15.204:3300/0 conn(0x7f25c806e850 0x7f25c8160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.264+0000 7f25c7200640 1 --2- 172.21.15.31:0/1176664166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25c8150020 0x7f25c80ad4e0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.264+0000 7f25c7200640 1 -- 172.21.15.31:0/1176664166 >> 172.21.15.31:0/1176664166 conn(0x7f25c8068f10 msgr2=0x7f25c80cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.264+0000 7f25c7200640 1 -- 172.21.15.31:0/1176664166 shutdown_connections 2024-09-11T00:47:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.264+0000 7f25c7200640 1 -- 172.21.15.31:0/1176664166 wait complete. 2024-09-11T00:47:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.266+0000 7f25c7200640 1 Processor -- start 2024-09-11T00:47:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.266+0000 7f25c7200640 1 -- start start 2024-09-11T00:47:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.267+0000 7f25c7200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25c806e850 0x7f25c81ef070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.267+0000 7f25c7200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25c80ada20 0x7f25c81df680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.267+0000 7f25c7200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25c8150020 0x7f25c81dfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.267+0000 7f25c7200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f25c8157550 con 0x7f25c80ada20 2024-09-11T00:47:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.267+0000 7f25c7200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f25c81573d0 con 0x7f25c8150020 2024-09-11T00:47:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.267+0000 7f25c7200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f25c8156960 con 0x7f25c806e850 2024-09-11T00:47:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.267+0000 7f25c5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25c80ada20 0x7f25c81df680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.267+0000 7f25c5400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25c80ada20 0x7f25c81df680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56532/0 (socket says 172.21.15.31:56532) 2024-09-11T00:47:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.267+0000 7f25c5400640 1 -- 172.21.15.31:0/4244593369 learned_addr learned my addr 172.21.15.31:0/4244593369 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:47:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.268+0000 7f25c5e00640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25c806e850 0x7f25c81ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.268+0000 7f25c6800640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25c8150020 0x7f25c81dfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.268+0000 7f25c5400640 1 -- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25c806e850 msgr2=0x7f25c81ef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.268+0000 7f25c5400640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25c806e850 0x7f25c81ef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.268+0000 7f25c5400640 1 -- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25c8150020 msgr2=0x7f25c81dfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.268+0000 7f25c5400640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25c8150020 0x7f25c81dfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.268+0000 7f25c5400640 1 -- 172.21.15.31:0/4244593369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c81e1fe0 con 0x7f25c80ada20 2024-09-11T00:47:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.268+0000 7f25c5400640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25c80ada20 0x7f25c81df680 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f25b400c420 tx=0x7f25b400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.268+0000 7f25be000640 1 -- 172.21.15.31:0/4244593369 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b40052d0 con 0x7f25c80ada20 2024-09-11T00:47:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.269+0000 7f25be000640 1 -- 172.21.15.31:0/4244593369 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25b4007020 con 0x7f25c80ada20 2024-09-11T00:47:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.269+0000 7f25be000640 1 -- 172.21.15.31:0/4244593369 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4007470 con 0x7f25c80ada20 2024-09-11T00:47:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.269+0000 7f25c7200640 1 -- 172.21.15.31:0/4244593369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c80ae680 con 0x7f25c80ada20 2024-09-11T00:47:58.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.269+0000 7f25c7200640 1 -- 172.21.15.31:0/4244593369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c80ae920 con 0x7f25c80ada20 2024-09-11T00:47:58.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.270+0000 7f25c7200640 1 -- 172.21.15.31:0/4244593369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25c80cc220 con 0x7f25c80ada20 2024-09-11T00:47:58.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.271+0000 7f25be000640 1 -- 172.21.15.31:0/4244593369 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f25b40067d0 con 0x7f25c80ada20 2024-09-11T00:47:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.272+0000 7f25be000640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f25a8077f70 0x7f25a807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:47:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.272+0000 7f25be000640 1 -- 172.21.15.31:0/4244593369 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8644+0+0 (secure 0 0 0) 0x7f25b4099dc0 con 0x7f25c80ada20 2024-09-11T00:47:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.273+0000 7f25c5e00640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f25a8077f70 0x7f25a807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:47:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.274+0000 7f25c5e00640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f25a8077f70 0x7f25a807a430 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f25b8033000 tx=0x7f25b8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:47:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.277+0000 7f25be000640 1 -- 172.21.15.31:0/4244593369 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f25b40616d0 con 0x7f25c80ada20 2024-09-11T00:47:58.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.557+0000 7f25c7200640 1 -- 172.21.15.31:0/4244593369 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25c80ace10 con 0x7f25c80ada20 2024-09-11T00:47:58.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.558+0000 7f25be000640 1 -- 172.21.15.31:0/4244593369 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6542 (secure 0 0 0) 0x7f25b4066100 con 0x7f25c80ada20 2024-09-11T00:47:58.560 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:47:58.561 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":127,"btime":"2024-09-11T00:47:30:455013+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6829,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:46:02.442423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5957":{"gid":5957,"name":"b","rank":0,"incarnation":118,"state":"up:active","state_seq":10,"addr":"172.21.15.204:6837/1082799504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1082799504},{"type":"v1","addr":"172.21.15.204:6837","nonce":1082799504}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5957,"qdb_cluster":[5957]},"id":2}]} 2024-09-11T00:47:58.561 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 127 2024-09-11T00:47:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.560+0000 7f25c7200640 1 -- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f25a8077f70 msgr2=0x7f25a807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.560+0000 7f25c7200640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f25a8077f70 0x7f25a807a430 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f25b8033000 tx=0x7f25b8004730 comp rx=0 tx=0).stop 2024-09-11T00:47:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.560+0000 7f25c7200640 1 -- 172.21.15.31:0/4244593369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25c80ada20 msgr2=0x7f25c81df680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:47:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.560+0000 7f25c7200640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25c80ada20 0x7f25c81df680 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f25b400c420 tx=0x7f25b400c8e0 comp rx=0 tx=0).stop 2024-09-11T00:47:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.561+0000 7f25c7200640 1 -- 172.21.15.31:0/4244593369 shutdown_connections 2024-09-11T00:47:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.561+0000 7f25c7200640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f25a8077f70 0x7f25a807a430 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.561+0000 7f25c7200640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25c8150020 0x7f25c81dfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.561+0000 7f25c7200640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25c80ada20 0x7f25c81df680 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.561+0000 7f25c7200640 1 --2- 172.21.15.31:0/4244593369 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f25c806e850 0x7f25c81ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:47:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.561+0000 7f25c7200640 1 -- 172.21.15.31:0/4244593369 >> 172.21.15.31:0/4244593369 conn(0x7f25c8068f10 msgr2=0x7f25c81583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:47:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.561+0000 7f25c7200640 1 -- 172.21.15.31:0/4244593369 shutdown_connections 2024-09-11T00:47:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:47:58.561+0000 7f25c7200640 1 -- 172.21.15.31:0/4244593369 wait complete. 2024-09-11T00:47:58.584 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:47:59.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.042+0000 7f3332600640 1 -- 172.21.15.31:0/423099201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f332c071100 msgr2=0x7f332c0bd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.042+0000 7f3332600640 1 --2- 172.21.15.31:0/423099201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f332c071100 0x7f332c0bd240 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3318009860 tx=0x7f331802a4d0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.042+0000 7f3332600640 1 -- 172.21.15.31:0/423099201 shutdown_connections 2024-09-11T00:48:00.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.042+0000 7f3332600640 1 --2- 172.21.15.31:0/423099201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f332c071100 0x7f332c0bd240 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.042+0000 7f3332600640 1 -- 172.21.15.31:0/423099201 >> 172.21.15.31:0/423099201 conn(0x7f332c0c4d20 msgr2=0x7f332c0c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:00.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.043+0000 7f3332600640 1 -- 172.21.15.31:0/423099201 shutdown_connections 2024-09-11T00:48:00.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.043+0000 7f3332600640 1 -- 172.21.15.31:0/423099201 wait complete. 2024-09-11T00:48:00.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.044+0000 7f3332600640 1 Processor -- start 2024-09-11T00:48:00.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.044+0000 7f3332600640 1 -- start start 2024-09-11T00:48:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.045+0000 7f3332600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f332c06fda0 0x7f332c149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.045+0000 7f3332600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f332c071100 0x7f332c1e1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.045+0000 7f3332600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f332c14cb80 0x7f332c1556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.045+0000 7f3332600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f332c06e390 con 0x7f332c071100 2024-09-11T00:48:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.045+0000 7f3332600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f332c06e210 con 0x7f332c14cb80 2024-09-11T00:48:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.045+0000 7f3332600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f332c14cf60 con 0x7f332c06fda0 2024-09-11T00:48:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.046+0000 7f332be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f332c071100 0x7f332c1e1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.046+0000 7f332be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f332c071100 0x7f332c1e1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56542/0 (socket says 172.21.15.31:56542) 2024-09-11T00:48:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.046+0000 7f332be00640 1 -- 172.21.15.31:0/3011913743 learned_addr learned my addr 172.21.15.31:0/3011913743 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:00.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.046+0000 7f3331c00640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f332c14cb80 0x7f332c1556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.046+0000 7f3331200640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f332c06fda0 0x7f332c149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.046+0000 7f332be00640 1 -- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f332c06fda0 msgr2=0x7f332c149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.046+0000 7f332be00640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f332c06fda0 0x7f332c149f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.046+0000 7f332be00640 1 -- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f332c14cb80 msgr2=0x7f332c1556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.046+0000 7f332be00640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f332c14cb80 0x7f332c1556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.046+0000 7f332be00640 1 -- 172.21.15.31:0/3011913743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332c14a450 con 0x7f332c071100 2024-09-11T00:48:00.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.047+0000 7f332be00640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f332c071100 0x7f332c1e1540 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f331802aac0 tx=0x7f3318032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:00.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.047+0000 7f3328c00640 1 -- 172.21.15.31:0/3011913743 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3318040070 con 0x7f332c071100 2024-09-11T00:48:00.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.047+0000 7f3328c00640 1 -- 172.21.15.31:0/3011913743 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3318005310 con 0x7f332c071100 2024-09-11T00:48:00.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.047+0000 7f3328c00640 1 -- 172.21.15.31:0/3011913743 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3318005c90 con 0x7f332c071100 2024-09-11T00:48:00.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.047+0000 7f3332600640 1 -- 172.21.15.31:0/3011913743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332c1df0d0 con 0x7f332c071100 2024-09-11T00:48:00.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.047+0000 7f3332600640 1 -- 172.21.15.31:0/3011913743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f332c1563e0 con 0x7f332c071100 2024-09-11T00:48:00.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.048+0000 7f3332600640 1 -- 172.21.15.31:0/3011913743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f332c0702f0 con 0x7f332c071100 2024-09-11T00:48:00.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.049+0000 7f3328c00640 1 -- 172.21.15.31:0/3011913743 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f331804c030 con 0x7f332c071100 2024-09-11T00:48:00.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.050+0000 7f3328c00640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f330c077f70 0x7f330c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.050+0000 7f3328c00640 1 -- 172.21.15.31:0/3011913743 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7f33180bb7e0 con 0x7f332c071100 2024-09-11T00:48:00.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.051+0000 7f3331200640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f330c077f70 0x7f330c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.052+0000 7f3331200640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f330c077f70 0x7f330c07a430 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3320007f30 tx=0x7f3320007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:00.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.055+0000 7f3328c00640 1 -- 172.21.15.31:0/3011913743 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f332c14a450 con 0x7f332c071100 2024-09-11T00:48:00.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.290+0000 7f3332600640 1 -- 172.21.15.31:0/3011913743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f332c154d50 con 0x7f332c071100 2024-09-11T00:48:00.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.291+0000 7f3328c00640 1 -- 172.21.15.31:0/3011913743 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) ==== 95+0+5705 (secure 0 0 0) 0x7f33180830a0 con 0x7f332c071100 2024-09-11T00:48:00.294 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:00.294 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":128,"btime":"2024-09-11T00:47:59:450189+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:47:59.450181+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:48:00.294 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 128 2024-09-11T00:48:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 -- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f330c077f70 msgr2=0x7f330c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f330c077f70 0x7f330c07a430 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3320007f30 tx=0x7f3320007ec0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 -- 172.21.15.31:0/3011913743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f332c071100 msgr2=0x7f332c1e1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f332c071100 0x7f332c1e1540 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f331802aac0 tx=0x7f3318032b90 comp rx=0 tx=0).stop 2024-09-11T00:48:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 -- 172.21.15.31:0/3011913743 shutdown_connections 2024-09-11T00:48:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f330c077f70 0x7f330c07a430 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f332c14cb80 0x7f332c1556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f332c071100 0x7f332c1e1540 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 --2- 172.21.15.31:0/3011913743 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f332c06fda0 0x7f332c149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 -- 172.21.15.31:0/3011913743 >> 172.21.15.31:0/3011913743 conn(0x7f332c0c4d20 msgr2=0x7f332c073670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:00.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 -- 172.21.15.31:0/3011913743 shutdown_connections 2024-09-11T00:48:00.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.294+0000 7f3332600640 1 -- 172.21.15.31:0/3011913743 wait complete. 2024-09-11T00:48:00.314 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-11T00:48:00.314 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-11T00:48:00.314 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:48:00.333 INFO:tasks.ceph.mds.c:Stopped 2024-09-11T00:48:00.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:00.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.553+0000 7fe9cbe00640 1 -- 172.21.15.31:0/3598982284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9c40c3f20 msgr2=0x7fe9c40c2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.553+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/3598982284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9c40c3f20 0x7fe9c40c2ab0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00acf0 tx=0x7fe9bc02bbb0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.553+0000 7fe9cbe00640 1 -- 172.21.15.31:0/3598982284 shutdown_connections 2024-09-11T00:48:00.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.553+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/3598982284 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9c40c3f20 0x7fe9c40c2ab0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.553+0000 7fe9cbe00640 1 -- 172.21.15.31:0/3598982284 >> 172.21.15.31:0/3598982284 conn(0x7fe9c40a9750 msgr2=0x7fe9c40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:00.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.554+0000 7fe9cbe00640 1 -- 172.21.15.31:0/3598982284 shutdown_connections 2024-09-11T00:48:00.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.554+0000 7fe9cbe00640 1 -- 172.21.15.31:0/3598982284 wait complete. 2024-09-11T00:48:00.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.555+0000 7fe9cbe00640 1 Processor -- start 2024-09-11T00:48:00.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.555+0000 7fe9cbe00640 1 -- start start 2024-09-11T00:48:00.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.556+0000 7fe9cbe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c406ff40 0x7fe9c40c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.556+0000 7fe9cbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9c40c3f20 0x7fe9c41ef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.556+0000 7fe9cbe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9c41532c0 0x7fe9c41efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.556+0000 7fe9cbe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c40c7650 con 0x7fe9c40c3f20 2024-09-11T00:48:00.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.556+0000 7fe9cbe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c406f340 con 0x7fe9c406ff40 2024-09-11T00:48:00.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.556+0000 7fe9cbe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe9c40c77d0 con 0x7fe9c41532c0 2024-09-11T00:48:00.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.557+0000 7fe9c8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9c40c3f20 0x7fe9c41ef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.557+0000 7fe9c8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9c40c3f20 0x7fe9c41ef8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56544/0 (socket says 172.21.15.31:56544) 2024-09-11T00:48:00.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.557+0000 7fe9c8a00640 1 -- 172.21.15.31:0/4170320360 learned_addr learned my addr 172.21.15.31:0/4170320360 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:00.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.557+0000 7fe9c9400640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c406ff40 0x7fe9c40c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.557+0000 7fe9c9e00640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9c41532c0 0x7fe9c41efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.557+0000 7fe9c8a00640 1 -- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9c41532c0 msgr2=0x7fe9c41efe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.557+0000 7fe9c8a00640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9c41532c0 0x7fe9c41efe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.557+0000 7fe9c8a00640 1 -- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c406ff40 msgr2=0x7fe9c40c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.557+0000 7fe9c8a00640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c406ff40 0x7fe9c40c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.557+0000 7fe9c8a00640 1 -- 172.21.15.31:0/4170320360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c4154cf0 con 0x7fe9c40c3f20 2024-09-11T00:48:00.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.558+0000 7fe9c8a00640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9c40c3f20 0x7fe9c41ef8f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000c4c0 tx=0x7fe9b000c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:00.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.558+0000 7fe9c1600640 1 -- 172.21.15.31:0/4170320360 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b00052d0 con 0x7fe9c40c3f20 2024-09-11T00:48:00.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.558+0000 7fe9c1600640 1 -- 172.21.15.31:0/4170320360 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9b0005490 con 0x7fe9c40c3f20 2024-09-11T00:48:00.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.558+0000 7fe9c1600640 1 -- 172.21.15.31:0/4170320360 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0012890 con 0x7fe9c40c3f20 2024-09-11T00:48:00.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.558+0000 7fe9cbe00640 1 -- 172.21.15.31:0/4170320360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c4156860 con 0x7fe9c40c3f20 2024-09-11T00:48:00.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.558+0000 7fe9cbe00640 1 -- 172.21.15.31:0/4170320360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c41dd480 con 0x7fe9c40c3f20 2024-09-11T00:48:00.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.559+0000 7fe9cbe00640 1 -- 172.21.15.31:0/4170320360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c40a73e0 con 0x7fe9c40c3f20 2024-09-11T00:48:00.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.561+0000 7fe9c1600640 1 -- 172.21.15.31:0/4170320360 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe9b0012a30 con 0x7fe9c40c3f20 2024-09-11T00:48:00.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.561+0000 7fe9c1600640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a4077f70 0x7fe9a407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.562+0000 7fe9c1600640 1 -- 172.21.15.31:0/4170320360 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7fe9b009a330 con 0x7fe9c40c3f20 2024-09-11T00:48:00.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.562+0000 7fe9c9400640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a4077f70 0x7fe9a407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.563+0000 7fe9c9400640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a4077f70 0x7fe9a407a430 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8000c00 tx=0x7fe9b8007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:00.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.566+0000 7fe9c1600640 1 -- 172.21.15.31:0/4170320360 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe9b0061bf0 con 0x7fe9c40c3f20 2024-09-11T00:48:00.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:00.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.772+0000 7f8583200640 1 -- 172.21.15.31:0/3264416207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8584150020 msgr2=0x7f85840ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.772+0000 7f8583200640 1 --2- 172.21.15.31:0/3264416207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8584150020 0x7f85840ad4e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8574005fd0 tx=0x7f857402a5d0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.772+0000 7f8583200640 1 -- 172.21.15.31:0/3264416207 shutdown_connections 2024-09-11T00:48:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.772+0000 7f8583200640 1 --2- 172.21.15.31:0/3264416207 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85840ada20 0x7f858406e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.772+0000 7f8583200640 1 --2- 172.21.15.31:0/3264416207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8584150020 0x7f85840ad4e0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.772+0000 7f8583200640 1 -- 172.21.15.31:0/3264416207 >> 172.21.15.31:0/3264416207 conn(0x7f8584068f10 msgr2=0x7f85840cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.772+0000 7f8583200640 1 -- 172.21.15.31:0/3264416207 shutdown_connections 2024-09-11T00:48:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.773+0000 7f8583200640 1 -- 172.21.15.31:0/3264416207 wait complete. 2024-09-11T00:48:00.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.774+0000 7f8583200640 1 Processor -- start 2024-09-11T00:48:00.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.775+0000 7f8583200640 1 -- start start 2024-09-11T00:48:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.775+0000 7f8583200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85840ada20 0x7f85840c4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.775+0000 7f8583200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8584150020 0x7f85840c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.775+0000 7f8583200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8584156960 0x7f85840c3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8583200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8584157550 con 0x7f8584150020 2024-09-11T00:48:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8583200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f85841573d0 con 0x7f85840ada20 2024-09-11T00:48:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8583200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f85840cc220 con 0x7f8584156960 2024-09-11T00:48:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8581400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8584150020 0x7f85840c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8581400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8584150020 0x7f85840c58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56558/0 (socket says 172.21.15.31:56558) 2024-09-11T00:48:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8581400640 1 -- 172.21.15.31:0/3917183499 learned_addr learned my addr 172.21.15.31:0/3917183499 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8581e00640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85840ada20 0x7f85840c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8582800640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8584156960 0x7f85840c3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8581400640 1 -- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8584156960 msgr2=0x7f85840c3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8581400640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8584156960 0x7f85840c3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8581400640 1 -- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85840ada20 msgr2=0x7f85840c4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8581400640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85840ada20 0x7f85840c4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.776+0000 7f8581400640 1 -- 172.21.15.31:0/3917183499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85840c4e30 con 0x7f8584150020 2024-09-11T00:48:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.777+0000 7f8581400640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8584150020 0x7f85840c58a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f857000d950 tx=0x7f857000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.777+0000 7f857a000640 1 -- 172.21.15.31:0/3917183499 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570017070 con 0x7f8584150020 2024-09-11T00:48:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.777+0000 7f857a000640 1 -- 172.21.15.31:0/3917183499 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8570009d20 con 0x7f8584150020 2024-09-11T00:48:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.777+0000 7f857a000640 1 -- 172.21.15.31:0/3917183499 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85700106c0 con 0x7f8584150020 2024-09-11T00:48:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.777+0000 7f8583200640 1 -- 172.21.15.31:0/3917183499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85840c2230 con 0x7f8584150020 2024-09-11T00:48:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.777+0000 7f8583200640 1 -- 172.21.15.31:0/3917183499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85840c5380 con 0x7f8584150020 2024-09-11T00:48:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.779+0000 7f8583200640 1 -- 172.21.15.31:0/3917183499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85840cb5c0 con 0x7f8584150020 2024-09-11T00:48:00.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.780+0000 7f857a000640 1 -- 172.21.15.31:0/3917183499 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8570010860 con 0x7f8584150020 2024-09-11T00:48:00.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.781+0000 7f857a000640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8564077f70 0x7f856407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:00.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.781+0000 7f857a000640 1 -- 172.21.15.31:0/3917183499 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7f857009ac80 con 0x7f8584150020 2024-09-11T00:48:00.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.782+0000 7f8581e00640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8564077f70 0x7f856407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:00.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.783+0000 7f8581e00640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8564077f70 0x7f856407a430 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8574033000 tx=0x7f8574004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:00.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.785+0000 7f857a000640 1 -- 172.21.15.31:0/3917183499 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8570062540 con 0x7f8584150020 2024-09-11T00:48:00.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.874+0000 7fe9cbe00640 1 -- 172.21.15.31:0/4170320360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9c40c6d20 con 0x7fe9c40c3f20 2024-09-11T00:48:00.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.875+0000 7fe9c1600640 1 -- 172.21.15.31:0/4170320360 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+5708 (secure 0 0 0) 0x7fe9b0066620 con 0x7fe9c40c3f20 2024-09-11T00:48:00.878 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:00.878 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":129,"btime":"2024-09-11T00:48:00:760966+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:47:59.792493+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:48:00.878 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 129 2024-09-11T00:48:00.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 -- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a4077f70 msgr2=0x7fe9a407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a4077f70 0x7fe9a407a430 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8000c00 tx=0x7fe9b8007ec0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 -- 172.21.15.31:0/4170320360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9c40c3f20 msgr2=0x7fe9c41ef8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:00.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9c40c3f20 0x7fe9c41ef8f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000c4c0 tx=0x7fe9b000c980 comp rx=0 tx=0).stop 2024-09-11T00:48:00.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 -- 172.21.15.31:0/4170320360 shutdown_connections 2024-09-11T00:48:00.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe9a4077f70 0x7fe9a407a430 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe9c41532c0 0x7fe9c41efe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe9c40c3f20 0x7fe9c41ef8f0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 --2- 172.21.15.31:0/4170320360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c406ff40 0x7fe9c40c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:00.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 -- 172.21.15.31:0/4170320360 >> 172.21.15.31:0/4170320360 conn(0x7fe9c40a9750 msgr2=0x7fe9c41515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:00.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 -- 172.21.15.31:0/4170320360 shutdown_connections 2024-09-11T00:48:00.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:00.878+0000 7fe9cbe00640 1 -- 172.21.15.31:0/4170320360 wait complete. 2024-09-11T00:48:00.898 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:48:01.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.034+0000 7f8583200640 1 -- 172.21.15.31:0/3917183499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85840ad4e0 con 0x7f8584150020 2024-09-11T00:48:01.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.035+0000 7f857a000640 1 -- 172.21.15.31:0/3917183499 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+5708 (secure 0 0 0) 0x7f8570066f70 con 0x7f8584150020 2024-09-11T00:48:01.038 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:01.038 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":129,"btime":"2024-09-11T00:48:00:760966+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:47:59.792493+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:48:01.038 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 129 2024-09-11T00:48:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.037+0000 7f8583200640 1 -- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8564077f70 msgr2=0x7f856407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.037+0000 7f8583200640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8564077f70 0x7f856407a430 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8574033000 tx=0x7f8574004730 comp rx=0 tx=0).stop 2024-09-11T00:48:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.037+0000 7f8583200640 1 -- 172.21.15.31:0/3917183499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8584150020 msgr2=0x7f85840c58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.037+0000 7f8583200640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8584150020 0x7f85840c58a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f857000d950 tx=0x7f857000de10 comp rx=0 tx=0).stop 2024-09-11T00:48:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.037+0000 7f8583200640 1 -- 172.21.15.31:0/3917183499 shutdown_connections 2024-09-11T00:48:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.037+0000 7f8583200640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8564077f70 0x7f856407a430 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.037+0000 7f8583200640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8584156960 0x7f85840c3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.037+0000 7f8583200640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8584150020 0x7f85840c58a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.037+0000 7f8583200640 1 --2- 172.21.15.31:0/3917183499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85840ada20 0x7f85840c4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.038+0000 7f8583200640 1 -- 172.21.15.31:0/3917183499 >> 172.21.15.31:0/3917183499 conn(0x7f8584068f10 msgr2=0x7f858414f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.038+0000 7f8583200640 1 -- 172.21.15.31:0/3917183499 shutdown_connections 2024-09-11T00:48:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.038+0000 7f8583200640 1 -- 172.21.15.31:0/3917183499 wait complete. 2024-09-11T00:48:01.058 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-11T00:48:01.058 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:48:01.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:01.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.241+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3706574800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5ac0b12f0 msgr2=0x7ff5ac14f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.241+0000 7ff5b1a00640 1 --2- 172.21.15.31:0/3706574800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5ac0b12f0 0x7ff5ac14f940 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0ae640 tx=0x7ff5a402aa30 comp rx=0 tx=0).stop 2024-09-11T00:48:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.241+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3706574800 shutdown_connections 2024-09-11T00:48:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.241+0000 7ff5b1a00640 1 --2- 172.21.15.31:0/3706574800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5ac0b12f0 0x7ff5ac14f940 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.241+0000 7ff5b1a00640 1 --2- 172.21.15.31:0/3706574800 >> v2:172.21.15.204:3300/0 conn(0x7ff5ac0c6730 0x7ff5ac1596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.241+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3706574800 >> 172.21.15.31:0/3706574800 conn(0x7ff5ac068ac0 msgr2=0x7ff5ac068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.242+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3706574800 shutdown_connections 2024-09-11T00:48:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.242+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3706574800 wait complete. 2024-09-11T00:48:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.243+0000 7ff5b1a00640 1 Processor -- start 2024-09-11T00:48:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.243+0000 7ff5b1a00640 1 -- start start 2024-09-11T00:48:01.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.244+0000 7ff5b1a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac0b12f0 0x7ff5ac150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:01.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.244+0000 7ff5b1a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff5ac0bf240 0x7ff5ac0c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:01.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.244+0000 7ff5b1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5ac0c6730 0x7ff5ac1e0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:01.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.244+0000 7ff5b1a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ac0c6b10 con 0x7ff5ac0c6730 2024-09-11T00:48:01.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.244+0000 7ff5b1a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ac156640 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.244+0000 7ff5b1a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff5ac0c6c90 con 0x7ff5ac0bf240 2024-09-11T00:48:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5b1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5ac0c6730 0x7ff5ac1e0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5abe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac0b12f0 0x7ff5ac150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5b1000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5ac0c6730 0x7ff5ac1e0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56578/0 (socket says 172.21.15.31:56578) 2024-09-11T00:48:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5b1000640 1 -- 172.21.15.31:0/3612781116 learned_addr learned my addr 172.21.15.31:0/3612781116 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5ab400640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff5ac0bf240 0x7ff5ac0c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5abe00640 1 -- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff5ac0bf240 msgr2=0x7ff5ac0c2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5abe00640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff5ac0bf240 0x7ff5ac0c2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5abe00640 1 -- 172.21.15.31:0/3612781116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5ac0c6730 msgr2=0x7ff5ac1e0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5abe00640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5ac0c6730 0x7ff5ac1e0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5abe00640 1 -- 172.21.15.31:0/3612781116 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ac1f04f0 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.245+0000 7ff5abe00640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac0b12f0 0x7ff5ac150bc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00e6d0 tx=0x7ff59c00eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.246+0000 7ff5a3e00640 1 -- 172.21.15.31:0/3612781116 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c00cab0 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.246+0000 7ff5a3e00640 1 -- 172.21.15.31:0/3612781116 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff59c00cd40 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.246+0000 7ff5a3e00640 1 -- 172.21.15.31:0/3612781116 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c010690 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.246+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3612781116 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5ac1f0210 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.246+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3612781116 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5ac1ec570 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.247+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3612781116 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5ac06e850 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.248+0000 7ff5a3e00640 1 -- 172.21.15.31:0/3612781116 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff59c010830 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.249+0000 7ff5a3e00640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff58c077f70 0x7ff58c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.250+0000 7ff5a3e00640 1 -- 172.21.15.31:0/3612781116 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7ff59c09b270 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.250+0000 7ff5ab400640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff58c077f70 0x7ff58c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:01.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.251+0000 7ff5ab400640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff58c077f70 0x7ff58c07a430 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0bfae0 tx=0x7ff598009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:01.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.254+0000 7ff5a3e00640 1 -- 172.21.15.31:0/3612781116 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff59c062b00 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.539+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3612781116 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5ac0cafd0 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.540+0000 7ff5a3e00640 1 -- 172.21.15.31:0/3612781116 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+5708 (secure 0 0 0) 0x7ff59c067530 con 0x7ff5ac0b12f0 2024-09-11T00:48:01.542 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:01.542 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":129,"btime":"2024-09-11T00:48:00:760966+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:47:59.792493+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:48:01.543 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 129 2024-09-11T00:48:01.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff58c077f70 msgr2=0x7ff58c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:01.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff58c077f70 0x7ff58c07a430 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0bfae0 tx=0x7ff598009040 comp rx=0 tx=0).stop 2024-09-11T00:48:01.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac0b12f0 msgr2=0x7ff5ac150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:01.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac0b12f0 0x7ff5ac150bc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00e6d0 tx=0x7ff59c00eb90 comp rx=0 tx=0).stop 2024-09-11T00:48:01.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3612781116 shutdown_connections 2024-09-11T00:48:01.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff58c077f70 0x7ff58c07a430 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5ac0c6730 0x7ff5ac1e0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff5ac0bf240 0x7ff5ac0c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 --2- 172.21.15.31:0/3612781116 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5ac0b12f0 0x7ff5ac150bc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:01.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3612781116 >> 172.21.15.31:0/3612781116 conn(0x7ff5ac068ac0 msgr2=0x7ff5ac152ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:01.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3612781116 shutdown_connections 2024-09-11T00:48:01.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:01.542+0000 7ff5b1a00640 1 -- 172.21.15.31:0/3612781116 wait complete. 2024-09-11T00:48:01.563 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:48:00:760966+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 129, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 126, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6804": { "addr": "172.21.15.31:6835/2163558126", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 2163558126, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 2163558126, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6804, "incarnation": 123, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:47:22.793059+0000", "qdb_cluster": [ 6804 ], "qdb_leader": 6804, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6804 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 129, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6829": { "addr": "172.21.15.204:6835/634556689", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 634556689, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 634556689, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6829, "incarnation": 128, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:47:59.792493+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6829 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:48:02.899 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.067+0000 7f78e6000640 1 -- 172.21.15.31:0/2887311342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78e00ac2c0 msgr2=0x7f78e00ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.067+0000 7f78e6000640 1 --2- 172.21.15.31:0/2887311342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78e00ac2c0 0x7f78e00ac6a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f78d4005fd0 tx=0x7f78d402a4b0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.068+0000 7f78e6000640 1 -- 172.21.15.31:0/2887311342 shutdown_connections 2024-09-11T00:48:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.068+0000 7f78e6000640 1 --2- 172.21.15.31:0/2887311342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78e00ac2c0 0x7f78e00ac6a0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.068+0000 7f78e6000640 1 -- 172.21.15.31:0/2887311342 >> 172.21.15.31:0/2887311342 conn(0x7f78e006caf0 msgr2=0x7f78e006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.068+0000 7f78e6000640 1 -- 172.21.15.31:0/2887311342 shutdown_connections 2024-09-11T00:48:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.068+0000 7f78e6000640 1 -- 172.21.15.31:0/2887311342 wait complete. 2024-09-11T00:48:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.069+0000 7f78e6000640 1 Processor -- start 2024-09-11T00:48:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.069+0000 7f78e6000640 1 -- start start 2024-09-11T00:48:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.070+0000 7f78e6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78e00a6e20 0x7f78e006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.070+0000 7f78e6000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78e00ac2c0 0x7f78e006ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.070+0000 7f78e6000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78e00acbe0 0x7f78e01486e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.070+0000 7f78e6000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f78e00c5d20 con 0x7f78e00a6e20 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.070+0000 7f78e6000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78e01519e0 con 0x7f78e00ac2c0 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.070+0000 7f78e6000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f78e00c68c0 con 0x7f78e00acbe0 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.070+0000 7f78e4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78e00a6e20 0x7f78e006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.070+0000 7f78e4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78e00a6e20 0x7f78e006e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56602/0 (socket says 172.21.15.31:56602) 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.070+0000 7f78e4c00640 1 -- 172.21.15.31:0/3253875687 learned_addr learned my addr 172.21.15.31:0/3253875687 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.071+0000 7f78dfe00640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78e00ac2c0 0x7f78e006ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.071+0000 7f78e5600640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78e00acbe0 0x7f78e01486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.071+0000 7f78e4c00640 1 -- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78e00acbe0 msgr2=0x7f78e01486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.071+0000 7f78e4c00640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78e00acbe0 0x7f78e01486e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.071+0000 7f78e4c00640 1 -- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78e00ac2c0 msgr2=0x7f78e006ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.071+0000 7f78e4c00640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78e00ac2c0 0x7f78e006ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.071+0000 7f78e4c00640 1 -- 172.21.15.31:0/3253875687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e01ef260 con 0x7f78e00a6e20 2024-09-11T00:48:03.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.071+0000 7f78e4c00640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78e00a6e20 0x7f78e006e3d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f78d4009850 tx=0x7f78d40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:03.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.072+0000 7f78dcc00640 1 -- 172.21.15.31:0/3253875687 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d403f070 con 0x7f78e00a6e20 2024-09-11T00:48:03.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.072+0000 7f78e6000640 1 -- 172.21.15.31:0/3253875687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e01ef500 con 0x7f78e00a6e20 2024-09-11T00:48:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.072+0000 7f78dcc00640 1 -- 172.21.15.31:0/3253875687 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78d4004890 con 0x7f78e00a6e20 2024-09-11T00:48:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.072+0000 7f78dcc00640 1 -- 172.21.15.31:0/3253875687 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d4035030 con 0x7f78e00a6e20 2024-09-11T00:48:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.072+0000 7f78e6000640 1 -- 172.21.15.31:0/3253875687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e006e910 con 0x7f78e00a6e20 2024-09-11T00:48:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.073+0000 7f78e6000640 1 -- 172.21.15.31:0/3253875687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78e006f3b0 con 0x7f78e00a6e20 2024-09-11T00:48:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.074+0000 7f78dcc00640 1 -- 172.21.15.31:0/3253875687 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f78d4004c60 con 0x7f78e00a6e20 2024-09-11T00:48:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.075+0000 7f78dcc00640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f78c0077f70 0x7f78c007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.075+0000 7f78dcc00640 1 -- 172.21.15.31:0/3253875687 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7f78d40bbcc0 con 0x7f78e00a6e20 2024-09-11T00:48:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.075+0000 7f78dfe00640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f78c0077f70 0x7f78c007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.076+0000 7f78dfe00640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f78c0077f70 0x7f78c007a430 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f78e01ecd40 tx=0x7f78cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:03.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.079+0000 7f78dcc00640 1 -- 172.21.15.31:0/3253875687 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f78d40c2080 con 0x7f78e00a6e20 2024-09-11T00:48:03.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.311+0000 7f78e6000640 1 -- 172.21.15.31:0/3253875687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78e00c7470 con 0x7f78e00a6e20 2024-09-11T00:48:03.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.312+0000 7f78dcc00640 1 -- 172.21.15.31:0/3253875687 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+5713 (secure 0 0 0) 0x7f78d40833e0 con 0x7f78e00a6e20 2024-09-11T00:48:03.314 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:03.314 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":131,"btime":"2024-09-11T00:48:02:775676+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:03.315 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 131 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 -- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f78c0077f70 msgr2=0x7f78c007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f78c0077f70 0x7f78c007a430 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f78e01ecd40 tx=0x7f78cc009040 comp rx=0 tx=0).stop 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 -- 172.21.15.31:0/3253875687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78e00a6e20 msgr2=0x7f78e006e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78e00a6e20 0x7f78e006e3d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f78d4009850 tx=0x7f78d40039e0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 -- 172.21.15.31:0/3253875687 shutdown_connections 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f78c0077f70 0x7f78c007a430 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f78e00acbe0 0x7f78e01486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78e00ac2c0 0x7f78e006ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 --2- 172.21.15.31:0/3253875687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78e00a6e20 0x7f78e006e3d0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 -- 172.21.15.31:0/3253875687 >> 172.21.15.31:0/3253875687 conn(0x7f78e006caf0 msgr2=0x7f78e0153290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 -- 172.21.15.31:0/3253875687 shutdown_connections 2024-09-11T00:48:03.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.314+0000 7f78e6000640 1 -- 172.21.15.31:0/3253875687 wait complete. 2024-09-11T00:48:03.332 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:48:03.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:03.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.723+0000 7fa46e800640 1 -- 172.21.15.31:0/4074019017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4681518b0 msgr2=0x7fa4680c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:03.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.723+0000 7fa46e800640 1 --2- 172.21.15.31:0/4074019017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4681518b0 0x7fa4680c3600 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa4680add10 tx=0x7fa46002ab50 comp rx=0 tx=0).stop 2024-09-11T00:48:03.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.723+0000 7fa46e800640 1 -- 172.21.15.31:0/4074019017 shutdown_connections 2024-09-11T00:48:03.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.723+0000 7fa46e800640 1 --2- 172.21.15.31:0/4074019017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4681518b0 0x7fa4680c3600 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.723+0000 7fa46e800640 1 --2- 172.21.15.31:0/4074019017 >> v2:172.21.15.204:3300/0 conn(0x7fa4680c23a0 0x7fa4680c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.723+0000 7fa46e800640 1 -- 172.21.15.31:0/4074019017 >> 172.21.15.31:0/4074019017 conn(0x7fa4680a9750 msgr2=0x7fa4680a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:03.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.723+0000 7fa46e800640 1 -- 172.21.15.31:0/4074019017 shutdown_connections 2024-09-11T00:48:03.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.723+0000 7fa46e800640 1 -- 172.21.15.31:0/4074019017 wait complete. 2024-09-11T00:48:03.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.724+0000 7fa46e800640 1 Processor -- start 2024-09-11T00:48:03.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.725+0000 7fa46e800640 1 -- start start 2024-09-11T00:48:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.725+0000 7fa46e800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa468072480 0x7fa4681ebe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.725+0000 7fa46e800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4680c23a0 0x7fa4681500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.725+0000 7fa46e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4681518b0 0x7fa4681531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.725+0000 7fa46e800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa468074200 con 0x7fa4680c23a0 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.725+0000 7fa46e800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa468074080 con 0x7fa4681518b0 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.725+0000 7fa46e800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa468072860 con 0x7fa468072480 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa466a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4680c23a0 0x7fa4681500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa466a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4680c23a0 0x7fa4681500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56616/0 (socket says 172.21.15.31:56616) 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa466a00640 1 -- 172.21.15.31:0/2691719484 learned_addr learned my addr 172.21.15.31:0/2691719484 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa467400640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa468072480 0x7fa4681ebe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa467e00640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4681518b0 0x7fa4681531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa466a00640 1 -- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa468072480 msgr2=0x7fa4681ebe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa466a00640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa468072480 0x7fa4681ebe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa466a00640 1 -- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4681518b0 msgr2=0x7fa4681531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa466a00640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4681518b0 0x7fa4681531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa466a00640 1 -- 172.21.15.31:0/2691719484 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4681529f0 con 0x7fa4680c23a0 2024-09-11T00:48:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa467e00640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4681518b0 0x7fa4681531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:48:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.726+0000 7fa466a00640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4680c23a0 0x7fa4681500f0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa45400e9f0 tx=0x7fa45400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.727+0000 7fa45f400640 1 -- 172.21.15.31:0/2691719484 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45400cd80 con 0x7fa4680c23a0 2024-09-11T00:48:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.727+0000 7fa45f400640 1 -- 172.21.15.31:0/2691719484 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa454005020 con 0x7fa4680c23a0 2024-09-11T00:48:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.727+0000 7fa46e800640 1 -- 172.21.15.31:0/2691719484 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46806fab0 con 0x7fa4680c23a0 2024-09-11T00:48:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.727+0000 7fa45f400640 1 -- 172.21.15.31:0/2691719484 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454005450 con 0x7fa4680c23a0 2024-09-11T00:48:03.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.727+0000 7fa46e800640 1 -- 172.21.15.31:0/2691719484 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa468070070 con 0x7fa4680c23a0 2024-09-11T00:48:03.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.728+0000 7fa46e800640 1 -- 172.21.15.31:0/2691719484 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa468153f20 con 0x7fa4680c23a0 2024-09-11T00:48:03.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.729+0000 7fa45f400640 1 -- 172.21.15.31:0/2691719484 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa4540047a0 con 0x7fa4680c23a0 2024-09-11T00:48:03.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.730+0000 7fa45f400640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa448077f70 0x7fa44807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:03.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.730+0000 7fa45f400640 1 -- 172.21.15.31:0/2691719484 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7fa45409b380 con 0x7fa4680c23a0 2024-09-11T00:48:03.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.730+0000 7fa467400640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa448077f70 0x7fa44807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:03.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.731+0000 7fa467400640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa448077f70 0x7fa44807a430 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa4580017c0 tx=0x7fa458009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:03.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.734+0000 7fa45f400640 1 -- 172.21.15.31:0/2691719484 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa4540632c0 con 0x7fa4680c23a0 2024-09-11T00:48:03.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.970+0000 7fa46e800640 1 -- 172.21.15.31:0/2691719484 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4680c3820 con 0x7fa4680c23a0 2024-09-11T00:48:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.970+0000 7fa45f400640 1 -- 172.21.15.31:0/2691719484 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+5713 (secure 0 0 0) 0x7fa454062bc0 con 0x7fa4680c23a0 2024-09-11T00:48:03.973 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:03.973 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":131,"btime":"2024-09-11T00:48:02:775676+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:03.974 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 131 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 -- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa448077f70 msgr2=0x7fa44807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa448077f70 0x7fa44807a430 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa4580017c0 tx=0x7fa458009040 comp rx=0 tx=0).stop 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 -- 172.21.15.31:0/2691719484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4680c23a0 msgr2=0x7fa4681500f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4680c23a0 0x7fa4681500f0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa45400e9f0 tx=0x7fa45400eeb0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 -- 172.21.15.31:0/2691719484 shutdown_connections 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa448077f70 0x7fa44807a430 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4681518b0 0x7fa4681531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4680c23a0 0x7fa4681500f0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 --2- 172.21.15.31:0/2691719484 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa468072480 0x7fa4681ebe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 -- 172.21.15.31:0/2691719484 >> 172.21.15.31:0/2691719484 conn(0x7fa4680a9750 msgr2=0x7fa46814c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 -- 172.21.15.31:0/2691719484 shutdown_connections 2024-09-11T00:48:03.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:03.973+0000 7fa46e800640 1 -- 172.21.15.31:0/2691719484 wait complete. 2024-09-11T00:48:03.991 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=5957 2024-09-11T00:48:03.991 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.b 2024-09-11T00:48:03.992 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-11T00:48:03.992 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-11T00:48:03.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-11T00:48:03.996 INFO:tasks.ceph.mds.b:Started 2024-09-11T00:48:03.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:04.141 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:48:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.157+0000 7f6c67400640 1 -- 172.21.15.31:0/3709861210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c60153f20 msgr2=0x7f6c600c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.157+0000 7f6c67400640 1 --2- 172.21.15.31:0/3709861210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c60153f20 0x7f6c600c30c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f6c600c4400 tx=0x7f6c5802aac0 comp rx=0 tx=0).stop 2024-09-11T00:48:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.157+0000 7f6c67400640 1 -- 172.21.15.31:0/3709861210 shutdown_connections 2024-09-11T00:48:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.157+0000 7f6c67400640 1 --2- 172.21.15.31:0/3709861210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c60153f20 0x7f6c600c30c0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.157+0000 7f6c67400640 1 --2- 172.21.15.31:0/3709861210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c600c23a0 0x7f6c601516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.157+0000 7f6c67400640 1 --2- 172.21.15.31:0/3709861210 >> v2:172.21.15.204:3300/0 conn(0x7f6c60072480 0x7f6c601511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.157+0000 7f6c67400640 1 -- 172.21.15.31:0/3709861210 >> 172.21.15.31:0/3709861210 conn(0x7f6c600a9750 msgr2=0x7f6c600a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.157+0000 7f6c67400640 1 -- 172.21.15.31:0/3709861210 shutdown_connections 2024-09-11T00:48:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.157+0000 7f6c67400640 1 -- 172.21.15.31:0/3709861210 wait complete. 2024-09-11T00:48:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.158+0000 7f6c67400640 1 Processor -- start 2024-09-11T00:48:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.159+0000 7f6c67400640 1 -- start start 2024-09-11T00:48:04.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.159+0000 7f6c67400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c60072480 0x7f6c60148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:04.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.159+0000 7f6c67400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c600c23a0 0x7f6c600c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:04.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.159+0000 7f6c67400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c60153f20 0x7f6c601eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:04.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.159+0000 7f6c67400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c60074200 con 0x7f6c60153f20 2024-09-11T00:48:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.159+0000 7f6c67400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6c60074080 con 0x7f6c600c23a0 2024-09-11T00:48:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.159+0000 7f6c67400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6c60072860 con 0x7f6c60072480 2024-09-11T00:48:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c65400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c60153f20 0x7f6c601eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c600c23a0 0x7f6c600c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c64a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c60072480 0x7f6c60148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c57e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c600c23a0 0x7f6c600c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:48990/0 (socket says 172.21.15.31:48990) 2024-09-11T00:48:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c57e00640 1 -- 172.21.15.31:0/3369932875 learned_addr learned my addr 172.21.15.31:0/3369932875 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:04.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c65400640 1 -- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c60072480 msgr2=0x7f6c60148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:04.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c65400640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c60072480 0x7f6c60148f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:04.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c65400640 1 -- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c600c23a0 msgr2=0x7f6c600c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:04.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c65400640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c600c23a0 0x7f6c600c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c65400640 1 -- 172.21.15.31:0/3369932875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c601494d0 con 0x7f6c60153f20 2024-09-11T00:48:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.160+0000 7f6c57e00640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c600c23a0 0x7f6c600c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:48:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.161+0000 7f6c65400640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c60153f20 0x7f6c601eed80 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f6c58032000 tx=0x7f6c58003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.161+0000 7f6c5d600640 1 -- 172.21.15.31:0/3369932875 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58044070 con 0x7f6c60153f20 2024-09-11T00:48:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.161+0000 7f6c5d600640 1 -- 172.21.15.31:0/3369932875 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c58004040 con 0x7f6c60153f20 2024-09-11T00:48:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.161+0000 7f6c5d600640 1 -- 172.21.15.31:0/3369932875 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58004450 con 0x7f6c60153f20 2024-09-11T00:48:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.161+0000 7f6c67400640 1 -- 172.21.15.31:0/3369932875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c60147b50 con 0x7f6c60153f20 2024-09-11T00:48:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.161+0000 7f6c67400640 1 -- 172.21.15.31:0/3369932875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c6006e3a0 con 0x7f6c60153f20 2024-09-11T00:48:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.162+0000 7f6c67400640 1 -- 172.21.15.31:0/3369932875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c600aa1b0 con 0x7f6c60153f20 2024-09-11T00:48:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.163+0000 7f6c5d600640 1 -- 172.21.15.31:0/3369932875 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6c580045f0 con 0x7f6c60153f20 2024-09-11T00:48:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.164+0000 7f6c5d600640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c3c077f70 0x7f6c3c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.165+0000 7f6c5d600640 1 -- 172.21.15.31:0/3369932875 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7f6c580bb1c0 con 0x7f6c60153f20 2024-09-11T00:48:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.165+0000 7f6c64a00640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c3c077f70 0x7f6c3c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.165+0000 7f6c64a00640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c3c077f70 0x7f6c3c07a430 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009bd0 tx=0x7f6c50005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:04.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.169+0000 7f6c5d600640 1 -- 172.21.15.31:0/3369932875 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6c600aa1b0 con 0x7f6c60153f20 2024-09-11T00:48:04.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.403+0000 7f6c67400640 1 -- 172.21.15.31:0/3369932875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c60148420 con 0x7f6c60153f20 2024-09-11T00:48:04.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.403+0000 7f6c5d600640 1 -- 172.21.15.31:0/3369932875 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+5713 (secure 0 0 0) 0x7f6c58032030 con 0x7f6c60153f20 2024-09-11T00:48:04.406 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:04.406 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":131,"btime":"2024-09-11T00:48:02:775676+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:04.407 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 131 2024-09-11T00:48:04.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 -- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c3c077f70 msgr2=0x7f6c3c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:04.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c3c077f70 0x7f6c3c07a430 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009bd0 tx=0x7f6c50005f90 comp rx=0 tx=0).stop 2024-09-11T00:48:04.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 -- 172.21.15.31:0/3369932875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c60153f20 msgr2=0x7f6c601eed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:04.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c60153f20 0x7f6c601eed80 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f6c58032000 tx=0x7f6c58003c10 comp rx=0 tx=0).stop 2024-09-11T00:48:04.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 -- 172.21.15.31:0/3369932875 shutdown_connections 2024-09-11T00:48:04.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6c3c077f70 0x7f6c3c07a430 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:04.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c60153f20 0x7f6c601eed80 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:04.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c600c23a0 0x7f6c600c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:04.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 --2- 172.21.15.31:0/3369932875 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6c60072480 0x7f6c60148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:04.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 -- 172.21.15.31:0/3369932875 >> 172.21.15.31:0/3369932875 conn(0x7f6c600a9750 msgr2=0x7f6c60153040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:04.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 -- 172.21.15.31:0/3369932875 shutdown_connections 2024-09-11T00:48:04.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:04.406+0000 7f6c67400640 1 -- 172.21.15.31:0/3369932875 wait complete. 2024-09-11T00:48:04.425 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-11T00:48:05.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:05.472 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:48:05.469+0000 7fb15c000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:48:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.498+0000 7f2c5be00640 1 -- 172.21.15.31:0/295813501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c5c0c6420 msgr2=0x7f2c5c153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.498+0000 7f2c5be00640 1 --2- 172.21.15.31:0/295813501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c5c0c6420 0x7f2c5c153310 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2c44005fd0 tx=0x7f2c4402a540 comp rx=0 tx=0).stop 2024-09-11T00:48:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.498+0000 7f2c5be00640 1 -- 172.21.15.31:0/295813501 shutdown_connections 2024-09-11T00:48:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.498+0000 7f2c5be00640 1 --2- 172.21.15.31:0/295813501 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c5c153850 0x7f2c5c156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.498+0000 7f2c5be00640 1 --2- 172.21.15.31:0/295813501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c5c0c6420 0x7f2c5c153310 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.498+0000 7f2c5be00640 1 -- 172.21.15.31:0/295813501 >> 172.21.15.31:0/295813501 conn(0x7f2c5c06caf0 msgr2=0x7f2c5c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.498+0000 7f2c5be00640 1 -- 172.21.15.31:0/295813501 shutdown_connections 2024-09-11T00:48:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.498+0000 7f2c5be00640 1 -- 172.21.15.31:0/295813501 wait complete. 2024-09-11T00:48:05.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.499+0000 7f2c5be00640 1 Processor -- start 2024-09-11T00:48:05.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.499+0000 7f2c5be00640 1 -- start start 2024-09-11T00:48:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.500+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c5c0c5aa0 0x7f2c5c0ab880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.500+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c5c0c6420 0x7f2c5c1e0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.500+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c5c153850 0x7f2c5c154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.500+0000 7f2c5be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5c0c5e80 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.500+0000 7f2c5be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5c0c2e80 con 0x7f2c5c0c6420 2024-09-11T00:48:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.500+0000 7f2c5be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2c5c0c6000 con 0x7f2c5c153850 2024-09-11T00:48:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c5aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c5c0c5aa0 0x7f2c5c0ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c5aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c5c0c5aa0 0x7f2c5c0ab880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56658/0 (socket says 172.21.15.31:56658) 2024-09-11T00:48:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c5aa00640 1 -- 172.21.15.31:0/1698697952 learned_addr learned my addr 172.21.15.31:0/1698697952 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c5b400640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c5c153850 0x7f2c5c154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c53e00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c5c0c6420 0x7f2c5c1e0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c5aa00640 1 -- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c5c153850 msgr2=0x7f2c5c154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c5aa00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c5c153850 0x7f2c5c154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c5aa00640 1 -- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c5c0c6420 msgr2=0x7f2c5c1e0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c5aa00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c5c0c6420 0x7f2c5c1e0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c5aa00640 1 -- 172.21.15.31:0/1698697952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c5c1e2180 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.501+0000 7f2c5aa00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c5c0c5aa0 0x7f2c5c0ab880 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00c500 tx=0x7f2c4c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:05.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.502+0000 7f2c53400640 1 -- 172.21.15.31:0/1698697952 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c017070 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.502+0000 7f2c53400640 1 -- 172.21.15.31:0/1698697952 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c4c0035f0 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.502+0000 7f2c53400640 1 -- 172.21.15.31:0/1698697952 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c003a40 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.502+0000 7f2c5be00640 1 -- 172.21.15.31:0/1698697952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c5c0ac5e0 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.502+0000 7f2c5be00640 1 -- 172.21.15.31:0/1698697952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c5c1e0360 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.503+0000 7f2c5be00640 1 -- 172.21.15.31:0/1698697952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c5c06a170 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.504+0000 7f2c53400640 1 -- 172.21.15.31:0/1698697952 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2c4c003be0 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.505+0000 7f2c53400640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c38077f70 0x7f2c3807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:05.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.506+0000 7f2c53400640 1 -- 172.21.15.31:0/1698697952 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7f2c4c09a0b0 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.506+0000 7f2c53e00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c38077f70 0x7f2c3807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.507+0000 7f2c53e00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c38077f70 0x7f2c3807a430 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2c44033000 tx=0x7f2c44004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:05.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.510+0000 7f2c53400640 1 -- 172.21.15.31:0/1698697952 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2c5c06a170 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.741+0000 7f2c5be00640 1 -- 172.21.15.31:0/1698697952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c5c152d80 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.742+0000 7f2c53400640 1 -- 172.21.15.31:0/1698697952 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6542 (secure 0 0 0) 0x7f2c4c097e20 con 0x7f2c5c0c5aa0 2024-09-11T00:48:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:05.745 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":132,"btime":"2024-09-11T00:48:04:788120+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:05.745 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 132 2024-09-11T00:48:05.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 -- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c38077f70 msgr2=0x7f2c3807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:05.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c38077f70 0x7f2c3807a430 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2c44033000 tx=0x7f2c44004730 comp rx=0 tx=0).stop 2024-09-11T00:48:05.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 -- 172.21.15.31:0/1698697952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c5c0c5aa0 msgr2=0x7f2c5c0ab880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:05.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c5c0c5aa0 0x7f2c5c0ab880 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00c500 tx=0x7f2c4c00c9c0 comp rx=0 tx=0).stop 2024-09-11T00:48:05.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 -- 172.21.15.31:0/1698697952 shutdown_connections 2024-09-11T00:48:05.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c38077f70 0x7f2c3807a430 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:05.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c5c153850 0x7f2c5c154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:05.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c5c0c6420 0x7f2c5c1e0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:05.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 --2- 172.21.15.31:0/1698697952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c5c0c5aa0 0x7f2c5c0ab880 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:05.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 -- 172.21.15.31:0/1698697952 >> 172.21.15.31:0/1698697952 conn(0x7f2c5c06caf0 msgr2=0x7f2c5c0a7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:05.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 -- 172.21.15.31:0/1698697952 shutdown_connections 2024-09-11T00:48:05.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:05.744+0000 7f2c5be00640 1 -- 172.21.15.31:0/1698697952 wait complete. 2024-09-11T00:48:05.763 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:48:06.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:06.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.590+0000 7fb01d200640 1 -- 172.21.15.31:0/2003779477 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0180a8bb0 msgr2=0x7fb0180bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:06.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.590+0000 7fb01d200640 1 --2- 172.21.15.31:0/2003779477 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0180a8bb0 0x7fb0180bf600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb004009860 tx=0x7fb00402a510 comp rx=0 tx=0).stop 2024-09-11T00:48:06.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.590+0000 7fb01d200640 1 -- 172.21.15.31:0/2003779477 shutdown_connections 2024-09-11T00:48:06.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.590+0000 7fb01d200640 1 --2- 172.21.15.31:0/2003779477 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0180a8bb0 0x7fb0180bf600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:06.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.590+0000 7fb01d200640 1 -- 172.21.15.31:0/2003779477 >> 172.21.15.31:0/2003779477 conn(0x7fb0180c9af0 msgr2=0x7fb0180c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:06.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.590+0000 7fb01d200640 1 -- 172.21.15.31:0/2003779477 shutdown_connections 2024-09-11T00:48:06.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.590+0000 7fb01d200640 1 -- 172.21.15.31:0/2003779477 wait complete. 2024-09-11T00:48:06.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.591+0000 7fb01d200640 1 Processor -- start 2024-09-11T00:48:06.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.592+0000 7fb01d200640 1 -- start start 2024-09-11T00:48:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.592+0000 7fb01d200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0180a8bb0 0x7fb01814dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.592+0000 7fb01d200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0180c67f0 0x7fb01814ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.592+0000 7fb01d200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0181578e0 0x7fb01814d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.592+0000 7fb01d200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb0180a9260 con 0x7fb0180a8bb0 2024-09-11T00:48:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.592+0000 7fb01d200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb0180a90e0 con 0x7fb0181578e0 2024-09-11T00:48:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.592+0000 7fb01d200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb0180c5b20 con 0x7fb0180c67f0 2024-09-11T00:48:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb017400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0180a8bb0 0x7fb01814dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:06.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb017400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0180a8bb0 0x7fb01814dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56668/0 (socket says 172.21.15.31:56668) 2024-09-11T00:48:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb016a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0180c67f0 0x7fb01814ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb017400640 1 -- 172.21.15.31:0/1606115183 learned_addr learned my addr 172.21.15.31:0/1606115183 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb017e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0181578e0 0x7fb01814d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb017400640 1 -- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0180c67f0 msgr2=0x7fb01814ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb017400640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0180c67f0 0x7fb01814ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb017400640 1 -- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0181578e0 msgr2=0x7fb01814d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb017400640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0181578e0 0x7fb01814d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb017400640 1 -- 172.21.15.31:0/1606115183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0180ac2c0 con 0x7fb0180a8bb0 2024-09-11T00:48:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.593+0000 7fb017e00640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0181578e0 0x7fb01814d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:48:06.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.594+0000 7fb017400640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0180a8bb0 0x7fb01814dda0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb00800c490 tx=0x7fb00800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.594+0000 7fb00f400640 1 -- 172.21.15.31:0/1606115183 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0080052d0 con 0x7fb0180a8bb0 2024-09-11T00:48:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.594+0000 7fb00f400640 1 -- 172.21.15.31:0/1606115183 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb008005490 con 0x7fb0180a8bb0 2024-09-11T00:48:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.594+0000 7fb00f400640 1 -- 172.21.15.31:0/1606115183 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008012890 con 0x7fb0180a8bb0 2024-09-11T00:48:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.594+0000 7fb01d200640 1 -- 172.21.15.31:0/1606115183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0180a9850 con 0x7fb0180a8bb0 2024-09-11T00:48:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.594+0000 7fb01d200640 1 -- 172.21.15.31:0/1606115183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb01814e2e0 con 0x7fb0180a8bb0 2024-09-11T00:48:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.595+0000 7fb01d200640 1 -- 172.21.15.31:0/1606115183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0180c7370 con 0x7fb0180a8bb0 2024-09-11T00:48:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.596+0000 7fb00f400640 1 -- 172.21.15.31:0/1606115183 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb008012a30 con 0x7fb0180a8bb0 2024-09-11T00:48:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.597+0000 7fb00f400640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faff8077f70 0x7faff807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.597+0000 7fb00f400640 1 -- 172.21.15.31:0/1606115183 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7fb00809a4c0 con 0x7fb0180a8bb0 2024-09-11T00:48:06.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.597+0000 7fb016a00640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faff8077f70 0x7faff807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:06.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.598+0000 7fb016a00640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faff8077f70 0x7faff807a430 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb004034000 tx=0x7fb004004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:06.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.601+0000 7fb00f400640 1 -- 172.21.15.31:0/1606115183 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb0180c7370 con 0x7fb0180a8bb0 2024-09-11T00:48:06.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.835+0000 7fb01d200640 1 -- 172.21.15.31:0/1606115183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb018070ce0 con 0x7fb0180a8bb0 2024-09-11T00:48:06.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.835+0000 7fb00f400640 1 -- 172.21.15.31:0/1606115183 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6542 (secure 0 0 0) 0x7fb018070ce0 con 0x7fb0180a8bb0 2024-09-11T00:48:06.838 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:06.838 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":132,"btime":"2024-09-11T00:48:04:788120+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:06.839 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 132 2024-09-11T00:48:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 -- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faff8077f70 msgr2=0x7faff807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faff8077f70 0x7faff807a430 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb004034000 tx=0x7fb004004730 comp rx=0 tx=0).stop 2024-09-11T00:48:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 -- 172.21.15.31:0/1606115183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0180a8bb0 msgr2=0x7fb01814dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0180a8bb0 0x7fb01814dda0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb00800c490 tx=0x7fb00800c950 comp rx=0 tx=0).stop 2024-09-11T00:48:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 -- 172.21.15.31:0/1606115183 shutdown_connections 2024-09-11T00:48:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faff8077f70 0x7faff807a430 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0181578e0 0x7fb01814d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0180c67f0 0x7fb01814ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 --2- 172.21.15.31:0/1606115183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0180a8bb0 0x7fb01814dda0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 -- 172.21.15.31:0/1606115183 >> 172.21.15.31:0/1606115183 conn(0x7fb0180c9af0 msgr2=0x7fb0180720f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 -- 172.21.15.31:0/1606115183 shutdown_connections 2024-09-11T00:48:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:06.838+0000 7fb01d200640 1 -- 172.21.15.31:0/1606115183 wait complete. 2024-09-11T00:48:06.857 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-11T00:48:06.857 INFO:tasks.mds_thrash.fs.[b]:waiting for 73 secs before thrashing 2024-09-11T00:48:07.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:07.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.925+0000 7f5ab4c00640 1 -- 172.21.15.31:0/3551287786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0153f20 msgr2=0x7f5ab01516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:07.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.925+0000 7f5ab4c00640 1 --2- 172.21.15.31:0/3551287786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0153f20 0x7f5ab01516e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00c4400 tx=0x7f5aa802ab40 comp rx=0 tx=0).stop 2024-09-11T00:48:07.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.925+0000 7f5ab4c00640 1 -- 172.21.15.31:0/3551287786 shutdown_connections 2024-09-11T00:48:07.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.925+0000 7f5ab4c00640 1 --2- 172.21.15.31:0/3551287786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0153f20 0x7f5ab01516e0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:07.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.925+0000 7f5ab4c00640 1 --2- 172.21.15.31:0/3551287786 >> v2:172.21.15.204:3300/0 conn(0x7f5ab0072480 0x7f5ab01511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:07.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.925+0000 7f5ab4c00640 1 -- 172.21.15.31:0/3551287786 >> 172.21.15.31:0/3551287786 conn(0x7f5ab00a9750 msgr2=0x7f5ab00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:07.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.925+0000 7f5ab4c00640 1 -- 172.21.15.31:0/3551287786 shutdown_connections 2024-09-11T00:48:07.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.925+0000 7f5ab4c00640 1 -- 172.21.15.31:0/3551287786 wait complete. 2024-09-11T00:48:07.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.927+0000 7f5ab4c00640 1 Processor -- start 2024-09-11T00:48:07.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.927+0000 7f5ab4c00640 1 -- start start 2024-09-11T00:48:07.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5ab4c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ab0072480 0x7f5ab00c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:07.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5ab4c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ab00c23a0 0x7f5ab00abba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:07.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5ab4c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0153f20 0x7f5ab00c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5ab4c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab0074200 con 0x7f5ab0153f20 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5ab4c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab0074080 con 0x7f5ab00c23a0 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5ab4c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab0072860 con 0x7f5ab0072480 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5aae800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0153f20 0x7f5ab00c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5aae800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0153f20 0x7f5ab00c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37268/0 (socket says 172.21.15.31:37268) 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5aae800640 1 -- 172.21.15.31:0/1509624389 learned_addr learned my addr 172.21.15.31:0/1509624389 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5aad400640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ab00c23a0 0x7f5ab00abba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.928+0000 7f5aade00640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ab0072480 0x7f5ab00c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5aae800640 1 -- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ab0072480 msgr2=0x7f5ab00c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5aae800640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ab0072480 0x7f5ab00c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5aae800640 1 -- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ab00c23a0 msgr2=0x7f5ab00abba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:07.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5aae800640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ab00c23a0 0x7f5ab00abba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:07.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5aae800640 1 -- 172.21.15.31:0/1509624389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab00ab4a0 con 0x7f5ab0153f20 2024-09-11T00:48:07.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5aade00640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ab0072480 0x7f5ab00c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:48:07.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5aae800640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0153f20 0x7f5ab00c6e10 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f5aa802b130 tx=0x7f5aa802c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:07.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5aa6000640 1 -- 172.21.15.31:0/1509624389 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8004250 con 0x7f5ab0153f20 2024-09-11T00:48:07.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5aa6000640 1 -- 172.21.15.31:0/1509624389 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5aa80043f0 con 0x7f5ab0153f20 2024-09-11T00:48:07.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5aa6000640 1 -- 172.21.15.31:0/1509624389 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8004800 con 0x7f5ab0153f20 2024-09-11T00:48:07.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5ab4c00640 1 -- 172.21.15.31:0/1509624389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab006f510 con 0x7f5ab0153f20 2024-09-11T00:48:07.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.929+0000 7f5ab4c00640 1 -- 172.21.15.31:0/1509624389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab01f3b80 con 0x7f5ab0153f20 2024-09-11T00:48:07.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.931+0000 7f5ab4c00640 1 -- 172.21.15.31:0/1509624389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab01511a0 con 0x7f5ab0153f20 2024-09-11T00:48:07.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.932+0000 7f5aa6000640 1 -- 172.21.15.31:0/1509624389 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5aa8049430 con 0x7f5ab0153f20 2024-09-11T00:48:07.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.933+0000 7f5aa6000640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a90077fa0 0x7f5a9007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:07.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.933+0000 7f5aa6000640 1 -- 172.21.15.31:0/1509624389 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7f5aa80bb6e0 con 0x7f5ab0153f20 2024-09-11T00:48:07.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.934+0000 7f5aade00640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a90077fa0 0x7f5a9007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:07.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.934+0000 7f5aade00640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a90077fa0 0x7f5a9007a460 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009c20 tx=0x7f5aa0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:07.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:07.941+0000 7f5aa6000640 1 -- 172.21.15.31:0/1509624389 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5aa8032030 con 0x7f5ab0153f20 2024-09-11T00:48:08.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.176+0000 7f5ab4c00640 1 -- 172.21.15.31:0/1509624389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ab0153170 con 0x7f5ab0153f20 2024-09-11T00:48:08.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.176+0000 7f5aa6000640 1 -- 172.21.15.31:0/1509624389 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6542 (secure 0 0 0) 0x7f5aa80835f0 con 0x7f5ab0153f20 2024-09-11T00:48:08.179 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:08.179 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":132,"btime":"2024-09-11T00:48:04:788120+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:08.179 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 132 2024-09-11T00:48:08.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.178+0000 7f5ab4c00640 1 -- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a90077fa0 msgr2=0x7f5a9007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:08.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.178+0000 7f5ab4c00640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a90077fa0 0x7f5a9007a460 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009c20 tx=0x7f5aa0005f90 comp rx=0 tx=0).stop 2024-09-11T00:48:08.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.178+0000 7f5ab4c00640 1 -- 172.21.15.31:0/1509624389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0153f20 msgr2=0x7f5ab00c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:08.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.178+0000 7f5ab4c00640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0153f20 0x7f5ab00c6e10 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f5aa802b130 tx=0x7f5aa802c8b0 comp rx=0 tx=0).stop 2024-09-11T00:48:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.178+0000 7f5ab4c00640 1 -- 172.21.15.31:0/1509624389 shutdown_connections 2024-09-11T00:48:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.178+0000 7f5ab4c00640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a90077fa0 0x7f5a9007a460 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.178+0000 7f5ab4c00640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab0153f20 0x7f5ab00c6e10 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.178+0000 7f5ab4c00640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ab00c23a0 0x7f5ab00abba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.178+0000 7f5ab4c00640 1 --2- 172.21.15.31:0/1509624389 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ab0072480 0x7f5ab00c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.178+0000 7f5ab4c00640 1 -- 172.21.15.31:0/1509624389 >> 172.21.15.31:0/1509624389 conn(0x7f5ab00a9750 msgr2=0x7f5ab014ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.179+0000 7f5ab4c00640 1 -- 172.21.15.31:0/1509624389 shutdown_connections 2024-09-11T00:48:08.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:08.179+0000 7f5ab4c00640 1 -- 172.21.15.31:0/1509624389 wait complete. 2024-09-11T00:48:08.197 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:48:10.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:10.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.369+0000 7f46d0e00640 1 -- 172.21.15.31:0/272217837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc157c40 msgr2=0x7f46cc15e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:10.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.369+0000 7f46d0e00640 1 --2- 172.21.15.31:0/272217837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc157c40 0x7f46cc15e8c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0774a0 tx=0x7f46c402a820 comp rx=0 tx=0).stop 2024-09-11T00:48:10.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.369+0000 7f46d0e00640 1 -- 172.21.15.31:0/272217837 shutdown_connections 2024-09-11T00:48:10.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.369+0000 7f46d0e00640 1 --2- 172.21.15.31:0/272217837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc157c40 0x7f46cc15e8c0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:10.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.369+0000 7f46d0e00640 1 --2- 172.21.15.31:0/272217837 >> v2:172.21.15.204:3300/0 conn(0x7f46cc0c44f0 0x7f46cc0c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:10.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.369+0000 7f46d0e00640 1 -- 172.21.15.31:0/272217837 >> 172.21.15.31:0/272217837 conn(0x7f46cc0ab8a0 msgr2=0x7f46cc0abcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:10.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.369+0000 7f46d0e00640 1 -- 172.21.15.31:0/272217837 shutdown_connections 2024-09-11T00:48:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.369+0000 7f46d0e00640 1 -- 172.21.15.31:0/272217837 wait complete. 2024-09-11T00:48:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.371+0000 7f46d0e00640 1 Processor -- start 2024-09-11T00:48:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.371+0000 7f46d0e00640 1 -- start start 2024-09-11T00:48:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46d0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c44f0 0x7f46cc157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46d0e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46cc0c4e10 0x7f46cc0aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46d0e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc157c40 0x7f46cc14e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46d0e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc151af0 con 0x7f46cc0c44f0 2024-09-11T00:48:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46d0e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc0c54e0 con 0x7f46cc0c4e10 2024-09-11T00:48:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46d0e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc0701b0 con 0x7f46cc157c40 2024-09-11T00:48:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c44f0 0x7f46cc157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46c9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c44f0 0x7f46cc157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37294/0 (socket says 172.21.15.31:37294) 2024-09-11T00:48:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46c9e00640 1 -- 172.21.15.31:0/2361244623 learned_addr learned my addr 172.21.15.31:0/2361244623 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46ca800640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc157c40 0x7f46cc14e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46c9400640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46cc0c4e10 0x7f46cc0aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46c9e00640 1 -- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc157c40 msgr2=0x7f46cc14e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46c9e00640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc157c40 0x7f46cc14e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46c9e00640 1 -- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46cc0c4e10 msgr2=0x7f46cc0aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46c9e00640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46cc0c4e10 0x7f46cc0aa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.372+0000 7f46c9e00640 1 -- 172.21.15.31:0/2361244623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc14eed0 con 0x7f46cc0c44f0 2024-09-11T00:48:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.373+0000 7f46c9e00640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c44f0 0x7f46cc157160 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00e9f0 tx=0x7f46bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.373+0000 7f46c2000640 1 -- 172.21.15.31:0/2361244623 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc00cd80 con 0x7f46cc0c44f0 2024-09-11T00:48:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.373+0000 7f46c2000640 1 -- 172.21.15.31:0/2361244623 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46bc010430 con 0x7f46cc0c44f0 2024-09-11T00:48:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.373+0000 7f46d0e00640 1 -- 172.21.15.31:0/2361244623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc1ec0b0 con 0x7f46cc0c44f0 2024-09-11T00:48:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.373+0000 7f46c2000640 1 -- 172.21.15.31:0/2361244623 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc010840 con 0x7f46cc0c44f0 2024-09-11T00:48:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.373+0000 7f46d0e00640 1 -- 172.21.15.31:0/2361244623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc1ec670 con 0x7f46cc0c44f0 2024-09-11T00:48:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.374+0000 7f46d0e00640 1 -- 172.21.15.31:0/2361244623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46cc157700 con 0x7f46cc0c44f0 2024-09-11T00:48:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.375+0000 7f46c2000640 1 -- 172.21.15.31:0/2361244623 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f46bc0109e0 con 0x7f46cc0c44f0 2024-09-11T00:48:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.376+0000 7f46c2000640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46ac077fa0 0x7f46ac07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.376+0000 7f46c2000640 1 -- 172.21.15.31:0/2361244623 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7f46bc09b430 con 0x7f46cc0c44f0 2024-09-11T00:48:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.376+0000 7f46c9400640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46ac077fa0 0x7f46ac07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.377+0000 7f46c9400640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46ac077fa0 0x7f46ac07a460 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f46cc14f880 tx=0x7f46b8007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.380+0000 7f46c2000640 1 -- 172.21.15.31:0/2361244623 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f46bc062db0 con 0x7f46cc0c44f0 2024-09-11T00:48:10.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.620+0000 7f46d0e00640 1 -- 172.21.15.31:0/2361244623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46cc0a9380 con 0x7f46cc0c44f0 2024-09-11T00:48:10.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.620+0000 7f46c2000640 1 -- 172.21.15.31:0/2361244623 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6542 (secure 0 0 0) 0x7f46bc0677e0 con 0x7f46cc0c44f0 2024-09-11T00:48:10.623 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:10.623 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":132,"btime":"2024-09-11T00:48:04:788120+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:10.624 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 132 2024-09-11T00:48:10.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 -- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46ac077fa0 msgr2=0x7f46ac07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:10.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46ac077fa0 0x7f46ac07a460 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f46cc14f880 tx=0x7f46b8007e90 comp rx=0 tx=0).stop 2024-09-11T00:48:10.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 -- 172.21.15.31:0/2361244623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c44f0 msgr2=0x7f46cc157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:10.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c44f0 0x7f46cc157160 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00e9f0 tx=0x7f46bc00eeb0 comp rx=0 tx=0).stop 2024-09-11T00:48:10.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 -- 172.21.15.31:0/2361244623 shutdown_connections 2024-09-11T00:48:10.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46ac077fa0 0x7f46ac07a460 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:10.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46cc157c40 0x7f46cc14e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:10.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46cc0c4e10 0x7f46cc0aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:10.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 --2- 172.21.15.31:0/2361244623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46cc0c44f0 0x7f46cc157160 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:10.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 -- 172.21.15.31:0/2361244623 >> 172.21.15.31:0/2361244623 conn(0x7f46cc0ab8a0 msgr2=0x7f46cc1dffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:10.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 -- 172.21.15.31:0/2361244623 shutdown_connections 2024-09-11T00:48:10.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:10.623+0000 7f46d0e00640 1 -- 172.21.15.31:0/2361244623 wait complete. 2024-09-11T00:48:10.642 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:48:12.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:12.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.819+0000 7f358c400640 1 -- 172.21.15.31:0/491027158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c23a0 msgr2=0x7f35841516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:12.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.819+0000 7f358c400640 1 --2- 172.21.15.31:0/491027158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c23a0 0x7f35841516e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3570005fd0 tx=0x7f357002a670 comp rx=0 tx=0).stop 2024-09-11T00:48:12.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.819+0000 7f358c400640 1 -- 172.21.15.31:0/491027158 shutdown_connections 2024-09-11T00:48:12.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.819+0000 7f358c400640 1 --2- 172.21.15.31:0/491027158 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35840c23a0 0x7f35841516e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:12.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.819+0000 7f358c400640 1 --2- 172.21.15.31:0/491027158 >> v2:172.21.15.204:3300/0 conn(0x7f3584072480 0x7f35841511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:12.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.819+0000 7f358c400640 1 -- 172.21.15.31:0/491027158 >> 172.21.15.31:0/491027158 conn(0x7f35840a9750 msgr2=0x7f35840a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:12.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.819+0000 7f358c400640 1 -- 172.21.15.31:0/491027158 shutdown_connections 2024-09-11T00:48:12.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.819+0000 7f358c400640 1 -- 172.21.15.31:0/491027158 wait complete. 2024-09-11T00:48:12.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.820+0000 7f358c400640 1 Processor -- start 2024-09-11T00:48:12.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.821+0000 7f358c400640 1 -- start start 2024-09-11T00:48:12.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.821+0000 7f358c400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584072480 0x7f358414be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:12.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.821+0000 7f358c400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c23a0 0x7f35840c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:12.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.821+0000 7f358c400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3584153f20 0x7f35841527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.821+0000 7f358c400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3584074200 con 0x7f35840c23a0 2024-09-11T00:48:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.821+0000 7f358c400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3584074080 con 0x7f3584072480 2024-09-11T00:48:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.821+0000 7f358c400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3584072860 con 0x7f3584153f20 2024-09-11T00:48:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f3589000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c23a0 0x7f35840c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f3589a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584072480 0x7f358414be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f358a400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3584153f20 0x7f35841527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f3589000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c23a0 0x7f35840c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37302/0 (socket says 172.21.15.31:37302) 2024-09-11T00:48:12.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f3589000640 1 -- 172.21.15.31:0/2467619517 learned_addr learned my addr 172.21.15.31:0/2467619517 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:12.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f3589a00640 1 -- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3584153f20 msgr2=0x7f35841527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:12.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f3589a00640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3584153f20 0x7f35841527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:12.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f3589a00640 1 -- 172.21.15.31:0/2467619517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c23a0 msgr2=0x7f35840c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:12.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f3589a00640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c23a0 0x7f35840c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:12.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f3589a00640 1 -- 172.21.15.31:0/2467619517 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35840c74a0 con 0x7f3584072480 2024-09-11T00:48:12.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.822+0000 7f3589000640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c23a0 0x7f35840c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:48:12.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.823+0000 7f3589a00640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584072480 0x7f358414be90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f357800e9f0 tx=0x7f357800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:12.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.823+0000 7f3581600640 1 -- 172.21.15.31:0/2467619517 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357800ccc0 con 0x7f3584072480 2024-09-11T00:48:12.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.823+0000 7f3581600640 1 -- 172.21.15.31:0/2467619517 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3578010430 con 0x7f3584072480 2024-09-11T00:48:12.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.823+0000 7f358c400640 1 -- 172.21.15.31:0/2467619517 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35841dda00 con 0x7f3584072480 2024-09-11T00:48:12.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.823+0000 7f3581600640 1 -- 172.21.15.31:0/2467619517 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3578010860 con 0x7f3584072480 2024-09-11T00:48:12.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.823+0000 7f358c400640 1 -- 172.21.15.31:0/2467619517 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35841dd690 con 0x7f3584072480 2024-09-11T00:48:12.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.824+0000 7f358c400640 1 -- 172.21.15.31:0/2467619517 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35840c3430 con 0x7f3584072480 2024-09-11T00:48:12.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.825+0000 7f3581600640 1 -- 172.21.15.31:0/2467619517 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3578010a00 con 0x7f3584072480 2024-09-11T00:48:12.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.826+0000 7f3581600640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564077f70 0x7f356407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:12.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.826+0000 7f3581600640 1 -- 172.21.15.31:0/2467619517 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8730+0+0 (secure 0 0 0) 0x7f357809b340 con 0x7f3584072480 2024-09-11T00:48:12.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.827+0000 7f3589000640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564077f70 0x7f356407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:12.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.827+0000 7f3589000640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564077f70 0x7f356407a430 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f358414f5f0 tx=0x7f3570004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:12.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:12.830+0000 7f3581600640 1 -- 172.21.15.31:0/2467619517 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f35840c3430 con 0x7f3584072480 2024-09-11T00:48:13.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.073+0000 7f358c400640 1 -- 172.21.15.31:0/2467619517 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35841511a0 con 0x7f3584072480 2024-09-11T00:48:13.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.074+0000 7f3581600640 1 -- 172.21.15.31:0/2467619517 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+6542 (secure 0 0 0) 0x7f3578062c00 con 0x7f3584072480 2024-09-11T00:48:13.076 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:13.077 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":132,"btime":"2024-09-11T00:48:04:788120+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":132}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:47:22.793059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"c","rank":0,"incarnation":123,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/2163558126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2163558126},{"type":"v1","addr":"172.21.15.31:6835","nonce":2163558126}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:13.077 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 132 2024-09-11T00:48:13.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.076+0000 7f358c400640 1 -- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564077f70 msgr2=0x7f356407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:13.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.076+0000 7f358c400640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564077f70 0x7f356407a430 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f358414f5f0 tx=0x7f3570004730 comp rx=0 tx=0).stop 2024-09-11T00:48:13.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.076+0000 7f358c400640 1 -- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584072480 msgr2=0x7f358414be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:13.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.076+0000 7f358c400640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584072480 0x7f358414be90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f357800e9f0 tx=0x7f357800eeb0 comp rx=0 tx=0).stop 2024-09-11T00:48:13.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.076+0000 7f358c400640 1 -- 172.21.15.31:0/2467619517 shutdown_connections 2024-09-11T00:48:13.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.076+0000 7f358c400640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3564077f70 0x7f356407a430 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:13.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.076+0000 7f358c400640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3584153f20 0x7f35841527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:13.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.076+0000 7f358c400640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35840c23a0 0x7f35840c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:13.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.076+0000 7f358c400640 1 --2- 172.21.15.31:0/2467619517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3584072480 0x7f358414be90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:13.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.076+0000 7f358c400640 1 -- 172.21.15.31:0/2467619517 >> 172.21.15.31:0/2467619517 conn(0x7f35840a9750 msgr2=0x7f358414cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:13.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.077+0000 7f358c400640 1 -- 172.21.15.31:0/2467619517 shutdown_connections 2024-09-11T00:48:13.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:13.077+0000 7f358c400640 1 -- 172.21.15.31:0/2467619517 wait complete. 2024-09-11T00:48:13.095 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:48:15.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:15.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.258+0000 7f41e0a00640 1 -- 172.21.15.31:0/3300400902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41dc0c5aa0 msgr2=0x7f41dc0c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:15.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.258+0000 7f41e0a00640 1 --2- 172.21.15.31:0/3300400902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41dc0c5aa0 0x7f41dc0c7150 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f41cc005fd0 tx=0x7f41cc02a5d0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.258+0000 7f41e0a00640 1 -- 172.21.15.31:0/3300400902 shutdown_connections 2024-09-11T00:48:15.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.258+0000 7f41e0a00640 1 --2- 172.21.15.31:0/3300400902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41dc0c5aa0 0x7f41dc0c7150 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.258+0000 7f41e0a00640 1 -- 172.21.15.31:0/3300400902 >> 172.21.15.31:0/3300400902 conn(0x7f41dc06caf0 msgr2=0x7f41dc06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:15.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.258+0000 7f41e0a00640 1 -- 172.21.15.31:0/3300400902 shutdown_connections 2024-09-11T00:48:15.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.258+0000 7f41e0a00640 1 -- 172.21.15.31:0/3300400902 wait complete. 2024-09-11T00:48:15.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.260+0000 7f41e0a00640 1 Processor -- start 2024-09-11T00:48:15.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.260+0000 7f41e0a00640 1 -- start start 2024-09-11T00:48:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41e0a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41dc0c5aa0 0x7f41dc074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41e0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41dc0c6420 0x7f41dc1ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41e0a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41dc156350 0x7f41dc1ec4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41e0a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc0c3160 con 0x7f41dc0c6420 2024-09-11T00:48:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41e0a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc0c2e80 con 0x7f41dc0c5aa0 2024-09-11T00:48:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41e0a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f41dc153a20 con 0x7f41dc156350 2024-09-11T00:48:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41db400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41dc0c5aa0 0x7f41dc074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41daa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41dc0c6420 0x7f41dc1ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:15.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41db400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41dc0c5aa0 0x7f41dc074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:55084/0 (socket says 172.21.15.31:55084) 2024-09-11T00:48:15.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41db400640 1 -- 172.21.15.31:0/2947281860 learned_addr learned my addr 172.21.15.31:0/2947281860 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:15.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.261+0000 7f41dbe00640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41dc156350 0x7f41dc1ec4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.262+0000 7f41daa00640 1 -- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41dc156350 msgr2=0x7f41dc1ec4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.262+0000 7f41daa00640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41dc156350 0x7f41dc1ec4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.262+0000 7f41daa00640 1 -- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41dc0c5aa0 msgr2=0x7f41dc074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.262+0000 7f41daa00640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41dc0c5aa0 0x7f41dc074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.262+0000 7f41daa00640 1 -- 172.21.15.31:0/2947281860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41dc14d320 con 0x7f41dc0c6420 2024-09-11T00:48:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.262+0000 7f41db400640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41dc0c5aa0 0x7f41dc074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:48:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.262+0000 7f41daa00640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41dc0c6420 0x7f41dc1ebf80 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c3f0 tx=0x7f41c800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.263+0000 7f41d3400640 1 -- 172.21.15.31:0/2947281860 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c8017070 con 0x7f41dc0c6420 2024-09-11T00:48:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.263+0000 7f41d3400640 1 -- 172.21.15.31:0/2947281860 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41c80053c0 con 0x7f41dc0c6420 2024-09-11T00:48:15.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.263+0000 7f41e0a00640 1 -- 172.21.15.31:0/2947281860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41dc1e1890 con 0x7f41dc0c6420 2024-09-11T00:48:15.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.263+0000 7f41d3400640 1 -- 172.21.15.31:0/2947281860 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c8003850 con 0x7f41dc0c6420 2024-09-11T00:48:15.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.263+0000 7f41e0a00640 1 -- 172.21.15.31:0/2947281860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41dc1ecda0 con 0x7f41dc0c6420 2024-09-11T00:48:15.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.264+0000 7f41e0a00640 1 -- 172.21.15.31:0/2947281860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41dc0c1f00 con 0x7f41dc0c6420 2024-09-11T00:48:15.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.265+0000 7f41d3400640 1 -- 172.21.15.31:0/2947281860 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f41c80039f0 con 0x7f41dc0c6420 2024-09-11T00:48:15.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.267+0000 7f41d3400640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f41bc077f70 0x7f41bc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:15.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.267+0000 7f41d3400640 1 -- 172.21.15.31:0/2947281860 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7f41c8099f00 con 0x7f41dc0c6420 2024-09-11T00:48:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.267+0000 7f41db400640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f41bc077f70 0x7f41bc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:15.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.268+0000 7f41db400640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f41bc077f70 0x7f41bc07a430 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f41dc152300 tx=0x7f41cc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:15.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.271+0000 7f41d3400640 1 -- 172.21.15.31:0/2947281860 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f41dc0c1f00 con 0x7f41dc0c6420 2024-09-11T00:48:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.502+0000 7f41e0a00640 1 -- 172.21.15.31:0/2947281860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41dc0a9000 con 0x7f41dc0c6420 2024-09-11T00:48:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.503+0000 7f41d3400640 1 -- 172.21.15.31:0/2947281860 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) ==== 95+0+5708 (secure 0 0 0) 0x7f41dc0a9000 con 0x7f41dc0c6420 2024-09-11T00:48:15.505 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:15.505 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":133,"btime":"2024-09-11T00:48:14:453450+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:14.453444+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 133 2024-09-11T00:48:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.505+0000 7f41e0a00640 1 -- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f41bc077f70 msgr2=0x7f41bc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.505+0000 7f41e0a00640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f41bc077f70 0x7f41bc07a430 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f41dc152300 tx=0x7f41cc004730 comp rx=0 tx=0).stop 2024-09-11T00:48:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.505+0000 7f41e0a00640 1 -- 172.21.15.31:0/2947281860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41dc0c6420 msgr2=0x7f41dc1ebf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.505+0000 7f41e0a00640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41dc0c6420 0x7f41dc1ebf80 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c3f0 tx=0x7f41c800c8b0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.506+0000 7f41e0a00640 1 -- 172.21.15.31:0/2947281860 shutdown_connections 2024-09-11T00:48:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.506+0000 7f41e0a00640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f41bc077f70 0x7f41bc07a430 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.506+0000 7f41e0a00640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41dc156350 0x7f41dc1ec4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.506+0000 7f41e0a00640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41dc0c6420 0x7f41dc1ebf80 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.506+0000 7f41e0a00640 1 --2- 172.21.15.31:0/2947281860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41dc0c5aa0 0x7f41dc074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.506+0000 7f41e0a00640 1 -- 172.21.15.31:0/2947281860 >> 172.21.15.31:0/2947281860 conn(0x7f41dc06caf0 msgr2=0x7f41dc14fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.506+0000 7f41e0a00640 1 -- 172.21.15.31:0/2947281860 shutdown_connections 2024-09-11T00:48:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.506+0000 7f41e0a00640 1 -- 172.21.15.31:0/2947281860 wait complete. 2024-09-11T00:48:15.524 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-11T00:48:15.524 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:48:15.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.683+0000 7f795cc00640 1 -- 172.21.15.31:0/1385983688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580a6e20 msgr2=0x7f79581510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.683+0000 7f795cc00640 1 --2- 172.21.15.31:0/1385983688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580a6e20 0x7f79581510c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f7940005fd0 tx=0x7f794002a4b0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.683+0000 7f795cc00640 1 -- 172.21.15.31:0/1385983688 shutdown_connections 2024-09-11T00:48:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.683+0000 7f795cc00640 1 --2- 172.21.15.31:0/1385983688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580a6e20 0x7f79581510c0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.683+0000 7f795cc00640 1 --2- 172.21.15.31:0/1385983688 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79580ac2c0 0x7f79580ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.683+0000 7f795cc00640 1 -- 172.21.15.31:0/1385983688 >> 172.21.15.31:0/1385983688 conn(0x7f795806caf0 msgr2=0x7f795806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.683+0000 7f795cc00640 1 -- 172.21.15.31:0/1385983688 shutdown_connections 2024-09-11T00:48:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.683+0000 7f795cc00640 1 -- 172.21.15.31:0/1385983688 wait complete. 2024-09-11T00:48:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.685+0000 7f795cc00640 1 Processor -- start 2024-09-11T00:48:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.685+0000 7f795cc00640 1 -- start start 2024-09-11T00:48:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.685+0000 7f795cc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580a6e20 0x7f79580c0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f795cc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580ac2c0 0x7f79580c2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f795cc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79580acbe0 0x7f7958143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f795cc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7958151780 con 0x7f79580ac2c0 2024-09-11T00:48:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f795cc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7958151600 con 0x7f79580a6e20 2024-09-11T00:48:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f795cc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7958151900 con 0x7f79580acbe0 2024-09-11T00:48:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f7957400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580a6e20 0x7f79580c0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f794ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580ac2c0 0x7f79580c2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f7957400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580a6e20 0x7f79580c0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:55096/0 (socket says 172.21.15.31:55096) 2024-09-11T00:48:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f7957400640 1 -- 172.21.15.31:0/3901208705 learned_addr learned my addr 172.21.15.31:0/3901208705 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f7957e00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79580acbe0 0x7f7958143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f7957400640 1 -- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79580acbe0 msgr2=0x7f7958143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f7957400640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79580acbe0 0x7f7958143d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f7957400640 1 -- 172.21.15.31:0/3901208705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580ac2c0 msgr2=0x7f79580c2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f7957400640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580ac2c0 0x7f79580c2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f7957400640 1 -- 172.21.15.31:0/3901208705 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79581dd430 con 0x7f79580a6e20 2024-09-11T00:48:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.686+0000 7f794ea00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580ac2c0 0x7f79580c2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:48:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.687+0000 7f7957400640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580a6e20 0x7f79580c0fd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f794800d950 tx=0x7f794800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.687+0000 7f794fe00640 1 -- 172.21.15.31:0/3901208705 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948017070 con 0x7f79580a6e20 2024-09-11T00:48:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.687+0000 7f794fe00640 1 -- 172.21.15.31:0/3901208705 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7948009d20 con 0x7f79580a6e20 2024-09-11T00:48:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.687+0000 7f794fe00640 1 -- 172.21.15.31:0/3901208705 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79480106c0 con 0x7f79580a6e20 2024-09-11T00:48:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.687+0000 7f795cc00640 1 -- 172.21.15.31:0/3901208705 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79581dd710 con 0x7f79580a6e20 2024-09-11T00:48:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.687+0000 7f795cc00640 1 -- 172.21.15.31:0/3901208705 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79580c1510 con 0x7f79580a6e20 2024-09-11T00:48:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.688+0000 7f795cc00640 1 -- 172.21.15.31:0/3901208705 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79581532b0 con 0x7f79580a6e20 2024-09-11T00:48:15.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.689+0000 7f794fe00640 1 -- 172.21.15.31:0/3901208705 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7948010860 con 0x7f79580a6e20 2024-09-11T00:48:15.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.690+0000 7f794fe00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7934077f70 0x7f793407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:15.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.690+0000 7f794fe00640 1 -- 172.21.15.31:0/3901208705 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7f794809ad10 con 0x7f79580a6e20 2024-09-11T00:48:15.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.691+0000 7f794ea00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7934077f70 0x7f793407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:15.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.691+0000 7f794ea00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7934077f70 0x7f793407a430 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7940033000 tx=0x7f7940004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:15.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.696+0000 7f794fe00640 1 -- 172.21.15.31:0/3901208705 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f79581532b0 con 0x7f79580a6e20 2024-09-11T00:48:15.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.930+0000 7f795cc00640 1 -- 172.21.15.31:0/3901208705 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79580c2950 con 0x7f79580a6e20 2024-09-11T00:48:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.931+0000 7f794fe00640 1 -- 172.21.15.31:0/3901208705 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5711 (secure 0 0 0) 0x7f79480a0020 con 0x7f79580a6e20 2024-09-11T00:48:15.932 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:15.932 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":134,"btime":"2024-09-11T00:48:15:820954+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:15.020500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:15.933 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 134 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 -- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7934077f70 msgr2=0x7f793407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7934077f70 0x7f793407a430 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7940033000 tx=0x7f7940004730 comp rx=0 tx=0).stop 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 -- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580a6e20 msgr2=0x7f79580c0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580a6e20 0x7f79580c0fd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f794800d950 tx=0x7f794800de10 comp rx=0 tx=0).stop 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 -- 172.21.15.31:0/3901208705 shutdown_connections 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7934077f70 0x7f793407a430 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79580acbe0 0x7f7958143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79580ac2c0 0x7f79580c2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 --2- 172.21.15.31:0/3901208705 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79580a6e20 0x7f79580c0fd0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 -- 172.21.15.31:0/3901208705 >> 172.21.15.31:0/3901208705 conn(0x7f795806caf0 msgr2=0x7f795814f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 -- 172.21.15.31:0/3901208705 shutdown_connections 2024-09-11T00:48:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:15.933+0000 7f795cc00640 1 -- 172.21.15.31:0/3901208705 wait complete. 2024-09-11T00:48:15.951 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:48:15:820954+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 134, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 134, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6934": { "addr": "172.21.15.204:6837/4164016077", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 4164016077, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 4164016077, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6934, "incarnation": 133, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:48:15.020500+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6934 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 131, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6829": { "addr": "172.21.15.204:6835/634556689", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 634556689, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 634556689, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6829, "incarnation": 128, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:48:02.775671+0000", "qdb_cluster": [ 6829 ], "qdb_leader": 6829, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6829 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:48:17.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:18.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.122+0000 7fd636000640 1 -- 172.21.15.31:0/4084289335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6300cc220 msgr2=0x7fd63006e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:18.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.122+0000 7fd636000640 1 --2- 172.21.15.31:0/4084289335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6300cc220 0x7fd63006e680 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd630153d20 tx=0x7fd62802aa10 comp rx=0 tx=0).stop 2024-09-11T00:48:18.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.122+0000 7fd636000640 1 -- 172.21.15.31:0/4084289335 shutdown_connections 2024-09-11T00:48:18.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.122+0000 7fd636000640 1 --2- 172.21.15.31:0/4084289335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6300cc220 0x7fd63006e680 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:18.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.122+0000 7fd636000640 1 --2- 172.21.15.31:0/4084289335 >> v2:172.21.15.204:3300/0 conn(0x7fd630156960 0x7fd63006e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:18.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.122+0000 7fd636000640 1 -- 172.21.15.31:0/4084289335 >> 172.21.15.31:0/4084289335 conn(0x7fd630068f10 msgr2=0x7fd6300cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:18.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.122+0000 7fd636000640 1 -- 172.21.15.31:0/4084289335 shutdown_connections 2024-09-11T00:48:18.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.122+0000 7fd636000640 1 -- 172.21.15.31:0/4084289335 wait complete. 2024-09-11T00:48:18.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.124+0000 7fd636000640 1 Processor -- start 2024-09-11T00:48:18.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.124+0000 7fd636000640 1 -- start start 2024-09-11T00:48:18.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.125+0000 7fd636000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6300cc220 0x7fd63014eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:18.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.125+0000 7fd636000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd630150020 0x7fd63014f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:18.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd636000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd630156960 0x7fd6300bf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:18.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd636000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd6300c6ca0 con 0x7fd630150020 2024-09-11T00:48:18.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd636000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd6300c6b20 con 0x7fd630156960 2024-09-11T00:48:18.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd636000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd6301573d0 con 0x7fd6300cc220 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd62fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd630150020 0x7fd63014f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd62fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd630150020 0x7fd63014f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51436/0 (socket says 172.21.15.31:51436) 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd62fe00640 1 -- 172.21.15.31:0/760981829 learned_addr learned my addr 172.21.15.31:0/760981829 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd635600640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd630156960 0x7fd6300bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd634c00640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6300cc220 0x7fd63014eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd62fe00640 1 -- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6300cc220 msgr2=0x7fd63014eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd62fe00640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6300cc220 0x7fd63014eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd62fe00640 1 -- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd630156960 msgr2=0x7fd6300bf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd62fe00640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd630156960 0x7fd6300bf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd62fe00640 1 -- 172.21.15.31:0/760981829 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6301ecb80 con 0x7fd630150020 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.126+0000 7fd634c00640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6300cc220 0x7fd63014eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:48:18.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.127+0000 7fd62fe00640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd630150020 0x7fd63014f410 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd61c00e7c0 tx=0x7fd61c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:18.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.127+0000 7fd62cc00640 1 -- 172.21.15.31:0/760981829 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c00ccc0 con 0x7fd630150020 2024-09-11T00:48:18.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.127+0000 7fd62cc00640 1 -- 172.21.15.31:0/760981829 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd61c010430 con 0x7fd630150020 2024-09-11T00:48:18.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.127+0000 7fd62cc00640 1 -- 172.21.15.31:0/760981829 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c010860 con 0x7fd630150020 2024-09-11T00:48:18.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.127+0000 7fd636000640 1 -- 172.21.15.31:0/760981829 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6300afab0 con 0x7fd630150020 2024-09-11T00:48:18.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.127+0000 7fd636000640 1 -- 172.21.15.31:0/760981829 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6301ecdf0 con 0x7fd630150020 2024-09-11T00:48:18.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.128+0000 7fd636000640 1 -- 172.21.15.31:0/760981829 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6300ad4e0 con 0x7fd630150020 2024-09-11T00:48:18.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.129+0000 7fd62cc00640 1 -- 172.21.15.31:0/760981829 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd61c010a00 con 0x7fd630150020 2024-09-11T00:48:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.130+0000 7fd62cc00640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd610077f70 0x7fd61007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.131+0000 7fd62cc00640 1 -- 172.21.15.31:0/760981829 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7fd61c09b2a0 con 0x7fd630150020 2024-09-11T00:48:18.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.131+0000 7fd634c00640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd610077f70 0x7fd61007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:18.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.132+0000 7fd634c00640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd610077f70 0x7fd61007a430 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd624007fd0 tx=0x7fd624009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:18.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.135+0000 7fd62cc00640 1 -- 172.21.15.31:0/760981829 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd61c062b70 con 0x7fd630150020 2024-09-11T00:48:18.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.465+0000 7fd636000640 1 -- 172.21.15.31:0/760981829 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6300b1820 con 0x7fd630150020 2024-09-11T00:48:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.466+0000 7fd62cc00640 1 -- 172.21.15.31:0/760981829 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+5715 (secure 0 0 0) 0x7fd61c0675a0 con 0x7fd630150020 2024-09-11T00:48:18.468 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:18.468 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":136,"btime":"2024-09-11T00:48:17:824502+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:18.468 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 136 2024-09-11T00:48:18.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.468+0000 7fd636000640 1 -- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd610077f70 msgr2=0x7fd61007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.468+0000 7fd636000640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd610077f70 0x7fd61007a430 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd624007fd0 tx=0x7fd624009040 comp rx=0 tx=0).stop 2024-09-11T00:48:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.468+0000 7fd636000640 1 -- 172.21.15.31:0/760981829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd630150020 msgr2=0x7fd63014f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.468+0000 7fd636000640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd630150020 0x7fd63014f410 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd61c00e7c0 tx=0x7fd61c00ec80 comp rx=0 tx=0).stop 2024-09-11T00:48:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.468+0000 7fd636000640 1 -- 172.21.15.31:0/760981829 shutdown_connections 2024-09-11T00:48:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.468+0000 7fd636000640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd610077f70 0x7fd61007a430 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.468+0000 7fd636000640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd630156960 0x7fd6300bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.468+0000 7fd636000640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd630150020 0x7fd63014f410 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.468+0000 7fd636000640 1 --2- 172.21.15.31:0/760981829 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6300cc220 0x7fd63014eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:18.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.468+0000 7fd636000640 1 -- 172.21.15.31:0/760981829 >> 172.21.15.31:0/760981829 conn(0x7fd630068f10 msgr2=0x7fd6301549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.469+0000 7fd636000640 1 -- 172.21.15.31:0/760981829 shutdown_connections 2024-09-11T00:48:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:18.469+0000 7fd636000640 1 -- 172.21.15.31:0/760981829 wait complete. 2024-09-11T00:48:18.488 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=6804 2024-09-11T00:48:18.488 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.c 2024-09-11T00:48:18.891 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:48:18.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:48:19.489 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-11T00:48:19.489 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-11T00:48:19.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-11T00:48:19.493 INFO:tasks.ceph.mds.c:Started 2024-09-11T00:48:19.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:48:19.650 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-11T00:48:19.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.679+0000 7f0691a00640 1 -- 172.21.15.31:0/2331030488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f068c0c23a0 msgr2=0x7f068c0c2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:19.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.679+0000 7f0691a00640 1 --2- 172.21.15.31:0/2331030488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f068c0c23a0 0x7f068c0c2850 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f0684009be0 tx=0x7f068402a850 comp rx=0 tx=0).stop 2024-09-11T00:48:19.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.680+0000 7f0691a00640 1 -- 172.21.15.31:0/2331030488 shutdown_connections 2024-09-11T00:48:19.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.680+0000 7f0691a00640 1 --2- 172.21.15.31:0/2331030488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f068c0c23a0 0x7f068c0c2850 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:19.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.680+0000 7f0691a00640 1 --2- 172.21.15.31:0/2331030488 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f068c06fa70 0x7f068c0c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:19.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.680+0000 7f0691a00640 1 -- 172.21.15.31:0/2331030488 >> 172.21.15.31:0/2331030488 conn(0x7f068c0a9750 msgr2=0x7f068c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:19.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.680+0000 7f0691a00640 1 -- 172.21.15.31:0/2331030488 shutdown_connections 2024-09-11T00:48:19.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.680+0000 7f0691a00640 1 -- 172.21.15.31:0/2331030488 wait complete. 2024-09-11T00:48:19.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.682+0000 7f0691a00640 1 Processor -- start 2024-09-11T00:48:19.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.682+0000 7f0691a00640 1 -- start start 2024-09-11T00:48:19.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.683+0000 7f0691a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f068c06fa70 0x7f068c150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:19.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.683+0000 7f0691a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f068c0c23a0 0x7f068c0ac480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:19.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.683+0000 7f0691a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f068c156300 0x7f068c0745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:19.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.683+0000 7f0691a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f068c0c77a0 con 0x7f068c06fa70 2024-09-11T00:48:19.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.683+0000 7f0691a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f068c0c7620 con 0x7f068c0c23a0 2024-09-11T00:48:19.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.683+0000 7f0691a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f068c156df0 con 0x7f068c156300 2024-09-11T00:48:19.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.684+0000 7f068b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f068c156300 0x7f068c0745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:19.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.684+0000 7f068b400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f068c156300 0x7f068c0745f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:42632/0 (socket says 172.21.15.31:42632) 2024-09-11T00:48:19.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.684+0000 7f068b400640 1 -- 172.21.15.31:0/3430832135 learned_addr learned my addr 172.21.15.31:0/3430832135 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:48:19.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.684+0000 7f068a000640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f068c0c23a0 0x7f068c0ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:19.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.684+0000 7f068aa00640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f068c06fa70 0x7f068c150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.684+0000 7f068a000640 1 -- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f068c156300 msgr2=0x7f068c0745f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.684+0000 7f068a000640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f068c156300 0x7f068c0745f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.684+0000 7f068a000640 1 -- 172.21.15.31:0/3430832135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f068c06fa70 msgr2=0x7f068c150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.684+0000 7f068a000640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f068c06fa70 0x7f068c150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.685+0000 7f068a000640 1 -- 172.21.15.31:0/3430832135 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c1505d0 con 0x7f068c0c23a0 2024-09-11T00:48:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.685+0000 7f068b400640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f068c156300 0x7f068c0745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:48:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.685+0000 7f068a000640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f068c0c23a0 0x7f068c0ac480 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f067400c940 tx=0x7f067400ce00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.685+0000 7f0682a00640 1 -- 172.21.15.31:0/3430832135 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674007bb0 con 0x7f068c0c23a0 2024-09-11T00:48:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.685+0000 7f0682a00640 1 -- 172.21.15.31:0/3430832135 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0674007e40 con 0x7f068c0c23a0 2024-09-11T00:48:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.685+0000 7f0682a00640 1 -- 172.21.15.31:0/3430832135 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674005280 con 0x7f068c0c23a0 2024-09-11T00:48:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.685+0000 7f0691a00640 1 -- 172.21.15.31:0/3430832135 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c1f0890 con 0x7f068c0c23a0 2024-09-11T00:48:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.685+0000 7f0691a00640 1 -- 172.21.15.31:0/3430832135 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c1f0e50 con 0x7f068c0c23a0 2024-09-11T00:48:19.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.687+0000 7f0691a00640 1 -- 172.21.15.31:0/3430832135 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068c0c2780 con 0x7f068c0c23a0 2024-09-11T00:48:19.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.688+0000 7f0682a00640 1 -- 172.21.15.31:0/3430832135 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0674005420 con 0x7f068c0c23a0 2024-09-11T00:48:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.689+0000 7f0682a00640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0668077f70 0x7f066807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:48:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.689+0000 7f0682a00640 1 -- 172.21.15.31:0/3430832135 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7f067409b080 con 0x7f068c0c23a0 2024-09-11T00:48:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.690+0000 7f068aa00640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0668077f70 0x7f066807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:48:19.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.690+0000 7f068aa00640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0668077f70 0x7f066807a430 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f067c001720 tx=0x7f067c007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:48:19.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.695+0000 7f0682a00640 1 -- 172.21.15.31:0/3430832135 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f068c0c2780 con 0x7f068c0c23a0 2024-09-11T00:48:19.802 INFO:tasks.ceph.mds.b.smithi204.stderr:2024-09-11T00:48:19.800+0000 7fecb4600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:48:19.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.977+0000 7f0691a00640 1 -- 172.21.15.31:0/3430832135 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f068c14d120 con 0x7f068c0c23a0 2024-09-11T00:48:19.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.978+0000 7f0682a00640 1 -- 172.21.15.31:0/3430832135 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6541 (secure 0 0 0) 0x7f0674014030 con 0x7f068c0c23a0 2024-09-11T00:48:19.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:48:19.979 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":137,"btime":"2024-09-11T00:48:19:837386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:48:19.980 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 137 2024-09-11T00:48:19.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.980+0000 7f0691a00640 1 -- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0668077f70 msgr2=0x7f066807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:19.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.980+0000 7f0691a00640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0668077f70 0x7f066807a430 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f067c001720 tx=0x7f067c007e90 comp rx=0 tx=0).stop 2024-09-11T00:48:19.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.980+0000 7f0691a00640 1 -- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f068c0c23a0 msgr2=0x7f068c0ac480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:48:19.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.980+0000 7f0691a00640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f068c0c23a0 0x7f068c0ac480 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f067400c940 tx=0x7f067400ce00 comp rx=0 tx=0).stop 2024-09-11T00:48:19.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.980+0000 7f0691a00640 1 -- 172.21.15.31:0/3430832135 shutdown_connections 2024-09-11T00:48:19.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.980+0000 7f0691a00640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0668077f70 0x7f066807a430 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:19.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.980+0000 7f0691a00640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f068c156300 0x7f068c0745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:19.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.980+0000 7f0691a00640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f068c0c23a0 0x7f068c0ac480 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:19.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.980+0000 7f0691a00640 1 --2- 172.21.15.31:0/3430832135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f068c06fa70 0x7f068c150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:48:19.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.980+0000 7f0691a00640 1 -- 172.21.15.31:0/3430832135 >> 172.21.15.31:0/3430832135 conn(0x7f068c0a9750 msgr2=0x7f068c0a8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:48:19.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.981+0000 7f0691a00640 1 -- 172.21.15.31:0/3430832135 shutdown_connections 2024-09-11T00:48:19.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:48:19.981+0000 7f0691a00640 1 -- 172.21.15.31:0/3430832135 wait complete. 2024-09-11T00:48:20.002 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-11T00:48:20.002 INFO:tasks.mds_thrash.fs.[a]:waiting for 98 secs before thrashing 2024-09-11T00:48:48.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:48:49.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:49:19.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:49:19.111 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:49:19.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:20.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.088+0000 7ff395a00640 1 -- 172.21.15.31:0/2783049086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3900c47a0 msgr2=0x7ff390163310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:20.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.088+0000 7ff395a00640 1 --2- 172.21.15.31:0/2783049086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3900c47a0 0x7ff390163310 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff388005fd0 tx=0x7ff38802a780 comp rx=0 tx=0).stop 2024-09-11T00:49:20.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.088+0000 7ff395a00640 1 -- 172.21.15.31:0/2783049086 shutdown_connections 2024-09-11T00:49:20.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.088+0000 7ff395a00640 1 --2- 172.21.15.31:0/2783049086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3900c47a0 0x7ff390163310 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.088+0000 7ff395a00640 1 --2- 172.21.15.31:0/2783049086 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff39008c9b0 0x7ff39006dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.088+0000 7ff395a00640 1 -- 172.21.15.31:0/2783049086 >> 172.21.15.31:0/2783049086 conn(0x7ff39006caf0 msgr2=0x7ff39006ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:20.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.089+0000 7ff395a00640 1 -- 172.21.15.31:0/2783049086 shutdown_connections 2024-09-11T00:49:20.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.089+0000 7ff395a00640 1 -- 172.21.15.31:0/2783049086 wait complete. 2024-09-11T00:49:20.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.091+0000 7ff395a00640 1 Processor -- start 2024-09-11T00:49:20.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.091+0000 7ff395a00640 1 -- start start 2024-09-11T00:49:20.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.092+0000 7ff395a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39008c9b0 0x7ff390161140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:20.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.092+0000 7ff395a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3900c43c0 0x7ff3900a6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:20.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.092+0000 7ff395a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3900c47a0 0x7ff3901fac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:20.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.092+0000 7ff395a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff39008c690 con 0x7ff39008c9b0 2024-09-11T00:49:20.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.092+0000 7ff395a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff3900c7550 con 0x7ff3900c43c0 2024-09-11T00:49:20.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.092+0000 7ff395a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff390162d80 con 0x7ff3900c47a0 2024-09-11T00:49:20.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.093+0000 7ff38fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39008c9b0 0x7ff390161140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:20.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.093+0000 7ff38fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39008c9b0 0x7ff390161140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53584/0 (socket says 172.21.15.31:53584) 2024-09-11T00:49:20.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.093+0000 7ff38fe00640 1 -- 172.21.15.31:0/3721907476 learned_addr learned my addr 172.21.15.31:0/3721907476 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:20.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.093+0000 7ff395000640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3900c47a0 0x7ff3901fac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:20.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.093+0000 7ff38f400640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3900c43c0 0x7ff3900a6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:20.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.093+0000 7ff38fe00640 1 -- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3900c47a0 msgr2=0x7ff3901fac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:20.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.093+0000 7ff38fe00640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3900c47a0 0x7ff3901fac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.093+0000 7ff38fe00640 1 -- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3900c43c0 msgr2=0x7ff3900a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:20.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.093+0000 7ff38fe00640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3900c43c0 0x7ff3900a6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.093+0000 7ff38fe00640 1 -- 172.21.15.31:0/3721907476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff390161680 con 0x7ff39008c9b0 2024-09-11T00:49:20.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.094+0000 7ff38fe00640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39008c9b0 0x7ff390161140 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff38000c450 tx=0x7ff38000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:20.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.094+0000 7ff387e00640 1 -- 172.21.15.31:0/3721907476 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380011790 con 0x7ff39008c9b0 2024-09-11T00:49:20.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.094+0000 7ff387e00640 1 -- 172.21.15.31:0/3721907476 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff380017070 con 0x7ff39008c9b0 2024-09-11T00:49:20.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.094+0000 7ff395a00640 1 -- 172.21.15.31:0/3721907476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff390087c40 con 0x7ff39008c9b0 2024-09-11T00:49:20.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.094+0000 7ff387e00640 1 -- 172.21.15.31:0/3721907476 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380011cc0 con 0x7ff39008c9b0 2024-09-11T00:49:20.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.094+0000 7ff395a00640 1 -- 172.21.15.31:0/3721907476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff390088200 con 0x7ff39008c9b0 2024-09-11T00:49:20.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.096+0000 7ff395a00640 1 -- 172.21.15.31:0/3721907476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3900700d0 con 0x7ff39008c9b0 2024-09-11T00:49:20.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.097+0000 7ff387e00640 1 -- 172.21.15.31:0/3721907476 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff38001c460 con 0x7ff39008c9b0 2024-09-11T00:49:20.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.098+0000 7ff387e00640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff370077f70 0x7ff37007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:20.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.099+0000 7ff387e00640 1 -- 172.21.15.31:0/3721907476 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7ff38009a660 con 0x7ff39008c9b0 2024-09-11T00:49:20.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.099+0000 7ff38f400640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff370077f70 0x7ff37007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:20.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.100+0000 7ff38f400640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff370077f70 0x7ff37007a430 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff3900c1b50 tx=0x7ff37c011040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:20.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.104+0000 7ff387e00640 1 -- 172.21.15.31:0/3721907476 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff380061e70 con 0x7ff39008c9b0 2024-09-11T00:49:20.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.384+0000 7ff395a00640 1 -- 172.21.15.31:0/3721907476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff39008b450 con 0x7ff39008c9b0 2024-09-11T00:49:20.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.384+0000 7ff387e00640 1 -- 172.21.15.31:0/3721907476 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6541 (secure 0 0 0) 0x7ff3800668a0 con 0x7ff39008c9b0 2024-09-11T00:49:20.386 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:20.386 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":137,"btime":"2024-09-11T00:48:19:837386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:49:20.386 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 137 2024-09-11T00:49:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 -- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff370077f70 msgr2=0x7ff37007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff370077f70 0x7ff37007a430 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff3900c1b50 tx=0x7ff37c011040 comp rx=0 tx=0).stop 2024-09-11T00:49:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 -- 172.21.15.31:0/3721907476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39008c9b0 msgr2=0x7ff390161140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39008c9b0 0x7ff390161140 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ff38000c450 tx=0x7ff38000c910 comp rx=0 tx=0).stop 2024-09-11T00:49:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 -- 172.21.15.31:0/3721907476 shutdown_connections 2024-09-11T00:49:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff370077f70 0x7ff37007a430 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff3900c47a0 0x7ff3901fac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3900c43c0 0x7ff3900a6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 --2- 172.21.15.31:0/3721907476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff39008c9b0 0x7ff390161140 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 -- 172.21.15.31:0/3721907476 >> 172.21.15.31:0/3721907476 conn(0x7ff39006caf0 msgr2=0x7ff39008a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:20.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 -- 172.21.15.31:0/3721907476 shutdown_connections 2024-09-11T00:49:20.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.387+0000 7ff395a00640 1 -- 172.21.15.31:0/3721907476 wait complete. 2024-09-11T00:49:20.407 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-11T00:49:20.407 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:49:20.407 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:49:20.424 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:49:20.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.607+0000 7fa730e00640 1 -- 172.21.15.31:0/3755357074 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa72c14c8e0 msgr2=0x7fa72c156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.607+0000 7fa730e00640 1 --2- 172.21.15.31:0/3755357074 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa72c14c8e0 0x7fa72c156b00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009830 tx=0x7fa71c02a510 comp rx=0 tx=0).stop 2024-09-11T00:49:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.607+0000 7fa730e00640 1 -- 172.21.15.31:0/3755357074 shutdown_connections 2024-09-11T00:49:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.607+0000 7fa730e00640 1 --2- 172.21.15.31:0/3755357074 >> v2:172.21.15.204:3300/0 conn(0x7fa72c0c9a10 0x7fa72c159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.607+0000 7fa730e00640 1 --2- 172.21.15.31:0/3755357074 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa72c14c8e0 0x7fa72c156b00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.607+0000 7fa730e00640 1 -- 172.21.15.31:0/3755357074 >> 172.21.15.31:0/3755357074 conn(0x7fa72c068ee0 msgr2=0x7fa72c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.607+0000 7fa730e00640 1 -- 172.21.15.31:0/3755357074 shutdown_connections 2024-09-11T00:49:20.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.607+0000 7fa730e00640 1 -- 172.21.15.31:0/3755357074 wait complete. 2024-09-11T00:49:20.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.608+0000 7fa730e00640 1 Processor -- start 2024-09-11T00:49:20.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.609+0000 7fa730e00640 1 -- start start 2024-09-11T00:49:20.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.609+0000 7fa730e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa72c0c9a10 0x7fa72c14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:20.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.609+0000 7fa730e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa72c14c8e0 0x7fa72c159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa730e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa72c14ccc0 0x7fa72c0c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa730e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c0c6c90 con 0x7fa72c0c9a10 2024-09-11T00:49:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa730e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa72c0c6490 con 0x7fa72c14ccc0 2024-09-11T00:49:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa730e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa72c0adca0 con 0x7fa72c14c8e0 2024-09-11T00:49:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa72c0c9a10 0x7fa72c14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa72c0c9a10 0x7fa72c14ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53588/0 (socket says 172.21.15.31:53588) 2024-09-11T00:49:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa72c14c8e0 0x7fa72c159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72b400640 1 -- 172.21.15.31:0/3940086925 learned_addr learned my addr 172.21.15.31:0/3940086925 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72be00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa72c14ccc0 0x7fa72c0c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:20.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72aa00640 1 -- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa72c14ccc0 msgr2=0x7fa72c0c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:20.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72aa00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa72c14ccc0 0x7fa72c0c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72aa00640 1 -- 172.21.15.31:0/3940086925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa72c0c9a10 msgr2=0x7fa72c14ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72aa00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa72c0c9a10 0x7fa72c14ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72aa00640 1 -- 172.21.15.31:0/3940086925 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72c14d920 con 0x7fa72c14c8e0 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72b400640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa72c0c9a10 0x7fa72c14ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.610+0000 7fa72be00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa72c14ccc0 0x7fa72c0c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.611+0000 7fa72aa00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa72c14c8e0 0x7fa72c159020 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa71800c390 tx=0x7fa71800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.611+0000 7fa723400640 1 -- 172.21.15.31:0/3940086925 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718017070 con 0x7fa72c14c8e0 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.611+0000 7fa723400640 1 -- 172.21.15.31:0/3940086925 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7180053e0 con 0x7fa72c14c8e0 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.611+0000 7fa723400640 1 -- 172.21.15.31:0/3940086925 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718003850 con 0x7fa72c14c8e0 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.611+0000 7fa730e00640 1 -- 172.21.15.31:0/3940086925 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa72c14d440 con 0x7fa72c14c8e0 2024-09-11T00:49:20.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.611+0000 7fa730e00640 1 -- 172.21.15.31:0/3940086925 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa72c159560 con 0x7fa72c14c8e0 2024-09-11T00:49:20.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.612+0000 7fa730e00640 1 -- 172.21.15.31:0/3940086925 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa72c0c6ef0 con 0x7fa72c14c8e0 2024-09-11T00:49:20.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.614+0000 7fa723400640 1 -- 172.21.15.31:0/3940086925 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa7180039f0 con 0x7fa72c14c8e0 2024-09-11T00:49:20.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.615+0000 7fa723400640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa70c077f70 0x7fa70c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:20.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.615+0000 7fa723400640 1 -- 172.21.15.31:0/3940086925 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7fa718099df0 con 0x7fa72c14c8e0 2024-09-11T00:49:20.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.615+0000 7fa72b400640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa70c077f70 0x7fa70c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:20.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.616+0000 7fa72b400640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa70c077f70 0x7fa70c07a430 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa71c0095f0 tx=0x7fa71c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:20.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.619+0000 7fa723400640 1 -- 172.21.15.31:0/3940086925 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa72c0c6ef0 con 0x7fa72c14c8e0 2024-09-11T00:49:20.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.898+0000 7fa730e00640 1 -- 172.21.15.31:0/3940086925 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa72c0ac220 con 0x7fa72c14c8e0 2024-09-11T00:49:20.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.899+0000 7fa723400640 1 -- 172.21.15.31:0/3940086925 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6541 (secure 0 0 0) 0x7fa72c0ac220 con 0x7fa72c14c8e0 2024-09-11T00:49:20.900 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:20.900 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":137,"btime":"2024-09-11T00:48:19:837386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:49:20.901 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 137 2024-09-11T00:49:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.901+0000 7fa730e00640 1 -- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa70c077f70 msgr2=0x7fa70c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.901+0000 7fa730e00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa70c077f70 0x7fa70c07a430 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa71c0095f0 tx=0x7fa71c004730 comp rx=0 tx=0).stop 2024-09-11T00:49:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.901+0000 7fa730e00640 1 -- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa72c14c8e0 msgr2=0x7fa72c159020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.901+0000 7fa730e00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa72c14c8e0 0x7fa72c159020 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa71800c390 tx=0x7fa71800c850 comp rx=0 tx=0).stop 2024-09-11T00:49:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.901+0000 7fa730e00640 1 -- 172.21.15.31:0/3940086925 shutdown_connections 2024-09-11T00:49:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.901+0000 7fa730e00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa70c077f70 0x7fa70c07a430 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.901+0000 7fa730e00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa72c14ccc0 0x7fa72c0c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.901+0000 7fa730e00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa72c14c8e0 0x7fa72c159020 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.901+0000 7fa730e00640 1 --2- 172.21.15.31:0/3940086925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa72c0c9a10 0x7fa72c14ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.901+0000 7fa730e00640 1 -- 172.21.15.31:0/3940086925 >> 172.21.15.31:0/3940086925 conn(0x7fa72c068ee0 msgr2=0x7fa72c0acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.902+0000 7fa730e00640 1 -- 172.21.15.31:0/3940086925 shutdown_connections 2024-09-11T00:49:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:20.902+0000 7fa730e00640 1 -- 172.21.15.31:0/3940086925 wait complete. 2024-09-11T00:49:20.919 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:49:22.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.087+0000 7f4c3e200640 1 -- 172.21.15.31:0/3470513066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c380c5aa0 msgr2=0x7f4c380c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.087+0000 7f4c3e200640 1 --2- 172.21.15.31:0/3470513066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c380c5aa0 0x7f4c380c7150 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4c28005fd0 tx=0x7f4c2802a5c0 comp rx=0 tx=0).stop 2024-09-11T00:49:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.087+0000 7f4c3e200640 1 -- 172.21.15.31:0/3470513066 shutdown_connections 2024-09-11T00:49:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.087+0000 7f4c3e200640 1 --2- 172.21.15.31:0/3470513066 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c380c6420 0x7f4c38153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.087+0000 7f4c3e200640 1 --2- 172.21.15.31:0/3470513066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c380c5aa0 0x7f4c380c7150 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.087+0000 7f4c3e200640 1 -- 172.21.15.31:0/3470513066 >> 172.21.15.31:0/3470513066 conn(0x7f4c3806caf0 msgr2=0x7f4c3806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.087+0000 7f4c3e200640 1 -- 172.21.15.31:0/3470513066 shutdown_connections 2024-09-11T00:49:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.087+0000 7f4c3e200640 1 -- 172.21.15.31:0/3470513066 wait complete. 2024-09-11T00:49:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.088+0000 7f4c3e200640 1 Processor -- start 2024-09-11T00:49:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.089+0000 7f4c3e200640 1 -- start start 2024-09-11T00:49:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.089+0000 7f4c3e200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c380c5aa0 0x7f4c380a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.089+0000 7f4c3e200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c380c6420 0x7f4c380a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.089+0000 7f4c3e200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c38153850 0x7f4c380ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.089+0000 7f4c3e200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4c380c2e80 con 0x7f4c380c5aa0 2024-09-11T00:49:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.089+0000 7f4c3e200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c380c2ba0 con 0x7f4c38153850 2024-09-11T00:49:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.089+0000 7f4c3e200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4c380c5e80 con 0x7f4c380c6420 2024-09-11T00:49:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.090+0000 7f4c3ce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c380c5aa0 0x7f4c380a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.090+0000 7f4c2fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c380c6420 0x7f4c380a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.090+0000 7f4c3d800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c38153850 0x7f4c380ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.090+0000 7f4c2fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c380c6420 0x7f4c380a52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:33760/0 (socket says 172.21.15.31:33760) 2024-09-11T00:49:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.090+0000 7f4c2fe00640 1 -- 172.21.15.31:0/814012016 learned_addr learned my addr 172.21.15.31:0/814012016 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.090+0000 7f4c2fe00640 1 -- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c38153850 msgr2=0x7f4c380ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.090+0000 7f4c2fe00640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c38153850 0x7f4c380ac6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.090+0000 7f4c2fe00640 1 -- 172.21.15.31:0/814012016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c380c5aa0 msgr2=0x7f4c380a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.090+0000 7f4c2fe00640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c380c5aa0 0x7f4c380a44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.090+0000 7f4c2fe00640 1 -- 172.21.15.31:0/814012016 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c380ab080 con 0x7f4c380c6420 2024-09-11T00:49:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.091+0000 7f4c2fe00640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c380c6420 0x7f4c380a52e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000e9f0 tx=0x7f4c2000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.091+0000 7f4c35600640 1 -- 172.21.15.31:0/814012016 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2000cde0 con 0x7f4c380c6420 2024-09-11T00:49:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.091+0000 7f4c35600640 1 -- 172.21.15.31:0/814012016 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c20010520 con 0x7f4c380c6420 2024-09-11T00:49:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.091+0000 7f4c35600640 1 -- 172.21.15.31:0/814012016 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c20010970 con 0x7f4c380c6420 2024-09-11T00:49:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.091+0000 7f4c3e200640 1 -- 172.21.15.31:0/814012016 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c380ab8e0 con 0x7f4c380c6420 2024-09-11T00:49:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.091+0000 7f4c3e200640 1 -- 172.21.15.31:0/814012016 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c380abea0 con 0x7f4c380c6420 2024-09-11T00:49:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.092+0000 7f4c3e200640 1 -- 172.21.15.31:0/814012016 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c380c7150 con 0x7f4c380c6420 2024-09-11T00:49:23.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.094+0000 7f4c35600640 1 -- 172.21.15.31:0/814012016 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4c20010b10 con 0x7f4c380c6420 2024-09-11T00:49:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.095+0000 7f4c35600640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c14077f70 0x7f4c1407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.095+0000 7f4c35600640 1 -- 172.21.15.31:0/814012016 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7f4c2009b3e0 con 0x7f4c380c6420 2024-09-11T00:49:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.095+0000 7f4c3ce00640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c14077f70 0x7f4c1407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:23.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.096+0000 7f4c3ce00640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c14077f70 0x7f4c1407a430 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4c28033000 tx=0x7f4c28004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.099+0000 7f4c35600640 1 -- 172.21.15.31:0/814012016 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4c20062c70 con 0x7f4c380c6420 2024-09-11T00:49:23.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.363+0000 7f4c3e200640 1 -- 172.21.15.31:0/814012016 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c380ac280 con 0x7f4c380c6420 2024-09-11T00:49:23.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.364+0000 7f4c35600640 1 -- 172.21.15.31:0/814012016 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6541 (secure 0 0 0) 0x7f4c200676a0 con 0x7f4c380c6420 2024-09-11T00:49:23.365 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:23.366 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":137,"btime":"2024-09-11T00:48:19:837386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:49:23.366 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 137 2024-09-11T00:49:23.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 -- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c14077f70 msgr2=0x7f4c1407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:23.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c14077f70 0x7f4c1407a430 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4c28033000 tx=0x7f4c28004730 comp rx=0 tx=0).stop 2024-09-11T00:49:23.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 -- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c380c6420 msgr2=0x7f4c380a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:23.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c380c6420 0x7f4c380a52e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000e9f0 tx=0x7f4c2000eeb0 comp rx=0 tx=0).stop 2024-09-11T00:49:23.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 -- 172.21.15.31:0/814012016 shutdown_connections 2024-09-11T00:49:23.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c14077f70 0x7f4c1407a430 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:23.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c38153850 0x7f4c380ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:23.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c380c6420 0x7f4c380a52e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:23.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 --2- 172.21.15.31:0/814012016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c380c5aa0 0x7f4c380a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:23.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 -- 172.21.15.31:0/814012016 >> 172.21.15.31:0/814012016 conn(0x7f4c3806caf0 msgr2=0x7f4c3814ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:23.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 -- 172.21.15.31:0/814012016 shutdown_connections 2024-09-11T00:49:23.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:23.366+0000 7f4c3e200640 1 -- 172.21.15.31:0/814012016 wait complete. 2024-09-11T00:49:23.384 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:49:25.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:25.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.567+0000 7f82e2000640 1 -- 172.21.15.31:0/4212919991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc0c8de0 msgr2=0x7f82dc06f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:25.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.567+0000 7f82e2000640 1 --2- 172.21.15.31:0/4212919991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc0c8de0 0x7f82dc06f0a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009ca0 tx=0x7f82d402aac0 comp rx=0 tx=0).stop 2024-09-11T00:49:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.567+0000 7f82e2000640 1 -- 172.21.15.31:0/4212919991 shutdown_connections 2024-09-11T00:49:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.567+0000 7f82e2000640 1 --2- 172.21.15.31:0/4212919991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc0c8de0 0x7f82dc06f0a0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.567+0000 7f82e2000640 1 --2- 172.21.15.31:0/4212919991 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82dc06f8d0 0x7f82dc06eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.567+0000 7f82e2000640 1 -- 172.21.15.31:0/4212919991 >> 172.21.15.31:0/4212919991 conn(0x7f82dc0694f0 msgr2=0x7f82dc0c9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.567+0000 7f82e2000640 1 -- 172.21.15.31:0/4212919991 shutdown_connections 2024-09-11T00:49:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.567+0000 7f82e2000640 1 -- 172.21.15.31:0/4212919991 wait complete. 2024-09-11T00:49:25.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.569+0000 7f82e2000640 1 Processor -- start 2024-09-11T00:49:25.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.569+0000 7f82e2000640 1 -- start start 2024-09-11T00:49:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82e2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82dc06f8d0 0x7f82dc1551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:25.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82e2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc0c67f0 0x7f82dc0aa140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:25.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82e2000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82dc0c8de0 0x7f82dc0aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:25.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82e2000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc0c5ab0 con 0x7f82dc0c67f0 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82e2000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc0c89d0 con 0x7f82dc06f8d0 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82e2000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f82dc0c5c30 con 0x7f82dc0c8de0 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82dbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc0c67f0 0x7f82dc0aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82dbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc0c67f0 0x7f82dc0aa140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53610/0 (socket says 172.21.15.31:53610) 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82dbe00640 1 -- 172.21.15.31:0/4111648470 learned_addr learned my addr 172.21.15.31:0/4111648470 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82e1600640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82dc0c8de0 0x7f82dc0aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82e0c00640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82dc06f8d0 0x7f82dc1551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82dbe00640 1 -- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82dc0c8de0 msgr2=0x7f82dc0aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82dbe00640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82dc0c8de0 0x7f82dc0aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82dbe00640 1 -- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82dc06f8d0 msgr2=0x7f82dc1551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:25.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.570+0000 7f82dbe00640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82dc06f8d0 0x7f82dc1551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:25.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.571+0000 7f82dbe00640 1 -- 172.21.15.31:0/4111648470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82dc155730 con 0x7f82dc0c67f0 2024-09-11T00:49:25.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.571+0000 7f82dbe00640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc0c67f0 0x7f82dc0aa140 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d7a0 tx=0x7f82c800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:25.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.571+0000 7f82d8c00640 1 -- 172.21.15.31:0/4111648470 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c8017070 con 0x7f82dc0c67f0 2024-09-11T00:49:25.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.571+0000 7f82d8c00640 1 -- 172.21.15.31:0/4111648470 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82c8009d20 con 0x7f82dc0c67f0 2024-09-11T00:49:25.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.571+0000 7f82e2000640 1 -- 172.21.15.31:0/4111648470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82dc1ecd60 con 0x7f82dc0c67f0 2024-09-11T00:49:25.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.571+0000 7f82d8c00640 1 -- 172.21.15.31:0/4111648470 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c8004280 con 0x7f82dc0c67f0 2024-09-11T00:49:25.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.571+0000 7f82e2000640 1 -- 172.21.15.31:0/4111648470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82dc0c9c20 con 0x7f82dc0c67f0 2024-09-11T00:49:25.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.572+0000 7f82e2000640 1 -- 172.21.15.31:0/4111648470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82dc0c6bd0 con 0x7f82dc0c67f0 2024-09-11T00:49:25.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.574+0000 7f82d8c00640 1 -- 172.21.15.31:0/4111648470 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f82c80035f0 con 0x7f82dc0c67f0 2024-09-11T00:49:25.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.575+0000 7f82d8c00640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f82bc077fa0 0x7f82bc07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:25.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.575+0000 7f82d8c00640 1 -- 172.21.15.31:0/4111648470 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7f82c809a9e0 con 0x7f82dc0c67f0 2024-09-11T00:49:25.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.575+0000 7f82e0c00640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f82bc077fa0 0x7f82bc07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:25.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.576+0000 7f82e0c00640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f82bc077fa0 0x7f82bc07a460 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f82d00017c0 tx=0x7f82d0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:25.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.580+0000 7f82d8c00640 1 -- 172.21.15.31:0/4111648470 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f82dc155730 con 0x7f82dc0c67f0 2024-09-11T00:49:25.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.881+0000 7f82e2000640 1 -- 172.21.15.31:0/4111648470 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82dc0ae640 con 0x7f82dc0c67f0 2024-09-11T00:49:25.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.882+0000 7f82d8c00640 1 -- 172.21.15.31:0/4111648470 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6541 (secure 0 0 0) 0x7f82c8062270 con 0x7f82dc0c67f0 2024-09-11T00:49:25.884 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:25.884 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":137,"btime":"2024-09-11T00:48:19:837386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:49:25.884 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 137 2024-09-11T00:49:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 -- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f82bc077fa0 msgr2=0x7f82bc07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f82bc077fa0 0x7f82bc07a460 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f82d00017c0 tx=0x7f82d0009040 comp rx=0 tx=0).stop 2024-09-11T00:49:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 -- 172.21.15.31:0/4111648470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc0c67f0 msgr2=0x7f82dc0aa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc0c67f0 0x7f82dc0aa140 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d7a0 tx=0x7f82c800dc60 comp rx=0 tx=0).stop 2024-09-11T00:49:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 -- 172.21.15.31:0/4111648470 shutdown_connections 2024-09-11T00:49:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f82bc077fa0 0x7f82bc07a460 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f82dc0c8de0 0x7f82dc0aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f82dc0c67f0 0x7f82dc0aa140 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 --2- 172.21.15.31:0/4111648470 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82dc06f8d0 0x7f82dc1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 -- 172.21.15.31:0/4111648470 >> 172.21.15.31:0/4111648470 conn(0x7f82dc0694f0 msgr2=0x7f82dc159e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:25.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 -- 172.21.15.31:0/4111648470 shutdown_connections 2024-09-11T00:49:25.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:25.885+0000 7f82e2000640 1 -- 172.21.15.31:0/4111648470 wait complete. 2024-09-11T00:49:25.902 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:49:27.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:28.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.072+0000 7f7067200640 1 -- 172.21.15.31:0/1415571547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70680ada20 msgr2=0x7f706806e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:28.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.072+0000 7f7067200640 1 --2- 172.21.15.31:0/1415571547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70680ada20 0x7f706806e140 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f7068153d20 tx=0x7f705402a5f0 comp rx=0 tx=0).stop 2024-09-11T00:49:28.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.073+0000 7f7067200640 1 -- 172.21.15.31:0/1415571547 shutdown_connections 2024-09-11T00:49:28.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.073+0000 7f7067200640 1 --2- 172.21.15.31:0/1415571547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70680ada20 0x7f706806e140 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:28.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.073+0000 7f7067200640 1 --2- 172.21.15.31:0/1415571547 >> v2:172.21.15.204:3300/0 conn(0x7f7068150020 0x7f70680ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:28.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.073+0000 7f7067200640 1 -- 172.21.15.31:0/1415571547 >> 172.21.15.31:0/1415571547 conn(0x7f7068068f10 msgr2=0x7f70680cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:28.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.073+0000 7f7067200640 1 -- 172.21.15.31:0/1415571547 shutdown_connections 2024-09-11T00:49:28.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.073+0000 7f7067200640 1 -- 172.21.15.31:0/1415571547 wait complete. 2024-09-11T00:49:28.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.074+0000 7f7067200640 1 Processor -- start 2024-09-11T00:49:28.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.075+0000 7f7067200640 1 -- start start 2024-09-11T00:49:28.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.075+0000 7f7067200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70680ada20 0x7f70680c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:28.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.075+0000 7f7067200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7068150020 0x7f70680affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:28.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.075+0000 7f7067200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7068156960 0x7f706814e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:28.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.075+0000 7f7067200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7068157550 con 0x7f70680ada20 2024-09-11T00:49:28.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.075+0000 7f7067200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f70681573d0 con 0x7f7068156960 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.075+0000 7f7067200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f70680cc220 con 0x7f7068150020 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7065e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70680ada20 0x7f70680c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7065e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70680ada20 0x7f70680c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38398/0 (socket says 172.21.15.31:38398) 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7065e00640 1 -- 172.21.15.31:0/226454946 learned_addr learned my addr 172.21.15.31:0/226454946 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7066800640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7068156960 0x7f706814e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7065400640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7068150020 0x7f70680affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7065e00640 1 -- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7068150020 msgr2=0x7f70680affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7065e00640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7068150020 0x7f70680affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7065e00640 1 -- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7068156960 msgr2=0x7f706814e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7065e00640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7068156960 0x7f706814e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7065e00640 1 -- 172.21.15.31:0/226454946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70680ae680 con 0x7f70680ada20 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.076+0000 7f7065400640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7068150020 0x7f70680affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:49:28.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.077+0000 7f7065e00640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70680ada20 0x7f70680c33e0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f705800c920 tx=0x7f705800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:28.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.077+0000 7f705e000640 1 -- 172.21.15.31:0/226454946 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058007c70 con 0x7f70680ada20 2024-09-11T00:49:28.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.077+0000 7f705e000640 1 -- 172.21.15.31:0/226454946 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f705800f450 con 0x7f70680ada20 2024-09-11T00:49:28.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.077+0000 7f705e000640 1 -- 172.21.15.31:0/226454946 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705800f8a0 con 0x7f70680ada20 2024-09-11T00:49:28.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.077+0000 7f7067200640 1 -- 172.21.15.31:0/226454946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70681e1250 con 0x7f70680ada20 2024-09-11T00:49:28.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.077+0000 7f7067200640 1 -- 172.21.15.31:0/226454946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70680c3920 con 0x7f70680ada20 2024-09-11T00:49:28.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.078+0000 7f7067200640 1 -- 172.21.15.31:0/226454946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70680ab280 con 0x7f70680ada20 2024-09-11T00:49:28.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.080+0000 7f705e000640 1 -- 172.21.15.31:0/226454946 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f705800fa40 con 0x7f70680ada20 2024-09-11T00:49:28.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.081+0000 7f705e000640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7048077f70 0x7f704807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:28.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.081+0000 7f705e000640 1 -- 172.21.15.31:0/226454946 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7f705809b180 con 0x7f70680ada20 2024-09-11T00:49:28.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.082+0000 7f7065400640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7048077f70 0x7f704807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:28.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.082+0000 7f7065400640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7048077f70 0x7f704807a430 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7054033000 tx=0x7f7054004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:28.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.086+0000 7f705e000640 1 -- 172.21.15.31:0/226454946 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f70680ab280 con 0x7f70680ada20 2024-09-11T00:49:28.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.341+0000 7f7067200640 1 -- 172.21.15.31:0/226454946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70680cc8e0 con 0x7f70680ada20 2024-09-11T00:49:28.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.342+0000 7f705e000640 1 -- 172.21.15.31:0/226454946 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6541 (secure 0 0 0) 0x7f7058014030 con 0x7f70680ada20 2024-09-11T00:49:28.344 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:28.344 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":137,"btime":"2024-09-11T00:48:19:837386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:49:28.344 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 137 2024-09-11T00:49:28.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.344+0000 7f7067200640 1 -- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7048077f70 msgr2=0x7f704807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:28.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.344+0000 7f7067200640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7048077f70 0x7f704807a430 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7054033000 tx=0x7f7054004730 comp rx=0 tx=0).stop 2024-09-11T00:49:28.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.344+0000 7f7067200640 1 -- 172.21.15.31:0/226454946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70680ada20 msgr2=0x7f70680c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:28.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.344+0000 7f7067200640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70680ada20 0x7f70680c33e0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f705800c920 tx=0x7f705800cde0 comp rx=0 tx=0).stop 2024-09-11T00:49:28.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.344+0000 7f7067200640 1 -- 172.21.15.31:0/226454946 shutdown_connections 2024-09-11T00:49:28.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.344+0000 7f7067200640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7048077f70 0x7f704807a430 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:28.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.344+0000 7f7067200640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7068156960 0x7f706814e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:28.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.344+0000 7f7067200640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7068150020 0x7f70680affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:28.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.344+0000 7f7067200640 1 --2- 172.21.15.31:0/226454946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f70680ada20 0x7f70680c33e0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:28.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.344+0000 7f7067200640 1 -- 172.21.15.31:0/226454946 >> 172.21.15.31:0/226454946 conn(0x7f7068068f10 msgr2=0x7f7068153f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:28.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.345+0000 7f7067200640 1 -- 172.21.15.31:0/226454946 shutdown_connections 2024-09-11T00:49:28.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:28.345+0000 7f7067200640 1 -- 172.21.15.31:0/226454946 wait complete. 2024-09-11T00:49:28.362 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:49:30.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:30.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.554+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2108789484 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e580c6760 msgr2=0x7f6e580c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:30.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.554+0000 7f6e5ee00640 1 --2- 172.21.15.31:0/2108789484 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e580c6760 0x7f6e580c8c80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009be0 tx=0x7f6e4c02a9b0 comp rx=0 tx=0).stop 2024-09-11T00:49:30.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.554+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2108789484 shutdown_connections 2024-09-11T00:49:30.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.554+0000 7f6e5ee00640 1 --2- 172.21.15.31:0/2108789484 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e580c6760 0x7f6e580c8c80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:30.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.554+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2108789484 >> 172.21.15.31:0/2108789484 conn(0x7f6e580694f0 msgr2=0x7f6e580c9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:30.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.554+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2108789484 shutdown_connections 2024-09-11T00:49:30.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.554+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2108789484 wait complete. 2024-09-11T00:49:30.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.556+0000 7f6e5ee00640 1 Processor -- start 2024-09-11T00:49:30.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.556+0000 7f6e5ee00640 1 -- start start 2024-09-11T00:49:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.557+0000 7f6e5ee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e580c5a20 0x7f6e581e0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5ee00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e580c6760 0x7f6e581df440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5ee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e580c6b40 0x7f6e581ef480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5ee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6e580c92e0 con 0x7f6e580c5a20 2024-09-11T00:49:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5ee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6e580c8760 con 0x7f6e580c6b40 2024-09-11T00:49:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5ee00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6e580c9460 con 0x7f6e580c6760 2024-09-11T00:49:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e580c5a20 0x7f6e581e0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5da00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e580c5a20 0x7f6e581e0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38404/0 (socket says 172.21.15.31:38404) 2024-09-11T00:49:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5da00640 1 -- 172.21.15.31:0/2121944103 learned_addr learned my addr 172.21.15.31:0/2121944103 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:30.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5d000640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e580c6760 0x7f6e581df440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:30.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5e400640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e580c6b40 0x7f6e581ef480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:30.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5da00640 1 -- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e580c6760 msgr2=0x7f6e581df440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:30.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5da00640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e580c6760 0x7f6e581df440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:30.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5da00640 1 -- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e580c6b40 msgr2=0x7f6e581ef480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:30.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5da00640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e580c6b40 0x7f6e581ef480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:30.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5da00640 1 -- 172.21.15.31:0/2121944103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e580aee10 con 0x7f6e580c5a20 2024-09-11T00:49:30.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.558+0000 7f6e5e400640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e580c6b40 0x7f6e581ef480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:49:30.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.559+0000 7f6e5da00640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e580c5a20 0x7f6e581e0350 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009d00 tx=0x7f6e4c030df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:30.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.559+0000 7f6e55600640 1 -- 172.21.15.31:0/2121944103 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c047070 con 0x7f6e580c5a20 2024-09-11T00:49:30.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.559+0000 7f6e55600640 1 -- 172.21.15.31:0/2121944103 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e4c004040 con 0x7f6e580c5a20 2024-09-11T00:49:30.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.559+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2121944103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e581ebf90 con 0x7f6e580c5a20 2024-09-11T00:49:30.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.559+0000 7f6e55600640 1 -- 172.21.15.31:0/2121944103 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c02f400 con 0x7f6e580c5a20 2024-09-11T00:49:30.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.559+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2121944103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e581ec830 con 0x7f6e580c5a20 2024-09-11T00:49:30.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.560+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2121944103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e581ec380 con 0x7f6e580c5a20 2024-09-11T00:49:30.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.561+0000 7f6e55600640 1 -- 172.21.15.31:0/2121944103 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6e4c004470 con 0x7f6e580c5a20 2024-09-11T00:49:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.563+0000 7f6e55600640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6e38077f70 0x7f6e3807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.563+0000 7f6e55600640 1 -- 172.21.15.31:0/2121944103 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7f6e4c0bb7f0 con 0x7f6e580c5a20 2024-09-11T00:49:30.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.563+0000 7f6e5d000640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6e38077f70 0x7f6e3807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:30.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.564+0000 7f6e5d000640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6e38077f70 0x7f6e3807a430 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6e58158280 tx=0x7f6e44007f50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:30.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.567+0000 7f6e55600640 1 -- 172.21.15.31:0/2121944103 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6e4c033020 con 0x7f6e580c5a20 2024-09-11T00:49:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.879+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2121944103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e580c8c80 con 0x7f6e580c5a20 2024-09-11T00:49:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.880+0000 7f6e55600640 1 -- 172.21.15.31:0/2121944103 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6541 (secure 0 0 0) 0x7f6e4c02c020 con 0x7f6e580c5a20 2024-09-11T00:49:30.882 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:30.882 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":137,"btime":"2024-09-11T00:48:19:837386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:49:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 137 2024-09-11T00:49:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.882+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6e38077f70 msgr2=0x7f6e3807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.882+0000 7f6e5ee00640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6e38077f70 0x7f6e3807a430 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6e58158280 tx=0x7f6e44007f50 comp rx=0 tx=0).stop 2024-09-11T00:49:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.883+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2121944103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e580c5a20 msgr2=0x7f6e581e0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.883+0000 7f6e5ee00640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e580c5a20 0x7f6e581e0350 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009d00 tx=0x7f6e4c030df0 comp rx=0 tx=0).stop 2024-09-11T00:49:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.883+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2121944103 shutdown_connections 2024-09-11T00:49:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.883+0000 7f6e5ee00640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6e38077f70 0x7f6e3807a430 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.883+0000 7f6e5ee00640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e580c6b40 0x7f6e581ef480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.883+0000 7f6e5ee00640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e580c6760 0x7f6e581df440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:30.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.883+0000 7f6e5ee00640 1 --2- 172.21.15.31:0/2121944103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e580c5a20 0x7f6e581e0350 secure :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009d00 tx=0x7f6e4c030df0 comp rx=0 tx=0).stop 2024-09-11T00:49:30.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.883+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2121944103 >> 172.21.15.31:0/2121944103 conn(0x7f6e580694f0 msgr2=0x7f6e581577a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:30.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.883+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2121944103 shutdown_connections 2024-09-11T00:49:30.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:30.883+0000 7f6e5ee00640 1 -- 172.21.15.31:0/2121944103 wait complete. 2024-09-11T00:49:30.902 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:49:32.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:33.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.100+0000 7fa751c00640 1 -- 172.21.15.31:0/1579541775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa730003fd0 msgr2=0x7fa730004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:33.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.100+0000 7fa751c00640 1 --2- 172.21.15.31:0/1579541775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa730003fd0 0x7fa730004800 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa738009b80 tx=0x7fa730005830 comp rx=0 tx=0).stop 2024-09-11T00:49:33.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.100+0000 7fa751200640 1 -- 172.21.15.31:0/1579541775 reap_dead start 2024-09-11T00:49:33.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.100+0000 7fa751c00640 1 -- 172.21.15.31:0/1579541775 shutdown_connections 2024-09-11T00:49:33.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.100+0000 7fa751c00640 1 -- 172.21.15.31:0/1579541775 >> 172.21.15.31:0/1579541775 conn(0x7fa74c068f10 msgr2=0x7fa74c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:33.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.100+0000 7fa751c00640 1 -- 172.21.15.31:0/1579541775 shutdown_connections 2024-09-11T00:49:33.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.100+0000 7fa751c00640 1 -- 172.21.15.31:0/1579541775 wait complete. 2024-09-11T00:49:33.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.102+0000 7fa751c00640 1 Processor -- start 2024-09-11T00:49:33.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.102+0000 7fa751c00640 1 -- start start 2024-09-11T00:49:33.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa751c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730003fd0 0x7fa74c14e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:33.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa751c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74c0c1130 0x7fa74c14acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:33.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa751c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c14b230 0x7fa74c0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:33.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa751c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c157550 con 0x7fa74c0c1130 2024-09-11T00:49:33.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa751c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa74c1573d0 con 0x7fa730003fd0 2024-09-11T00:49:33.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa751c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa74c0cc220 con 0x7fa74c14b230 2024-09-11T00:49:33.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa74b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74c0c1130 0x7fa74c14acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:33.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa74b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74c0c1130 0x7fa74c14acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38420/0 (socket says 172.21.15.31:38420) 2024-09-11T00:49:33.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa74b400640 1 -- 172.21.15.31:0/517361401 learned_addr learned my addr 172.21.15.31:0/517361401 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:33.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa751200640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c14b230 0x7fa74c0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:33.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa74be00640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730003fd0 0x7fa74c14e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:33.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa74b400640 1 -- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c14b230 msgr2=0x7fa74c0b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:33.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa74b400640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c14b230 0x7fa74c0b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:33.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa74b400640 1 -- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730003fd0 msgr2=0x7fa74c14e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:33.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa74b400640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730003fd0 0x7fa74c14e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:33.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.103+0000 7fa74b400640 1 -- 172.21.15.31:0/517361401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74c0b04c0 con 0x7fa74c0c1130 2024-09-11T00:49:33.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.104+0000 7fa74b400640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74c0c1130 0x7fa74c14acf0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa73802b4f0 tx=0x7fa738033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:33.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.104+0000 7fa743e00640 1 -- 172.21.15.31:0/517361401 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738038040 con 0x7fa74c0c1130 2024-09-11T00:49:33.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.104+0000 7fa743e00640 1 -- 172.21.15.31:0/517361401 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa738004020 con 0x7fa74c0c1130 2024-09-11T00:49:33.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.104+0000 7fa743e00640 1 -- 172.21.15.31:0/517361401 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738004470 con 0x7fa74c0c1130 2024-09-11T00:49:33.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.104+0000 7fa751c00640 1 -- 172.21.15.31:0/517361401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa74c0af130 con 0x7fa74c0c1130 2024-09-11T00:49:33.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.104+0000 7fa751c00640 1 -- 172.21.15.31:0/517361401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa74c0c1d10 con 0x7fa74c0c1130 2024-09-11T00:49:33.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.106+0000 7fa751c00640 1 -- 172.21.15.31:0/517361401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa74c14b610 con 0x7fa74c0c1130 2024-09-11T00:49:33.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.107+0000 7fa743e00640 1 -- 172.21.15.31:0/517361401 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa73803b030 con 0x7fa74c0c1130 2024-09-11T00:49:33.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.108+0000 7fa743e00640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa72c077fa0 0x7fa72c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:33.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.109+0000 7fa743e00640 1 -- 172.21.15.31:0/517361401 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8816+0+0 (secure 0 0 0) 0x7fa7380baf60 con 0x7fa74c0c1130 2024-09-11T00:49:33.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.109+0000 7fa74be00640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa72c077fa0 0x7fa72c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.110+0000 7fa74be00640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa72c077fa0 0x7fa72c07a460 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa74c1f04f0 tx=0x7fa73c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:33.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.113+0000 7fa743e00640 1 -- 172.21.15.31:0/517361401 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa738082eb0 con 0x7fa74c0c1130 2024-09-11T00:49:33.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.385+0000 7fa751c00640 1 -- 172.21.15.31:0/517361401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa74c0cc8e0 con 0x7fa74c0c1130 2024-09-11T00:49:33.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.386+0000 7fa743e00640 1 -- 172.21.15.31:0/517361401 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6541 (secure 0 0 0) 0x7fa7380827b0 con 0x7fa74c0c1130 2024-09-11T00:49:33.387 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:33.388 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":137,"btime":"2024-09-11T00:48:19:837386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7065,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:48:02.775671+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6829":{"gid":6829,"name":"d","rank":0,"incarnation":128,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/634556689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":634556689},{"type":"v1","addr":"172.21.15.204:6835","nonce":634556689}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6829,"qdb_cluster":[6829]},"id":2}]} 2024-09-11T00:49:33.388 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 137 2024-09-11T00:49:33.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.388+0000 7fa751c00640 1 -- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa72c077fa0 msgr2=0x7fa72c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa72c077fa0 0x7fa72c07a460 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa74c1f04f0 tx=0x7fa73c005f90 comp rx=0 tx=0).stop 2024-09-11T00:49:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 -- 172.21.15.31:0/517361401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74c0c1130 msgr2=0x7fa74c14acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74c0c1130 0x7fa74c14acf0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa73802b4f0 tx=0x7fa738033670 comp rx=0 tx=0).stop 2024-09-11T00:49:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 -- 172.21.15.31:0/517361401 shutdown_connections 2024-09-11T00:49:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa72c077fa0 0x7fa72c07a460 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa74c14b230 0x7fa74c0b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa74c0c1130 0x7fa74c14acf0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 --2- 172.21.15.31:0/517361401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa730003fd0 0x7fa74c14e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:33.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 -- 172.21.15.31:0/517361401 >> 172.21.15.31:0/517361401 conn(0x7fa74c068f10 msgr2=0x7fa74c158730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:33.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 -- 172.21.15.31:0/517361401 shutdown_connections 2024-09-11T00:49:33.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:33.389+0000 7fa751c00640 1 -- 172.21.15.31:0/517361401 wait complete. 2024-09-11T00:49:33.410 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:49:35.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:35.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.583+0000 7f8ef8c00640 1 -- 172.21.15.31:0/85134274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ef4150020 msgr2=0x7f8ef40ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.583+0000 7f8ef8c00640 1 --2- 172.21.15.31:0/85134274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ef4150020 0x7f8ef40ad4e0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4153d20 tx=0x7f8ee402a5f0 comp rx=0 tx=0).stop 2024-09-11T00:49:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.583+0000 7f8ef8c00640 1 -- 172.21.15.31:0/85134274 shutdown_connections 2024-09-11T00:49:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.583+0000 7f8ef8c00640 1 --2- 172.21.15.31:0/85134274 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ef406e850 0x7f8ef4160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.583+0000 7f8ef8c00640 1 --2- 172.21.15.31:0/85134274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ef4150020 0x7f8ef40ad4e0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.583+0000 7f8ef8c00640 1 -- 172.21.15.31:0/85134274 >> 172.21.15.31:0/85134274 conn(0x7f8ef4068f10 msgr2=0x7f8ef40cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.583+0000 7f8ef8c00640 1 -- 172.21.15.31:0/85134274 shutdown_connections 2024-09-11T00:49:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.583+0000 7f8ef8c00640 1 -- 172.21.15.31:0/85134274 wait complete. 2024-09-11T00:49:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.585+0000 7f8ef8c00640 1 Processor -- start 2024-09-11T00:49:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.585+0000 7f8ef8c00640 1 -- start start 2024-09-11T00:49:35.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef8c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef406e850 0x7f8ef40af170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:35.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef8c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ef40ada20 0x7f8ef40b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:35.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef8c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ef4150020 0x7f8ef414cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:35.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef8c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef4157550 con 0x7f8ef4150020 2024-09-11T00:49:35.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef8c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef41573d0 con 0x7f8ef406e850 2024-09-11T00:49:35.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef8c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef4156960 con 0x7f8ef40ada20 2024-09-11T00:49:35.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef406e850 0x7f8ef40af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:35.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef2a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ef40ada20 0x7f8ef40b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:35.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ef4150020 0x7f8ef414cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:35.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef3400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef406e850 0x7f8ef40af170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:53780/0 (socket says 172.21.15.31:53780) 2024-09-11T00:49:35.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.586+0000 7f8ef3400640 1 -- 172.21.15.31:0/2055433329 learned_addr learned my addr 172.21.15.31:0/2055433329 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.587+0000 7f8ef2a00640 1 -- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef406e850 msgr2=0x7f8ef40af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.587+0000 7f8ef2a00640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef406e850 0x7f8ef40af170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.587+0000 7f8ef2a00640 1 -- 172.21.15.31:0/2055433329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ef4150020 msgr2=0x7f8ef414cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.587+0000 7f8ef2a00640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ef4150020 0x7f8ef414cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.587+0000 7f8ef2a00640 1 -- 172.21.15.31:0/2055433329 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef41ef410 con 0x7f8ef40ada20 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.587+0000 7f8ef2a00640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ef40ada20 0x7f8ef40b0d00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000c3d0 tx=0x7f8ee000c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.588+0000 7f8eeb400640 1 -- 172.21.15.31:0/2055433329 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0017070 con 0x7f8ef40ada20 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.588+0000 7f8eeb400640 1 -- 172.21.15.31:0/2055433329 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ee00035f0 con 0x7f8ef40ada20 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.588+0000 7f8eeb400640 1 -- 172.21.15.31:0/2055433329 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0003a40 con 0x7f8ef40ada20 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.588+0000 7f8ef8c00640 1 -- 172.21.15.31:0/2055433329 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef41ef6f0 con 0x7f8ef40ada20 2024-09-11T00:49:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.588+0000 7f8ef8c00640 1 -- 172.21.15.31:0/2055433329 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef41efcb0 con 0x7f8ef40ada20 2024-09-11T00:49:35.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.589+0000 7f8ef8c00640 1 -- 172.21.15.31:0/2055433329 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ef40cc220 con 0x7f8ef40ada20 2024-09-11T00:49:35.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.590+0000 7f8eeb400640 1 -- 172.21.15.31:0/2055433329 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8ee0003be0 con 0x7f8ef40ada20 2024-09-11T00:49:35.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.591+0000 7f8eeb400640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ed4077fa0 0x7f8ed407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:35.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.591+0000 7f8eeb400640 1 -- 172.21.15.31:0/2055433329 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f8ee0099eb0 con 0x7f8ef40ada20 2024-09-11T00:49:35.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.591+0000 7f8ef3400640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ed4077fa0 0x7f8ed407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:35.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.592+0000 7f8ef3400640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ed4077fa0 0x7f8ed407a460 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40096c0 tx=0x7f8ee4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:35.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.595+0000 7f8eeb400640 1 -- 172.21.15.31:0/2055433329 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8ee00616c0 con 0x7f8ef40ada20 2024-09-11T00:49:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.893+0000 7f8ef8c00640 1 -- 172.21.15.31:0/2055433329 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ef4158520 con 0x7f8ef40ada20 2024-09-11T00:49:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.894+0000 7f8eeb400640 1 -- 172.21.15.31:0/2055433329 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+5711 (secure 0 0 0) 0x7f8ee006a430 con 0x7f8ef40ada20 2024-09-11T00:49:35.896 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:35.896 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":139,"btime":"2024-09-11T00:49:35:499632+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:34.875310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:49:35.896 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 139 2024-09-11T00:49:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.896+0000 7f8ef8c00640 1 -- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ed4077fa0 msgr2=0x7f8ed407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.896+0000 7f8ef8c00640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ed4077fa0 0x7f8ed407a460 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40096c0 tx=0x7f8ee4004730 comp rx=0 tx=0).stop 2024-09-11T00:49:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.897+0000 7f8ef8c00640 1 -- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ef40ada20 msgr2=0x7f8ef40b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.897+0000 7f8ef8c00640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ef40ada20 0x7f8ef40b0d00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000c3d0 tx=0x7f8ee000c890 comp rx=0 tx=0).stop 2024-09-11T00:49:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.897+0000 7f8ef8c00640 1 -- 172.21.15.31:0/2055433329 shutdown_connections 2024-09-11T00:49:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.897+0000 7f8ef8c00640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8ed4077fa0 0x7f8ed407a460 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.897+0000 7f8ef8c00640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ef4150020 0x7f8ef414cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.897+0000 7f8ef8c00640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8ef40ada20 0x7f8ef40b0d00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.897+0000 7f8ef8c00640 1 --2- 172.21.15.31:0/2055433329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef406e850 0x7f8ef40af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.897+0000 7f8ef8c00640 1 -- 172.21.15.31:0/2055433329 >> 172.21.15.31:0/2055433329 conn(0x7f8ef4068f10 msgr2=0x7f8ef41e29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.897+0000 7f8ef8c00640 1 -- 172.21.15.31:0/2055433329 shutdown_connections 2024-09-11T00:49:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:35.897+0000 7f8ef8c00640 1 -- 172.21.15.31:0/2055433329 wait complete. 2024-09-11T00:49:35.915 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-11T00:49:35.916 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:49:35.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:36.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.095+0000 7f7bd0e00640 1 -- 172.21.15.31:0/1708476886 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bcc14ccc0 msgr2=0x7f7bcc0ad590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:36.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.096+0000 7f7bd0e00640 1 --2- 172.21.15.31:0/1708476886 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bcc14ccc0 0x7f7bcc0ad590 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc153bc0 tx=0x7f7bb802a540 comp rx=0 tx=0).stop 2024-09-11T00:49:36.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.096+0000 7f7bd0e00640 1 -- 172.21.15.31:0/1708476886 shutdown_connections 2024-09-11T00:49:36.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.096+0000 7f7bd0e00640 1 --2- 172.21.15.31:0/1708476886 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bcc14ccc0 0x7f7bcc0ad590 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:36.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.096+0000 7f7bd0e00640 1 --2- 172.21.15.31:0/1708476886 >> v2:172.21.15.31:3300/0 conn(0x7f7bcc14c8e0 0x7f7bcc156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:36.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.096+0000 7f7bd0e00640 1 -- 172.21.15.31:0/1708476886 >> 172.21.15.31:0/1708476886 conn(0x7f7bcc068ee0 msgr2=0x7f7bcc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:36.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.096+0000 7f7bd0e00640 1 -- 172.21.15.31:0/1708476886 shutdown_connections 2024-09-11T00:49:36.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.096+0000 7f7bd0e00640 1 -- 172.21.15.31:0/1708476886 wait complete. 2024-09-11T00:49:36.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.098+0000 7f7bd0e00640 1 Processor -- start 2024-09-11T00:49:36.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.098+0000 7f7bd0e00640 1 -- start start 2024-09-11T00:49:36.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bd0e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bcc0c9a10 0x7f7bcc158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:36.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bd0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bcc14c8e0 0x7f7bcc1efa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:36.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bd0e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bcc14ccc0 0x7f7bcc14b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:36.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bd0e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc0c6490 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bd0e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc0c6800 con 0x7f7bcc14ccc0 2024-09-11T00:49:36.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bd0e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7bcc0c6c90 con 0x7f7bcc0c9a10 2024-09-11T00:49:36.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bcaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bcc14c8e0 0x7f7bcc1efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bcaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bcc14c8e0 0x7f7bcc1efa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38450/0 (socket says 172.21.15.31:38450) 2024-09-11T00:49:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bcaa00640 1 -- 172.21.15.31:0/805660972 learned_addr learned my addr 172.21.15.31:0/805660972 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bcb400640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bcc0c9a10 0x7f7bcc158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.099+0000 7f7bcbe00640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bcc14ccc0 0x7f7bcc14b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.100+0000 7f7bcaa00640 1 -- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bcc0c9a10 msgr2=0x7f7bcc158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.100+0000 7f7bcaa00640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bcc0c9a10 0x7f7bcc158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.100+0000 7f7bcaa00640 1 -- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bcc14ccc0 msgr2=0x7f7bcc14b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.100+0000 7f7bcaa00640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bcc14ccc0 0x7f7bcc14b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.100+0000 7f7bcaa00640 1 -- 172.21.15.31:0/805660972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bcc14c230 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.100+0000 7f7bcaa00640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bcc14c8e0 0x7f7bcc1efa20 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f7bb802ab30 tx=0x7f7bb8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.100+0000 7f7bc3400640 1 -- 172.21.15.31:0/805660972 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8005c90 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.100+0000 7f7bc3400640 1 -- 172.21.15.31:0/805660972 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7bb8031e50 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.100+0000 7f7bd0e00640 1 -- 172.21.15.31:0/805660972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bcc1eca60 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.100+0000 7f7bc3400640 1 -- 172.21.15.31:0/805660972 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8005260 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.101+0000 7f7bd0e00640 1 -- 172.21.15.31:0/805660972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bcc158c70 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.102+0000 7f7bd0e00640 1 -- 172.21.15.31:0/805660972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bcc0c7a70 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.103+0000 7f7bc3400640 1 -- 172.21.15.31:0/805660972 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7bb8005400 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.104+0000 7f7bc3400640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7bac077f70 0x7f7bac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:36.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.104+0000 7f7bc3400640 1 -- 172.21.15.31:0/805660972 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f7bb80bb8a0 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.104+0000 7f7bcb400640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7bac077f70 0x7f7bac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:36.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.105+0000 7f7bcb400640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7bac077f70 0x7f7bac07a430 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc1547f0 tx=0x7f7bbc005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:36.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.109+0000 7f7bc3400640 1 -- 172.21.15.31:0/805660972 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7bcc0c7a70 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.350+0000 7f7bd0e00640 1 -- 172.21.15.31:0/805660972 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bcc0adc90 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.351+0000 7f7bc3400640 1 -- 172.21.15.31:0/805660972 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+5711 (secure 0 0 0) 0x7f7bb803f030 con 0x7f7bcc14c8e0 2024-09-11T00:49:36.352 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:36.353 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":139,"btime":"2024-09-11T00:49:35:499632+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:34.875310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:49:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 139 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 -- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7bac077f70 msgr2=0x7f7bac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7bac077f70 0x7f7bac07a430 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc1547f0 tx=0x7f7bbc005f90 comp rx=0 tx=0).stop 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 -- 172.21.15.31:0/805660972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bcc14c8e0 msgr2=0x7f7bcc1efa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bcc14c8e0 0x7f7bcc1efa20 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f7bb802ab30 tx=0x7f7bb8033c20 comp rx=0 tx=0).stop 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 -- 172.21.15.31:0/805660972 shutdown_connections 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7bac077f70 0x7f7bac07a430 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7bcc14ccc0 0x7f7bcc14b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7bcc14c8e0 0x7f7bcc1efa20 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 --2- 172.21.15.31:0/805660972 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7bcc0c9a10 0x7f7bcc158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 -- 172.21.15.31:0/805660972 >> 172.21.15.31:0/805660972 conn(0x7f7bcc068ee0 msgr2=0x7f7bcc155840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:36.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 -- 172.21.15.31:0/805660972 shutdown_connections 2024-09-11T00:49:36.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:36.354+0000 7f7bd0e00640 1 -- 172.21.15.31:0/805660972 wait complete. 2024-09-11T00:49:36.374 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:49:35:499632+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 139, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 136, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6934": { "addr": "172.21.15.204:6837/4164016077", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 4164016077, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 4164016077, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6934, "incarnation": 133, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:48:17.824495+0000", "qdb_cluster": [ 6934 ], "qdb_leader": 6934, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6934 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 139, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7065": { "addr": "172.21.15.31:6835/3725011623", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3725011623, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3725011623, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7065, "incarnation": 138, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:49:34.875310+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7065 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:49:38.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:38.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.548+0000 7f55dce00640 1 -- 172.21.15.31:0/295803072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d815a540 msgr2=0x7f55d806eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:38.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.549+0000 7f55dce00640 1 --2- 172.21.15.31:0/295803072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d815a540 0x7f55d806eea0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f55d0005fd0 tx=0x7f55d002a820 comp rx=0 tx=0).stop 2024-09-11T00:49:38.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.549+0000 7f55dce00640 1 -- 172.21.15.31:0/295803072 shutdown_connections 2024-09-11T00:49:38.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.549+0000 7f55dce00640 1 --2- 172.21.15.31:0/295803072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d815a540 0x7f55d806eea0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:38.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.549+0000 7f55dce00640 1 --2- 172.21.15.31:0/295803072 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55d80c69e0 0x7f55d806e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:38.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.549+0000 7f55dce00640 1 -- 172.21.15.31:0/295803072 >> 172.21.15.31:0/295803072 conn(0x7f55d80c9af0 msgr2=0x7f55d80c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:38.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.549+0000 7f55dce00640 1 -- 172.21.15.31:0/295803072 shutdown_connections 2024-09-11T00:49:38.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.549+0000 7f55dce00640 1 -- 172.21.15.31:0/295803072 wait complete. 2024-09-11T00:49:38.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.551+0000 7f55dce00640 1 Processor -- start 2024-09-11T00:49:38.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.551+0000 7f55dce00640 1 -- start start 2024-09-11T00:49:38.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.552+0000 7f55dce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d80c6600 0x7f55d81508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:38.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.552+0000 7f55dce00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55d80c69e0 0x7f55d8157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.552+0000 7f55dce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55d815a540 0x7f55d80bf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.552+0000 7f55dce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f55d806f3e0 con 0x7f55d80c6600 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.552+0000 7f55dce00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f55d806fa60 con 0x7f55d815a540 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.552+0000 7f55dce00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f55d806f560 con 0x7f55d80c69e0 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.552+0000 7f55d7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d80c6600 0x7f55d81508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.552+0000 7f55d7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d80c6600 0x7f55d81508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57264/0 (socket says 172.21.15.31:57264) 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.552+0000 7f55d7400640 1 -- 172.21.15.31:0/2605288654 learned_addr learned my addr 172.21.15.31:0/2605288654 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.553+0000 7f55d6a00640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55d80c69e0 0x7f55d8157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.553+0000 7f55d7e00640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55d815a540 0x7f55d80bf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.553+0000 7f55d7400640 1 -- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55d80c69e0 msgr2=0x7f55d8157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:38.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.553+0000 7f55d7400640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55d80c69e0 0x7f55d8157930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.553+0000 7f55d7400640 1 -- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55d815a540 msgr2=0x7f55d80bf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.553+0000 7f55d7400640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55d815a540 0x7f55d80bf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.553+0000 7f55d7400640 1 -- 172.21.15.31:0/2605288654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d8159490 con 0x7f55d80c6600 2024-09-11T00:49:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.553+0000 7f55d7400640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d80c6600 0x7f55d81508c0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c3c0 tx=0x7f55c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.553+0000 7f55cf400640 1 -- 172.21.15.31:0/2605288654 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8017070 con 0x7f55d80c6600 2024-09-11T00:49:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.554+0000 7f55cf400640 1 -- 172.21.15.31:0/2605288654 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55c80035f0 con 0x7f55d80c6600 2024-09-11T00:49:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.554+0000 7f55cf400640 1 -- 172.21.15.31:0/2605288654 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8003a80 con 0x7f55d80c6600 2024-09-11T00:49:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.554+0000 7f55dce00640 1 -- 172.21.15.31:0/2605288654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d8159770 con 0x7f55d80c6600 2024-09-11T00:49:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.554+0000 7f55dce00640 1 -- 172.21.15.31:0/2605288654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d80c0a30 con 0x7f55d80c6600 2024-09-11T00:49:38.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.555+0000 7f55dce00640 1 -- 172.21.15.31:0/2605288654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d80c9270 con 0x7f55d80c6600 2024-09-11T00:49:38.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.556+0000 7f55cf400640 1 -- 172.21.15.31:0/2605288654 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f55c8003c20 con 0x7f55d80c6600 2024-09-11T00:49:38.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.557+0000 7f55cf400640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55b8077f70 0x7f55b807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:38.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.557+0000 7f55cf400640 1 -- 172.21.15.31:0/2605288654 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f55c8099f30 con 0x7f55d80c6600 2024-09-11T00:49:38.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.558+0000 7f55d6a00640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55b8077f70 0x7f55b807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:38.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.558+0000 7f55d6a00640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55b8077f70 0x7f55b807a430 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f55d81e28d0 tx=0x7f55c400b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:38.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.562+0000 7f55cf400640 1 -- 172.21.15.31:0/2605288654 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f55c8061740 con 0x7f55d80c6600 2024-09-11T00:49:38.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.839+0000 7f55dce00640 1 -- 172.21.15.31:0/2605288654 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55d80ae9d0 con 0x7f55d80c6600 2024-09-11T00:49:38.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.839+0000 7f55cf400640 1 -- 172.21.15.31:0/2605288654 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+5715 (secure 0 0 0) 0x7f55c8066170 con 0x7f55d80c6600 2024-09-11T00:49:38.841 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:38.841 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":141,"btime":"2024-09-11T00:49:37:512748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:49:38.842 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 141 2024-09-11T00:49:38.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 -- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55b8077f70 msgr2=0x7f55b807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:38.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55b8077f70 0x7f55b807a430 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f55d81e28d0 tx=0x7f55c400b2e0 comp rx=0 tx=0).stop 2024-09-11T00:49:38.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 -- 172.21.15.31:0/2605288654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d80c6600 msgr2=0x7f55d81508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:38.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d80c6600 0x7f55d81508c0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c3c0 tx=0x7f55c800c880 comp rx=0 tx=0).stop 2024-09-11T00:49:38.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 -- 172.21.15.31:0/2605288654 shutdown_connections 2024-09-11T00:49:38.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f55b8077f70 0x7f55b807a430 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:38.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f55d815a540 0x7f55d80bf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:38.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f55d80c69e0 0x7f55d8157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:38.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 --2- 172.21.15.31:0/2605288654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55d80c6600 0x7f55d81508c0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:38.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 -- 172.21.15.31:0/2605288654 >> 172.21.15.31:0/2605288654 conn(0x7f55d80c9af0 msgr2=0x7f55d8159e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:38.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 -- 172.21.15.31:0/2605288654 shutdown_connections 2024-09-11T00:49:38.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:38.842+0000 7f55dce00640 1 -- 172.21.15.31:0/2605288654 wait complete. 2024-09-11T00:49:38.860 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=6829 2024-09-11T00:49:38.861 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.d 2024-09-11T00:49:39.857 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-11T00:49:39.856+0000 7fba4ea00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:49:44.861 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-11T00:49:44.861 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:49:44.862 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:49:44.866 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:49:44.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:44.998 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:49:45.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.055+0000 7f1fce200640 1 -- 172.21.15.31:0/3670506368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fc81567f0 msgr2=0x7f1fc8156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:45.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.055+0000 7f1fce200640 1 --2- 172.21.15.31:0/3670506368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fc81567f0 0x7f1fc8156c60 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1fc806dad0 tx=0x7f1fc002a7a0 comp rx=0 tx=0).stop 2024-09-11T00:49:45.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.055+0000 7f1fce200640 1 -- 172.21.15.31:0/3670506368 shutdown_connections 2024-09-11T00:49:45.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.055+0000 7f1fce200640 1 --2- 172.21.15.31:0/3670506368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fc81567f0 0x7f1fc8156c60 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:45.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.055+0000 7f1fce200640 1 --2- 172.21.15.31:0/3670506368 >> v2:172.21.15.204:3300/0 conn(0x7f1fc80c6bb0 0x7f1fc806e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:45.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.055+0000 7f1fce200640 1 -- 172.21.15.31:0/3670506368 >> 172.21.15.31:0/3670506368 conn(0x7f1fc806caf0 msgr2=0x7f1fc806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:45.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.055+0000 7f1fce200640 1 -- 172.21.15.31:0/3670506368 shutdown_connections 2024-09-11T00:49:45.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.055+0000 7f1fce200640 1 -- 172.21.15.31:0/3670506368 wait complete. 2024-09-11T00:49:45.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.057+0000 7f1fce200640 1 Processor -- start 2024-09-11T00:49:45.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.057+0000 7f1fce200640 1 -- start start 2024-09-11T00:49:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fce200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc80c3280 0x7f1fc8152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fce200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fc80c6bb0 0x7f1fc814fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fce200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fc81567f0 0x7f1fc8154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fce200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc81571a0 con 0x7f1fc80c6bb0 2024-09-11T00:49:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fce200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc806e5b0 con 0x7f1fc80c3280 2024-09-11T00:49:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fce200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc8152d80 con 0x7f1fc81567f0 2024-09-11T00:49:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fcce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc80c3280 0x7f1fc8152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fc7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fc80c6bb0 0x7f1fc814fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fcce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc80c3280 0x7f1fc8152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33422/0 (socket says 172.21.15.31:33422) 2024-09-11T00:49:45.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fcce00640 1 -- 172.21.15.31:0/3417991677 learned_addr learned my addr 172.21.15.31:0/3417991677 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:45.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.058+0000 7f1fcd800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fc81567f0 0x7f1fc8154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.059+0000 7f1fcce00640 1 -- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fc81567f0 msgr2=0x7f1fc8154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.059+0000 7f1fcce00640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fc81567f0 0x7f1fc8154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.059+0000 7f1fcce00640 1 -- 172.21.15.31:0/3417991677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fc80c6bb0 msgr2=0x7f1fc814fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.059+0000 7f1fcce00640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fc80c6bb0 0x7f1fc814fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.059+0000 7f1fcce00640 1 -- 172.21.15.31:0/3417991677 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc80bf970 con 0x7f1fc80c3280 2024-09-11T00:49:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.059+0000 7f1fcce00640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc80c3280 0x7f1fc8152210 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00e9c0 tx=0x7f1fbc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.059+0000 7f1fc4c00640 1 -- 172.21.15.31:0/3417991677 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc00ccc0 con 0x7f1fc80c3280 2024-09-11T00:49:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.059+0000 7f1fc4c00640 1 -- 172.21.15.31:0/3417991677 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fbc005020 con 0x7f1fc80c3280 2024-09-11T00:49:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.059+0000 7f1fc4c00640 1 -- 172.21.15.31:0/3417991677 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc00ce60 con 0x7f1fc80c3280 2024-09-11T00:49:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.059+0000 7f1fce200640 1 -- 172.21.15.31:0/3417991677 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc80bfc50 con 0x7f1fc80c3280 2024-09-11T00:49:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.060+0000 7f1fce200640 1 -- 172.21.15.31:0/3417991677 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc8152750 con 0x7f1fc80c3280 2024-09-11T00:49:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.061+0000 7f1fce200640 1 -- 172.21.15.31:0/3417991677 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc80c6f90 con 0x7f1fc80c3280 2024-09-11T00:49:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.062+0000 7f1fc4c00640 1 -- 172.21.15.31:0/3417991677 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1fbc005450 con 0x7f1fc80c3280 2024-09-11T00:49:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.063+0000 7f1fc4c00640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1fa8077f70 0x7f1fa807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.063+0000 7f1fc4c00640 1 -- 172.21.15.31:0/3417991677 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f1fbc09b2c0 con 0x7f1fc80c3280 2024-09-11T00:49:45.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.064+0000 7f1fc7e00640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1fa8077f70 0x7f1fa807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:45.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.065+0000 7f1fc7e00640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1fa8077f70 0x7f1fa807a430 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80c2180 tx=0x7f1fb4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:45.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.068+0000 7f1fc4c00640 1 -- 172.21.15.31:0/3417991677 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1fbc062ad0 con 0x7f1fc80c3280 2024-09-11T00:49:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.329+0000 7f1fce200640 1 -- 172.21.15.31:0/3417991677 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1fc80c4150 con 0x7f1fc80c3280 2024-09-11T00:49:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.330+0000 7f1fc4c00640 1 -- 172.21.15.31:0/3417991677 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+5715 (secure 0 0 0) 0x7f1fbc067500 con 0x7f1fc80c3280 2024-09-11T00:49:45.331 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:45.331 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":141,"btime":"2024-09-11T00:49:37:512748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:49:45.332 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 141 2024-09-11T00:49:45.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.332+0000 7f1fce200640 1 -- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1fa8077f70 msgr2=0x7f1fa807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:45.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.332+0000 7f1fce200640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1fa8077f70 0x7f1fa807a430 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80c2180 tx=0x7f1fb4009040 comp rx=0 tx=0).stop 2024-09-11T00:49:45.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.332+0000 7f1fce200640 1 -- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc80c3280 msgr2=0x7f1fc8152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.332+0000 7f1fce200640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc80c3280 0x7f1fc8152210 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00e9c0 tx=0x7f1fbc00ee80 comp rx=0 tx=0).stop 2024-09-11T00:49:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.332+0000 7f1fce200640 1 -- 172.21.15.31:0/3417991677 shutdown_connections 2024-09-11T00:49:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.332+0000 7f1fce200640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1fa8077f70 0x7f1fa807a430 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.332+0000 7f1fce200640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1fc81567f0 0x7f1fc8154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.332+0000 7f1fce200640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1fc80c6bb0 0x7f1fc814fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.332+0000 7f1fce200640 1 --2- 172.21.15.31:0/3417991677 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc80c3280 0x7f1fc8152210 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.332+0000 7f1fce200640 1 -- 172.21.15.31:0/3417991677 >> 172.21.15.31:0/3417991677 conn(0x7f1fc806caf0 msgr2=0x7f1fc806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.333+0000 7f1fce200640 1 -- 172.21.15.31:0/3417991677 shutdown_connections 2024-09-11T00:49:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:45.333+0000 7f1fce200640 1 -- 172.21.15.31:0/3417991677 wait complete. 2024-09-11T00:49:45.352 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:49:47.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:47.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.535+0000 7f5a80000640 1 -- 172.21.15.31:0/2745164790 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a781518b0 msgr2=0x7f5a780c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:47.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.535+0000 7f5a80000640 1 --2- 172.21.15.31:0/2745164790 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a781518b0 0x7f5a780c3600 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5a70005fd0 tx=0x7f5a7002a930 comp rx=0 tx=0).stop 2024-09-11T00:49:47.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.536+0000 7f5a80000640 1 -- 172.21.15.31:0/2745164790 shutdown_connections 2024-09-11T00:49:47.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.536+0000 7f5a80000640 1 --2- 172.21.15.31:0/2745164790 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a781518b0 0x7f5a780c3600 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:47.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.536+0000 7f5a80000640 1 --2- 172.21.15.31:0/2745164790 >> v2:172.21.15.204:3300/0 conn(0x7f5a780c23a0 0x7f5a780c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:47.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.536+0000 7f5a80000640 1 -- 172.21.15.31:0/2745164790 >> 172.21.15.31:0/2745164790 conn(0x7f5a780a9750 msgr2=0x7f5a780a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:47.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.536+0000 7f5a80000640 1 -- 172.21.15.31:0/2745164790 shutdown_connections 2024-09-11T00:49:47.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.536+0000 7f5a80000640 1 -- 172.21.15.31:0/2745164790 wait complete. 2024-09-11T00:49:47.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.537+0000 7f5a80000640 1 Processor -- start 2024-09-11T00:49:47.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.537+0000 7f5a80000640 1 -- start start 2024-09-11T00:49:47.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a80000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a78072480 0x7f5a781575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:47.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a80000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a780c23a0 0x7f5a78157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:47.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a80000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a781518b0 0x7f5a780ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:47.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a80000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5a78074200 con 0x7f5a78072480 2024-09-11T00:49:47.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a80000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5a78074080 con 0x7f5a781518b0 2024-09-11T00:49:47.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a80000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5a78072860 con 0x7f5a780c23a0 2024-09-11T00:49:47.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a7d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a78072480 0x7f5a781575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:47.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a7cc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a780c23a0 0x7f5a78157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:47.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a7d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a78072480 0x7f5a781575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57278/0 (socket says 172.21.15.31:57278) 2024-09-11T00:49:47.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a7d600640 1 -- 172.21.15.31:0/675925177 learned_addr learned my addr 172.21.15.31:0/675925177 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:47.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.538+0000 7f5a7e000640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a781518b0 0x7f5a780ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:47.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.539+0000 7f5a7cc00640 1 -- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a781518b0 msgr2=0x7f5a780ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:47.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.539+0000 7f5a7cc00640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a781518b0 0x7f5a780ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:47.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.539+0000 7f5a7cc00640 1 -- 172.21.15.31:0/675925177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a78072480 msgr2=0x7f5a781575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:47.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.539+0000 7f5a7cc00640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a78072480 0x7f5a781575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:47.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.539+0000 7f5a7cc00640 1 -- 172.21.15.31:0/675925177 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a781567c0 con 0x7f5a780c23a0 2024-09-11T00:49:47.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.539+0000 7f5a7cc00640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a780c23a0 0x7f5a78157fa0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400e9c0 tx=0x7f5a6400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:47.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.540+0000 7f5a75600640 1 -- 172.21.15.31:0/675925177 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400ccc0 con 0x7f5a780c23a0 2024-09-11T00:49:47.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.540+0000 7f5a75600640 1 -- 172.21.15.31:0/675925177 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a64010430 con 0x7f5a780c23a0 2024-09-11T00:49:47.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.540+0000 7f5a75600640 1 -- 172.21.15.31:0/675925177 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a64010860 con 0x7f5a780c23a0 2024-09-11T00:49:47.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.540+0000 7f5a80000640 1 -- 172.21.15.31:0/675925177 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a781553b0 con 0x7f5a780c23a0 2024-09-11T00:49:47.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.540+0000 7f5a80000640 1 -- 172.21.15.31:0/675925177 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a78155690 con 0x7f5a780c23a0 2024-09-11T00:49:47.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.541+0000 7f5a80000640 1 -- 172.21.15.31:0/675925177 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a780c3b90 con 0x7f5a780c23a0 2024-09-11T00:49:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.543+0000 7f5a75600640 1 -- 172.21.15.31:0/675925177 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5a64010a00 con 0x7f5a780c23a0 2024-09-11T00:49:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.544+0000 7f5a75600640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a58077fc0 0x7f5a5807a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.544+0000 7f5a75600640 1 -- 172.21.15.31:0/675925177 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f5a6409b3c0 con 0x7f5a780c23a0 2024-09-11T00:49:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.544+0000 7f5a7d600640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a58077fc0 0x7f5a5807a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:47.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.545+0000 7f5a7d600640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a58077fc0 0x7f5a5807a480 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c007fd0 tx=0x7f5a6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:47.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.550+0000 7f5a75600640 1 -- 172.21.15.31:0/675925177 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5a64062c50 con 0x7f5a780c23a0 2024-09-11T00:49:47.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.856+0000 7f5a80000640 1 -- 172.21.15.31:0/675925177 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a7814d4c0 con 0x7f5a780c23a0 2024-09-11T00:49:47.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.857+0000 7f5a75600640 1 -- 172.21.15.31:0/675925177 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7f5a64067680 con 0x7f5a780c23a0 2024-09-11T00:49:47.859 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:47.859 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:49:47.859 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:49:47.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.859+0000 7f5a80000640 1 -- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a58077fc0 msgr2=0x7f5a5807a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:47.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.859+0000 7f5a80000640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a58077fc0 0x7f5a5807a480 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c007fd0 tx=0x7f5a6c009040 comp rx=0 tx=0).stop 2024-09-11T00:49:47.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.859+0000 7f5a80000640 1 -- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a780c23a0 msgr2=0x7f5a78157fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:47.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.859+0000 7f5a80000640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a780c23a0 0x7f5a78157fa0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400e9c0 tx=0x7f5a6400ee80 comp rx=0 tx=0).stop 2024-09-11T00:49:47.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.860+0000 7f5a80000640 1 -- 172.21.15.31:0/675925177 shutdown_connections 2024-09-11T00:49:47.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.860+0000 7f5a80000640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5a58077fc0 0x7f5a5807a480 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:47.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.860+0000 7f5a80000640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a781518b0 0x7f5a780ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:47.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.860+0000 7f5a80000640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5a780c23a0 0x7f5a78157fa0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:47.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.860+0000 7f5a80000640 1 --2- 172.21.15.31:0/675925177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a78072480 0x7f5a781575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:47.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.860+0000 7f5a80000640 1 -- 172.21.15.31:0/675925177 >> 172.21.15.31:0/675925177 conn(0x7f5a780a9750 msgr2=0x7f5a781e0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:47.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.860+0000 7f5a80000640 1 -- 172.21.15.31:0/675925177 shutdown_connections 2024-09-11T00:49:47.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:47.860+0000 7f5a80000640 1 -- 172.21.15.31:0/675925177 wait complete. 2024-09-11T00:49:47.878 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-11T00:49:47.878 INFO:tasks.mds_thrash.fs.[b]:waiting for 19 secs before thrashing 2024-09-11T00:49:50.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:49:50.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:49:57.988 INFO:tasks.workunit.client.0.smithi031.stdout:fs setup took 995 secs 2024-09-11T00:49:58.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:58.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.168+0000 7f2559800640 1 -- 172.21.15.31:0/2865984681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f255406e2b0 msgr2=0x7f255414f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:58.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.168+0000 7f2559800640 1 --2- 172.21.15.31:0/2865984681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f255406e2b0 0x7f255414f710 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2540005fd0 tx=0x7f254002a4b0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.169+0000 7f2559800640 1 -- 172.21.15.31:0/2865984681 shutdown_connections 2024-09-11T00:49:58.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.169+0000 7f2559800640 1 --2- 172.21.15.31:0/2865984681 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f255406e2b0 0x7f255414f710 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.169+0000 7f2559800640 1 --2- 172.21.15.31:0/2865984681 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f255406ded0 0x7f25540ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.169+0000 7f2559800640 1 -- 172.21.15.31:0/2865984681 >> 172.21.15.31:0/2865984681 conn(0x7f255406caf0 msgr2=0x7f255406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:58.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.169+0000 7f2559800640 1 -- 172.21.15.31:0/2865984681 shutdown_connections 2024-09-11T00:49:58.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.169+0000 7f2559800640 1 -- 172.21.15.31:0/2865984681 wait complete. 2024-09-11T00:49:58.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.170+0000 7f2559800640 1 Processor -- start 2024-09-11T00:49:58.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.170+0000 7f2559800640 1 -- start start 2024-09-11T00:49:58.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.171+0000 7f2559800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f255406ded0 0x7f255414bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:58.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.171+0000 7f2559800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f255406e2b0 0x7f25541ef010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.171+0000 7f2559800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f255414fc50 0x7f25541ef550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.171+0000 7f2559800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2554156800 con 0x7f255406e2b0 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.171+0000 7f2559800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f25540c52a0 con 0x7f255406ded0 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.171+0000 7f2559800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f255406e710 con 0x7f255414fc50 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.171+0000 7f2553400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f255406e2b0 0x7f25541ef010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.172+0000 7f2553400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f255406e2b0 0x7f25541ef010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54816/0 (socket says 172.21.15.31:54816) 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.172+0000 7f2553400640 1 -- 172.21.15.31:0/2928449065 learned_addr learned my addr 172.21.15.31:0/2928449065 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.172+0000 7f2553e00640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f255406ded0 0x7f255414bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.172+0000 7f2558e00640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f255414fc50 0x7f25541ef550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.172+0000 7f2553e00640 1 -- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f255414fc50 msgr2=0x7f25541ef550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.172+0000 7f2553e00640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f255414fc50 0x7f25541ef550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.172+0000 7f2553e00640 1 -- 172.21.15.31:0/2928449065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f255406e2b0 msgr2=0x7f25541ef010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:58.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.172+0000 7f2553e00640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f255406e2b0 0x7f25541ef010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.172+0000 7f2553e00640 1 -- 172.21.15.31:0/2928449065 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255414c0b0 con 0x7f255406ded0 2024-09-11T00:49:58.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.172+0000 7f2553e00640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f255406ded0 0x7f255414bb70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f254400eac0 tx=0x7f254400ef80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.173+0000 7f254be00640 1 -- 172.21.15.31:0/2928449065 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254401b070 con 0x7f255406ded0 2024-09-11T00:49:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.173+0000 7f254be00640 1 -- 172.21.15.31:0/2928449065 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f254400cd70 con 0x7f255406ded0 2024-09-11T00:49:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.173+0000 7f254be00640 1 -- 172.21.15.31:0/2928449065 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544016670 con 0x7f255406ded0 2024-09-11T00:49:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.173+0000 7f2559800640 1 -- 172.21.15.31:0/2928449065 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25540a6f40 con 0x7f255406ded0 2024-09-11T00:49:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.173+0000 7f2559800640 1 -- 172.21.15.31:0/2928449065 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2554152a10 con 0x7f255406ded0 2024-09-11T00:49:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.174+0000 7f2559800640 1 -- 172.21.15.31:0/2928449065 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25541efa90 con 0x7f255406ded0 2024-09-11T00:49:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.175+0000 7f254be00640 1 -- 172.21.15.31:0/2928449065 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f254400b040 con 0x7f255406ded0 2024-09-11T00:49:58.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.176+0000 7f254be00640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 0x7f253407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:58.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.176+0000 7f254be00640 1 -- 172.21.15.31:0/2928449065 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f254409acb0 con 0x7f255406ded0 2024-09-11T00:49:58.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.176+0000 7f2553400640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 0x7f253407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:58.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.177+0000 7f2553400640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 0x7f253407a430 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2540000c00 tx=0x7f2540004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:58.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.181+0000 7f254be00640 1 -- 172.21.15.31:0/2928449065 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f25541efa90 con 0x7f255406ded0 2024-09-11T00:49:58.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.411+0000 7f2559800640 1 -- 172.21.15.31:0/2928449065 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25540ac8d0 con 0x7f255406ded0 2024-09-11T00:49:58.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.412+0000 7f254be00640 1 -- 172.21.15.31:0/2928449065 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7f25440624c0 con 0x7f255406ded0 2024-09-11T00:49:58.413 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:58.413 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:49:58.414 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:49:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 -- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 msgr2=0x7f253407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 0x7f253407a430 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2540000c00 tx=0x7f2540004730 comp rx=0 tx=0).stop 2024-09-11T00:49:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 -- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f255406ded0 msgr2=0x7f255414bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f255406ded0 0x7f255414bb70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f254400eac0 tx=0x7f254400ef80 comp rx=0 tx=0).stop 2024-09-11T00:49:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 -- 172.21.15.31:0/2928449065 shutdown_connections 2024-09-11T00:49:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2534077f70 0x7f253407a430 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f255414fc50 0x7f25541ef550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f255406e2b0 0x7f25541ef010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 --2- 172.21.15.31:0/2928449065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f255406ded0 0x7f255414bb70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 -- 172.21.15.31:0/2928449065 >> 172.21.15.31:0/2928449065 conn(0x7f255406caf0 msgr2=0x7f2554153540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:58.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 -- 172.21.15.31:0/2928449065 shutdown_connections 2024-09-11T00:49:58.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.414+0000 7f2559800640 1 -- 172.21.15.31:0/2928449065 wait complete. 2024-09-11T00:49:58.431 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-09-11T00:49:58.432 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-11T00:49:58.432 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:49:58.449 INFO:tasks.ceph.mds.b:Stopped 2024-09-11T00:49:58.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:49:58.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.613+0000 7eff0be00640 1 -- 172.21.15.31:0/3422889308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff0c0c6690 msgr2=0x7eff0c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:58.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.613+0000 7eff0be00640 1 --2- 172.21.15.31:0/3422889308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff0c0c6690 0x7eff0c06dd20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7efefc005fd0 tx=0x7efefc02a540 comp rx=0 tx=0).stop 2024-09-11T00:49:58.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.613+0000 7eff0be00640 1 -- 172.21.15.31:0/3422889308 shutdown_connections 2024-09-11T00:49:58.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.613+0000 7eff0be00640 1 --2- 172.21.15.31:0/3422889308 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff0c0bf1a0 0x7eff0c0b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.613+0000 7eff0be00640 1 --2- 172.21.15.31:0/3422889308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff0c0c6690 0x7eff0c06dd20 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.613+0000 7eff0be00640 1 -- 172.21.15.31:0/3422889308 >> 172.21.15.31:0/3422889308 conn(0x7eff0c0cbec0 msgr2=0x7eff0c0cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:58.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.613+0000 7eff0be00640 1 -- 172.21.15.31:0/3422889308 shutdown_connections 2024-09-11T00:49:58.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.613+0000 7eff0be00640 1 -- 172.21.15.31:0/3422889308 wait complete. 2024-09-11T00:49:58.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.615+0000 7eff0be00640 1 Processor -- start 2024-09-11T00:49:58.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.615+0000 7eff0be00640 1 -- start start 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff0be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff0c0bf1a0 0x7eff0c14f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff0be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff0c0c6690 0x7eff0c149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff0be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff0c14a1d0 0x7eff0c1ec170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff0be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c0c6a70 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff0be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff0c155f90 con 0x7eff0c14a1d0 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff0be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7eff0c0c6bf0 con 0x7eff0c0c6690 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff0aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff0c0bf1a0 0x7eff0c14f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff0b400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff0c14a1d0 0x7eff0c1ec170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff0aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff0c0bf1a0 0x7eff0c14f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54820/0 (socket says 172.21.15.31:54820) 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff0aa00640 1 -- 172.21.15.31:0/3979762017 learned_addr learned my addr 172.21.15.31:0/3979762017 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:49:58.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.616+0000 7eff02000640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff0c0c6690 0x7eff0c149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff0aa00640 1 -- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff0c0c6690 msgr2=0x7eff0c149c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff0aa00640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff0c0c6690 0x7eff0c149c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff0aa00640 1 -- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff0c14a1d0 msgr2=0x7eff0c1ec170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff0aa00640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff0c14a1d0 0x7eff0c1ec170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff0aa00640 1 -- 172.21.15.31:0/3979762017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0c1dfeb0 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff0b400640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff0c14a1d0 0x7eff0c1ec170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff0aa00640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff0c0bf1a0 0x7eff0c14f550 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7efefc009850 tx=0x7efefc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff03400640 1 -- 172.21.15.31:0/3979762017 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc03f070 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff03400640 1 -- 172.21.15.31:0/3979762017 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efefc0047a0 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff0be00640 1 -- 172.21.15.31:0/3979762017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff0c1df5f0 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff03400640 1 -- 172.21.15.31:0/3979762017 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efefc035030 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.617+0000 7eff0be00640 1 -- 172.21.15.31:0/3979762017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff0c150b80 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.618+0000 7eff0be00640 1 -- 172.21.15.31:0/3979762017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff0c0c9b60 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.620+0000 7eff03400640 1 -- 172.21.15.31:0/3979762017 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7efefc004b90 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.620+0000 7eff03400640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efee8077fa0 0x7efee807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:49:58.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.621+0000 7eff03400640 1 -- 172.21.15.31:0/3979762017 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7efefc0bb980 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.621+0000 7eff02000640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efee8077fa0 0x7efee807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:49:58.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.621+0000 7eff02000640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efee8077fa0 0x7efee807a460 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7eff0c1552f0 tx=0x7efef4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:49:58.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.625+0000 7eff03400640 1 -- 172.21.15.31:0/3979762017 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7efefc083190 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.767 INFO:tasks.workunit.client.0.smithi031.stdout:Syncing()...0 sec 2024-09-11T00:49:58.769 INFO:tasks.workunit.client.0.smithi031.stdout:Starting Actual Benchmark At: Wed Sep 11 00:49:58 2024 2024-09-11T00:49:58.769 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:49:58.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.864+0000 7eff0be00640 1 -- 172.21.15.31:0/3979762017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff0c155320 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.865+0000 7eff03400640 1 -- 172.21.15.31:0/3979762017 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7efefc08a0e0 con 0x7eff0c0bf1a0 2024-09-11T00:49:58.867 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:49:58.867 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:49:58.867 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:49:58.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 -- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efee8077fa0 msgr2=0x7efee807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:58.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efee8077fa0 0x7efee807a460 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7eff0c1552f0 tx=0x7efef4009040 comp rx=0 tx=0).stop 2024-09-11T00:49:58.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 -- 172.21.15.31:0/3979762017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff0c0bf1a0 msgr2=0x7eff0c14f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:49:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff0c0bf1a0 0x7eff0c14f550 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7efefc009850 tx=0x7efefc0038a0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 -- 172.21.15.31:0/3979762017 shutdown_connections 2024-09-11T00:49:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efee8077fa0 0x7efee807a460 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff0c14a1d0 0x7eff0c1ec170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff0c0c6690 0x7eff0c149c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 --2- 172.21.15.31:0/3979762017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff0c0bf1a0 0x7eff0c14f550 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:49:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 -- 172.21.15.31:0/3979762017 >> 172.21.15.31:0/3979762017 conn(0x7eff0c0cbec0 msgr2=0x7eff0c155840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:49:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 -- 172.21.15.31:0/3979762017 shutdown_connections 2024-09-11T00:49:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:49:58.867+0000 7eff0be00640 1 -- 172.21.15.31:0/3979762017 wait complete. 2024-09-11T00:49:58.884 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:50:00.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:01.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.064+0000 7f3c93e00640 1 -- 172.21.15.31:0/1576348114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c8c1539c0 msgr2=0x7f3c8c159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:01.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.064+0000 7f3c93e00640 1 --2- 172.21.15.31:0/1576348114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c8c1539c0 0x7f3c8c159670 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c06e7f0 tx=0x7f3c8402a8b0 comp rx=0 tx=0).stop 2024-09-11T00:50:01.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.064+0000 7f3c93e00640 1 -- 172.21.15.31:0/1576348114 shutdown_connections 2024-09-11T00:50:01.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.064+0000 7f3c93e00640 1 --2- 172.21.15.31:0/1576348114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c8c1539c0 0x7f3c8c159670 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:01.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.064+0000 7f3c93e00640 1 --2- 172.21.15.31:0/1576348114 >> v2:172.21.15.204:3300/0 conn(0x7f3c8c156ad0 0x7f3c8c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:01.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.064+0000 7f3c93e00640 1 -- 172.21.15.31:0/1576348114 >> 172.21.15.31:0/1576348114 conn(0x7f3c8c0a9750 msgr2=0x7f3c8c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:01.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.064+0000 7f3c93e00640 1 -- 172.21.15.31:0/1576348114 shutdown_connections 2024-09-11T00:50:01.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.064+0000 7f3c93e00640 1 -- 172.21.15.31:0/1576348114 wait complete. 2024-09-11T00:50:01.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.066+0000 7f3c93e00640 1 Processor -- start 2024-09-11T00:50:01.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.066+0000 7f3c93e00640 1 -- start start 2024-09-11T00:50:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c93e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c8c0c5b10 0x7f3c8c1525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c93e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c8c1539c0 0x7f3c8c152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c93e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c8c156ad0 0x7f3c8c14a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:01.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c93e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c156f20 con 0x7f3c8c1539c0 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c93e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c0c7320 con 0x7f3c8c0c5b10 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c93e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8c0c3950 con 0x7f3c8c156ad0 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c90a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c8c1539c0 0x7f3c8c152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c91400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c8c0c5b10 0x7f3c8c1525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c91400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c8c0c5b10 0x7f3c8c1525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:33598/0 (socket says 172.21.15.31:33598) 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c91400640 1 -- 172.21.15.31:0/2226663597 learned_addr learned my addr 172.21.15.31:0/2226663597 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.067+0000 7f3c91e00640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c8c156ad0 0x7f3c8c14a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c90a00640 1 -- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c8c156ad0 msgr2=0x7f3c8c14a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c90a00640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c8c156ad0 0x7f3c8c14a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c90a00640 1 -- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c8c0c5b10 msgr2=0x7f3c8c1525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c90a00640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c8c0c5b10 0x7f3c8c1525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:01.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c90a00640 1 -- 172.21.15.31:0/2226663597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8c1e1700 con 0x7f3c8c1539c0 2024-09-11T00:50:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c90a00640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c8c1539c0 0x7f3c8c152b00 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800d700 tx=0x7f3c7800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c89600640 1 -- 172.21.15.31:0/2226663597 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78017070 con 0x7f3c8c1539c0 2024-09-11T00:50:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c89600640 1 -- 172.21.15.31:0/2226663597 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c78005310 con 0x7f3c8c1539c0 2024-09-11T00:50:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c89600640 1 -- 172.21.15.31:0/2226663597 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78009820 con 0x7f3c8c1539c0 2024-09-11T00:50:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c93e00640 1 -- 172.21.15.31:0/2226663597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c8c1e09a0 con 0x7f3c8c1539c0 2024-09-11T00:50:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.068+0000 7f3c93e00640 1 -- 172.21.15.31:0/2226663597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c8c1e0c20 con 0x7f3c8c1539c0 2024-09-11T00:50:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.070+0000 7f3c93e00640 1 -- 172.21.15.31:0/2226663597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c8c159670 con 0x7f3c8c1539c0 2024-09-11T00:50:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.071+0000 7f3c89600640 1 -- 172.21.15.31:0/2226663597 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3c780099c0 con 0x7f3c8c1539c0 2024-09-11T00:50:01.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.071+0000 7f3c89600640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c6c077f70 0x7f3c6c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:01.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.072+0000 7f3c89600640 1 -- 172.21.15.31:0/2226663597 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f3c7809aa90 con 0x7f3c8c1539c0 2024-09-11T00:50:01.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.072+0000 7f3c91400640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c6c077f70 0x7f3c6c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:01.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.072+0000 7f3c91400640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c6c077f70 0x7f3c6c07a430 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c14fa40 tx=0x7f3c80007f50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:01.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.076+0000 7f3c89600640 1 -- 172.21.15.31:0/2226663597 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3c8c1e09a0 con 0x7f3c8c1539c0 2024-09-11T00:50:01.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.308+0000 7f3c93e00640 1 -- 172.21.15.31:0/2226663597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c8c0c63a0 con 0x7f3c8c1539c0 2024-09-11T00:50:01.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.309+0000 7f3c89600640 1 -- 172.21.15.31:0/2226663597 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7f3c8c1e1700 con 0x7f3c8c1539c0 2024-09-11T00:50:01.311 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:01.311 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:01.311 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:50:01.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 -- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c6c077f70 msgr2=0x7f3c6c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:01.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c6c077f70 0x7f3c6c07a430 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c14fa40 tx=0x7f3c80007f50 comp rx=0 tx=0).stop 2024-09-11T00:50:01.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 -- 172.21.15.31:0/2226663597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c8c1539c0 msgr2=0x7f3c8c152b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:01.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c8c1539c0 0x7f3c8c152b00 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800d700 tx=0x7f3c7800dbc0 comp rx=0 tx=0).stop 2024-09-11T00:50:01.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 -- 172.21.15.31:0/2226663597 shutdown_connections 2024-09-11T00:50:01.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3c6c077f70 0x7f3c6c07a430 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3c8c156ad0 0x7f3c8c14a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c8c1539c0 0x7f3c8c152b00 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 --2- 172.21.15.31:0/2226663597 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c8c0c5b10 0x7f3c8c1525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 -- 172.21.15.31:0/2226663597 >> 172.21.15.31:0/2226663597 conn(0x7f3c8c0a9750 msgr2=0x7f3c8c0c15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 -- 172.21.15.31:0/2226663597 shutdown_connections 2024-09-11T00:50:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:01.311+0000 7f3c93e00640 1 -- 172.21.15.31:0/2226663597 wait complete. 2024-09-11T00:50:01.329 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:50:03.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:03.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.502+0000 7f1d99800640 1 -- 172.21.15.31:0/963835039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94072480 msgr2=0x7f1d94074a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:03.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.502+0000 7f1d99800640 1 --2- 172.21.15.31:0/963835039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94072480 0x7f1d94074a90 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f1d84009830 tx=0x7f1d8402a4d0 comp rx=0 tx=0).stop 2024-09-11T00:50:03.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.502+0000 7f1d99800640 1 -- 172.21.15.31:0/963835039 shutdown_connections 2024-09-11T00:50:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.502+0000 7f1d99800640 1 --2- 172.21.15.31:0/963835039 >> v2:172.21.15.204:3300/0 conn(0x7f1d941511c0 0x7f1d94151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.502+0000 7f1d99800640 1 --2- 172.21.15.31:0/963835039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94072480 0x7f1d94074a90 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.502+0000 7f1d99800640 1 -- 172.21.15.31:0/963835039 >> 172.21.15.31:0/963835039 conn(0x7f1d940a9750 msgr2=0x7f1d940a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.502+0000 7f1d99800640 1 -- 172.21.15.31:0/963835039 shutdown_connections 2024-09-11T00:50:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.502+0000 7f1d99800640 1 -- 172.21.15.31:0/963835039 wait complete. 2024-09-11T00:50:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.504+0000 7f1d99800640 1 Processor -- start 2024-09-11T00:50:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.504+0000 7f1d99800640 1 -- start start 2024-09-11T00:50:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d99800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d94072480 0x7f1d941e9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d99800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d941511c0 0x7f1d941dd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d99800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d94154b60 0x7f1d940c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d99800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1d940c23a0 con 0x7f1d941511c0 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d99800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1d94154640 con 0x7f1d94154b60 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d99800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1d940a7140 con 0x7f1d94072480 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d8a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d941511c0 0x7f1d941dd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d8a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d941511c0 0x7f1d941dd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54860/0 (socket says 172.21.15.31:54860) 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d8a000640 1 -- 172.21.15.31:0/4122398624 learned_addr learned my addr 172.21.15.31:0/4122398624 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d93400640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d94154b60 0x7f1d940c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.505+0000 7f1d92a00640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d94072480 0x7f1d941e9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.506+0000 7f1d8a000640 1 -- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d94072480 msgr2=0x7f1d941e9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.506+0000 7f1d8a000640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d94072480 0x7f1d941e9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.506+0000 7f1d8a000640 1 -- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d94154b60 msgr2=0x7f1d940c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.506+0000 7f1d8a000640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d94154b60 0x7f1d940c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.506+0000 7f1d8a000640 1 -- 172.21.15.31:0/4122398624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d94140ec0 con 0x7f1d941511c0 2024-09-11T00:50:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.506+0000 7f1d8a000640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d941511c0 0x7f1d941dd810 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00c3a0 tx=0x7f1d7c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.506+0000 7f1d8b400640 1 -- 172.21.15.31:0/4122398624 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c011790 con 0x7f1d941511c0 2024-09-11T00:50:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.506+0000 7f1d8b400640 1 -- 172.21.15.31:0/4122398624 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d7c017070 con 0x7f1d941511c0 2024-09-11T00:50:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.506+0000 7f1d8b400640 1 -- 172.21.15.31:0/4122398624 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c011cc0 con 0x7f1d941511c0 2024-09-11T00:50:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.506+0000 7f1d99800640 1 -- 172.21.15.31:0/4122398624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d94140be0 con 0x7f1d941511c0 2024-09-11T00:50:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.507+0000 7f1d99800640 1 -- 172.21.15.31:0/4122398624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d941ee2b0 con 0x7f1d941511c0 2024-09-11T00:50:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.508+0000 7f1d99800640 1 -- 172.21.15.31:0/4122398624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d94072860 con 0x7f1d941511c0 2024-09-11T00:50:03.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.509+0000 7f1d8b400640 1 -- 172.21.15.31:0/4122398624 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1d7c01c460 con 0x7f1d941511c0 2024-09-11T00:50:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.510+0000 7f1d8b400640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d70077fa0 0x7f1d7007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.510+0000 7f1d8b400640 1 -- 172.21.15.31:0/4122398624 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f1d7c09a720 con 0x7f1d941511c0 2024-09-11T00:50:03.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.510+0000 7f1d92a00640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d70077fa0 0x7f1d7007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:03.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.512+0000 7f1d92a00640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d70077fa0 0x7f1d7007a460 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1d84034000 tx=0x7f1d84004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.514+0000 7f1d8b400640 1 -- 172.21.15.31:0/4122398624 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1d7c061e20 con 0x7f1d941511c0 2024-09-11T00:50:03.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.750+0000 7f1d99800640 1 -- 172.21.15.31:0/4122398624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d940a6f20 con 0x7f1d941511c0 2024-09-11T00:50:03.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.751+0000 7f1d8b400640 1 -- 172.21.15.31:0/4122398624 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7f1d7c066850 con 0x7f1d941511c0 2024-09-11T00:50:03.754 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:03.754 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:03.754 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 -- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d70077fa0 msgr2=0x7f1d7007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d70077fa0 0x7f1d7007a460 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1d84034000 tx=0x7f1d84004730 comp rx=0 tx=0).stop 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 -- 172.21.15.31:0/4122398624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d941511c0 msgr2=0x7f1d941dd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d941511c0 0x7f1d941dd810 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00c3a0 tx=0x7f1d7c00c860 comp rx=0 tx=0).stop 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 -- 172.21.15.31:0/4122398624 shutdown_connections 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1d70077fa0 0x7f1d7007a460 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d94154b60 0x7f1d940c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d941511c0 0x7f1d941dd810 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 --2- 172.21.15.31:0/4122398624 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1d94072480 0x7f1d941e9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 -- 172.21.15.31:0/4122398624 >> 172.21.15.31:0/4122398624 conn(0x7f1d940a9750 msgr2=0x7f1d9414a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 -- 172.21.15.31:0/4122398624 shutdown_connections 2024-09-11T00:50:03.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:03.754+0000 7f1d99800640 1 -- 172.21.15.31:0/4122398624 wait complete. 2024-09-11T00:50:03.773 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:50:05.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.946+0000 7f8843a00640 1 -- 172.21.15.31:0/1717701624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f883c153f20 msgr2=0x7f883c0c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.946+0000 7f8843a00640 1 --2- 172.21.15.31:0/1717701624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f883c153f20 0x7f883c0c30c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f883c071d90 tx=0x7f883402aac0 comp rx=0 tx=0).stop 2024-09-11T00:50:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.946+0000 7f8843a00640 1 -- 172.21.15.31:0/1717701624 shutdown_connections 2024-09-11T00:50:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.946+0000 7f8843a00640 1 --2- 172.21.15.31:0/1717701624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f883c153f20 0x7f883c0c30c0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.946+0000 7f8843a00640 1 --2- 172.21.15.31:0/1717701624 >> v2:172.21.15.204:3300/0 conn(0x7f883c0c23a0 0x7f883c1516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.946+0000 7f8843a00640 1 --2- 172.21.15.31:0/1717701624 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f883c072480 0x7f883c1511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.946+0000 7f8843a00640 1 -- 172.21.15.31:0/1717701624 >> 172.21.15.31:0/1717701624 conn(0x7f883c0a9750 msgr2=0x7f883c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.946+0000 7f8843a00640 1 -- 172.21.15.31:0/1717701624 shutdown_connections 2024-09-11T00:50:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.946+0000 7f8843a00640 1 -- 172.21.15.31:0/1717701624 wait complete. 2024-09-11T00:50:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.948+0000 7f8843a00640 1 Processor -- start 2024-09-11T00:50:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.948+0000 7f8843a00640 1 -- start start 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f8843a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f883c072480 0x7f883c146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f8843a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f883c0c23a0 0x7f883c146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f8843a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f883c153f20 0x7f883c1df050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f8843a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f883c074200 con 0x7f883c153f20 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f8843a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f883c074080 con 0x7f883c072480 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f8843a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f883c072860 con 0x7f883c0c23a0 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f8841a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f883c153f20 0x7f883c1df050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f8841000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f883c072480 0x7f883c146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f8841a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f883c153f20 0x7f883c1df050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54884/0 (socket says 172.21.15.31:54884) 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f8841a00640 1 -- 172.21.15.31:0/1081944976 learned_addr learned my addr 172.21.15.31:0/1081944976 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.949+0000 7f883be00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f883c0c23a0 0x7f883c146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:05.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8841a00640 1 -- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f883c0c23a0 msgr2=0x7f883c146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:05.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8841a00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f883c0c23a0 0x7f883c146490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8841a00640 1 -- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f883c072480 msgr2=0x7f883c146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8841a00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f883c072480 0x7f883c146f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8841a00640 1 -- 172.21.15.31:0/1081944976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883c0c71e0 con 0x7f883c153f20 2024-09-11T00:50:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f883be00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f883c0c23a0 0x7f883c146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:50:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8841a00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f883c153f20 0x7f883c1df050 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8834009d60 tx=0x7f8834003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8838c00640 1 -- 172.21.15.31:0/1081944976 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8834044070 con 0x7f883c153f20 2024-09-11T00:50:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8838c00640 1 -- 172.21.15.31:0/1081944976 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8834004040 con 0x7f883c153f20 2024-09-11T00:50:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8843a00640 1 -- 172.21.15.31:0/1081944976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f883c0c74c0 con 0x7f883c153f20 2024-09-11T00:50:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8838c00640 1 -- 172.21.15.31:0/1081944976 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8834004470 con 0x7f883c153f20 2024-09-11T00:50:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.950+0000 7f8843a00640 1 -- 172.21.15.31:0/1081944976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f883c146bb0 con 0x7f883c153f20 2024-09-11T00:50:05.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.952+0000 7f8843a00640 1 -- 172.21.15.31:0/1081944976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f883c1511a0 con 0x7f883c153f20 2024-09-11T00:50:05.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.953+0000 7f8838c00640 1 -- 172.21.15.31:0/1081944976 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8834004610 con 0x7f883c153f20 2024-09-11T00:50:05.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.954+0000 7f8838c00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f881c077f70 0x7f881c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:05.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.954+0000 7f8838c00640 1 -- 172.21.15.31:0/1081944976 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f88340bafc0 con 0x7f883c153f20 2024-09-11T00:50:05.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.954+0000 7f8841000640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f881c077f70 0x7f881c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:05.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.955+0000 7f8841000640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f881c077f70 0x7f881c07a430 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f882c009bd0 tx=0x7f882c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:05.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:05.959+0000 7f8838c00640 1 -- 172.21.15.31:0/1081944976 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8834032030 con 0x7f883c153f20 2024-09-11T00:50:06.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.191+0000 7f8843a00640 1 -- 172.21.15.31:0/1081944976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f883c0c7690 con 0x7f883c153f20 2024-09-11T00:50:06.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.192+0000 7f8838c00640 1 -- 172.21.15.31:0/1081944976 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7f88340826b0 con 0x7f883c153f20 2024-09-11T00:50:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:06.195 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:06.195 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:50:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.194+0000 7f8843a00640 1 -- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f881c077f70 msgr2=0x7f881c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.194+0000 7f8843a00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f881c077f70 0x7f881c07a430 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f882c009bd0 tx=0x7f882c005f90 comp rx=0 tx=0).stop 2024-09-11T00:50:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.194+0000 7f8843a00640 1 -- 172.21.15.31:0/1081944976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f883c153f20 msgr2=0x7f883c1df050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:06.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.194+0000 7f8843a00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f883c153f20 0x7f883c1df050 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8834009d60 tx=0x7f8834003c10 comp rx=0 tx=0).stop 2024-09-11T00:50:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.194+0000 7f8843a00640 1 -- 172.21.15.31:0/1081944976 shutdown_connections 2024-09-11T00:50:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.194+0000 7f8843a00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f881c077f70 0x7f881c07a430 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.194+0000 7f8843a00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f883c153f20 0x7f883c1df050 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.194+0000 7f8843a00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f883c0c23a0 0x7f883c146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.194+0000 7f8843a00640 1 --2- 172.21.15.31:0/1081944976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f883c072480 0x7f883c146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.194+0000 7f8843a00640 1 -- 172.21.15.31:0/1081944976 >> 172.21.15.31:0/1081944976 conn(0x7f883c0a9750 msgr2=0x7f883c0a8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.195+0000 7f8843a00640 1 -- 172.21.15.31:0/1081944976 shutdown_connections 2024-09-11T00:50:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:06.195+0000 7f8843a00640 1 -- 172.21.15.31:0/1081944976 wait complete. 2024-09-11T00:50:06.212 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:50:06.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:07.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.042+0000 7f0dadc00640 1 -- 172.21.15.31:0/1503021303 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0da806d440 msgr2=0x7f0da8074590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:07.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.042+0000 7f0dadc00640 1 --2- 172.21.15.31:0/1503021303 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0da806d440 0x7f0da8074590 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0da0005fd0 tx=0x7f0da002a840 comp rx=0 tx=0).stop 2024-09-11T00:50:07.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.042+0000 7f0dadc00640 1 -- 172.21.15.31:0/1503021303 shutdown_connections 2024-09-11T00:50:07.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.042+0000 7f0dadc00640 1 --2- 172.21.15.31:0/1503021303 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0da806d440 0x7f0da8074590 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.042+0000 7f0dadc00640 1 --2- 172.21.15.31:0/1503021303 >> v2:172.21.15.204:3300/0 conn(0x7f0da8070360 0x7f0da8074050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.042+0000 7f0dadc00640 1 -- 172.21.15.31:0/1503021303 >> 172.21.15.31:0/1503021303 conn(0x7f0da806dc00 msgr2=0x7f0da806e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:07.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.042+0000 7f0dadc00640 1 -- 172.21.15.31:0/1503021303 shutdown_connections 2024-09-11T00:50:07.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.042+0000 7f0dadc00640 1 -- 172.21.15.31:0/1503021303 wait complete. 2024-09-11T00:50:07.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.044+0000 7f0dadc00640 1 Processor -- start 2024-09-11T00:50:07.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.044+0000 7f0dadc00640 1 -- start start 2024-09-11T00:50:07.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0dadc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0da806d440 0x7f0da81e0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:07.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0dadc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0da8070360 0x7f0da81ec360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:07.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0dadc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da8153b30 0x7f0da81ec8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:07.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0dadc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0da80710d0 con 0x7f0da8153b30 2024-09-11T00:50:07.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0dadc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0da8070f50 con 0x7f0da806d440 2024-09-11T00:50:07.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0dadc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0da8074ad0 con 0x7f0da8070360 2024-09-11T00:50:07.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da8153b30 0x7f0da81ec8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:07.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da6000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0da8070360 0x7f0da81ec360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:07.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da8153b30 0x7f0da81ec8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54896/0 (socket says 172.21.15.31:54896) 2024-09-11T00:50:07.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da7400640 1 -- 172.21.15.31:0/1958950932 learned_addr learned my addr 172.21.15.31:0/1958950932 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:07.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da6a00640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0da806d440 0x7f0da81e0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:07.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da6000640 1 -- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0da806d440 msgr2=0x7f0da81e0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:07.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da6000640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0da806d440 0x7f0da81e0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da6000640 1 -- 172.21.15.31:0/1958950932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da8153b30 msgr2=0x7f0da81ec8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:07.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da6000640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da8153b30 0x7f0da81ec8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da6000640 1 -- 172.21.15.31:0/1958950932 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da81f0280 con 0x7f0da8070360 2024-09-11T00:50:07.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.045+0000 7f0da7400640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da8153b30 0x7f0da81ec8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:50:07.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.046+0000 7f0da6000640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0da8070360 0x7f0da81ec360 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d9400e9f0 tx=0x7f0d9400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:07.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.046+0000 7f0d9ea00640 1 -- 172.21.15.31:0/1958950932 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9400ccc0 con 0x7f0da8070360 2024-09-11T00:50:07.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.046+0000 7f0d9ea00640 1 -- 172.21.15.31:0/1958950932 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d94010430 con 0x7f0da8070360 2024-09-11T00:50:07.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.046+0000 7f0dadc00640 1 -- 172.21.15.31:0/1958950932 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da81f0560 con 0x7f0da8070360 2024-09-11T00:50:07.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.046+0000 7f0d9ea00640 1 -- 172.21.15.31:0/1958950932 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d94010860 con 0x7f0da8070360 2024-09-11T00:50:07.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.046+0000 7f0dadc00640 1 -- 172.21.15.31:0/1958950932 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da8151b30 con 0x7f0da8070360 2024-09-11T00:50:07.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.047+0000 7f0dadc00640 1 -- 172.21.15.31:0/1958950932 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0da8070840 con 0x7f0da8070360 2024-09-11T00:50:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.048+0000 7f0d9ea00640 1 -- 172.21.15.31:0/1958950932 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0d94010a00 con 0x7f0da8070360 2024-09-11T00:50:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.049+0000 7f0d9ea00640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d88077f70 0x7f0d8807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.049+0000 7f0d9ea00640 1 -- 172.21.15.31:0/1958950932 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f0d9409b420 con 0x7f0da8070360 2024-09-11T00:50:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.050+0000 7f0da6a00640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d88077f70 0x7f0d8807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.050+0000 7f0da6a00640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d88077f70 0x7f0d8807a430 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0d98000f60 tx=0x7f0d980016e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.055+0000 7f0d9ea00640 1 -- 172.21.15.31:0/1958950932 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0d94062c30 con 0x7f0da8070360 2024-09-11T00:50:07.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.288+0000 7f0dadc00640 1 -- 172.21.15.31:0/1958950932 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0da806d220 con 0x7f0da8070360 2024-09-11T00:50:07.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.290+0000 7f0d9ea00640 1 -- 172.21.15.31:0/1958950932 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7f0d94067660 con 0x7f0da8070360 2024-09-11T00:50:07.292 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:07.292 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:07.292 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:50:07.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 -- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d88077f70 msgr2=0x7f0d8807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d88077f70 0x7f0d8807a430 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0d98000f60 tx=0x7f0d980016e0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 -- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0da8070360 msgr2=0x7f0da81ec360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0da8070360 0x7f0da81ec360 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d9400e9f0 tx=0x7f0d9400eeb0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 -- 172.21.15.31:0/1958950932 shutdown_connections 2024-09-11T00:50:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0d88077f70 0x7f0d8807a430 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da8153b30 0x7f0da81ec8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0da8070360 0x7f0da81ec360 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 --2- 172.21.15.31:0/1958950932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0da806d440 0x7f0da81e0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 -- 172.21.15.31:0/1958950932 >> 172.21.15.31:0/1958950932 conn(0x7f0da806dc00 msgr2=0x7f0da80c68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:07.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 -- 172.21.15.31:0/1958950932 shutdown_connections 2024-09-11T00:50:07.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.292+0000 7f0dadc00640 1 -- 172.21.15.31:0/1958950932 wait complete. 2024-09-11T00:50:07.310 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-11T00:50:07.310 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-11T00:50:07.311 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:50:07.327 INFO:tasks.ceph.mds.c:Stopped 2024-09-11T00:50:07.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:07.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.528+0000 7fb9f3e00640 1 -- 172.21.15.31:0/802750460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f40ab640 msgr2=0x7fb9f4152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:07.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.528+0000 7fb9f3e00640 1 --2- 172.21.15.31:0/802750460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f40ab640 0x7fb9f4152ba0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009830 tx=0x7fb9dc02a4a0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.529+0000 7fb9f3e00640 1 -- 172.21.15.31:0/802750460 shutdown_connections 2024-09-11T00:50:07.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.529+0000 7fb9f3e00640 1 --2- 172.21.15.31:0/802750460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f40ab640 0x7fb9f4152ba0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.529+0000 7fb9f3e00640 1 -- 172.21.15.31:0/802750460 >> 172.21.15.31:0/802750460 conn(0x7fb9f40c48c0 msgr2=0x7fb9f40c4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:07.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.529+0000 7fb9f3e00640 1 -- 172.21.15.31:0/802750460 shutdown_connections 2024-09-11T00:50:07.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.529+0000 7fb9f3e00640 1 -- 172.21.15.31:0/802750460 wait complete. 2024-09-11T00:50:07.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.530+0000 7fb9f3e00640 1 Processor -- start 2024-09-11T00:50:07.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.530+0000 7fb9f3e00640 1 -- start start 2024-09-11T00:50:07.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.531+0000 7fb9f3e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f40ab640 0x7fb9f406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:07.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.531+0000 7fb9f3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f40c37d0 0x7fb9f4147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:07.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.531+0000 7fb9f3e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9f414cc70 0x7fb9f4147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:07.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.531+0000 7fb9f3e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f4156330 con 0x7fb9f40c37d0 2024-09-11T00:50:07.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.531+0000 7fb9f3e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f406e8d0 con 0x7fb9f40ab640 2024-09-11T00:50:07.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.531+0000 7fb9f3e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f40adeb0 con 0x7fb9f414cc70 2024-09-11T00:50:07.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9ea000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f40c37d0 0x7fb9f4147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:07.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9f2a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f40ab640 0x7fb9f406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9ea000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f40c37d0 0x7fb9f4147660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54898/0 (socket says 172.21.15.31:54898) 2024-09-11T00:50:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9ea000640 1 -- 172.21.15.31:0/413357372 learned_addr learned my addr 172.21.15.31:0/413357372 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9f3400640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9f414cc70 0x7fb9f4147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9f2a00640 1 -- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9f414cc70 msgr2=0x7fb9f4147ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9f2a00640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9f414cc70 0x7fb9f4147ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9f2a00640 1 -- 172.21.15.31:0/413357372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f40c37d0 msgr2=0x7fb9f4147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9f2a00640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f40c37d0 0x7fb9f4147660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9f2a00640 1 -- 172.21.15.31:0/413357372 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f41de630 con 0x7fb9f40ab640 2024-09-11T00:50:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9ea000640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f40c37d0 0x7fb9f4147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:50:07.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.532+0000 7fb9f2a00640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f40ab640 0x7fb9f406ecc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400c340 tx=0x7fb9e400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:07.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.533+0000 7fb9ebe00640 1 -- 172.21.15.31:0/413357372 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4017070 con 0x7fb9f40ab640 2024-09-11T00:50:07.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.533+0000 7fb9ebe00640 1 -- 172.21.15.31:0/413357372 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9e40053c0 con 0x7fb9f40ab640 2024-09-11T00:50:07.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.533+0000 7fb9ebe00640 1 -- 172.21.15.31:0/413357372 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4003850 con 0x7fb9f40ab640 2024-09-11T00:50:07.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.533+0000 7fb9f3e00640 1 -- 172.21.15.31:0/413357372 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f406f670 con 0x7fb9f40ab640 2024-09-11T00:50:07.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.533+0000 7fb9f3e00640 1 -- 172.21.15.31:0/413357372 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f4074010 con 0x7fb9f40ab640 2024-09-11T00:50:07.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.534+0000 7fb9f3e00640 1 -- 172.21.15.31:0/413357372 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9f4152ba0 con 0x7fb9f40ab640 2024-09-11T00:50:07.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.535+0000 7fb9ebe00640 1 -- 172.21.15.31:0/413357372 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb9e40039f0 con 0x7fb9f40ab640 2024-09-11T00:50:07.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.536+0000 7fb9ebe00640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 0x7fb9d007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:07.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.536+0000 7fb9ebe00640 1 -- 172.21.15.31:0/413357372 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7fb9e4099fe0 con 0x7fb9f40ab640 2024-09-11T00:50:07.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.536+0000 7fb9ea000640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 0x7fb9d007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:07.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.537+0000 7fb9ea000640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 0x7fb9d007a460 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40743a0 tx=0x7fb9dc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:07.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.543+0000 7fb9ebe00640 1 -- 172.21.15.31:0/413357372 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb9e4061710 con 0x7fb9f40ab640 2024-09-11T00:50:07.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.774+0000 7fb9f3e00640 1 -- 172.21.15.31:0/413357372 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9f41511b0 con 0x7fb9f40ab640 2024-09-11T00:50:07.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.775+0000 7fb9ebe00640 1 -- 172.21.15.31:0/413357372 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7fb9e4066140 con 0x7fb9f40ab640 2024-09-11T00:50:07.778 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:07.778 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:07.778 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:50:07.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.777+0000 7fb9f3e00640 1 -- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 msgr2=0x7fb9d007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.777+0000 7fb9f3e00640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 0x7fb9d007a460 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40743a0 tx=0x7fb9dc004730 comp rx=0 tx=0).stop 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.777+0000 7fb9f3e00640 1 -- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f40ab640 msgr2=0x7fb9f406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.777+0000 7fb9f3e00640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f40ab640 0x7fb9f406ecc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400c340 tx=0x7fb9e400c800 comp rx=0 tx=0).stop 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.778+0000 7fb9f3e00640 1 -- 172.21.15.31:0/413357372 shutdown_connections 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.778+0000 7fb9f3e00640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb9d0077fa0 0x7fb9d007a460 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.778+0000 7fb9f3e00640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb9f414cc70 0x7fb9f4147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.778+0000 7fb9f3e00640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb9f40c37d0 0x7fb9f4147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.778+0000 7fb9f3e00640 1 --2- 172.21.15.31:0/413357372 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb9f40ab640 0x7fb9f406ecc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.778+0000 7fb9f3e00640 1 -- 172.21.15.31:0/413357372 >> 172.21.15.31:0/413357372 conn(0x7fb9f40c48c0 msgr2=0x7fb9f41567c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.778+0000 7fb9f3e00640 1 -- 172.21.15.31:0/413357372 shutdown_connections 2024-09-11T00:50:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:07.778+0000 7fb9f3e00640 1 -- 172.21.15.31:0/413357372 wait complete. 2024-09-11T00:50:07.795 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:50:08.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.371+0000 7f1dd1800640 1 -- 172.21.15.31:0/2578677949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc0ac2c0 msgr2=0x7f1dcc0ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.371+0000 7f1dd1800640 1 --2- 172.21.15.31:0/2578677949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc0ac2c0 0x7f1dcc0ac6a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009830 tx=0x7f1dbc02a3f0 comp rx=0 tx=0).stop 2024-09-11T00:50:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.371+0000 7f1dd1800640 1 -- 172.21.15.31:0/2578677949 shutdown_connections 2024-09-11T00:50:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.371+0000 7f1dd1800640 1 --2- 172.21.15.31:0/2578677949 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1dcc0acbe0 0x7f1dcc0c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.371+0000 7f1dd1800640 1 --2- 172.21.15.31:0/2578677949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc0ac2c0 0x7f1dcc0ac6a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.371+0000 7f1dd1800640 1 -- 172.21.15.31:0/2578677949 >> 172.21.15.31:0/2578677949 conn(0x7f1dcc06caf0 msgr2=0x7f1dcc06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.371+0000 7f1dd1800640 1 -- 172.21.15.31:0/2578677949 shutdown_connections 2024-09-11T00:50:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.371+0000 7f1dd1800640 1 -- 172.21.15.31:0/2578677949 wait complete. 2024-09-11T00:50:08.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.373+0000 7f1dd1800640 1 Processor -- start 2024-09-11T00:50:08.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.373+0000 7f1dd1800640 1 -- start start 2024-09-11T00:50:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dd1800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dcc0a6e20 0x7f1dcc1eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dd1800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc0ac2c0 0x7f1dcc1dd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dd1800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1dcc0acbe0 0x7f1dcc0c0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dd1800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc151950 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dd1800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc1517d0 con 0x7f1dcc0ac2c0 2024-09-11T00:50:08.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dd1800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1dcc0c66f0 con 0x7f1dcc0acbe0 2024-09-11T00:50:08.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dcbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dcc0a6e20 0x7f1dcc1eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:08.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dcbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dcc0a6e20 0x7f1dcc1eebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40760/0 (socket says 172.21.15.31:40760) 2024-09-11T00:50:08.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dcbe00640 1 -- 172.21.15.31:0/1960433425 learned_addr learned my addr 172.21.15.31:0/1960433425 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:08.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dd0e00640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1dcc0acbe0 0x7f1dcc0c0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:08.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dcb400640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc0ac2c0 0x7f1dcc1dd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:08.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dcbe00640 1 -- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1dcc0acbe0 msgr2=0x7f1dcc0c0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:08.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dcbe00640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1dcc0acbe0 0x7f1dcc0c0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:08.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dcbe00640 1 -- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc0ac2c0 msgr2=0x7f1dcc1dd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:08.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.374+0000 7f1dcbe00640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc0ac2c0 0x7f1dcc1dd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:08.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.375+0000 7f1dcbe00640 1 -- 172.21.15.31:0/1960433425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dcc1478b0 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.375+0000 7f1dcbe00640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dcc0a6e20 0x7f1dcc1eebb0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009950 tx=0x7f1dbc032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:08.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.375+0000 7f1dc3e00640 1 -- 172.21.15.31:0/1960433425 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc040070 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.375+0000 7f1dc3e00640 1 -- 172.21.15.31:0/1960433425 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1dbc005310 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.375+0000 7f1dd1800640 1 -- 172.21.15.31:0/1960433425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dcc0c26c0 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.375+0000 7f1dc3e00640 1 -- 172.21.15.31:0/1960433425 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc005c90 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.375+0000 7f1dd1800640 1 -- 172.21.15.31:0/1960433425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dcc06ea10 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.376+0000 7f1dd1800640 1 -- 172.21.15.31:0/1960433425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dcc0c6b50 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.377+0000 7f1dc3e00640 1 -- 172.21.15.31:0/1960433425 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1dbc0049b0 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.379+0000 7f1dc3e00640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1dac077f70 0x7f1dac07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:08.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.379+0000 7f1dc3e00640 1 -- 172.21.15.31:0/1960433425 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f1dbc0bbe50 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.380+0000 7f1dcb400640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1dac077f70 0x7f1dac07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:08.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.380+0000 7f1dcb400640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1dac077f70 0x7f1dac07a430 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0c2c30 tx=0x7f1db8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:08.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.383+0000 7f1dc3e00640 1 -- 172.21.15.31:0/1960433425 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1dbc083c30 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.617+0000 7f1dd1800640 1 -- 172.21.15.31:0/1960433425 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1dcc0c1840 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.618+0000 7f1dc3e00640 1 -- 172.21.15.31:0/1960433425 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7f1dbc083530 con 0x7f1dcc0a6e20 2024-09-11T00:50:08.620 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:08.620 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:08.621 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:50:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 -- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1dac077f70 msgr2=0x7f1dac07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1dac077f70 0x7f1dac07a430 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0c2c30 tx=0x7f1db8009040 comp rx=0 tx=0).stop 2024-09-11T00:50:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 -- 172.21.15.31:0/1960433425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dcc0a6e20 msgr2=0x7f1dcc1eebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dcc0a6e20 0x7f1dcc1eebb0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009950 tx=0x7f1dbc032c10 comp rx=0 tx=0).stop 2024-09-11T00:50:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 -- 172.21.15.31:0/1960433425 shutdown_connections 2024-09-11T00:50:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1dac077f70 0x7f1dac07a430 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1dcc0acbe0 0x7f1dcc0c0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dcc0ac2c0 0x7f1dcc1dd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:08.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 --2- 172.21.15.31:0/1960433425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dcc0a6e20 0x7f1dcc1eebb0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:08.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 -- 172.21.15.31:0/1960433425 >> 172.21.15.31:0/1960433425 conn(0x7f1dcc06caf0 msgr2=0x7f1dcc1e05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:08.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 -- 172.21.15.31:0/1960433425 shutdown_connections 2024-09-11T00:50:08.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:08.620+0000 7f1dd1800640 1 -- 172.21.15.31:0/1960433425 wait complete. 2024-09-11T00:50:08.640 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:50:09.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.966+0000 7fc8ad000640 1 -- 172.21.15.31:0/1395963177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8a80c6b40 msgr2=0x7fc8a8074260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.966+0000 7fc8ad000640 1 --2- 172.21.15.31:0/1395963177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8a80c6b40 0x7fc8a8074260 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8075350 tx=0x7fc8a002aac0 comp rx=0 tx=0).stop 2024-09-11T00:50:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.966+0000 7fc8ad000640 1 -- 172.21.15.31:0/1395963177 shutdown_connections 2024-09-11T00:50:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.966+0000 7fc8ad000640 1 --2- 172.21.15.31:0/1395963177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8a80c6b40 0x7fc8a8074260 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.966+0000 7fc8ad000640 1 --2- 172.21.15.31:0/1395963177 >> v2:172.21.15.204:3300/0 conn(0x7fc8a80c5a20 0x7fc8a80bf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.966+0000 7fc8ad000640 1 --2- 172.21.15.31:0/1395963177 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8a80c6760 0x7fc8a80c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.966+0000 7fc8ad000640 1 -- 172.21.15.31:0/1395963177 >> 172.21.15.31:0/1395963177 conn(0x7fc8a80694f0 msgr2=0x7fc8a80c9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:09.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.966+0000 7fc8ad000640 1 -- 172.21.15.31:0/1395963177 shutdown_connections 2024-09-11T00:50:09.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.966+0000 7fc8ad000640 1 -- 172.21.15.31:0/1395963177 wait complete. 2024-09-11T00:50:09.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.967+0000 7fc8ad000640 1 Processor -- start 2024-09-11T00:50:09.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.968+0000 7fc8ad000640 1 -- start start 2024-09-11T00:50:09.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.968+0000 7fc8ad000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8a80c5a20 0x7fc8a814fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:09.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.968+0000 7fc8ad000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8a80c6760 0x7fc8a81ddea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:09.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.968+0000 7fc8ad000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8a80c6b40 0x7fc8a81dfd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:09.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8ad000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a80c9460 con 0x7fc8a80c6b40 2024-09-11T00:50:09.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8ad000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a80c92e0 con 0x7fc8a80c6760 2024-09-11T00:50:09.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8ad000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc8a8072ab0 con 0x7fc8a80c5a20 2024-09-11T00:50:09.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8a80c6b40 0x7fc8a81dfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:09.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8a80c6b40 0x7fc8a81dfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40790/0 (socket says 172.21.15.31:40790) 2024-09-11T00:50:09.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7e00640 1 -- 172.21.15.31:0/754755868 learned_addr learned my addr 172.21.15.31:0/754755868 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:09.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7400640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8a80c5a20 0x7fc8a814fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a6a00640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8a80c6760 0x7fc8a81ddea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7e00640 1 -- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8a80c5a20 msgr2=0x7fc8a814fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7e00640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8a80c5a20 0x7fc8a814fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7e00640 1 -- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8a80c6760 msgr2=0x7fc8a81ddea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7e00640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8a80c6760 0x7fc8a81ddea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7e00640 1 -- 172.21.15.31:0/754755868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a81deed0 con 0x7fc8a80c6b40 2024-09-11T00:50:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a6a00640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8a80c6760 0x7fc8a81ddea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:50:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7400640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8a80c5a20 0x7fc8a814fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:50:09.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.969+0000 7fc8a7e00640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8a80c6b40 0x7fc8a81dfd90 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc8a002b0b0 tx=0x7fc8a0003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.970+0000 7fc89f400640 1 -- 172.21.15.31:0/754755868 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0044070 con 0x7fc8a80c6b40 2024-09-11T00:50:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.970+0000 7fc89f400640 1 -- 172.21.15.31:0/754755868 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8a0003e50 con 0x7fc8a80c6b40 2024-09-11T00:50:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.970+0000 7fc89f400640 1 -- 172.21.15.31:0/754755868 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a00044b0 con 0x7fc8a80c6b40 2024-09-11T00:50:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.970+0000 7fc8ad000640 1 -- 172.21.15.31:0/754755868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8a814ec80 con 0x7fc8a80c6b40 2024-09-11T00:50:09.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.970+0000 7fc8ad000640 1 -- 172.21.15.31:0/754755868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8a81deb60 con 0x7fc8a80c6b40 2024-09-11T00:50:09.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.971+0000 7fc8ad000640 1 -- 172.21.15.31:0/754755868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8a80bfc70 con 0x7fc8a80c6b40 2024-09-11T00:50:09.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.972+0000 7fc89f400640 1 -- 172.21.15.31:0/754755868 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc8a0004650 con 0x7fc8a80c6b40 2024-09-11T00:50:09.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.973+0000 7fc89f400640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc888077f70 0x7fc88807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:09.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.973+0000 7fc89f400640 1 -- 172.21.15.31:0/754755868 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7fc8a00bafe0 con 0x7fc8a80c6b40 2024-09-11T00:50:09.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.974+0000 7fc8a7400640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc888077f70 0x7fc88807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:09.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.974+0000 7fc8a7400640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc888077f70 0x7fc88807a430 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc898009ba0 tx=0x7fc898005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:09.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:09.977+0000 7fc89f400640 1 -- 172.21.15.31:0/754755868 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc8a0032030 con 0x7fc8a80c6b40 2024-09-11T00:50:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.210+0000 7fc8ad000640 1 -- 172.21.15.31:0/754755868 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8a81dfaf0 con 0x7fc8a80c6b40 2024-09-11T00:50:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.210+0000 7fc89f400640 1 -- 172.21.15.31:0/754755868 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7fc8a0082e80 con 0x7fc8a80c6b40 2024-09-11T00:50:10.213 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:10.213 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:50:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 -- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc888077f70 msgr2=0x7fc88807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc888077f70 0x7fc88807a430 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc898009ba0 tx=0x7fc898005f90 comp rx=0 tx=0).stop 2024-09-11T00:50:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 -- 172.21.15.31:0/754755868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8a80c6b40 msgr2=0x7fc8a81dfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8a80c6b40 0x7fc8a81dfd90 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc8a002b0b0 tx=0x7fc8a0003ce0 comp rx=0 tx=0).stop 2024-09-11T00:50:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 -- 172.21.15.31:0/754755868 shutdown_connections 2024-09-11T00:50:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc888077f70 0x7fc88807a430 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8a80c6b40 0x7fc8a81dfd90 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc8a80c6760 0x7fc8a81ddea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:10.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 --2- 172.21.15.31:0/754755868 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc8a80c5a20 0x7fc8a814fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:10.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 -- 172.21.15.31:0/754755868 >> 172.21.15.31:0/754755868 conn(0x7fc8a80694f0 msgr2=0x7fc8a8156e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:10.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 -- 172.21.15.31:0/754755868 shutdown_connections 2024-09-11T00:50:10.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.213+0000 7fc8ad000640 1 -- 172.21.15.31:0/754755868 wait complete. 2024-09-11T00:50:10.232 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:50:10.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:10.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.801+0000 7fbb6b800640 1 -- 172.21.15.31:0/3379918079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb48004d40 msgr2=0x7fbb480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:10.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.801+0000 7fbb6b800640 1 --2- 172.21.15.31:0/3379918079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb48004d40 0x7fbb480051c0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c009f00 tx=0x7fbb48005830 comp rx=0 tx=0).stop 2024-09-11T00:50:10.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.801+0000 7fbb69800640 1 -- 172.21.15.31:0/3379918079 reap_dead start 2024-09-11T00:50:10.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.801+0000 7fbb6b800640 1 -- 172.21.15.31:0/3379918079 shutdown_connections 2024-09-11T00:50:10.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.801+0000 7fbb6b800640 1 -- 172.21.15.31:0/3379918079 >> 172.21.15.31:0/3379918079 conn(0x7fbb640a9750 msgr2=0x7fbb640a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:10.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.801+0000 7fbb6b800640 1 -- 172.21.15.31:0/3379918079 shutdown_connections 2024-09-11T00:50:10.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.801+0000 7fbb6b800640 1 -- 172.21.15.31:0/3379918079 wait complete. 2024-09-11T00:50:10.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.803+0000 7fbb6b800640 1 Processor -- start 2024-09-11T00:50:10.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.803+0000 7fbb6b800640 1 -- start start 2024-09-11T00:50:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb6b800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb48004d40 0x7fbb640c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb6b800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb64146740 0x7fbb6406f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb6b800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb640c6d90 0x7fbb640c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:10.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb6b800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbb64074200 con 0x7fbb64146740 2024-09-11T00:50:10.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb6b800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbb64074080 con 0x7fbb48004d40 2024-09-11T00:50:10.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb6b800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbb64072860 con 0x7fbb640c6d90 2024-09-11T00:50:10.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb5be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb64146740 0x7fbb6406f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:10.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb5be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb64146740 0x7fbb6406f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40806/0 (socket says 172.21.15.31:40806) 2024-09-11T00:50:10.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb5be00640 1 -- 172.21.15.31:0/4062026954 learned_addr learned my addr 172.21.15.31:0/4062026954 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:10.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb69800640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb640c6d90 0x7fbb640c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:10.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb68e00640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb48004d40 0x7fbb640c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:10.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb5be00640 1 -- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb640c6d90 msgr2=0x7fbb640c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb5be00640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb640c6d90 0x7fbb640c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb5be00640 1 -- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb48004d40 msgr2=0x7fbb640c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb5be00640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb48004d40 0x7fbb640c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.804+0000 7fbb5be00640 1 -- 172.21.15.31:0/4062026954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb640c1750 con 0x7fbb64146740 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.805+0000 7fbb5be00640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb64146740 0x7fbb6406f3d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c004d60 tx=0x7fbb4c001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.805+0000 7fbb61600640 1 -- 172.21.15.31:0/4062026954 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0107d0 con 0x7fbb64146740 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.805+0000 7fbb61600640 1 -- 172.21.15.31:0/4062026954 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb4c010a60 con 0x7fbb64146740 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.805+0000 7fbb61600640 1 -- 172.21.15.31:0/4062026954 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c011460 con 0x7fbb64146740 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.805+0000 7fbb6b800640 1 -- 172.21.15.31:0/4062026954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb641f01b0 con 0x7fbb64146740 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.805+0000 7fbb6b800640 1 -- 172.21.15.31:0/4062026954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb641508f0 con 0x7fbb64146740 2024-09-11T00:50:10.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.806+0000 7fbb6b800640 1 -- 172.21.15.31:0/4062026954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb64073630 con 0x7fbb64146740 2024-09-11T00:50:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.807+0000 7fbb61600640 1 -- 172.21.15.31:0/4062026954 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbb4c011600 con 0x7fbb64146740 2024-09-11T00:50:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.808+0000 7fbb61600640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb40077fa0 0x7fbb4007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:10.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.808+0000 7fbb61600640 1 -- 172.21.15.31:0/4062026954 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7fbb4c09b4e0 con 0x7fbb64146740 2024-09-11T00:50:10.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.808+0000 7fbb68e00640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb40077fa0 0x7fbb4007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:10.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.809+0000 7fbb68e00640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb40077fa0 0x7fbb4007a460 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbb54001860 tx=0x7fbb54000ef0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:10.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:10.812+0000 7fbb61600640 1 -- 172.21.15.31:0/4062026954 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbb64073630 con 0x7fbb64146740 2024-09-11T00:50:11.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.047+0000 7fbb6b800640 1 -- 172.21.15.31:0/4062026954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb641f06d0 con 0x7fbb64146740 2024-09-11T00:50:11.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.048+0000 7fbb61600640 1 -- 172.21.15.31:0/4062026954 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7fbb4c063340 con 0x7fbb64146740 2024-09-11T00:50:11.050 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:11.050 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:11.051 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:50:11.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 -- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb40077fa0 msgr2=0x7fbb4007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:11.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb40077fa0 0x7fbb4007a460 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbb54001860 tx=0x7fbb54000ef0 comp rx=0 tx=0).stop 2024-09-11T00:50:11.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 -- 172.21.15.31:0/4062026954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb64146740 msgr2=0x7fbb6406f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:11.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb64146740 0x7fbb6406f3d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c004d60 tx=0x7fbb4c001800 comp rx=0 tx=0).stop 2024-09-11T00:50:11.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 -- 172.21.15.31:0/4062026954 shutdown_connections 2024-09-11T00:50:11.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbb40077fa0 0x7fbb4007a460 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:11.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbb640c6d90 0x7fbb640c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:11.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb64146740 0x7fbb6406f3d0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 --2- 172.21.15.31:0/4062026954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb48004d40 0x7fbb640c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 -- 172.21.15.31:0/4062026954 >> 172.21.15.31:0/4062026954 conn(0x7fbb640a9750 msgr2=0x7fbb640a8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 -- 172.21.15.31:0/4062026954 shutdown_connections 2024-09-11T00:50:11.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:11.050+0000 7fbb6b800640 1 -- 172.21.15.31:0/4062026954 wait complete. 2024-09-11T00:50:11.069 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:50:12.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:12.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.401+0000 7f842c200640 1 -- 172.21.15.31:0/4001002111 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8424156300 msgr2=0x7f84240c5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:12.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.402+0000 7f842c200640 1 --2- 172.21.15.31:0/4001002111 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8424156300 0x7f84240c5b40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8418009860 tx=0x7f841802a530 comp rx=0 tx=0).stop 2024-09-11T00:50:12.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.402+0000 7f842c200640 1 -- 172.21.15.31:0/4001002111 shutdown_connections 2024-09-11T00:50:12.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.402+0000 7f842c200640 1 --2- 172.21.15.31:0/4001002111 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8424156300 0x7f84240c5b40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:12.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.402+0000 7f842c200640 1 -- 172.21.15.31:0/4001002111 >> 172.21.15.31:0/4001002111 conn(0x7f84240a9750 msgr2=0x7f84240a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:12.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.402+0000 7f842c200640 1 -- 172.21.15.31:0/4001002111 shutdown_connections 2024-09-11T00:50:12.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.402+0000 7f842c200640 1 -- 172.21.15.31:0/4001002111 wait complete. 2024-09-11T00:50:12.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.403+0000 7f842c200640 1 Processor -- start 2024-09-11T00:50:12.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.403+0000 7f842c200640 1 -- start start 2024-09-11T00:50:12.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.404+0000 7f842c200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f842406fa70 0x7f84241defd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:12.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.404+0000 7f842c200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84240c23a0 0x7f842414ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:12.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.404+0000 7f842c200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8424156300 0x7f842414b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:12.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.404+0000 7f842c200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8424156df0 con 0x7f8424156300 2024-09-11T00:50:12.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.404+0000 7f842c200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f842406f5d0 con 0x7f842406fa70 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.404+0000 7f842c200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8424156f70 con 0x7f84240c23a0 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f842a200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8424156300 0x7f842414b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f842a200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8424156300 0x7f842414b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40814/0 (socket says 172.21.15.31:40814) 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f842a200640 1 -- 172.21.15.31:0/4071469741 learned_addr learned my addr 172.21.15.31:0/4071469741 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f8429800640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f842406fa70 0x7f84241defd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f8428e00640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84240c23a0 0x7f842414ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f842a200640 1 -- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84240c23a0 msgr2=0x7f842414ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f842a200640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84240c23a0 0x7f842414ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f842a200640 1 -- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f842406fa70 msgr2=0x7f84241defd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f842a200640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f842406fa70 0x7f84241defd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f842a200640 1 -- 172.21.15.31:0/4071469741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84241df510 con 0x7f8424156300 2024-09-11T00:50:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.405+0000 7f842a200640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8424156300 0x7f842414b1c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f841c00cbf0 tx=0x7f841c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.406+0000 7f8421600640 1 -- 172.21.15.31:0/4071469741 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c016020 con 0x7f8424156300 2024-09-11T00:50:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.406+0000 7f842c200640 1 -- 172.21.15.31:0/4071469741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f842414bc20 con 0x7f8424156300 2024-09-11T00:50:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.406+0000 7f8421600640 1 -- 172.21.15.31:0/4071469741 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f841c00acf0 con 0x7f8424156300 2024-09-11T00:50:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.406+0000 7f8421600640 1 -- 172.21.15.31:0/4071469741 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c012680 con 0x7f8424156300 2024-09-11T00:50:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.406+0000 7f842c200640 1 -- 172.21.15.31:0/4071469741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84241543e0 con 0x7f8424156300 2024-09-11T00:50:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.407+0000 7f842c200640 1 -- 172.21.15.31:0/4071469741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84240c5b40 con 0x7f8424156300 2024-09-11T00:50:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.408+0000 7f8421600640 1 -- 172.21.15.31:0/4071469741 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f841c012820 con 0x7f8424156300 2024-09-11T00:50:12.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.409+0000 7f8421600640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8404077f70 0x7f840407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:12.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.409+0000 7f8421600640 1 -- 172.21.15.31:0/4071469741 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f841c09b3c0 con 0x7f8424156300 2024-09-11T00:50:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.409+0000 7f8429800640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8404077f70 0x7f840407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:12.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.410+0000 7f8429800640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8404077f70 0x7f840407a430 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8418034000 tx=0x7f8418004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:12.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.413+0000 7f8421600640 1 -- 172.21.15.31:0/4071469741 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f841c020020 con 0x7f8424156300 2024-09-11T00:50:12.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.650+0000 7f842c200640 1 -- 172.21.15.31:0/4071469741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84240c7240 con 0x7f8424156300 2024-09-11T00:50:12.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.650+0000 7f8421600640 1 -- 172.21.15.31:0/4071469741 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7f841c063200 con 0x7f8424156300 2024-09-11T00:50:12.653 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:12.653 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:12.653 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:50:12.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 -- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8404077f70 msgr2=0x7f840407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:12.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8404077f70 0x7f840407a430 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8418034000 tx=0x7f8418004730 comp rx=0 tx=0).stop 2024-09-11T00:50:12.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 -- 172.21.15.31:0/4071469741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8424156300 msgr2=0x7f842414b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:12.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8424156300 0x7f842414b1c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f841c00cbf0 tx=0x7f841c00a430 comp rx=0 tx=0).stop 2024-09-11T00:50:12.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 -- 172.21.15.31:0/4071469741 shutdown_connections 2024-09-11T00:50:12.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8404077f70 0x7f840407a430 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:12.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8424156300 0x7f842414b1c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f84240c23a0 0x7f842414ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 --2- 172.21.15.31:0/4071469741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f842406fa70 0x7f84241defd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 -- 172.21.15.31:0/4071469741 >> 172.21.15.31:0/4071469741 conn(0x7f84240a9750 msgr2=0x7f84240c3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 -- 172.21.15.31:0/4071469741 shutdown_connections 2024-09-11T00:50:12.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:12.653+0000 7f842c200640 1 -- 172.21.15.31:0/4071469741 wait complete. 2024-09-11T00:50:12.671 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:50:13.071 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:13.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.229+0000 7f1311200640 1 -- 172.21.15.31:0/1475670862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f130c1578e0 msgr2=0x7f130c157d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:13.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.229+0000 7f1311200640 1 --2- 172.21.15.31:0/1475670862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f130c1578e0 0x7f130c157d70 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f130c0c4820 tx=0x7f130402aa30 comp rx=0 tx=0).stop 2024-09-11T00:50:13.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.230+0000 7f1311200640 1 -- 172.21.15.31:0/1475670862 shutdown_connections 2024-09-11T00:50:13.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.230+0000 7f1311200640 1 --2- 172.21.15.31:0/1475670862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f130c1578e0 0x7f130c157d70 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:13.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.230+0000 7f1311200640 1 --2- 172.21.15.31:0/1475670862 >> v2:172.21.15.204:3300/0 conn(0x7f130c0a8bb0 0x7f130c0bf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:13.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.230+0000 7f1311200640 1 -- 172.21.15.31:0/1475670862 >> 172.21.15.31:0/1475670862 conn(0x7f130c0c9af0 msgr2=0x7f130c0c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:13.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.230+0000 7f1311200640 1 -- 172.21.15.31:0/1475670862 shutdown_connections 2024-09-11T00:50:13.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.230+0000 7f1311200640 1 -- 172.21.15.31:0/1475670862 wait complete. 2024-09-11T00:50:13.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.231+0000 7f1311200640 1 Processor -- start 2024-09-11T00:50:13.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.231+0000 7f1311200640 1 -- start start 2024-09-11T00:50:13.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.232+0000 7f1311200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f130c0a8bb0 0x7f130c1ece40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:13.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.232+0000 7f1311200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f130c0c67f0 0x7f130c14f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:13.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.232+0000 7f1311200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f130c1578e0 0x7f130c071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:13.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.232+0000 7f1311200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f130c0a9260 con 0x7f130c0c67f0 2024-09-11T00:50:13.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.232+0000 7f1311200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f130c0a90e0 con 0x7f130c0a8bb0 2024-09-11T00:50:13.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.232+0000 7f1311200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f130c0c5b20 con 0x7f130c1578e0 2024-09-11T00:50:13.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f1302a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f130c0c67f0 0x7f130c14f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:13.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f1302a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f130c0c67f0 0x7f130c14f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40828/0 (socket says 172.21.15.31:40828) 2024-09-11T00:50:13.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f1302a00640 1 -- 172.21.15.31:0/1292141959 learned_addr learned my addr 172.21.15.31:0/1292141959 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:13.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f130b400640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f130c0a8bb0 0x7f130c1ece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:13.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f130be00640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f130c1578e0 0x7f130c071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:13.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f1302a00640 1 -- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f130c1578e0 msgr2=0x7f130c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:13.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f1302a00640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f130c1578e0 0x7f130c071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:13.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f1302a00640 1 -- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f130c0a8bb0 msgr2=0x7f130c1ece40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:13.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f1302a00640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f130c0a8bb0 0x7f130c1ece40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:13.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f1302a00640 1 -- 172.21.15.31:0/1292141959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c1551d0 con 0x7f130c0c67f0 2024-09-11T00:50:13.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.233+0000 7f1302a00640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f130c0c67f0 0x7f130c14f120 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f12f400c920 tx=0x7f12f400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:13.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.234+0000 7f1303e00640 1 -- 172.21.15.31:0/1292141959 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f4007c70 con 0x7f130c0c67f0 2024-09-11T00:50:13.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.234+0000 7f1303e00640 1 -- 172.21.15.31:0/1292141959 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12f400f450 con 0x7f130c0c67f0 2024-09-11T00:50:13.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.234+0000 7f1303e00640 1 -- 172.21.15.31:0/1292141959 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f400f8a0 con 0x7f130c0c67f0 2024-09-11T00:50:13.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.234+0000 7f1311200640 1 -- 172.21.15.31:0/1292141959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130c152450 con 0x7f130c0c67f0 2024-09-11T00:50:13.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.234+0000 7f1311200640 1 -- 172.21.15.31:0/1292141959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f130c1ec360 con 0x7f130c0c67f0 2024-09-11T00:50:13.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.235+0000 7f1311200640 1 -- 172.21.15.31:0/1292141959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f130c0c6bd0 con 0x7f130c0c67f0 2024-09-11T00:50:13.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.236+0000 7f1303e00640 1 -- 172.21.15.31:0/1292141959 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f12f4020030 con 0x7f130c0c67f0 2024-09-11T00:50:13.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.237+0000 7f1303e00640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12e8077ff0 0x7f12e807a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:13.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.237+0000 7f1303e00640 1 -- 172.21.15.31:0/1292141959 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8902+0+0 (secure 0 0 0) 0x7f12f409aeb0 con 0x7f130c0c67f0 2024-09-11T00:50:13.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.238+0000 7f130b400640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12e8077ff0 0x7f12e807a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:13.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.238+0000 7f130b400640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12e8077ff0 0x7f12e807a4b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0016e0 tx=0x7f12fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:13.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.241+0000 7f1303e00640 1 -- 172.21.15.31:0/1292141959 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f12f4014030 con 0x7f130c0c67f0 2024-09-11T00:50:13.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.473+0000 7f1311200640 1 -- 172.21.15.31:0/1292141959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f130c0c5900 con 0x7f130c0c67f0 2024-09-11T00:50:13.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.474+0000 7f1303e00640 1 -- 172.21.15.31:0/1292141959 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6544 (secure 0 0 0) 0x7f12f4062d90 con 0x7f130c0c67f0 2024-09-11T00:50:13.477 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:13.477 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":142,"btime":"2024-09-11T00:49:45:742700+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:48:17.824495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6934},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6934":{"gid":6934,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":6,"addr":"172.21.15.204:6837/4164016077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":4164016077},{"type":"v1","addr":"172.21.15.204:6837","nonce":4164016077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6934,"qdb_cluster":[6934]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:13.477 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 142 2024-09-11T00:50:13.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.476+0000 7f1311200640 1 -- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12e8077ff0 msgr2=0x7f12e807a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:13.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.476+0000 7f1311200640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12e8077ff0 0x7f12e807a4b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0016e0 tx=0x7f12fc009040 comp rx=0 tx=0).stop 2024-09-11T00:50:13.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.476+0000 7f1311200640 1 -- 172.21.15.31:0/1292141959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f130c0c67f0 msgr2=0x7f130c14f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:13.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.476+0000 7f1311200640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f130c0c67f0 0x7f130c14f120 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f12f400c920 tx=0x7f12f400cde0 comp rx=0 tx=0).stop 2024-09-11T00:50:13.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.477+0000 7f1311200640 1 -- 172.21.15.31:0/1292141959 shutdown_connections 2024-09-11T00:50:13.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.477+0000 7f1311200640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f12e8077ff0 0x7f12e807a4b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:13.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.477+0000 7f1311200640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f130c1578e0 0x7f130c071430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:13.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.477+0000 7f1311200640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f130c0c67f0 0x7f130c14f120 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:13.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.477+0000 7f1311200640 1 --2- 172.21.15.31:0/1292141959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f130c0a8bb0 0x7f130c1ece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:13.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.477+0000 7f1311200640 1 -- 172.21.15.31:0/1292141959 >> 172.21.15.31:0/1292141959 conn(0x7f130c0c9af0 msgr2=0x7f130c0ae980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:13.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.477+0000 7f1311200640 1 -- 172.21.15.31:0/1292141959 shutdown_connections 2024-09-11T00:50:13.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:13.477+0000 7f1311200640 1 -- 172.21.15.31:0/1292141959 wait complete. 2024-09-11T00:50:13.497 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:50:14.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:14.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.835+0000 7f8abb200640 1 -- 172.21.15.31:0/3813700876 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8aa0003a00 msgr2=0x7f8aa0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.835+0000 7f8abb200640 1 --2- 172.21.15.31:0/3813700876 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8aa0003a00 0x7f8aa0003e00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00c6f0 tx=0x7f8aa0005830 comp rx=0 tx=0).stop 2024-09-11T00:50:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.835+0000 7f8abb200640 1 -- 172.21.15.31:0/3813700876 shutdown_connections 2024-09-11T00:50:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.835+0000 7f8abb200640 1 --2- 172.21.15.31:0/3813700876 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8aa0003a00 0x7f8aa0003e00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.835+0000 7f8abb200640 1 -- 172.21.15.31:0/3813700876 >> 172.21.15.31:0/3813700876 conn(0x7f8abc06caf0 msgr2=0x7f8abc06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.836+0000 7f8abb200640 1 -- 172.21.15.31:0/3813700876 shutdown_connections 2024-09-11T00:50:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.836+0000 7f8abb200640 1 -- 172.21.15.31:0/3813700876 wait complete. 2024-09-11T00:50:14.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.837+0000 7f8abb200640 1 Processor -- start 2024-09-11T00:50:14.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.837+0000 7f8abb200640 1 -- start start 2024-09-11T00:50:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.838+0000 7f8abb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8aa0003a00 0x7f8abc14c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:14.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.838+0000 7f8abb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8abc14fd80 0x7f8abc14e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.838+0000 7f8abb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8abc14bc10 0x7f8abc14e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.838+0000 7f8abb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc0c7620 con 0x7f8aa0003a00 2024-09-11T00:50:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.838+0000 7f8abb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc0700f0 con 0x7f8abc14fd80 2024-09-11T00:50:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.838+0000 7f8abb200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8abc1505e0 con 0x7f8abc14bc10 2024-09-11T00:50:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.838+0000 7f8ab9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8aa0003a00 0x7f8abc14c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.838+0000 7f8ab9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8aa0003a00 0x7f8abc14c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40850/0 (socket says 172.21.15.31:40850) 2024-09-11T00:50:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.838+0000 7f8ab9e00640 1 -- 172.21.15.31:0/348858983 learned_addr learned my addr 172.21.15.31:0/348858983 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8aba800640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8abc14bc10 0x7f8abc14e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab9400640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8abc14fd80 0x7f8abc14e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab9e00640 1 -- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8abc14bc10 msgr2=0x7f8abc14e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab9e00640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8abc14bc10 0x7f8abc14e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab9e00640 1 -- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8abc14fd80 msgr2=0x7f8abc14e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab9e00640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8abc14fd80 0x7f8abc14e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab9e00640 1 -- 172.21.15.31:0/348858983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8abc074810 con 0x7f8aa0003a00 2024-09-11T00:50:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab9400640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8abc14fd80 0x7f8abc14e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:50:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8aba800640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8abc14bc10 0x7f8abc14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:50:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab9e00640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8aa0003a00 0x7f8abc14c5f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8aac02a820 tx=0x7f8aac006150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:14.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab2000640 1 -- 172.21.15.31:0/348858983 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac003810 con 0x7f8aa0003a00 2024-09-11T00:50:14.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab2000640 1 -- 172.21.15.31:0/348858983 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8aac003aa0 con 0x7f8aa0003a00 2024-09-11T00:50:14.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8ab2000640 1 -- 172.21.15.31:0/348858983 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac03e440 con 0x7f8aa0003a00 2024-09-11T00:50:14.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.839+0000 7f8abb200640 1 -- 172.21.15.31:0/348858983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8abc14a800 con 0x7f8aa0003a00 2024-09-11T00:50:14.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.840+0000 7f8abb200640 1 -- 172.21.15.31:0/348858983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8abc14adc0 con 0x7f8aa0003a00 2024-09-11T00:50:14.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.841+0000 7f8abb200640 1 -- 172.21.15.31:0/348858983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8abc0705d0 con 0x7f8aa0003a00 2024-09-11T00:50:14.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.842+0000 7f8ab2000640 1 -- 172.21.15.31:0/348858983 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8aac03c020 con 0x7f8aa0003a00 2024-09-11T00:50:14.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.843+0000 7f8ab2000640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a9c077f70 0x7f8a9c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:14.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.843+0000 7f8ab2000640 1 -- 172.21.15.31:0/348858983 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8988+0+0 (secure 0 0 0) 0x7f8aac0ba340 con 0x7f8aa0003a00 2024-09-11T00:50:14.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.843+0000 7f8ab9400640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a9c077f70 0x7f8a9c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:14.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.844+0000 7f8ab9400640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a9c077f70 0x7f8a9c07a430 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8abc150180 tx=0x7f8aa8005e30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:14.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:14.847+0000 7f8ab2000640 1 -- 172.21.15.31:0/348858983 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8aac043060 con 0x7f8aa0003a00 2024-09-11T00:50:15.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.081+0000 7f8abb200640 1 -- 172.21.15.31:0/348858983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8abc14ea80 con 0x7f8aa0003a00 2024-09-11T00:50:15.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.081+0000 7f8ab2000640 1 -- 172.21.15.31:0/348858983 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) ==== 95+0+5708 (secure 0 0 0) 0x7f8aac0821b0 con 0x7f8aa0003a00 2024-09-11T00:50:15.084 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:15.085 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":143,"btime":"2024-09-11T00:50:14:476559+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:14.476553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:15.085 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 143 2024-09-11T00:50:15.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.083+0000 7f8abb200640 1 -- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a9c077f70 msgr2=0x7f8a9c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:15.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.083+0000 7f8abb200640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a9c077f70 0x7f8a9c07a430 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8abc150180 tx=0x7f8aa8005e30 comp rx=0 tx=0).stop 2024-09-11T00:50:15.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.083+0000 7f8abb200640 1 -- 172.21.15.31:0/348858983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8aa0003a00 msgr2=0x7f8abc14c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:15.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.083+0000 7f8abb200640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8aa0003a00 0x7f8abc14c5f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8aac02a820 tx=0x7f8aac006150 comp rx=0 tx=0).stop 2024-09-11T00:50:15.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.084+0000 7f8abb200640 1 -- 172.21.15.31:0/348858983 shutdown_connections 2024-09-11T00:50:15.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.084+0000 7f8abb200640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8a9c077f70 0x7f8a9c07a430 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.084+0000 7f8abb200640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8abc14bc10 0x7f8abc14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.084+0000 7f8abb200640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8abc14fd80 0x7f8abc14e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.084+0000 7f8abb200640 1 --2- 172.21.15.31:0/348858983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8aa0003a00 0x7f8abc14c5f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.084+0000 7f8abb200640 1 -- 172.21.15.31:0/348858983 >> 172.21.15.31:0/348858983 conn(0x7f8abc06caf0 msgr2=0x7f8abc0c3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:15.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.084+0000 7f8abb200640 1 -- 172.21.15.31:0/348858983 shutdown_connections 2024-09-11T00:50:15.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.084+0000 7f8abb200640 1 -- 172.21.15.31:0/348858983 wait complete. 2024-09-11T00:50:15.102 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:50:15.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.657+0000 7f09d6000640 1 -- 172.21.15.31:0/3901006286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00c6310 msgr2=0x7f09d01539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.657+0000 7f09d6000640 1 --2- 172.21.15.31:0/3901006286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00c6310 0x7f09d01539c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f09d0071d90 tx=0x7f09c802aa30 comp rx=0 tx=0).stop 2024-09-11T00:50:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.657+0000 7f09d6000640 1 -- 172.21.15.31:0/3901006286 shutdown_connections 2024-09-11T00:50:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.657+0000 7f09d6000640 1 --2- 172.21.15.31:0/3901006286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00c6310 0x7f09d01539c0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.657+0000 7f09d6000640 1 --2- 172.21.15.31:0/3901006286 >> v2:172.21.15.204:3300/0 conn(0x7f09d00c5ef0 0x7f09d0153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.657+0000 7f09d6000640 1 --2- 172.21.15.31:0/3901006286 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09d00c5b10 0x7f09d0156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.657+0000 7f09d6000640 1 -- 172.21.15.31:0/3901006286 >> 172.21.15.31:0/3901006286 conn(0x7f09d00a9750 msgr2=0x7f09d00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:15.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.657+0000 7f09d6000640 1 -- 172.21.15.31:0/3901006286 shutdown_connections 2024-09-11T00:50:15.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.657+0000 7f09d6000640 1 -- 172.21.15.31:0/3901006286 wait complete. 2024-09-11T00:50:15.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.659+0000 7f09d6000640 1 Processor -- start 2024-09-11T00:50:15.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.659+0000 7f09d6000640 1 -- start start 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09d6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00c5b10 0x7f09d0152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09d6000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09d00c5ef0 0x7f09d014cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09d6000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09d00c6310 0x7f09d014d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09d6000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01597f0 con 0x7f09d00c5b10 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09d6000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f09d0159670 con 0x7f09d00c5ef0 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09d6000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f09d0159970 con 0x7f09d00c6310 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09cf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00c5b10 0x7f09d0152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09cf400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00c5b10 0x7f09d0152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40860/0 (socket says 172.21.15.31:40860) 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09cf400640 1 -- 172.21.15.31:0/556814360 learned_addr learned my addr 172.21.15.31:0/556814360 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09c6a00640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09d00c5ef0 0x7f09d014cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.660+0000 7f09cfe00640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09d00c6310 0x7f09d014d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:15.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09cf400640 1 -- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09d00c6310 msgr2=0x7f09d014d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09cf400640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09d00c6310 0x7f09d014d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09cf400640 1 -- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09d00c5ef0 msgr2=0x7f09d014cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09cf400640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09d00c5ef0 0x7f09d014cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09cf400640 1 -- 172.21.15.31:0/556814360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d0152d40 con 0x7f09d00c5b10 2024-09-11T00:50:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09cf400640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00c5b10 0x7f09d0152130 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f09c000d980 tx=0x7f09c000de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09c7e00640 1 -- 172.21.15.31:0/556814360 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0017070 con 0x7f09d00c5b10 2024-09-11T00:50:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09c7e00640 1 -- 172.21.15.31:0/556814360 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09c0009d20 con 0x7f09d00c5b10 2024-09-11T00:50:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09c7e00640 1 -- 172.21.15.31:0/556814360 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c00106c0 con 0x7f09d00c5b10 2024-09-11T00:50:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09d6000640 1 -- 172.21.15.31:0/556814360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d0151090 con 0x7f09d00c5b10 2024-09-11T00:50:15.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.661+0000 7f09d6000640 1 -- 172.21.15.31:0/556814360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d0151350 con 0x7f09d00c5b10 2024-09-11T00:50:15.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.663+0000 7f09d6000640 1 -- 172.21.15.31:0/556814360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09d0156ad0 con 0x7f09d00c5b10 2024-09-11T00:50:15.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.664+0000 7f09c7e00640 1 -- 172.21.15.31:0/556814360 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f09c0010860 con 0x7f09d00c5b10 2024-09-11T00:50:15.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.665+0000 7f09c7e00640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f09ac077fa0 0x7f09ac07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:15.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.665+0000 7f09c7e00640 1 -- 172.21.15.31:0/556814360 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8988+0+0 (secure 0 0 0) 0x7f09c009ad50 con 0x7f09d00c5b10 2024-09-11T00:50:15.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.665+0000 7f09c6a00640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f09ac077fa0 0x7f09ac07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:15.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.666+0000 7f09c6a00640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f09ac077fa0 0x7f09ac07a460 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f09d006e7f0 tx=0x7f09b800b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:15.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.669+0000 7f09c7e00640 1 -- 172.21.15.31:0/556814360 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f09c009e030 con 0x7f09d00c5b10 2024-09-11T00:50:15.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.899+0000 7f09d6000640 1 -- 172.21.15.31:0/556814360 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09d00c1e70 con 0x7f09d00c5b10 2024-09-11T00:50:15.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.900+0000 7f09c7e00640 1 -- 172.21.15.31:0/556814360 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5711 (secure 0 0 0) 0x7f09c0062570 con 0x7f09d00c5b10 2024-09-11T00:50:15.903 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:15.903 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":144,"btime":"2024-09-11T00:50:15:479276+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:14.831090+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:15.903 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 144 2024-09-11T00:50:15.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.902+0000 7f09d6000640 1 -- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f09ac077fa0 msgr2=0x7f09ac07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:15.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.902+0000 7f09d6000640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f09ac077fa0 0x7f09ac07a460 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f09d006e7f0 tx=0x7f09b800b390 comp rx=0 tx=0).stop 2024-09-11T00:50:15.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.902+0000 7f09d6000640 1 -- 172.21.15.31:0/556814360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00c5b10 msgr2=0x7f09d0152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:15.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.902+0000 7f09d6000640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00c5b10 0x7f09d0152130 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f09c000d980 tx=0x7f09c000de40 comp rx=0 tx=0).stop 2024-09-11T00:50:15.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.902+0000 7f09d6000640 1 -- 172.21.15.31:0/556814360 shutdown_connections 2024-09-11T00:50:15.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.902+0000 7f09d6000640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f09ac077fa0 0x7f09ac07a460 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.903+0000 7f09d6000640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f09d00c6310 0x7f09d014d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.903+0000 7f09d6000640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09d00c5ef0 0x7f09d014cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.903+0000 7f09d6000640 1 --2- 172.21.15.31:0/556814360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d00c5b10 0x7f09d0152130 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:15.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.903+0000 7f09d6000640 1 -- 172.21.15.31:0/556814360 >> 172.21.15.31:0/556814360 conn(0x7f09d00a9750 msgr2=0x7f09d006f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:15.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.903+0000 7f09d6000640 1 -- 172.21.15.31:0/556814360 shutdown_connections 2024-09-11T00:50:15.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:15.903+0000 7f09d6000640 1 -- 172.21.15.31:0/556814360 wait complete. 2024-09-11T00:50:15.921 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-09-11T00:50:15.922 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:50:15.922 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:16.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.080+0000 7fa46c600640 1 -- 172.21.15.31:0/1027311394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa464072480 msgr2=0x7fa4641511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:16.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.080+0000 7fa46c600640 1 --2- 172.21.15.31:0/1027311394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa464072480 0x7fa4641511a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa4640add10 tx=0x7fa45802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:50:16.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.080+0000 7fa46c600640 1 -- 172.21.15.31:0/1027311394 shutdown_connections 2024-09-11T00:50:16.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.080+0000 7fa46c600640 1 --2- 172.21.15.31:0/1027311394 >> v2:172.21.15.204:3300/0 conn(0x7fa4640c23a0 0x7fa4641516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:16.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.080+0000 7fa46c600640 1 --2- 172.21.15.31:0/1027311394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa464072480 0x7fa4641511a0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:16.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.080+0000 7fa46c600640 1 -- 172.21.15.31:0/1027311394 >> 172.21.15.31:0/1027311394 conn(0x7fa4640a9750 msgr2=0x7fa4640a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:16.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.080+0000 7fa46c600640 1 -- 172.21.15.31:0/1027311394 shutdown_connections 2024-09-11T00:50:16.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.080+0000 7fa46c600640 1 -- 172.21.15.31:0/1027311394 wait complete. 2024-09-11T00:50:16.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.082+0000 7fa46c600640 1 Processor -- start 2024-09-11T00:50:16.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.082+0000 7fa46c600640 1 -- start start 2024-09-11T00:50:16.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46c600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa464072480 0x7fa46414a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46c600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4640c23a0 0x7fa4641de2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa464153f20 0x7fa4641de800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46c600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa464074200 con 0x7fa464153f20 2024-09-11T00:50:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46c600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa464074080 con 0x7fa4640c23a0 2024-09-11T00:50:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46c600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa464072860 con 0x7fa464072480 2024-09-11T00:50:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa464153f20 0x7fa4641de800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa464153f20 0x7fa4641de800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40878/0 (socket says 172.21.15.31:40878) 2024-09-11T00:50:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46a600640 1 -- 172.21.15.31:0/2367371811 learned_addr learned my addr 172.21.15.31:0/2367371811 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa469c00640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa464072480 0x7fa46414a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:16.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa469200640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4640c23a0 0x7fa4641de2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:16.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46a600640 1 -- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa464072480 msgr2=0x7fa46414a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:16.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46a600640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa464072480 0x7fa46414a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:16.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46a600640 1 -- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4640c23a0 msgr2=0x7fa4641de2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:16.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46a600640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4640c23a0 0x7fa4641de2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:16.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.083+0000 7fa46a600640 1 -- 172.21.15.31:0/2367371811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4641dede0 con 0x7fa464153f20 2024-09-11T00:50:16.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.084+0000 7fa46a600640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa464153f20 0x7fa4641de800 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fa45c00b970 tx=0x7fa45c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:16.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.084+0000 7fa461600640 1 -- 172.21.15.31:0/2367371811 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c019070 con 0x7fa464153f20 2024-09-11T00:50:16.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.084+0000 7fa461600640 1 -- 172.21.15.31:0/2367371811 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa45c00cc30 con 0x7fa464153f20 2024-09-11T00:50:16.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.084+0000 7fa461600640 1 -- 172.21.15.31:0/2367371811 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0146a0 con 0x7fa464153f20 2024-09-11T00:50:16.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.084+0000 7fa46c600640 1 -- 172.21.15.31:0/2367371811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4641df0c0 con 0x7fa464153f20 2024-09-11T00:50:16.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.084+0000 7fa46c600640 1 -- 172.21.15.31:0/2367371811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4641f09b0 con 0x7fa464153f20 2024-09-11T00:50:16.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.085+0000 7fa46c600640 1 -- 172.21.15.31:0/2367371811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa464077240 con 0x7fa464153f20 2024-09-11T00:50:16.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.086+0000 7fa461600640 1 -- 172.21.15.31:0/2367371811 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa45c01e440 con 0x7fa464153f20 2024-09-11T00:50:16.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.087+0000 7fa461600640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa444077fa0 0x7fa44407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:16.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.087+0000 7fa461600640 1 -- 172.21.15.31:0/2367371811 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8988+0+0 (secure 0 0 0) 0x7fa45c09a9a0 con 0x7fa464153f20 2024-09-11T00:50:16.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.088+0000 7fa469c00640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa444077fa0 0x7fa44407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.089+0000 7fa469c00640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa444077fa0 0x7fa44407a460 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa458033000 tx=0x7fa458004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:16.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.091+0000 7fa461600640 1 -- 172.21.15.31:0/2367371811 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa464077240 con 0x7fa464153f20 2024-09-11T00:50:16.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.328+0000 7fa46c600640 1 -- 172.21.15.31:0/2367371811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa46414cda0 con 0x7fa464153f20 2024-09-11T00:50:16.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.329+0000 7fa461600640 1 -- 172.21.15.31:0/2367371811 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5711 (secure 0 0 0) 0x7fa45c0620e0 con 0x7fa464153f20 2024-09-11T00:50:16.332 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:16.332 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":144,"btime":"2024-09-11T00:50:15:479276+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:14.831090+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:16.332 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 144 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 -- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa444077fa0 msgr2=0x7fa44407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa444077fa0 0x7fa44407a460 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa458033000 tx=0x7fa458004730 comp rx=0 tx=0).stop 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 -- 172.21.15.31:0/2367371811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa464153f20 msgr2=0x7fa4641de800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa464153f20 0x7fa4641de800 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fa45c00b970 tx=0x7fa45c00be30 comp rx=0 tx=0).stop 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 -- 172.21.15.31:0/2367371811 shutdown_connections 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa444077fa0 0x7fa44407a460 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa464153f20 0x7fa4641de800 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa4640c23a0 0x7fa4641de2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 --2- 172.21.15.31:0/2367371811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa464072480 0x7fa46414a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 -- 172.21.15.31:0/2367371811 >> 172.21.15.31:0/2367371811 conn(0x7fa4640a9750 msgr2=0x7fa4640c0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 -- 172.21.15.31:0/2367371811 shutdown_connections 2024-09-11T00:50:16.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:16.332+0000 7fa46c600640 1 -- 172.21.15.31:0/2367371811 wait complete. 2024-09-11T00:50:16.353 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:50:15:479276+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 144, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7143": { "addr": "172.21.15.204:6835/1981755707", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1981755707, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 1981755707, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7143, "incarnation": 143, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:50:14.831090+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7143 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 141, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7065": { "addr": "172.21.15.31:6835/3725011623", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 3725011623, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 3725011623, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7065, "incarnation": 138, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:49:37.512743+0000", "qdb_cluster": [ 7065 ], "qdb_leader": 7065, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7065 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:50:17.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:17.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.269+0000 7f0978e00640 1 -- 172.21.15.31:0/1532110812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f097406e850 msgr2=0x7f0974161870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:17.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.269+0000 7f0978e00640 1 --2- 172.21.15.31:0/1532110812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f097406e850 0x7f0974161870 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f096c005fd0 tx=0x7f096c02a8b0 comp rx=0 tx=0).stop 2024-09-11T00:50:17.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.270+0000 7f0978e00640 1 -- 172.21.15.31:0/1532110812 shutdown_connections 2024-09-11T00:50:17.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.270+0000 7f0978e00640 1 --2- 172.21.15.31:0/1532110812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f097406e850 0x7f0974161870 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:17.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.270+0000 7f0978e00640 1 -- 172.21.15.31:0/1532110812 >> 172.21.15.31:0/1532110812 conn(0x7f0974068f10 msgr2=0x7f09740cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:17.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.270+0000 7f0978e00640 1 -- 172.21.15.31:0/1532110812 shutdown_connections 2024-09-11T00:50:17.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.270+0000 7f0978e00640 1 -- 172.21.15.31:0/1532110812 wait complete. 2024-09-11T00:50:17.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.271+0000 7f0978e00640 1 Processor -- start 2024-09-11T00:50:17.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.271+0000 7f0978e00640 1 -- start start 2024-09-11T00:50:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.272+0000 7f0978e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f097406e850 0x7f097414b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.272+0000 7f0978e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0974150020 0x7f0974155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.272+0000 7f0978e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0974156960 0x7f09740cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.272+0000 7f0978e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09740c6ca0 con 0x7f0974150020 2024-09-11T00:50:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.272+0000 7f0978e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f09740c6b20 con 0x7f0974156960 2024-09-11T00:50:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.272+0000 7f0978e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f09741573d0 con 0x7f097406e850 2024-09-11T00:50:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.272+0000 7f096be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0974150020 0x7f0974155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.273+0000 7f096be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0974150020 0x7f0974155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40892/0 (socket says 172.21.15.31:40892) 2024-09-11T00:50:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.273+0000 7f096be00640 1 -- 172.21.15.31:0/2590127386 learned_addr learned my addr 172.21.15.31:0/2590127386 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:17.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.273+0000 7f0973400640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f097406e850 0x7f097414b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:17.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.273+0000 7f0973e00640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0974156960 0x7f09740cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:17.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.273+0000 7f096be00640 1 -- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f097406e850 msgr2=0x7f097414b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:17.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.273+0000 7f096be00640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f097406e850 0x7f097414b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:17.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.273+0000 7f096be00640 1 -- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0974156960 msgr2=0x7f09740cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:17.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.273+0000 7f096be00640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0974156960 0x7f09740cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:17.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.273+0000 7f096be00640 1 -- 172.21.15.31:0/2590127386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09741ec160 con 0x7f0974150020 2024-09-11T00:50:17.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.273+0000 7f096be00640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0974150020 0x7f0974155360 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f095c00b3f0 tx=0x7f095c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.274+0000 7f096b400640 1 -- 172.21.15.31:0/2590127386 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c015020 con 0x7f0974150020 2024-09-11T00:50:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.274+0000 7f096b400640 1 -- 172.21.15.31:0/2590127386 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f095c011920 con 0x7f0974150020 2024-09-11T00:50:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.274+0000 7f096b400640 1 -- 172.21.15.31:0/2590127386 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c011d70 con 0x7f0974150020 2024-09-11T00:50:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.274+0000 7f0978e00640 1 -- 172.21.15.31:0/2590127386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09740ca3c0 con 0x7f0974150020 2024-09-11T00:50:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.274+0000 7f0978e00640 1 -- 172.21.15.31:0/2590127386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09740c9d50 con 0x7f0974150020 2024-09-11T00:50:17.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.275+0000 7f0978e00640 1 -- 172.21.15.31:0/2590127386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09740cc220 con 0x7f0974150020 2024-09-11T00:50:17.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.276+0000 7f096b400640 1 -- 172.21.15.31:0/2590127386 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f095c01a400 con 0x7f0974150020 2024-09-11T00:50:17.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.277+0000 7f096b400640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0950077f70 0x7f095007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:17.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.277+0000 7f096b400640 1 -- 172.21.15.31:0/2590127386 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8988+0+0 (secure 0 0 0) 0x7f095c09a7c0 con 0x7f0974150020 2024-09-11T00:50:17.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.278+0000 7f0973400640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0950077f70 0x7f095007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:17.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.278+0000 7f0973400640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0950077f70 0x7f095007a430 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f097406de30 tx=0x7f0964007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:17.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.281+0000 7f096b400640 1 -- 172.21.15.31:0/2590127386 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f095c01f030 con 0x7f0974150020 2024-09-11T00:50:17.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.528+0000 7f0978e00640 1 -- 172.21.15.31:0/2590127386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09740cab60 con 0x7f0974150020 2024-09-11T00:50:17.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.529+0000 7f096b400640 1 -- 172.21.15.31:0/2590127386 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+5716 (secure 0 0 0) 0x7f095c061fe0 con 0x7f0974150020 2024-09-11T00:50:17.533 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:17.533 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":146,"btime":"2024-09-11T00:50:17:486440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:17.533 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 146 2024-09-11T00:50:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.533+0000 7f0978e00640 1 -- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0950077f70 msgr2=0x7f095007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.533+0000 7f0978e00640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0950077f70 0x7f095007a430 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f097406de30 tx=0x7f0964007eb0 comp rx=0 tx=0).stop 2024-09-11T00:50:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.533+0000 7f0978e00640 1 -- 172.21.15.31:0/2590127386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0974150020 msgr2=0x7f0974155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.533+0000 7f0978e00640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0974150020 0x7f0974155360 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f095c00b3f0 tx=0x7f095c00b8b0 comp rx=0 tx=0).stop 2024-09-11T00:50:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.533+0000 7f0978e00640 1 -- 172.21.15.31:0/2590127386 shutdown_connections 2024-09-11T00:50:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.533+0000 7f0978e00640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0950077f70 0x7f095007a430 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.533+0000 7f0978e00640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0974156960 0x7f09740cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.533+0000 7f0978e00640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0974150020 0x7f0974155360 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.533+0000 7f0978e00640 1 --2- 172.21.15.31:0/2590127386 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f097406e850 0x7f097414b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:17.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.533+0000 7f0978e00640 1 -- 172.21.15.31:0/2590127386 >> 172.21.15.31:0/2590127386 conn(0x7f0974068f10 msgr2=0x7f097414ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:17.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.534+0000 7f0978e00640 1 -- 172.21.15.31:0/2590127386 shutdown_connections 2024-09-11T00:50:17.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:17.534+0000 7f0978e00640 1 -- 172.21.15.31:0/2590127386 wait complete. 2024-09-11T00:50:17.562 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:50:18.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.601+0000 7f9921e00640 1 -- 172.21.15.31:0/204705717 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99000051c0 msgr2=0x7f990000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.601+0000 7f9921e00640 1 --2- 172.21.15.31:0/204705717 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99000051c0 0x7f990000b9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f991400c680 tx=0x7f9900005830 comp rx=0 tx=0).stop 2024-09-11T00:50:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.601+0000 7f9921e00640 1 -- 172.21.15.31:0/204705717 shutdown_connections 2024-09-11T00:50:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.601+0000 7f9921e00640 1 --2- 172.21.15.31:0/204705717 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99000051c0 0x7f990000b9c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.601+0000 7f9921e00640 1 -- 172.21.15.31:0/204705717 >> 172.21.15.31:0/204705717 conn(0x7f991c0c8320 msgr2=0x7f991c0c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.601+0000 7f9921e00640 1 -- 172.21.15.31:0/204705717 shutdown_connections 2024-09-11T00:50:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.601+0000 7f9921e00640 1 -- 172.21.15.31:0/204705717 wait complete. 2024-09-11T00:50:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.604+0000 7f9921e00640 1 Processor -- start 2024-09-11T00:50:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.604+0000 7f9921e00640 1 -- start start 2024-09-11T00:50:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.605+0000 7f9921e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99000051c0 0x7f991c074140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.605+0000 7f9921e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f991c158bc0 0x7f991c1e3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.605+0000 7f9921e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991c159270 0x7f991c159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.605+0000 7f9921e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f991c0c2dc0 con 0x7f991c159270 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.605+0000 7f9921e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f991c15a3d0 con 0x7f991c158bc0 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.605+0000 7f9921e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f991c14ce00 con 0x7f99000051c0 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.606+0000 7f991aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99000051c0 0x7f991c074140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.606+0000 7f991aa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99000051c0 0x7f991c074140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:56524/0 (socket says 172.21.15.31:56524) 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.606+0000 7f991aa00640 1 -- 172.21.15.31:0/1584283828 learned_addr learned my addr 172.21.15.31:0/1584283828 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.606+0000 7f991b400640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991c159270 0x7f991c159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.607+0000 7f991a000640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f991c158bc0 0x7f991c1e3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.607+0000 7f991b400640 1 -- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99000051c0 msgr2=0x7f991c074140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.607+0000 7f991b400640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99000051c0 0x7f991c074140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.607+0000 7f991b400640 1 -- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f991c158bc0 msgr2=0x7f991c1e3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.607+0000 7f991b400640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f991c158bc0 0x7f991c1e3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.607+0000 7f991b400640 1 -- 172.21.15.31:0/1584283828 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991c1f2f30 con 0x7f991c159270 2024-09-11T00:50:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.608+0000 7f991b400640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991c159270 0x7f991c159720 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f9914001ca0 tx=0x7f9914004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.608+0000 7f9913400640 1 -- 172.21.15.31:0/1584283828 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991402bd80 con 0x7f991c159270 2024-09-11T00:50:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.608+0000 7f9913400640 1 -- 172.21.15.31:0/1584283828 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9914006030 con 0x7f991c159270 2024-09-11T00:50:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.608+0000 7f9913400640 1 -- 172.21.15.31:0/1584283828 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9914006440 con 0x7f991c159270 2024-09-11T00:50:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.609+0000 7f9921e00640 1 -- 172.21.15.31:0/1584283828 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991c1f31a0 con 0x7f991c159270 2024-09-11T00:50:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.609+0000 7f9921e00640 1 -- 172.21.15.31:0/1584283828 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f991c1f3760 con 0x7f991c159270 2024-09-11T00:50:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.610+0000 7f9921e00640 1 -- 172.21.15.31:0/1584283828 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f991c06caf0 con 0x7f991c159270 2024-09-11T00:50:18.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.615+0000 7f9913400640 1 -- 172.21.15.31:0/1584283828 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f99140065e0 con 0x7f991c159270 2024-09-11T00:50:18.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.617+0000 7f9913400640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f98fc077fa0 0x7f98fc07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:18.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.617+0000 7f9913400640 1 -- 172.21.15.31:0/1584283828 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8988+0+0 (secure 0 0 0) 0x7f99140ba4a0 con 0x7f991c159270 2024-09-11T00:50:18.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.618+0000 7f991aa00640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f98fc077fa0 0x7f98fc07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:18.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.619+0000 7f991aa00640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f98fc077fa0 0x7f98fc07a460 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f990c009e30 tx=0x7f990c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:18.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:18.627+0000 7f9913400640 1 -- 172.21.15.31:0/1584283828 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9914044070 con 0x7f991c159270 2024-09-11T00:50:19.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.066+0000 7f9921e00640 1 -- 172.21.15.31:0/1584283828 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f991c069500 con 0x7f991c159270 2024-09-11T00:50:19.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.067+0000 7f9913400640 1 -- 172.21.15.31:0/1584283828 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+5716 (secure 0 0 0) 0x7f99140822b0 con 0x7f991c159270 2024-09-11T00:50:19.070 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:19.070 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":146,"btime":"2024-09-11T00:50:17:486440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:19.071 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 146 2024-09-11T00:50:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 -- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f98fc077fa0 msgr2=0x7f98fc07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f98fc077fa0 0x7f98fc07a460 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f990c009e30 tx=0x7f990c005f90 comp rx=0 tx=0).stop 2024-09-11T00:50:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 -- 172.21.15.31:0/1584283828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991c159270 msgr2=0x7f991c159720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991c159270 0x7f991c159720 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f9914001ca0 tx=0x7f9914004bb0 comp rx=0 tx=0).stop 2024-09-11T00:50:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 -- 172.21.15.31:0/1584283828 shutdown_connections 2024-09-11T00:50:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f98fc077fa0 0x7f98fc07a460 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f991c159270 0x7f991c159720 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f991c158bc0 0x7f991c1e3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:19.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 --2- 172.21.15.31:0/1584283828 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f99000051c0 0x7f991c074140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 -- 172.21.15.31:0/1584283828 >> 172.21.15.31:0/1584283828 conn(0x7f991c0c8320 msgr2=0x7f991c06cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 -- 172.21.15.31:0/1584283828 shutdown_connections 2024-09-11T00:50:19.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.072+0000 7f9921e00640 1 -- 172.21.15.31:0/1584283828 wait complete. 2024-09-11T00:50:19.100 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=6934 2024-09-11T00:50:19.101 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.b 2024-09-11T00:50:19.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:19.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.804+0000 7f01da000640 1 -- 172.21.15.31:0/3330944898 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01d40cc220 msgr2=0x7f01d406e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:19.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.804+0000 7f01da000640 1 --2- 172.21.15.31:0/3330944898 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01d40cc220 0x7f01d406e680 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009830 tx=0x7f01cc02a480 comp rx=0 tx=0).stop 2024-09-11T00:50:19.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.804+0000 7f01da000640 1 -- 172.21.15.31:0/3330944898 shutdown_connections 2024-09-11T00:50:19.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.804+0000 7f01da000640 1 --2- 172.21.15.31:0/3330944898 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01d40cc220 0x7f01d406e680 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:19.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.804+0000 7f01da000640 1 --2- 172.21.15.31:0/3330944898 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01d4156960 0x7f01d406e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:19.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.804+0000 7f01da000640 1 -- 172.21.15.31:0/3330944898 >> 172.21.15.31:0/3330944898 conn(0x7f01d4068f10 msgr2=0x7f01d40cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:19.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.804+0000 7f01da000640 1 -- 172.21.15.31:0/3330944898 shutdown_connections 2024-09-11T00:50:19.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.804+0000 7f01da000640 1 -- 172.21.15.31:0/3330944898 wait complete. 2024-09-11T00:50:19.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.807+0000 7f01da000640 1 Processor -- start 2024-09-11T00:50:19.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.808+0000 7f01da000640 1 -- start start 2024-09-11T00:50:19.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.809+0000 7f01da000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01d40cc220 0x7f01d40b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:19.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.810+0000 7f01da000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01d4150020 0x7f01d41f8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:19.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.810+0000 7f01da000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01d4156960 0x7f01d40b1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:19.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.810+0000 7f01da000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f01d40c6ca0 con 0x7f01d4150020 2024-09-11T00:50:19.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.810+0000 7f01da000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f01d40c6b20 con 0x7f01d40cc220 2024-09-11T00:50:19.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.810+0000 7f01da000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f01d41573d0 con 0x7f01d4156960 2024-09-11T00:50:19.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.810+0000 7f01d8c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01d40cc220 0x7f01d40b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:19.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.811+0000 7f01d8c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01d40cc220 0x7f01d40b09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:43922/0 (socket says 172.21.15.31:43922) 2024-09-11T00:50:19.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.811+0000 7f01d8c00640 1 -- 172.21.15.31:0/3817167084 learned_addr learned my addr 172.21.15.31:0/3817167084 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:19.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.811+0000 7f01d3e00640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01d4150020 0x7f01d41f8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:19.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.811+0000 7f01d8c00640 1 -- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01d4156960 msgr2=0x7f01d40b1210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:50:19.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.811+0000 7f01d8c00640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01d4156960 0x7f01d40b1210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:19.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.811+0000 7f01d8c00640 1 -- 172.21.15.31:0/3817167084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01d4150020 msgr2=0x7f01d41f8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:19.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.812+0000 7f01d8c00640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01d4150020 0x7f01d41f8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:19.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.812+0000 7f01d8c00640 1 -- 172.21.15.31:0/3817167084 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d4152df0 con 0x7f01d40cc220 2024-09-11T00:50:19.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.813+0000 7f01d3e00640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01d4150020 0x7f01d41f8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:50:19.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.813+0000 7f01d8c00640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01d40cc220 0x7f01d40b09f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f01c800cc50 tx=0x7f01c800a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:19.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.814+0000 7f01d0c00640 1 -- 172.21.15.31:0/3817167084 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8016020 con 0x7f01d40cc220 2024-09-11T00:50:19.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.814+0000 7f01d0c00640 1 -- 172.21.15.31:0/3817167084 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01c8012420 con 0x7f01d40cc220 2024-09-11T00:50:19.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.814+0000 7f01d0c00640 1 -- 172.21.15.31:0/3817167084 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8012850 con 0x7f01d40cc220 2024-09-11T00:50:19.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.814+0000 7f01da000640 1 -- 172.21.15.31:0/3817167084 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d40c1f40 con 0x7f01d40cc220 2024-09-11T00:50:19.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.814+0000 7f01da000640 1 -- 172.21.15.31:0/3817167084 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d40c2500 con 0x7f01d40cc220 2024-09-11T00:50:19.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.817+0000 7f01da000640 1 -- 172.21.15.31:0/3817167084 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d4157670 con 0x7f01d40cc220 2024-09-11T00:50:19.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.817+0000 7f01d0c00640 1 -- 172.21.15.31:0/3817167084 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f01c801b400 con 0x7f01d40cc220 2024-09-11T00:50:19.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.818+0000 7f01d0c00640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f01b4077fa0 0x7f01b407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:19.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.819+0000 7f01d0c00640 1 -- 172.21.15.31:0/3817167084 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8988+0+0 (secure 0 0 0) 0x7f01c809b4a0 con 0x7f01d40cc220 2024-09-11T00:50:19.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.819+0000 7f01d3e00640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f01b4077fa0 0x7f01b407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:19.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.820+0000 7f01d3e00640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f01b4077fa0 0x7f01b407a460 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f01d40741f0 tx=0x7f01c0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:19.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:19.823+0000 7f01d0c00640 1 -- 172.21.15.31:0/3817167084 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f01d4157670 con 0x7f01d40cc220 2024-09-11T00:50:20.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.245+0000 7f01da000640 1 -- 172.21.15.31:0/3817167084 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01d40ad4e0 con 0x7f01d40cc220 2024-09-11T00:50:20.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.246+0000 7f01d0c00640 1 -- 172.21.15.31:0/3817167084 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+5716 (secure 0 0 0) 0x7f01c8020020 con 0x7f01d40cc220 2024-09-11T00:50:20.249 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:20.249 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":146,"btime":"2024-09-11T00:50:17:486440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:20.249 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 146 2024-09-11T00:50:20.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.250+0000 7f01da000640 1 -- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f01b4077fa0 msgr2=0x7f01b407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:20.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.250+0000 7f01da000640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f01b4077fa0 0x7f01b407a460 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f01d40741f0 tx=0x7f01c0005f90 comp rx=0 tx=0).stop 2024-09-11T00:50:20.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.250+0000 7f01da000640 1 -- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01d40cc220 msgr2=0x7f01d40b09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:20.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.250+0000 7f01da000640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01d40cc220 0x7f01d40b09f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f01c800cc50 tx=0x7f01c800a430 comp rx=0 tx=0).stop 2024-09-11T00:50:20.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.250+0000 7f01da000640 1 -- 172.21.15.31:0/3817167084 shutdown_connections 2024-09-11T00:50:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.250+0000 7f01da000640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f01b4077fa0 0x7f01b407a460 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.250+0000 7f01da000640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f01d4156960 0x7f01d40b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.250+0000 7f01da000640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01d4150020 0x7f01d41f8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.250+0000 7f01da000640 1 --2- 172.21.15.31:0/3817167084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01d40cc220 0x7f01d40b09f0 secure :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f01c800cc50 tx=0x7f01c800a430 comp rx=0 tx=0).stop 2024-09-11T00:50:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.250+0000 7f01da000640 1 -- 172.21.15.31:0/3817167084 >> 172.21.15.31:0/3817167084 conn(0x7f01d4068f10 msgr2=0x7f01d40ab8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.251+0000 7f01da000640 1 -- 172.21.15.31:0/3817167084 shutdown_connections 2024-09-11T00:50:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:20.251+0000 7f01da000640 1 -- 172.21.15.31:0/3817167084 wait complete. 2024-09-11T00:50:20.277 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:50:20.627 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:50:20.633 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:50:21.753 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:50:21.751+0000 7f100d000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:50:22.102 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-09-11T00:50:22.102 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-11T00:50:22.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-11T00:50:22.108 INFO:tasks.ceph.mds.b:Started 2024-09-11T00:50:22.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:22.261 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:50:22.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:22.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.348+0000 7fdaf5000640 1 -- 172.21.15.31:0/2266624590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaf006e9c0 msgr2=0x7fdaf0155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:22.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.348+0000 7fdaf5000640 1 --2- 172.21.15.31:0/2266624590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaf006e9c0 0x7fdaf0155560 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009830 tx=0x7fdadc02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.349+0000 7fdaf5000640 1 -- 172.21.15.31:0/2266624590 shutdown_connections 2024-09-11T00:50:22.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.349+0000 7fdaf5000640 1 --2- 172.21.15.31:0/2266624590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaf006e9c0 0x7fdaf0155560 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.349+0000 7fdaf5000640 1 --2- 172.21.15.31:0/2266624590 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdaf0070570 0x7fdaf006e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.349+0000 7fdaf5000640 1 -- 172.21.15.31:0/2266624590 >> 172.21.15.31:0/2266624590 conn(0x7fdaf00a9750 msgr2=0x7fdaf00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:22.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.349+0000 7fdaf5000640 1 -- 172.21.15.31:0/2266624590 shutdown_connections 2024-09-11T00:50:22.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.350+0000 7fdaf5000640 1 -- 172.21.15.31:0/2266624590 wait complete. 2024-09-11T00:50:22.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.353+0000 7fdaf5000640 1 Processor -- start 2024-09-11T00:50:22.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.354+0000 7fdaf5000640 1 -- start start 2024-09-11T00:50:22.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.356+0000 7fdaf5000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdaf006e9c0 0x7fdaf01ef8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:22.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.356+0000 7fdaf5000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaf0070570 0x7fdaf0143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:22.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.356+0000 7fdaf5000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdaf00c2ff0 0x7fdaf0143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:22.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.356+0000 7fdaf5000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf014fb20 con 0x7fdaf0070570 2024-09-11T00:50:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.356+0000 7fdaf5000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf006ef90 con 0x7fdaf006e9c0 2024-09-11T00:50:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.356+0000 7fdaf5000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdaf00c7690 con 0x7fdaf00c2ff0 2024-09-11T00:50:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.356+0000 7fdaed400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaf0070570 0x7fdaf0143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.356+0000 7fdaede00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdaf006e9c0 0x7fdaf01ef8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.356+0000 7fdaede00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdaf006e9c0 0x7fdaf01ef8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:43924/0 (socket says 172.21.15.31:43924) 2024-09-11T00:50:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.356+0000 7fdaede00640 1 -- 172.21.15.31:0/2337542734 learned_addr learned my addr 172.21.15.31:0/2337542734 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.357+0000 7fdaede00640 1 -- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdaf00c2ff0 msgr2=0x7fdaf0143fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.357+0000 7fdaede00640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdaf00c2ff0 0x7fdaf0143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.357+0000 7fdaede00640 1 -- 172.21.15.31:0/2337542734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaf0070570 msgr2=0x7fdaf0143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.357+0000 7fdaede00640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaf0070570 0x7fdaf0143a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.357+0000 7fdaede00640 1 -- 172.21.15.31:0/2337542734 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf00adb60 con 0x7fdaf006e9c0 2024-09-11T00:50:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.357+0000 7fdaede00640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdaf006e9c0 0x7fdaf01ef8c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fdae000e9f0 tx=0x7fdae000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.357+0000 7fdae6000640 1 -- 172.21.15.31:0/2337542734 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae000cd80 con 0x7fdaf006e9c0 2024-09-11T00:50:22.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.357+0000 7fdae6000640 1 -- 172.21.15.31:0/2337542734 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdae0010430 con 0x7fdaf006e9c0 2024-09-11T00:50:22.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.357+0000 7fdae6000640 1 -- 172.21.15.31:0/2337542734 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0010800 con 0x7fdaf006e9c0 2024-09-11T00:50:22.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.357+0000 7fdaf5000640 1 -- 172.21.15.31:0/2337542734 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf0144500 con 0x7fdaf006e9c0 2024-09-11T00:50:22.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.358+0000 7fdaf5000640 1 -- 172.21.15.31:0/2337542734 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf0147f10 con 0x7fdaf006e9c0 2024-09-11T00:50:22.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.360+0000 7fdaf5000640 1 -- 172.21.15.31:0/2337542734 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaf0148150 con 0x7fdaf006e9c0 2024-09-11T00:50:22.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.361+0000 7fdae6000640 1 -- 172.21.15.31:0/2337542734 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdae00109a0 con 0x7fdaf006e9c0 2024-09-11T00:50:22.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.362+0000 7fdae6000640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdad0077fa0 0x7fdad007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:22.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.362+0000 7fdae6000640 1 -- 172.21.15.31:0/2337542734 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8988+0+0 (secure 0 0 0) 0x7fdae009b590 con 0x7fdaf006e9c0 2024-09-11T00:50:22.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.362+0000 7fdaed400640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdad0077fa0 0x7fdad007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:22.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.364+0000 7fdaed400640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdad0077fa0 0x7fdad007a460 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fdadc034000 tx=0x7fdadc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.367+0000 7fdae6000640 1 -- 172.21.15.31:0/2337542734 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdaf0148150 con 0x7fdaf006e9c0 2024-09-11T00:50:22.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.562+0000 7f910d000640 1 -- 172.21.15.31:0/843324804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91080b1230 msgr2=0x7f91081596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:22.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.562+0000 7f910d000640 1 --2- 172.21.15.31:0/843324804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91080b1230 0x7f91081596b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9108069820 tx=0x7f90f802a9a0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.563+0000 7f910d000640 1 -- 172.21.15.31:0/843324804 shutdown_connections 2024-09-11T00:50:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.563+0000 7f910d000640 1 --2- 172.21.15.31:0/843324804 >> v2:172.21.15.31:3300/0 conn(0x7f91080c9fe0 0x7f91080cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.563+0000 7f910d000640 1 --2- 172.21.15.31:0/843324804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91080b1230 0x7f91081596b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.563+0000 7f910d000640 1 -- 172.21.15.31:0/843324804 >> 172.21.15.31:0/843324804 conn(0x7f9108068f10 msgr2=0x7f91080cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.564+0000 7f910d000640 1 -- 172.21.15.31:0/843324804 shutdown_connections 2024-09-11T00:50:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.564+0000 7f910d000640 1 -- 172.21.15.31:0/843324804 wait complete. 2024-09-11T00:50:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.566+0000 7f910d000640 1 Processor -- start 2024-09-11T00:50:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.567+0000 7f910d000640 1 -- start start 2024-09-11T00:50:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.568+0000 7f910d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91080b1230 0x7f91080c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.568+0000 7f910d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91080c9b00 0x7f91081585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.569+0000 7f910d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91080c9fe0 0x7f91081efbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.569+0000 7f910d000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9108156540 con 0x7f91080c9b00 2024-09-11T00:50:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.569+0000 7f910d000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9108156260 con 0x7f91080c9fe0 2024-09-11T00:50:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.569+0000 7f910d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f91080bf390 con 0x7f91080b1230 2024-09-11T00:50:22.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.569+0000 7f9106800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91080c9fe0 0x7f91081efbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:22.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.569+0000 7f9106800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91080c9fe0 0x7f91081efbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:43932/0 (socket says 172.21.15.31:43932) 2024-09-11T00:50:22.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.569+0000 7f9106800640 1 -- 172.21.15.31:0/898966844 learned_addr learned my addr 172.21.15.31:0/898966844 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.570+0000 7f9105e00640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91080b1230 0x7f91080c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.570+0000 7f9106800640 1 -- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91080b1230 msgr2=0x7f91080c5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.570+0000 7f9106800640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91080b1230 0x7f91080c5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.570+0000 7f9106800640 1 -- 172.21.15.31:0/898966844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91080c9b00 msgr2=0x7f91081585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.570+0000 7f9105400640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91080c9b00 0x7f91081585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.571+0000 7f9106800640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91080c9b00 0x7f91081585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.571+0000 7f9106800640 1 -- 172.21.15.31:0/898966844 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91081ef700 con 0x7f91080c9fe0 2024-09-11T00:50:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.571+0000 7f9105e00640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91080b1230 0x7f91080c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:50:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.571+0000 7f9105400640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91080c9b00 0x7f91081585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:50:22.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.571+0000 7f9106800640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91080c9fe0 0x7f91081efbe0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f910000d660 tx=0x7f910000db20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:22.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.571+0000 7f90fe000640 1 -- 172.21.15.31:0/898966844 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100017070 con 0x7f91080c9fe0 2024-09-11T00:50:22.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.571+0000 7f90fe000640 1 -- 172.21.15.31:0/898966844 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9100005310 con 0x7f91080c9fe0 2024-09-11T00:50:22.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.571+0000 7f90fe000640 1 -- 172.21.15.31:0/898966844 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100009820 con 0x7f91080c9fe0 2024-09-11T00:50:22.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.572+0000 7f910d000640 1 -- 172.21.15.31:0/898966844 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91080c5590 con 0x7f91080c9fe0 2024-09-11T00:50:22.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.573+0000 7f910d000640 1 -- 172.21.15.31:0/898966844 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91081ecfc0 con 0x7f91080c9fe0 2024-09-11T00:50:22.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.575+0000 7f90fe000640 1 -- 172.21.15.31:0/898966844 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9100038b10 con 0x7f91080c9fe0 2024-09-11T00:50:22.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.575+0000 7f910d000640 1 -- 172.21.15.31:0/898966844 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91080766d0 con 0x7f91080c9fe0 2024-09-11T00:50:22.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.577+0000 7f90fe000640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90e8077f70 0x7f90e807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:22.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.577+0000 7f90fe000640 1 -- 172.21.15.31:0/898966844 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8988+0+0 (secure 0 0 0) 0x7f910009a2c0 con 0x7f91080c9fe0 2024-09-11T00:50:22.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.579+0000 7f9105e00640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90e8077f70 0x7f90e807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:22.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.580+0000 7f9105e00640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90e8077f70 0x7f90e807a430 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f90f8035000 tx=0x7f90f8004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:22.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.585+0000 7f90fe000640 1 -- 172.21.15.31:0/898966844 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f91000620d0 con 0x7f91080c9fe0 2024-09-11T00:50:22.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.795+0000 7fdaf5000640 1 -- 172.21.15.31:0/2337542734 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdaf014f740 con 0x7fdaf006e9c0 2024-09-11T00:50:22.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.795+0000 7fdae6000640 1 -- 172.21.15.31:0/2337542734 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+5716 (secure 0 0 0) 0x7fdae0063400 con 0x7fdaf006e9c0 2024-09-11T00:50:22.797 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:22.798 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":146,"btime":"2024-09-11T00:50:17:486440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:22.798 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 146 2024-09-11T00:50:22.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.798+0000 7fdaf5000640 1 -- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdad0077fa0 msgr2=0x7fdad007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:22.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.798+0000 7fdaf5000640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdad0077fa0 0x7fdad007a460 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fdadc034000 tx=0x7fdadc004730 comp rx=0 tx=0).stop 2024-09-11T00:50:22.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.798+0000 7fdaf5000640 1 -- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdaf006e9c0 msgr2=0x7fdaf01ef8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:22.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.798+0000 7fdaf5000640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdaf006e9c0 0x7fdaf01ef8c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fdae000e9f0 tx=0x7fdae000eeb0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.799+0000 7fdaf5000640 1 -- 172.21.15.31:0/2337542734 shutdown_connections 2024-09-11T00:50:22.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.799+0000 7fdaf5000640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdad0077fa0 0x7fdad007a460 secure :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fdadc034000 tx=0x7fdadc004730 comp rx=0 tx=0).stop 2024-09-11T00:50:22.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.799+0000 7fdaf5000640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdaf00c2ff0 0x7fdaf0143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.799+0000 7fdaf5000640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaf0070570 0x7fdaf0143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.799+0000 7fdaf5000640 1 --2- 172.21.15.31:0/2337542734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdaf006e9c0 0x7fdaf01ef8c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:22.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.799+0000 7fdaf5000640 1 -- 172.21.15.31:0/2337542734 >> 172.21.15.31:0/2337542734 conn(0x7fdaf00a9750 msgr2=0x7fdaf00a8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:22.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.799+0000 7fdaf5000640 1 -- 172.21.15.31:0/2337542734 shutdown_connections 2024-09-11T00:50:22.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:22.799+0000 7fdaf5000640 1 -- 172.21.15.31:0/2337542734 wait complete. 2024-09-11T00:50:22.825 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-11T00:50:23.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.017+0000 7f910d000640 1 -- 172.21.15.31:0/898966844 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91081596b0 con 0x7f91080c9fe0 2024-09-11T00:50:23.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.017+0000 7f90fe000640 1 -- 172.21.15.31:0/898966844 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+5716 (secure 0 0 0) 0x7f91000619d0 con 0x7f91080c9fe0 2024-09-11T00:50:23.019 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:23.019 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":146,"btime":"2024-09-11T00:50:17:486440+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:49:37.512743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7065},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7065":{"gid":7065,"name":"c","rank":0,"incarnation":138,"state":"up:active","state_seq":22,"addr":"172.21.15.31:6835/3725011623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3725011623},{"type":"v1","addr":"172.21.15.31:6835","nonce":3725011623}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7065,"qdb_cluster":[7065]},"id":2}]} 2024-09-11T00:50:23.020 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 146 2024-09-11T00:50:23.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 -- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90e8077f70 msgr2=0x7f90e807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:23.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90e8077f70 0x7f90e807a430 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f90f8035000 tx=0x7f90f8004ab0 comp rx=0 tx=0).stop 2024-09-11T00:50:23.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 -- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91080c9fe0 msgr2=0x7f91081efbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:23.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91080c9fe0 0x7f91081efbe0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f910000d660 tx=0x7f910000db20 comp rx=0 tx=0).stop 2024-09-11T00:50:23.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 -- 172.21.15.31:0/898966844 shutdown_connections 2024-09-11T00:50:23.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f90e8077f70 0x7f90e807a430 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:23.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91080c9fe0 0x7f91081efbe0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:23.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91080c9b00 0x7f91081585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:23.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 --2- 172.21.15.31:0/898966844 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91080b1230 0x7f91080c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:23.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 -- 172.21.15.31:0/898966844 >> 172.21.15.31:0/898966844 conn(0x7f9108068f10 msgr2=0x7f9108159dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:23.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.021+0000 7f910d000640 1 -- 172.21.15.31:0/898966844 shutdown_connections 2024-09-11T00:50:23.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:23.022+0000 7f910d000640 1 -- 172.21.15.31:0/898966844 wait complete. 2024-09-11T00:50:23.050 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:50:24.827 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:25.052 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.078+0000 7ff19f400640 1 -- 172.21.15.31:0/539455217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1980cb620 msgr2=0x7ff1980b11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.078+0000 7ff19f400640 1 --2- 172.21.15.31:0/539455217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1980cb620 0x7ff1980b11c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff190005fd0 tx=0x7ff19002aaa0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.078+0000 7ff19f400640 1 -- 172.21.15.31:0/539455217 shutdown_connections 2024-09-11T00:50:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.078+0000 7ff19f400640 1 --2- 172.21.15.31:0/539455217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1980cb620 0x7ff1980b11c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.078+0000 7ff19f400640 1 -- 172.21.15.31:0/539455217 >> 172.21.15.31:0/539455217 conn(0x7ff198068f10 msgr2=0x7ff1980ad750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.078+0000 7ff19f400640 1 -- 172.21.15.31:0/539455217 shutdown_connections 2024-09-11T00:50:25.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.079+0000 7ff19f400640 1 -- 172.21.15.31:0/539455217 wait complete. 2024-09-11T00:50:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.081+0000 7ff19f400640 1 Processor -- start 2024-09-11T00:50:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.081+0000 7ff19f400640 1 -- start start 2024-09-11T00:50:25.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.082+0000 7ff19f400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1980c9b40 0x7ff198152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:25.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.082+0000 7ff19f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1980cb240 0x7ff1981dff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:25.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.082+0000 7ff19f400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1980cb620 0x7ff19814bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:25.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.082+0000 7ff19f400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff198156b70 con 0x7ff1980cb240 2024-09-11T00:50:25.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.082+0000 7ff19f400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff1980c6ad0 con 0x7ff1980cb620 2024-09-11T00:50:25.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.082+0000 7ff19f400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff198156cf0 con 0x7ff1980c9b40 2024-09-11T00:50:25.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.083+0000 7ff197e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1980cb240 0x7ff1981dff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.083+0000 7ff197e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1980cb240 0x7ff1981dff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34866/0 (socket says 172.21.15.31:34866) 2024-09-11T00:50:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.083+0000 7ff197e00640 1 -- 172.21.15.31:0/1295228548 learned_addr learned my addr 172.21.15.31:0/1295228548 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.083+0000 7ff19d400640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1980cb620 0x7ff19814bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.083+0000 7ff197e00640 1 -- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1980c9b40 msgr2=0x7ff198152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.083+0000 7ff197e00640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1980c9b40 0x7ff198152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.083+0000 7ff197e00640 1 -- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1980cb620 msgr2=0x7ff19814bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:25.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.084+0000 7ff197e00640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1980cb620 0x7ff19814bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.084+0000 7ff197e00640 1 -- 172.21.15.31:0/1295228548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1980c5570 con 0x7ff1980cb240 2024-09-11T00:50:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.084+0000 7ff19d400640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1980cb620 0x7ff19814bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:50:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.085+0000 7ff197e00640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1980cb240 0x7ff1981dff50 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff18400c430 tx=0x7ff18400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.085+0000 7ff194c00640 1 -- 172.21.15.31:0/1295228548 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff184017070 con 0x7ff1980cb240 2024-09-11T00:50:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.085+0000 7ff19f400640 1 -- 172.21.15.31:0/1295228548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1980c5990 con 0x7ff1980cb240 2024-09-11T00:50:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.085+0000 7ff194c00640 1 -- 172.21.15.31:0/1295228548 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1840053c0 con 0x7ff1980cb240 2024-09-11T00:50:25.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.085+0000 7ff194c00640 1 -- 172.21.15.31:0/1295228548 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff184003850 con 0x7ff1980cb240 2024-09-11T00:50:25.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.090+0000 7ff19f400640 1 -- 172.21.15.31:0/1295228548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1980c5c30 con 0x7ff1980cb240 2024-09-11T00:50:25.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.090+0000 7ff194c00640 1 -- 172.21.15.31:0/1295228548 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff1840039f0 con 0x7ff1980cb240 2024-09-11T00:50:25.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.092+0000 7ff194c00640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff178077fa0 0x7ff17807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:25.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.092+0000 7ff194c00640 1 -- 172.21.15.31:0/1295228548 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7ff18409d170 con 0x7ff1980cb240 2024-09-11T00:50:25.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.092+0000 7ff19ca00640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff178077fa0 0x7ff17807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:25.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.093+0000 7ff19f400640 1 -- 172.21.15.31:0/1295228548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff19806ef30 con 0x7ff1980cb240 2024-09-11T00:50:25.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.093+0000 7ff19ca00640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff178077fa0 0x7ff17807a460 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff18c003cc0 tx=0x7ff18c0079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:25.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.102+0000 7ff194c00640 1 -- 172.21.15.31:0/1295228548 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff184061d30 con 0x7ff1980cb240 2024-09-11T00:50:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.334+0000 7f9eb7800640 1 -- 172.21.15.31:0/109957270 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9eb00c2ff0 msgr2=0x7f9eb015a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.334+0000 7f9eb7800640 1 --2- 172.21.15.31:0/109957270 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9eb00c2ff0 0x7f9eb015a8e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800a940 tx=0x7f9e9802b8f0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.334+0000 7f9eb7800640 1 -- 172.21.15.31:0/109957270 shutdown_connections 2024-09-11T00:50:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.334+0000 7f9eb7800640 1 --2- 172.21.15.31:0/109957270 >> v2:172.21.15.31:3300/0 conn(0x7f9eb00c3d10 0x7f9eb015ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.334+0000 7f9eb7800640 1 --2- 172.21.15.31:0/109957270 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9eb00c2ff0 0x7f9eb015a8e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.334+0000 7f9eb7800640 1 -- 172.21.15.31:0/109957270 >> 172.21.15.31:0/109957270 conn(0x7f9eb00a9750 msgr2=0x7f9eb00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.335+0000 7f9eb7800640 1 -- 172.21.15.31:0/109957270 shutdown_connections 2024-09-11T00:50:25.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.335+0000 7f9eb7800640 1 -- 172.21.15.31:0/109957270 wait complete. 2024-09-11T00:50:25.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.337+0000 7f9eb7800640 1 Processor -- start 2024-09-11T00:50:25.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.338+0000 7f9eb7800640 1 -- start start 2024-09-11T00:50:25.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.339+0000 7f9eb7800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eb0070570 0x7f9eb00c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:25.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.339+0000 7f9eb7800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eb00c2ff0 0x7f9eb0147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:25.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.339+0000 7f9eb7800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9eb00c3d10 0x7f9eb0148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:25.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.340+0000 7f9eafe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eb00c2ff0 0x7f9eb0147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:25.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.340+0000 7f9eafe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eb00c2ff0 0x7f9eb0147ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34880/0 (socket says 172.21.15.31:34880) 2024-09-11T00:50:25.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.340+0000 7f9eafe00640 1 -- 172.21.15.31:0/4063907364 learned_addr learned my addr 172.21.15.31:0/4063907364 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:25.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.340+0000 7f9eb5800640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9eb00c3d10 0x7f9eb0148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:25.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.340+0000 7f9eb4e00640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eb0070570 0x7f9eb00c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:25.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.340+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb014fb20 con 0x7f9eb00c2ff0 2024-09-11T00:50:25.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.340+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb006ef90 con 0x7f9eb0070570 2024-09-11T00:50:25.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.340+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9eb006eb90 con 0x7f9eb00c3d10 2024-09-11T00:50:25.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.342+0000 7f9eb4e00640 1 -- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9eb00c3d10 msgr2=0x7f9eb0148430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:25.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.342+0000 7f9eb4e00640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9eb00c3d10 0x7f9eb0148430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.342+0000 7f9eb4e00640 1 -- 172.21.15.31:0/4063907364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eb00c2ff0 msgr2=0x7f9eb0147ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:25.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.342+0000 7f9eb4e00640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eb00c2ff0 0x7f9eb0147ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.342+0000 7f9eb4e00640 1 -- 172.21.15.31:0/4063907364 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb00ab7c0 con 0x7f9eb0070570 2024-09-11T00:50:25.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.342+0000 7f9eafe00640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eb00c2ff0 0x7f9eb0147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:50:25.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.343+0000 7f9eb4e00640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eb0070570 0x7f9eb00c1040 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0002230 tx=0x7f9ea000cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:25.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.343+0000 7f9eacc00640 1 -- 172.21.15.31:0/4063907364 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0003b70 con 0x7f9eb0070570 2024-09-11T00:50:25.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.343+0000 7f9eacc00640 1 -- 172.21.15.31:0/4063907364 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ea0003d10 con 0x7f9eb0070570 2024-09-11T00:50:25.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.343+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb0072370 con 0x7f9eb0070570 2024-09-11T00:50:25.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.343+0000 7f9eacc00640 1 -- 172.21.15.31:0/4063907364 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea00052d0 con 0x7f9eb0070570 2024-09-11T00:50:25.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.343+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb00c1580 con 0x7f9eb0070570 2024-09-11T00:50:25.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.344+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb00c7690 con 0x7f9eb0070570 2024-09-11T00:50:25.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.347+0000 7f9eacc00640 1 -- 172.21.15.31:0/4063907364 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9ea0005680 con 0x7f9eb0070570 2024-09-11T00:50:25.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.349+0000 7f9eacc00640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9e8c077f70 0x7f9e8c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:25.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.350+0000 7f9eacc00640 1 -- 172.21.15.31:0/4063907364 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7f9ea009aad0 con 0x7f9eb0070570 2024-09-11T00:50:25.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.351+0000 7f9eafe00640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9e8c077f70 0x7f9e8c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:25.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.351+0000 7f9eafe00640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9e8c077f70 0x7f9e8c07a430 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9e98034040 tx=0x7f9e98004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:25.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.354+0000 7f9eacc00640 1 -- 172.21.15.31:0/4063907364 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9ea0062880 con 0x7f9eb0070570 2024-09-11T00:50:25.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.544+0000 7ff19f400640 1 -- 172.21.15.31:0/1295228548 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff19806ec50 con 0x7ff1980cb240 2024-09-11T00:50:25.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.546+0000 7ff194c00640 1 -- 172.21.15.31:0/1295228548 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) ==== 95+0+5714 (secure 0 0 0) 0x7ff184061630 con 0x7ff1980cb240 2024-09-11T00:50:25.549 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:25.550 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":149,"btime":"2024-09-11T00:50:25:112739+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:24.143474+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:50:25.550 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 149 2024-09-11T00:50:25.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 -- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff178077fa0 msgr2=0x7ff17807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:25.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff178077fa0 0x7ff17807a460 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff18c003cc0 tx=0x7ff18c0079b0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 -- 172.21.15.31:0/1295228548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1980cb240 msgr2=0x7ff1981dff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:25.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1980cb240 0x7ff1981dff50 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff18400c430 tx=0x7ff18400c8f0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 -- 172.21.15.31:0/1295228548 shutdown_connections 2024-09-11T00:50:25.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff178077fa0 0x7ff17807a460 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1980cb620 0x7ff19814bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1980cb240 0x7ff1981dff50 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 --2- 172.21.15.31:0/1295228548 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff1980c9b40 0x7ff198152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 -- 172.21.15.31:0/1295228548 >> 172.21.15.31:0/1295228548 conn(0x7ff198068f10 msgr2=0x7ff19806df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:25.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 -- 172.21.15.31:0/1295228548 shutdown_connections 2024-09-11T00:50:25.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.549+0000 7ff19f400640 1 -- 172.21.15.31:0/1295228548 wait complete. 2024-09-11T00:50:25.571 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-11T00:50:25.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.809+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9eb00aa5e0 con 0x7f9eb0070570 2024-09-11T00:50:25.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.810+0000 7f9eacc00640 1 -- 172.21.15.31:0/4063907364 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) ==== 95+0+5714 (secure 0 0 0) 0x7f9ea0062180 con 0x7f9eb0070570 2024-09-11T00:50:25.813 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:25.813 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":149,"btime":"2024-09-11T00:50:25:112739+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:24.143474+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:50:25.813 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 149 2024-09-11T00:50:25.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.814+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9e8c077f70 msgr2=0x7f9e8c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.814+0000 7f9eb7800640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9e8c077f70 0x7f9e8c07a430 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9e98034040 tx=0x7f9e98004730 comp rx=0 tx=0).stop 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.815+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eb0070570 msgr2=0x7f9eb00c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.815+0000 7f9eb7800640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eb0070570 0x7f9eb00c1040 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0002230 tx=0x7f9ea000cb90 comp rx=0 tx=0).stop 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.815+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 shutdown_connections 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.815+0000 7f9eb7800640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9e8c077f70 0x7f9e8c07a430 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.815+0000 7f9eb7800640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9eb00c3d10 0x7f9eb0148430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.815+0000 7f9eb7800640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9eb00c2ff0 0x7f9eb0147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.815+0000 7f9eb7800640 1 --2- 172.21.15.31:0/4063907364 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eb0070570 0x7f9eb00c1040 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.815+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 >> 172.21.15.31:0/4063907364 conn(0x7f9eb00a9750 msgr2=0x7f9eb00c5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.815+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 shutdown_connections 2024-09-11T00:50:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:25.815+0000 7f9eb7800640 1 -- 172.21.15.31:0/4063907364 wait complete. 2024-09-11T00:50:25.837 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-11T00:50:25.838 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:50:25.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.068+0000 7fd8a4a00640 1 -- 172.21.15.31:0/2431426752 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a00c34e0 msgr2=0x7fd8a01511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.068+0000 7fd8a4a00640 1 --2- 172.21.15.31:0/2431426752 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a00c34e0 0x7fd8a01511c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd898009be0 tx=0x7fd89802a850 comp rx=0 tx=0).stop 2024-09-11T00:50:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.068+0000 7fd8a4a00640 1 -- 172.21.15.31:0/2431426752 shutdown_connections 2024-09-11T00:50:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.068+0000 7fd8a4a00640 1 --2- 172.21.15.31:0/2431426752 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a00c34e0 0x7fd8a01511c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.068+0000 7fd8a4a00640 1 --2- 172.21.15.31:0/2431426752 >> v2:172.21.15.204:3300/0 conn(0x7fd8a0154b60 0x7fd8a00c2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.068+0000 7fd8a4a00640 1 -- 172.21.15.31:0/2431426752 >> 172.21.15.31:0/2431426752 conn(0x7fd8a00a9750 msgr2=0x7fd8a00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:26.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.068+0000 7fd8a4a00640 1 -- 172.21.15.31:0/2431426752 shutdown_connections 2024-09-11T00:50:26.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.068+0000 7fd8a4a00640 1 -- 172.21.15.31:0/2431426752 wait complete. 2024-09-11T00:50:26.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.072+0000 7fd8a4a00640 1 Processor -- start 2024-09-11T00:50:26.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.072+0000 7fd8a4a00640 1 -- start start 2024-09-11T00:50:26.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.073+0000 7fd8a4a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8a0072480 0x7fd8a00c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:26.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.073+0000 7fd8a4a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a00c34e0 0x7fd8a014a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:26.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.073+0000 7fd8a4a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a0154b60 0x7fd8a014adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:26.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.073+0000 7fd8a4a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a00a7140 con 0x7fd8a0072480 2024-09-11T00:50:26.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.073+0000 7fd8a4a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a00c23a0 con 0x7fd8a0154b60 2024-09-11T00:50:26.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.073+0000 7fd8a4a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a0072860 con 0x7fd8a00c34e0 2024-09-11T00:50:26.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.073+0000 7fd89de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8a0072480 0x7fd8a00c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a0154b60 0x7fd8a014adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89d400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a00c34e0 0x7fd8a014a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89e800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a0154b60 0x7fd8a014adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:43970/0 (socket says 172.21.15.31:43970) 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89e800640 1 -- 172.21.15.31:0/57820156 learned_addr learned my addr 172.21.15.31:0/57820156 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89e800640 1 -- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a00c34e0 msgr2=0x7fd8a014a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89e800640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a00c34e0 0x7fd8a014a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89e800640 1 -- 172.21.15.31:0/57820156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8a0072480 msgr2=0x7fd8a00c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89e800640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8a0072480 0x7fd8a00c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89e800640 1 -- 172.21.15.31:0/57820156 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a01dd520 con 0x7fd8a0154b60 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89d400640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a00c34e0 0x7fd8a014a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.074+0000 7fd89de00640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8a0072480 0x7fd8a00c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:50:26.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.075+0000 7fd89e800640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a0154b60 0x7fd8a014adf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd898006010 tx=0x7fd8980315d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:26.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.075+0000 7fd896000640 1 -- 172.21.15.31:0/57820156 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898041020 con 0x7fd8a0154b60 2024-09-11T00:50:26.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.075+0000 7fd896000640 1 -- 172.21.15.31:0/57820156 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd898031ac0 con 0x7fd8a0154b60 2024-09-11T00:50:26.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.075+0000 7fd896000640 1 -- 172.21.15.31:0/57820156 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898037040 con 0x7fd8a0154b60 2024-09-11T00:50:26.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.076+0000 7fd8a4a00640 1 -- 172.21.15.31:0/57820156 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a00ad970 con 0x7fd8a0154b60 2024-09-11T00:50:26.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.077+0000 7fd8a4a00640 1 -- 172.21.15.31:0/57820156 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a01ed210 con 0x7fd8a0154b60 2024-09-11T00:50:26.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.079+0000 7fd8a4a00640 1 -- 172.21.15.31:0/57820156 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a00a6f20 con 0x7fd8a0154b60 2024-09-11T00:50:26.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.081+0000 7fd896000640 1 -- 172.21.15.31:0/57820156 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd89803c460 con 0x7fd8a0154b60 2024-09-11T00:50:26.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.082+0000 7fd896000640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd88007c3d0 0x7fd88007e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:26.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.083+0000 7fd896000640 1 -- 172.21.15.31:0/57820156 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7fd898032640 con 0x7fd8a0154b60 2024-09-11T00:50:26.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.083+0000 7fd89de00640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd88007c3d0 0x7fd88007e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:26.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.084+0000 7fd89de00640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd88007c3d0 0x7fd88007e890 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd890007a50 tx=0x7fd890009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.089+0000 7fd896000640 1 -- 172.21.15.31:0/57820156 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd89804b030 con 0x7fd8a0154b60 2024-09-11T00:50:26.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.522+0000 7fd8a4a00640 1 -- 172.21.15.31:0/57820156 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8a00c2fa0 con 0x7fd8a0154b60 2024-09-11T00:50:26.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.523+0000 7fd896000640 1 -- 172.21.15.31:0/57820156 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+5711 (secure 0 0 0) 0x7fd898083960 con 0x7fd8a0154b60 2024-09-11T00:50:26.528 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:26.528 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":150,"btime":"2024-09-11T00:50:26:123307+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:25.129428+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:50:26.528 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 150 2024-09-11T00:50:26.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.526+0000 7fd8a4a00640 1 -- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd88007c3d0 msgr2=0x7fd88007e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:26.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.526+0000 7fd8a4a00640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd88007c3d0 0x7fd88007e890 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd890007a50 tx=0x7fd890009040 comp rx=0 tx=0).stop 2024-09-11T00:50:26.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.526+0000 7fd8a4a00640 1 -- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a0154b60 msgr2=0x7fd8a014adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.526+0000 7fd8a4a00640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a0154b60 0x7fd8a014adf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd898006010 tx=0x7fd8980315d0 comp rx=0 tx=0).stop 2024-09-11T00:50:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.527+0000 7fd8a4a00640 1 -- 172.21.15.31:0/57820156 shutdown_connections 2024-09-11T00:50:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.527+0000 7fd8a4a00640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd88007c3d0 0x7fd88007e890 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.527+0000 7fd8a4a00640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd8a0154b60 0x7fd8a014adf0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:26.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.527+0000 7fd8a4a00640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd8a00c34e0 0x7fd8a014a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:26.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.527+0000 7fd8a4a00640 1 --2- 172.21.15.31:0/57820156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8a0072480 0x7fd8a00c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:26.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.527+0000 7fd8a4a00640 1 -- 172.21.15.31:0/57820156 >> 172.21.15.31:0/57820156 conn(0x7fd8a00a9750 msgr2=0x7fd8a014d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:26.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.527+0000 7fd8a4a00640 1 -- 172.21.15.31:0/57820156 shutdown_connections 2024-09-11T00:50:26.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:26.528+0000 7fd8a4a00640 1 -- 172.21.15.31:0/57820156 wait complete. 2024-09-11T00:50:26.554 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:50:26:123307+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 150, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 146, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7143": { "addr": "172.21.15.204:6835/1981755707", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1981755707, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 1981755707, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7143, "incarnation": 143, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:50:17.486435+0000", "qdb_cluster": [ 7143 ], "qdb_leader": 7143, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7143 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 150, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6350": { "addr": "172.21.15.204:6837/2630216714", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 2630216714, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 2630216714, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6350, "incarnation": 148, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:50:25.129428+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6350 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:50:27.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:27.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.857+0000 7f9b69000640 1 -- 172.21.15.31:0/3367671361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b64072480 msgr2=0x7f9b641511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:27.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.857+0000 7f9b69000640 1 --2- 172.21.15.31:0/3367671361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b64072480 0x7f9b641511c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9b54005fd0 tx=0x7f9b5402a660 comp rx=0 tx=0).stop 2024-09-11T00:50:27.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.858+0000 7f9b69000640 1 -- 172.21.15.31:0/3367671361 shutdown_connections 2024-09-11T00:50:27.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.858+0000 7f9b69000640 1 --2- 172.21.15.31:0/3367671361 >> v2:172.21.15.204:3300/0 conn(0x7f9b640c2410 0x7f9b64151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:27.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.858+0000 7f9b69000640 1 --2- 172.21.15.31:0/3367671361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b64072480 0x7f9b641511c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:27.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.858+0000 7f9b69000640 1 -- 172.21.15.31:0/3367671361 >> 172.21.15.31:0/3367671361 conn(0x7f9b640a9750 msgr2=0x7f9b640a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:27.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.858+0000 7f9b69000640 1 -- 172.21.15.31:0/3367671361 shutdown_connections 2024-09-11T00:50:27.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.859+0000 7f9b69000640 1 -- 172.21.15.31:0/3367671361 wait complete. 2024-09-11T00:50:27.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.863+0000 7f9b69000640 1 Processor -- start 2024-09-11T00:50:27.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.864+0000 7f9b69000640 1 -- start start 2024-09-11T00:50:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.866+0000 7f9b69000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b64072480 0x7f9b64152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.866+0000 7f9b69000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b64074080 0x7f9b6414a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.866+0000 7f9b69000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b640c2410 0x7f9b6414f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b69000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9b640729e0 con 0x7f9b640c2410 2024-09-11T00:50:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b69000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9b64072860 con 0x7f9b64074080 2024-09-11T00:50:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b69000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9b64153f20 con 0x7f9b64072480 2024-09-11T00:50:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b5be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b64074080 0x7f9b6414a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b5be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b64074080 0x7f9b6414a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:43984/0 (socket says 172.21.15.31:43984) 2024-09-11T00:50:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b5be00640 1 -- 172.21.15.31:0/1092156697 learned_addr learned my addr 172.21.15.31:0/1092156697 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b62800640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b640c2410 0x7f9b6414f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b5be00640 1 -- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b64072480 msgr2=0x7f9b64152100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:50:27.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b5be00640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b64072480 0x7f9b64152100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b5be00640 1 -- 172.21.15.31:0/1092156697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b640c2410 msgr2=0x7f9b6414f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.867+0000 7f9b5be00640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b640c2410 0x7f9b6414f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.868+0000 7f9b5be00640 1 -- 172.21.15.31:0/1092156697 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b64150900 con 0x7f9b64074080 2024-09-11T00:50:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.868+0000 7f9b5be00640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b64074080 0x7f9b6414a6b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00d740 tx=0x7f9b4c00dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.869+0000 7f9b5a000640 1 -- 172.21.15.31:0/1092156697 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c017070 con 0x7f9b64074080 2024-09-11T00:50:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.869+0000 7f9b5a000640 1 -- 172.21.15.31:0/1092156697 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b4c00de50 con 0x7f9b64074080 2024-09-11T00:50:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.869+0000 7f9b5a000640 1 -- 172.21.15.31:0/1092156697 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c004020 con 0x7f9b64074080 2024-09-11T00:50:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.870+0000 7f9b69000640 1 -- 172.21.15.31:0/1092156697 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6414f060 con 0x7f9b64074080 2024-09-11T00:50:27.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.870+0000 7f9b69000640 1 -- 172.21.15.31:0/1092156697 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6414abf0 con 0x7f9b64074080 2024-09-11T00:50:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.872+0000 7f9b69000640 1 -- 172.21.15.31:0/1092156697 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b641511c0 con 0x7f9b64074080 2024-09-11T00:50:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.873+0000 7f9b5a000640 1 -- 172.21.15.31:0/1092156697 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9b4c0041c0 con 0x7f9b64074080 2024-09-11T00:50:27.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.874+0000 7f9b5a000640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b40077f70 0x7f9b4007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:27.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.874+0000 7f9b5a000640 1 -- 172.21.15.31:0/1092156697 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7f9b4c09ae60 con 0x7f9b64074080 2024-09-11T00:50:27.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.875+0000 7f9b61e00640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b40077f70 0x7f9b4007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:27.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.876+0000 7f9b61e00640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b40077f70 0x7f9b4007a430 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9b54033000 tx=0x7f9b54004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:27.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:27.880+0000 7f9b5a000640 1 -- 172.21.15.31:0/1092156697 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9b4c020020 con 0x7f9b64074080 2024-09-11T00:50:28.124 INFO:tasks.ceph.mds.b.smithi204.stderr:2024-09-11T00:50:28.122+0000 7ff0adc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:50:28.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.246+0000 7f9b69000640 1 -- 172.21.15.31:0/1092156697 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b6414cd90 con 0x7f9b64074080 2024-09-11T00:50:28.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.247+0000 7f9b5a000640 1 -- 172.21.15.31:0/1092156697 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5718 (secure 0 0 0) 0x7f9b4c062c10 con 0x7f9b64074080 2024-09-11T00:50:28.251 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:28.251 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":151,"btime":"2024-09-11T00:50:27:135970+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:50:28.252 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 151 2024-09-11T00:50:28.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.250+0000 7f9b69000640 1 -- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b40077f70 msgr2=0x7f9b4007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:28.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.250+0000 7f9b69000640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b40077f70 0x7f9b4007a430 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9b54033000 tx=0x7f9b54004730 comp rx=0 tx=0).stop 2024-09-11T00:50:28.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.250+0000 7f9b69000640 1 -- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b64074080 msgr2=0x7f9b6414a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:28.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.250+0000 7f9b69000640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b64074080 0x7f9b6414a6b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00d740 tx=0x7f9b4c00dc00 comp rx=0 tx=0).stop 2024-09-11T00:50:28.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.251+0000 7f9b69000640 1 -- 172.21.15.31:0/1092156697 shutdown_connections 2024-09-11T00:50:28.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.251+0000 7f9b69000640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9b40077f70 0x7f9b4007a430 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:28.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.251+0000 7f9b69000640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b640c2410 0x7f9b6414f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:28.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.251+0000 7f9b69000640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b64074080 0x7f9b6414a6b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:28.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.251+0000 7f9b69000640 1 --2- 172.21.15.31:0/1092156697 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9b64072480 0x7f9b64152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:28.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.251+0000 7f9b69000640 1 -- 172.21.15.31:0/1092156697 >> 172.21.15.31:0/1092156697 conn(0x7f9b640a9750 msgr2=0x7f9b6414d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:28.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.251+0000 7f9b69000640 1 -- 172.21.15.31:0/1092156697 shutdown_connections 2024-09-11T00:50:28.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.252+0000 7f9b69000640 1 -- 172.21.15.31:0/1092156697 wait complete. 2024-09-11T00:50:28.281 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:active state 2024-09-11T00:50:28.281 INFO:tasks.mds_thrash.fs.[a]:waiting for 112 secs before thrashing 2024-09-11T00:50:28.555 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:28.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.808+0000 7f06aba00640 1 -- 172.21.15.31:0/3042530852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06a40ad7e0 msgr2=0x7f06a40ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:28.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.808+0000 7f06aba00640 1 --2- 172.21.15.31:0/3042530852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06a40ad7e0 0x7f06a40ca6f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f06a408ddf0 tx=0x7f069802a560 comp rx=0 tx=0).stop 2024-09-11T00:50:28.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.808+0000 7f06aba00640 1 -- 172.21.15.31:0/3042530852 shutdown_connections 2024-09-11T00:50:28.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.808+0000 7f06aba00640 1 --2- 172.21.15.31:0/3042530852 >> v2:172.21.15.31:3300/0 conn(0x7f06a40c9f00 0x7f06a414ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:28.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.808+0000 7f06aba00640 1 --2- 172.21.15.31:0/3042530852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06a40ad7e0 0x7f06a40ca6f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:28.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.808+0000 7f06aba00640 1 -- 172.21.15.31:0/3042530852 >> 172.21.15.31:0/3042530852 conn(0x7f06a4068f10 msgr2=0x7f06a40cbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:28.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.809+0000 7f06aba00640 1 -- 172.21.15.31:0/3042530852 shutdown_connections 2024-09-11T00:50:28.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.809+0000 7f06aba00640 1 -- 172.21.15.31:0/3042530852 wait complete. 2024-09-11T00:50:28.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.811+0000 7f06aba00640 1 Processor -- start 2024-09-11T00:50:28.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.813+0000 7f06aba00640 1 -- start start 2024-09-11T00:50:28.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.816+0000 7f06aba00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06a40ad7e0 0x7f06a40c1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:28.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.816+0000 7f06aba00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06a40c99f0 0x7f06a41576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:28.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.816+0000 7f06aba00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06a40c9f00 0x7f06a4151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:28.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.816+0000 7f06aba00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f06a40adbc0 con 0x7f06a40c9f00 2024-09-11T00:50:28.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.816+0000 7f06aba00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f06a4156e30 con 0x7f06a40c99f0 2024-09-11T00:50:28.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.816+0000 7f06aba00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f06a40add40 con 0x7f06a40ad7e0 2024-09-11T00:50:28.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.816+0000 7f06a9000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06a40ad7e0 0x7f06a40c1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.816+0000 7f06a9000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06a40ad7e0 0x7f06a40c1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:52592/0 (socket says 172.21.15.31:52592) 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.816+0000 7f06a9000640 1 -- 172.21.15.31:0/2582517264 learned_addr learned my addr 172.21.15.31:0/2582517264 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.817+0000 7f06a9a00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06a40c9f00 0x7f06a4151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.817+0000 7f06a3e00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06a40c99f0 0x7f06a41576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.817+0000 7f06a9000640 1 -- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06a40c99f0 msgr2=0x7f06a41576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.817+0000 7f06a9000640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06a40c99f0 0x7f06a41576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.817+0000 7f06a9000640 1 -- 172.21.15.31:0/2582517264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06a40c9f00 msgr2=0x7f06a4151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.817+0000 7f06a9000640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06a40c9f00 0x7f06a4151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.817+0000 7f06a9000640 1 -- 172.21.15.31:0/2582517264 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a4152840 con 0x7f06a40ad7e0 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.817+0000 7f06a9a00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06a40c9f00 0x7f06a4151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:50:28.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.818+0000 7f06a9000640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06a40ad7e0 0x7f06a40c1be0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0698009850 tx=0x7f06980039e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:28.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.818+0000 7f06a0c00640 1 -- 172.21.15.31:0/2582517264 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069803f070 con 0x7f06a40ad7e0 2024-09-11T00:50:28.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.818+0000 7f06a0c00640 1 -- 172.21.15.31:0/2582517264 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0698004890 con 0x7f06a40ad7e0 2024-09-11T00:50:28.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.818+0000 7f06a0c00640 1 -- 172.21.15.31:0/2582517264 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0698035030 con 0x7f06a40ad7e0 2024-09-11T00:50:28.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.818+0000 7f06aba00640 1 -- 172.21.15.31:0/2582517264 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a4152b20 con 0x7f06a40ad7e0 2024-09-11T00:50:28.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.819+0000 7f06aba00640 1 -- 172.21.15.31:0/2582517264 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a40c2120 con 0x7f06a40ad7e0 2024-09-11T00:50:28.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.822+0000 7f06a0c00640 1 -- 172.21.15.31:0/2582517264 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0698004ca0 con 0x7f06a40ad7e0 2024-09-11T00:50:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.822+0000 7f06aba00640 1 -- 172.21.15.31:0/2582517264 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a415a110 con 0x7f06a40ad7e0 2024-09-11T00:50:28.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.823+0000 7f06a0c00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0684078070 0x7f068407a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.823+0000 7f06a0c00640 1 -- 172.21.15.31:0/2582517264 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7f06980bbc80 con 0x7f06a40ad7e0 2024-09-11T00:50:28.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.824+0000 7f06a3e00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0684078070 0x7f068407a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:28.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.825+0000 7f06a3e00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0684078070 0x7f068407a530 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f06a4069d80 tx=0x7f0690009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:28.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:28.829+0000 7f06a0c00640 1 -- 172.21.15.31:0/2582517264 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0698083ab0 con 0x7f06a40ad7e0 2024-09-11T00:50:29.278 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:29.278 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":151,"btime":"2024-09-11T00:50:27:135970+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:50:29.279 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 151 2024-09-11T00:50:29.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.275+0000 7f06aba00640 1 -- 172.21.15.31:0/2582517264 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06a40ca6f0 con 0x7f06a40ad7e0 2024-09-11T00:50:29.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.276+0000 7f06a0c00640 1 -- 172.21.15.31:0/2582517264 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5718 (secure 0 0 0) 0x7f06980833b0 con 0x7f06a40ad7e0 2024-09-11T00:50:29.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.278+0000 7f06aba00640 1 -- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0684078070 msgr2=0x7f068407a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:29.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.278+0000 7f06aba00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0684078070 0x7f068407a530 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f06a4069d80 tx=0x7f0690009040 comp rx=0 tx=0).stop 2024-09-11T00:50:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.278+0000 7f06aba00640 1 -- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06a40ad7e0 msgr2=0x7f06a40c1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.278+0000 7f06aba00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06a40ad7e0 0x7f06a40c1be0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0698009850 tx=0x7f06980039e0 comp rx=0 tx=0).stop 2024-09-11T00:50:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.279+0000 7f06aba00640 1 -- 172.21.15.31:0/2582517264 shutdown_connections 2024-09-11T00:50:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.279+0000 7f06aba00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0684078070 0x7f068407a530 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.279+0000 7f06aba00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06a40c9f00 0x7f06a4151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.279+0000 7f06aba00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06a40c99f0 0x7f06a41576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.279+0000 7f06aba00640 1 --2- 172.21.15.31:0/2582517264 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f06a40ad7e0 0x7f06a40c1be0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.279+0000 7f06aba00640 1 -- 172.21.15.31:0/2582517264 >> 172.21.15.31:0/2582517264 conn(0x7f06a4068f10 msgr2=0x7f06a40c5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.279+0000 7f06aba00640 1 -- 172.21.15.31:0/2582517264 shutdown_connections 2024-09-11T00:50:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:29.279+0000 7f06aba00640 1 -- 172.21.15.31:0/2582517264 wait complete. 2024-09-11T00:50:29.306 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=7065 2024-09-11T00:50:29.306 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.c 2024-09-11T00:50:31.307 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-11T00:50:31.307 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-11T00:50:31.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-11T00:50:31.312 INFO:tasks.ceph.mds.c:Started 2024-09-11T00:50:31.313 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:31.541 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-11T00:50:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.588+0000 7fd66c600640 1 -- 172.21.15.31:0/1234175910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6640c84b0 msgr2=0x7fd6640c2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.588+0000 7fd66c600640 1 --2- 172.21.15.31:0/1234175910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6640c84b0 0x7fd6640c2f20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd6640c3b50 tx=0x7fd65002a4f0 comp rx=0 tx=0).stop 2024-09-11T00:50:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.588+0000 7fd66c600640 1 -- 172.21.15.31:0/1234175910 shutdown_connections 2024-09-11T00:50:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.588+0000 7fd66c600640 1 --2- 172.21.15.31:0/1234175910 >> v2:172.21.15.31:3300/0 conn(0x7fd6640c3460 0x7fd66414f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.588+0000 7fd66c600640 1 --2- 172.21.15.31:0/1234175910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6640c84b0 0x7fd6640c2f20 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.588+0000 7fd66c600640 1 -- 172.21.15.31:0/1234175910 >> 172.21.15.31:0/1234175910 conn(0x7fd6640ab8a0 msgr2=0x7fd6640abcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.588+0000 7fd66c600640 1 -- 172.21.15.31:0/1234175910 shutdown_connections 2024-09-11T00:50:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.588+0000 7fd66c600640 1 -- 172.21.15.31:0/1234175910 wait complete. 2024-09-11T00:50:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.591+0000 7fd66c600640 1 Processor -- start 2024-09-11T00:50:31.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.591+0000 7fd66c600640 1 -- start start 2024-09-11T00:50:31.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.593+0000 7fd66c600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6640a90e0 0x7fd6641dfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:31.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.593+0000 7fd66c600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6640c3460 0x7fd6641f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:31.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.593+0000 7fd66c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6640c84b0 0x7fd664071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:31.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.593+0000 7fd66c600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd6640c7cb0 con 0x7fd6640c84b0 2024-09-11T00:50:31.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.593+0000 7fd66c600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd664156340 con 0x7fd6640a90e0 2024-09-11T00:50:31.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.593+0000 7fd66c600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd6640c7f90 con 0x7fd6640c3460 2024-09-11T00:50:31.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.594+0000 7fd669c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6640a90e0 0x7fd6641dfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:31.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.594+0000 7fd669200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6640c3460 0x7fd6641f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:31.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.594+0000 7fd66a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6640c84b0 0x7fd664071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:31.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.594+0000 7fd66a600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6640c84b0 0x7fd664071990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42408/0 (socket says 172.21.15.31:42408) 2024-09-11T00:50:31.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.594+0000 7fd66a600640 1 -- 172.21.15.31:0/3261131640 learned_addr learned my addr 172.21.15.31:0/3261131640 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:31.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.595+0000 7fd669c00640 1 -- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6640c3460 msgr2=0x7fd6641f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:31.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.595+0000 7fd669c00640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6640c3460 0x7fd6641f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:31.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.595+0000 7fd669c00640 1 -- 172.21.15.31:0/3261131640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6640c84b0 msgr2=0x7fd664071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:31.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.595+0000 7fd669c00640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6640c84b0 0x7fd664071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:31.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.595+0000 7fd669c00640 1 -- 172.21.15.31:0/3261131640 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6641e1f80 con 0x7fd6640a90e0 2024-09-11T00:50:31.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.596+0000 7fd669c00640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6640a90e0 0x7fd6641dfcb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd658001eb0 tx=0x7fd65800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:31.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.597+0000 7fd661600640 1 -- 172.21.15.31:0/3261131640 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658017070 con 0x7fd6640a90e0 2024-09-11T00:50:31.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.598+0000 7fd661600640 1 -- 172.21.15.31:0/3261131640 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6580035f0 con 0x7fd6640a90e0 2024-09-11T00:50:31.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.598+0000 7fd661600640 1 -- 172.21.15.31:0/3261131640 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658003a40 con 0x7fd6640a90e0 2024-09-11T00:50:31.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.598+0000 7fd66c600640 1 -- 172.21.15.31:0/3261131640 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd664072bf0 con 0x7fd6640a90e0 2024-09-11T00:50:31.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.598+0000 7fd66c600640 1 -- 172.21.15.31:0/3261131640 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6641e2220 con 0x7fd6640a90e0 2024-09-11T00:50:31.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.600+0000 7fd66c600640 1 -- 172.21.15.31:0/3261131640 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6640c9720 con 0x7fd6640a90e0 2024-09-11T00:50:31.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.605+0000 7fd661600640 1 -- 172.21.15.31:0/3261131640 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd658003be0 con 0x7fd6640a90e0 2024-09-11T00:50:31.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.609+0000 7fd661600640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd644078040 0x7fd64407a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:31.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.609+0000 7fd661600640 1 -- 172.21.15.31:0/3261131640 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7fd658099f20 con 0x7fd6640a90e0 2024-09-11T00:50:31.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.609+0000 7fd669200640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd644078040 0x7fd64407a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:31.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.610+0000 7fd669200640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd644078040 0x7fd64407a500 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd650034000 tx=0x7fd650004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:31.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:31.614+0000 7fd661600640 1 -- 172.21.15.31:0/3261131640 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd6641560c0 con 0x7fd6640a90e0 2024-09-11T00:50:32.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.020+0000 7fd66c600640 1 -- 172.21.15.31:0/3261131640 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6641df8d0 con 0x7fd6640a90e0 2024-09-11T00:50:32.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.022+0000 7fd661600640 1 -- 172.21.15.31:0/3261131640 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5718 (secure 0 0 0) 0x7fd658061700 con 0x7fd6640a90e0 2024-09-11T00:50:32.025 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:32.025 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":151,"btime":"2024-09-11T00:50:27:135970+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:50:32.025 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 151 2024-09-11T00:50:32.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 -- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd644078040 msgr2=0x7fd64407a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:32.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd644078040 0x7fd64407a500 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd650034000 tx=0x7fd650004730 comp rx=0 tx=0).stop 2024-09-11T00:50:32.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 -- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6640a90e0 msgr2=0x7fd6641dfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:32.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6640a90e0 0x7fd6641dfcb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd658001eb0 tx=0x7fd65800c820 comp rx=0 tx=0).stop 2024-09-11T00:50:32.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 -- 172.21.15.31:0/3261131640 shutdown_connections 2024-09-11T00:50:32.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd644078040 0x7fd64407a500 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:32.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6640c84b0 0x7fd664071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:32.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd6640c3460 0x7fd6641f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:32.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 --2- 172.21.15.31:0/3261131640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd6640a90e0 0x7fd6641dfcb0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:32.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 -- 172.21.15.31:0/3261131640 >> 172.21.15.31:0/3261131640 conn(0x7fd6640ab8a0 msgr2=0x7fd664155680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:32.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 -- 172.21.15.31:0/3261131640 shutdown_connections 2024-09-11T00:50:32.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:32.026+0000 7fd66c600640 1 -- 172.21.15.31:0/3261131640 wait complete. 2024-09-11T00:50:32.056 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-11T00:50:34.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:50:34.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.311+0000 7f1257e00640 1 -- 172.21.15.31:0/3201181325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12500ae080 msgr2=0x7f1250159830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:34.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.311+0000 7f1257e00640 1 --2- 172.21.15.31:0/3201181325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12500ae080 0x7f1250159830 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f125008dfb0 tx=0x7f124403b520 comp rx=0 tx=0).stop 2024-09-11T00:50:34.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.311+0000 7f1257e00640 1 -- 172.21.15.31:0/3201181325 shutdown_connections 2024-09-11T00:50:34.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.311+0000 7f1257e00640 1 --2- 172.21.15.31:0/3201181325 >> v2:172.21.15.31:3300/0 conn(0x7f125006cf40 0x7f12500cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:34.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.311+0000 7f1257e00640 1 --2- 172.21.15.31:0/3201181325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12500ae080 0x7f1250159830 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:34.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.311+0000 7f1257e00640 1 -- 172.21.15.31:0/3201181325 >> 172.21.15.31:0/3201181325 conn(0x7f12500c8cc0 msgr2=0x7f12500ad700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:34.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.311+0000 7f1257e00640 1 -- 172.21.15.31:0/3201181325 shutdown_connections 2024-09-11T00:50:34.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.312+0000 7f1257e00640 1 -- 172.21.15.31:0/3201181325 wait complete. 2024-09-11T00:50:34.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.314+0000 7f1257e00640 1 Processor -- start 2024-09-11T00:50:34.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.314+0000 7f1257e00640 1 -- start start 2024-09-11T00:50:34.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.315+0000 7f1257e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125006cf40 0x7f12500bfab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:34.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.315+0000 7f1257e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12500ae080 0x7f125014b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:34.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.315+0000 7f1257e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12500c7330 0x7f12500af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:34.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1255400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125006cf40 0x7f12500bfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:34.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1255400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125006cf40 0x7f12500bfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:52620/0 (socket says 172.21.15.31:52620) 2024-09-11T00:50:34.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1255400640 1 -- 172.21.15.31:0/2957891860 learned_addr learned my addr 172.21.15.31:0/2957891860 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f12500c79f0 con 0x7f12500ae080 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f12500bf3c0 con 0x7f12500c7330 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f12500c7b70 con 0x7f125006cf40 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1254a00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12500ae080 0x7f125014b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1255e00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12500c7330 0x7f12500af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1255400640 1 -- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12500c7330 msgr2=0x7f12500af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1255400640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12500c7330 0x7f12500af7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1255400640 1 -- 172.21.15.31:0/2957891860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12500ae080 msgr2=0x7f125014b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1255400640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12500ae080 0x7f125014b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1255400640 1 -- 172.21.15.31:0/2957891860 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125014b620 con 0x7f125006cf40 2024-09-11T00:50:34.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.316+0000 7f1255e00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12500c7330 0x7f12500af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:50:34.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.317+0000 7f1255400640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125006cf40 0x7f12500bfab0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f12440099c0 tx=0x7f12440039e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:34.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.317+0000 7f124d600640 1 -- 172.21.15.31:0/2957891860 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1244050070 con 0x7f125006cf40 2024-09-11T00:50:34.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.318+0000 7f124d600640 1 -- 172.21.15.31:0/2957891860 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1244004890 con 0x7f125006cf40 2024-09-11T00:50:34.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.318+0000 7f124d600640 1 -- 172.21.15.31:0/2957891860 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1244046030 con 0x7f125006cf40 2024-09-11T00:50:34.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.318+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125014ae40 con 0x7f125006cf40 2024-09-11T00:50:34.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.318+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12501ebed0 con 0x7f125006cf40 2024-09-11T00:50:34.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.320+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f125006e2e0 con 0x7f125006cf40 2024-09-11T00:50:34.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.323+0000 7f124d600640 1 -- 172.21.15.31:0/2957891860 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1244004ca0 con 0x7f125006cf40 2024-09-11T00:50:34.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.324+0000 7f124d600640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1230077f70 0x7f123007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:50:34.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.325+0000 7f124d600640 1 -- 172.21.15.31:0/2957891860 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7f12440cd150 con 0x7f125006cf40 2024-09-11T00:50:34.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.325+0000 7f1254a00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1230077f70 0x7f123007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:50:34.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.327+0000 7f124d600640 1 -- 172.21.15.31:0/2957891860 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f125006e2e0 con 0x7f125006cf40 2024-09-11T00:50:34.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.328+0000 7f1254a00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1230077f70 0x7f123007a430 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1250074060 tx=0x7f123c0090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:50:34.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.744+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12500af5e0 con 0x7f125006cf40 2024-09-11T00:50:34.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:50:34.747 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":152,"btime":"2024-09-11T00:50:32:153473+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:50:34.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.745+0000 7f124d600640 1 -- 172.21.15.31:0/2957891860 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6544 (secure 0 0 0) 0x7f125014b620 con 0x7f125006cf40 2024-09-11T00:50:34.748 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 152 2024-09-11T00:50:34.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.749+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1230077f70 msgr2=0x7f123007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:34.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.749+0000 7f1257e00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1230077f70 0x7f123007a430 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1250074060 tx=0x7f123c0090e0 comp rx=0 tx=0).stop 2024-09-11T00:50:34.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.749+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125006cf40 msgr2=0x7f12500bfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:50:34.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.749+0000 7f1257e00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125006cf40 0x7f12500bfab0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f12440099c0 tx=0x7f12440039e0 comp rx=0 tx=0).stop 2024-09-11T00:50:34.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.750+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 shutdown_connections 2024-09-11T00:50:34.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.750+0000 7f1257e00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1230077f70 0x7f123007a430 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:34.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.750+0000 7f1257e00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f12500c7330 0x7f12500af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:34.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.750+0000 7f1257e00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12500ae080 0x7f125014b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:34.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.750+0000 7f1257e00640 1 --2- 172.21.15.31:0/2957891860 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f125006cf40 0x7f12500bfab0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:50:34.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.750+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 >> 172.21.15.31:0/2957891860 conn(0x7f12500c8cc0 msgr2=0x7f125014c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:50:34.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.750+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 shutdown_connections 2024-09-11T00:50:34.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:50:34.750+0000 7f1257e00640 1 -- 172.21.15.31:0/2957891860 wait complete. 2024-09-11T00:50:34.778 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-11T00:50:34.778 INFO:tasks.mds_thrash.fs.[b]:waiting for 88 secs before thrashing 2024-09-11T00:50:48.847 INFO:tasks.workunit.client.1.smithi204.stdout:fs setup took 1063 secs 2024-09-11T00:50:51.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:50:51.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:50:53.195 INFO:tasks.workunit.client.1.smithi204.stdout:Syncing()...4 sec 2024-09-11T00:50:53.198 INFO:tasks.workunit.client.1.smithi204.stdout:Starting Actual Benchmark At: Wed Sep 11 00:50:53 2024 2024-09-11T00:50:53.198 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:51:22.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:51:22.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:51:52.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:51:52.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:52:02.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.061+0000 7fe04f200640 1 -- 172.21.15.31:0/1424129283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe050070410 msgr2=0x7fe05006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.061+0000 7fe04f200640 1 --2- 172.21.15.31:0/1424129283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe050070410 0x7fe05006dc60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009860 tx=0x7fe03c02a560 comp rx=0 tx=0).stop 2024-09-11T00:52:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.062+0000 7fe04f200640 1 -- 172.21.15.31:0/1424129283 shutdown_connections 2024-09-11T00:52:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.062+0000 7fe04f200640 1 --2- 172.21.15.31:0/1424129283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe050070410 0x7fe05006dc60 secure :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009860 tx=0x7fe03c02a560 comp rx=0 tx=0).stop 2024-09-11T00:52:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.062+0000 7fe04f200640 1 -- 172.21.15.31:0/1424129283 >> 172.21.15.31:0/1424129283 conn(0x7fe05006caf0 msgr2=0x7fe05006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.066+0000 7fe04f200640 1 -- 172.21.15.31:0/1424129283 shutdown_connections 2024-09-11T00:52:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.066+0000 7fe04f200640 1 -- 172.21.15.31:0/1424129283 wait complete. 2024-09-11T00:52:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.068+0000 7fe04f200640 1 Processor -- start 2024-09-11T00:52:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.069+0000 7fe04f200640 1 -- start start 2024-09-11T00:52:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0500acaf0 0x7fe0501f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0500c64b0 0x7fe0501f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:03.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe05014f350 0x7fe05014eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0500c4ad0 con 0x7fe0500acaf0 2024-09-11T00:52:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe0500700f0 con 0x7fe0500c64b0 2024-09-11T00:52:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe0500c4c50 con 0x7fe05014f350 2024-09-11T00:52:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0500acaf0 0x7fe0501f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04d400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0500c64b0 0x7fe0501f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0500acaf0 0x7fe0501f38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59658/0 (socket says 172.21.15.31:59658) 2024-09-11T00:52:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04d400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0500c64b0 0x7fe0501f0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:49734/0 (socket says 172.21.15.31:49734) 2024-09-11T00:52:03.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.073+0000 7fe04de00640 1 -- 172.21.15.31:0/466832910 learned_addr learned my addr 172.21.15.31:0/466832910 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:03.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.074+0000 7fe04d400640 1 -- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe05014f350 msgr2=0x7fe05014eba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:52:03.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.074+0000 7fe04d400640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe05014f350 0x7fe05014eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:03.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.074+0000 7fe04d400640 1 -- 172.21.15.31:0/466832910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0500acaf0 msgr2=0x7fe0501f38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:03.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.074+0000 7fe04d400640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0500acaf0 0x7fe0501f38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:03.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.074+0000 7fe04d400640 1 -- 172.21.15.31:0/466832910 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0501f3e10 con 0x7fe0500c64b0 2024-09-11T00:52:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.075+0000 7fe04d400640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0500c64b0 0x7fe0501f0c80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe04000c3c0 tx=0x7fe04000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.076+0000 7fe046000640 1 -- 172.21.15.31:0/466832910 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe040017070 con 0x7fe0500c64b0 2024-09-11T00:52:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.076+0000 7fe046000640 1 -- 172.21.15.31:0/466832910 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0400053c0 con 0x7fe0500c64b0 2024-09-11T00:52:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.076+0000 7fe046000640 1 -- 172.21.15.31:0/466832910 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe040003850 con 0x7fe0500c64b0 2024-09-11T00:52:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.076+0000 7fe04f200640 1 -- 172.21.15.31:0/466832910 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe050072250 con 0x7fe0500c64b0 2024-09-11T00:52:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.076+0000 7fe04f200640 1 -- 172.21.15.31:0/466832910 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0501f0480 con 0x7fe0500c64b0 2024-09-11T00:52:03.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.078+0000 7fe04f200640 1 -- 172.21.15.31:0/466832910 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0500c50d0 con 0x7fe0500c64b0 2024-09-11T00:52:03.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.080+0000 7fe046000640 1 -- 172.21.15.31:0/466832910 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe040038ac0 con 0x7fe0500c64b0 2024-09-11T00:52:03.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.082+0000 7fe046000640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe030077f70 0x7fe03007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:03.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.082+0000 7fe046000640 1 -- 172.21.15.31:0/466832910 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7fe04009a040 con 0x7fe0500c64b0 2024-09-11T00:52:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.083+0000 7fe04de00640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe030077f70 0x7fe03007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.084+0000 7fe04de00640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe030077f70 0x7fe03007a430 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe03c0095f0 tx=0x7fe03c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:03.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.089+0000 7fe046000640 1 -- 172.21.15.31:0/466832910 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe0500c50d0 con 0x7fe0500c64b0 2024-09-11T00:52:03.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.593+0000 7fe04f200640 1 -- 172.21.15.31:0/466832910 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe05006dc60 con 0x7fe0500c64b0 2024-09-11T00:52:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.595+0000 7fe046000640 1 -- 172.21.15.31:0/466832910 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6544 (secure 0 0 0) 0x7fe0400617f0 con 0x7fe0500c64b0 2024-09-11T00:52:03.600 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:03.600 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":152,"btime":"2024-09-11T00:50:32:153473+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:52:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 152 2024-09-11T00:52:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.603+0000 7fe04f200640 1 -- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe030077f70 msgr2=0x7fe03007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.603+0000 7fe04f200640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe030077f70 0x7fe03007a430 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe03c0095f0 tx=0x7fe03c004730 comp rx=0 tx=0).stop 2024-09-11T00:52:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.603+0000 7fe04f200640 1 -- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0500c64b0 msgr2=0x7fe0501f0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.603+0000 7fe04f200640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0500c64b0 0x7fe0501f0c80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe04000c3c0 tx=0x7fe04000c880 comp rx=0 tx=0).stop 2024-09-11T00:52:03.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.603+0000 7fe04f200640 1 -- 172.21.15.31:0/466832910 shutdown_connections 2024-09-11T00:52:03.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.603+0000 7fe04f200640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe030077f70 0x7fe03007a430 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:03.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.603+0000 7fe04f200640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe05014f350 0x7fe05014eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:03.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.603+0000 7fe04f200640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0500c64b0 0x7fe0501f0c80 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:03.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.603+0000 7fe04f200640 1 --2- 172.21.15.31:0/466832910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0500acaf0 0x7fe0501f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:03.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.603+0000 7fe04f200640 1 -- 172.21.15.31:0/466832910 >> 172.21.15.31:0/466832910 conn(0x7fe05006caf0 msgr2=0x7fe0500a8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:03.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.607+0000 7fe04f200640 1 -- 172.21.15.31:0/466832910 shutdown_connections 2024-09-11T00:52:03.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.607+0000 7fe04f200640 1 -- 172.21.15.31:0/466832910 wait complete. 2024-09-11T00:52:03.642 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-11T00:52:03.642 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-11T00:52:03.642 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:52:03.669 INFO:tasks.ceph.mds.b:Stopped 2024-09-11T00:52:03.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.957+0000 7f1948000640 1 -- 172.21.15.31:0/112714760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f194014f9f0 msgr2=0x7f194006ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.957+0000 7f1948000640 1 --2- 172.21.15.31:0/112714760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f194014f9f0 0x7f194006ec10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1934005fd0 tx=0x7f193402a410 comp rx=0 tx=0).stop 2024-09-11T00:52:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.958+0000 7f1948000640 1 -- 172.21.15.31:0/112714760 shutdown_connections 2024-09-11T00:52:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.958+0000 7f1948000640 1 --2- 172.21.15.31:0/112714760 >> v2:172.21.15.204:3300/0 conn(0x7f19400c5ae0 0x7f19400c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.958+0000 7f1948000640 1 --2- 172.21.15.31:0/112714760 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f194014f9f0 0x7f194006ec10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.958+0000 7f1948000640 1 -- 172.21.15.31:0/112714760 >> 172.21.15.31:0/112714760 conn(0x7f19400a9750 msgr2=0x7f19400a9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.958+0000 7f1948000640 1 -- 172.21.15.31:0/112714760 shutdown_connections 2024-09-11T00:52:03.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.958+0000 7f1948000640 1 -- 172.21.15.31:0/112714760 wait complete. 2024-09-11T00:52:03.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.961+0000 7f1948000640 1 Processor -- start 2024-09-11T00:52:03.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.961+0000 7f1948000640 1 -- start start 2024-09-11T00:52:03.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.962+0000 7f1948000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19400c5ae0 0x7f194013eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:03.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.962+0000 7f1948000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19400c6480 0x7f1940149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:03.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.964+0000 7f1945600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19400c5ae0 0x7f194013eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:03.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.965+0000 7f1944c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19400c6480 0x7f1940149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:03.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.965+0000 7f1948000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f194014f9f0 0x7f19400734a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.965+0000 7f1948000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f19400c3570 con 0x7f19400c6480 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.965+0000 7f1948000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f19400c38e0 con 0x7f19400c5ae0 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.965+0000 7f1948000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f194014fdd0 con 0x7f194014f9f0 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.965+0000 7f1944c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19400c6480 0x7f1940149120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59674/0 (socket says 172.21.15.31:59674) 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.965+0000 7f1944c00640 1 -- 172.21.15.31:0/4224708687 learned_addr learned my addr 172.21.15.31:0/4224708687 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.966+0000 7f1944c00640 1 -- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f194014f9f0 msgr2=0x7f19400734a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.966+0000 7f1944c00640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f194014f9f0 0x7f19400734a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.966+0000 7f1944c00640 1 -- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19400c5ae0 msgr2=0x7f194013eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.967+0000 7f1944c00640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19400c5ae0 0x7f194013eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.967+0000 7f1944c00640 1 -- 172.21.15.31:0/4224708687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1940146fa0 con 0x7f19400c6480 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.967+0000 7f1945600640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19400c5ae0 0x7f194013eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.968+0000 7f1944c00640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19400c6480 0x7f1940149120 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e9c0 tx=0x7f192c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:03.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.968+0000 7f193d600640 1 -- 172.21.15.31:0/4224708687 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c00cd80 con 0x7f19400c6480 2024-09-11T00:52:03.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.968+0000 7f193d600640 1 -- 172.21.15.31:0/4224708687 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f192c004030 con 0x7f19400c6480 2024-09-11T00:52:03.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.968+0000 7f193d600640 1 -- 172.21.15.31:0/4224708687 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c004480 con 0x7f19400c6480 2024-09-11T00:52:03.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.968+0000 7f1948000640 1 -- 172.21.15.31:0/4224708687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1940149660 con 0x7f19400c6480 2024-09-11T00:52:03.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.969+0000 7f1948000640 1 -- 172.21.15.31:0/4224708687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1940149ea0 con 0x7f19400c6480 2024-09-11T00:52:03.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.970+0000 7f1948000640 1 -- 172.21.15.31:0/4224708687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19400aa270 con 0x7f19400c6480 2024-09-11T00:52:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.972+0000 7f193d600640 1 -- 172.21.15.31:0/4224708687 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f192c0035f0 con 0x7f19400c6480 2024-09-11T00:52:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.974+0000 7f193d600640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1920077f70 0x7f192007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:03.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.974+0000 7f193d600640 1 -- 172.21.15.31:0/4224708687 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7f192c09b5f0 con 0x7f19400c6480 2024-09-11T00:52:03.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.975+0000 7f1945600640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1920077f70 0x7f192007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:03.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.976+0000 7f1945600640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1920077f70 0x7f192007a430 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1934033000 tx=0x7f1934004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:03.984+0000 7f193d600640 1 -- 172.21.15.31:0/4224708687 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f192c063400 con 0x7f19400c6480 2024-09-11T00:52:04.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.493+0000 7f1948000640 1 -- 172.21.15.31:0/4224708687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1940151200 con 0x7f19400c6480 2024-09-11T00:52:04.512 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:04.512 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":152,"btime":"2024-09-11T00:50:32:153473+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:52:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 152 2024-09-11T00:52:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.507+0000 7f193d600640 1 -- 172.21.15.31:0/4224708687 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6544 (secure 0 0 0) 0x7f192c062d00 con 0x7f19400c6480 2024-09-11T00:52:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.512+0000 7f1948000640 1 -- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1920077f70 msgr2=0x7f192007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.512+0000 7f1948000640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1920077f70 0x7f192007a430 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1934033000 tx=0x7f1934004730 comp rx=0 tx=0).stop 2024-09-11T00:52:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.512+0000 7f1948000640 1 -- 172.21.15.31:0/4224708687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19400c6480 msgr2=0x7f1940149120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.512+0000 7f1948000640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19400c6480 0x7f1940149120 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e9c0 tx=0x7f192c00ee80 comp rx=0 tx=0).stop 2024-09-11T00:52:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.513+0000 7f1948000640 1 -- 172.21.15.31:0/4224708687 shutdown_connections 2024-09-11T00:52:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.513+0000 7f1948000640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1920077f70 0x7f192007a430 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.513+0000 7f1948000640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f194014f9f0 0x7f19400734a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.513+0000 7f1948000640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19400c6480 0x7f1940149120 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.513+0000 7f1948000640 1 --2- 172.21.15.31:0/4224708687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19400c5ae0 0x7f194013eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.513+0000 7f1948000640 1 -- 172.21.15.31:0/4224708687 >> 172.21.15.31:0/4224708687 conn(0x7f19400a9750 msgr2=0x7f1940151a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.513+0000 7f1948000640 1 -- 172.21.15.31:0/4224708687 shutdown_connections 2024-09-11T00:52:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:04.513+0000 7f1948000640 1 -- 172.21.15.31:0/4224708687 wait complete. 2024-09-11T00:52:04.547 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:52:06.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:06.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.834+0000 7ffaeb200640 1 -- 172.21.15.31:0/464160528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae4070570 msgr2=0x7ffae414f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:06.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.834+0000 7ffaeb200640 1 --2- 172.21.15.31:0/464160528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae4070570 0x7ffae414f4d0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ffae40a7630 tx=0x7ffadc02caa0 comp rx=0 tx=0).stop 2024-09-11T00:52:06.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.834+0000 7ffaeb200640 1 -- 172.21.15.31:0/464160528 shutdown_connections 2024-09-11T00:52:06.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.834+0000 7ffaeb200640 1 --2- 172.21.15.31:0/464160528 >> v2:172.21.15.204:3300/0 conn(0x7ffae40c2ff0 0x7ffae406e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:06.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.834+0000 7ffaeb200640 1 --2- 172.21.15.31:0/464160528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae4070570 0x7ffae414f4d0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:06.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.834+0000 7ffaeb200640 1 -- 172.21.15.31:0/464160528 >> 172.21.15.31:0/464160528 conn(0x7ffae40a9750 msgr2=0x7ffae40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:06.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.836+0000 7ffaeb200640 1 -- 172.21.15.31:0/464160528 shutdown_connections 2024-09-11T00:52:06.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.836+0000 7ffaeb200640 1 -- 172.21.15.31:0/464160528 wait complete. 2024-09-11T00:52:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.840+0000 7ffaeb200640 1 Processor -- start 2024-09-11T00:52:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.841+0000 7ffaeb200640 1 -- start start 2024-09-11T00:52:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.844+0000 7ffaeb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae406e9c0 0x7ffae41dc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.845+0000 7ffaeb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae40c2ff0 0x7ffae41f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.845+0000 7ffaeb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffae4072770 0x7ffae41f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.845+0000 7ffaeb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffae40c7690 con 0x7ffae406e9c0 2024-09-11T00:52:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.845+0000 7ffaeb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffae414fb20 con 0x7ffae4072770 2024-09-11T00:52:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.845+0000 7ffaeb200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ffae41de3b0 con 0x7ffae40c2ff0 2024-09-11T00:52:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.845+0000 7ffae3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae406e9c0 0x7ffae41dc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.845+0000 7ffae3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae406e9c0 0x7ffae41dc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59692/0 (socket says 172.21.15.31:59692) 2024-09-11T00:52:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.845+0000 7ffae3e00640 1 -- 172.21.15.31:0/3862133587 learned_addr learned my addr 172.21.15.31:0/3862133587 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:06.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.845+0000 7ffae9200640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffae4072770 0x7ffae41f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:06.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.846+0000 7ffae3400640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae40c2ff0 0x7ffae41f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:06.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.846+0000 7ffae3e00640 1 -- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae40c2ff0 msgr2=0x7ffae41f2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:06.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.846+0000 7ffae3e00640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae40c2ff0 0x7ffae41f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:06.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.846+0000 7ffae3e00640 1 -- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffae4072770 msgr2=0x7ffae41f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:06.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.846+0000 7ffae3e00640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffae4072770 0x7ffae41f30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:06.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.846+0000 7ffae3e00640 1 -- 172.21.15.31:0/3862133587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae41dddf0 con 0x7ffae406e9c0 2024-09-11T00:52:06.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.847+0000 7ffae3e00640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae406e9c0 0x7ffae41dc880 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000f80 tx=0x7ffadc003c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:06.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.848+0000 7ffadbe00640 1 -- 172.21.15.31:0/3862133587 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc004450 con 0x7ffae406e9c0 2024-09-11T00:52:06.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.848+0000 7ffadbe00640 1 -- 172.21.15.31:0/3862133587 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffadc0045f0 con 0x7ffae406e9c0 2024-09-11T00:52:06.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.848+0000 7ffadbe00640 1 -- 172.21.15.31:0/3862133587 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc043440 con 0x7ffae406e9c0 2024-09-11T00:52:06.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.848+0000 7ffaeb200640 1 -- 172.21.15.31:0/3862133587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae41de0d0 con 0x7ffae406e9c0 2024-09-11T00:52:06.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.848+0000 7ffaeb200640 1 -- 172.21.15.31:0/3862133587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae41dc360 con 0x7ffae406e9c0 2024-09-11T00:52:06.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.850+0000 7ffaeb200640 1 -- 172.21.15.31:0/3862133587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffabc005290 con 0x7ffae406e9c0 2024-09-11T00:52:06.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.852+0000 7ffadbe00640 1 -- 172.21.15.31:0/3862133587 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ffadc041070 con 0x7ffae406e9c0 2024-09-11T00:52:06.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.854+0000 7ffadbe00640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffac4077ff0 0x7ffac407a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:06.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.854+0000 7ffadbe00640 1 -- 172.21.15.31:0/3862133587 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7ffadc0bbe20 con 0x7ffae406e9c0 2024-09-11T00:52:06.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.854+0000 7ffae3400640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffac4077ff0 0x7ffac407a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:06.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.856+0000 7ffae3400640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffac4077ff0 0x7ffac407a4b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ffae40ad4b0 tx=0x7ffad0005cc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:06.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:06.864+0000 7ffadbe00640 1 -- 172.21.15.31:0/3862133587 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ffadc035020 con 0x7ffae406e9c0 2024-09-11T00:52:07.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.319+0000 7ffaeb200640 1 -- 172.21.15.31:0/3862133587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffabc005850 con 0x7ffae406e9c0 2024-09-11T00:52:07.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.321+0000 7ffadbe00640 1 -- 172.21.15.31:0/3862133587 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6544 (secure 0 0 0) 0x7ffadc083cb0 con 0x7ffae406e9c0 2024-09-11T00:52:07.325 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:07.325 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":152,"btime":"2024-09-11T00:50:32:153473+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:52:07.326 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 152 2024-09-11T00:52:07.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.324+0000 7ffaeb200640 1 -- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffac4077ff0 msgr2=0x7ffac407a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:07.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.324+0000 7ffaeb200640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffac4077ff0 0x7ffac407a4b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ffae40ad4b0 tx=0x7ffad0005cc0 comp rx=0 tx=0).stop 2024-09-11T00:52:07.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.324+0000 7ffaeb200640 1 -- 172.21.15.31:0/3862133587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae406e9c0 msgr2=0x7ffae41dc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:07.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.324+0000 7ffaeb200640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae406e9c0 0x7ffae41dc880 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000f80 tx=0x7ffadc003c90 comp rx=0 tx=0).stop 2024-09-11T00:52:07.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.324+0000 7ffaeb200640 1 -- 172.21.15.31:0/3862133587 shutdown_connections 2024-09-11T00:52:07.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.324+0000 7ffaeb200640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ffac4077ff0 0x7ffac407a4b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:07.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.324+0000 7ffaeb200640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffae4072770 0x7ffae41f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:07.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.324+0000 7ffaeb200640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ffae40c2ff0 0x7ffae41f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:07.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.324+0000 7ffaeb200640 1 --2- 172.21.15.31:0/3862133587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffae406e9c0 0x7ffae41dc880 secure :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000f80 tx=0x7ffadc003c90 comp rx=0 tx=0).stop 2024-09-11T00:52:07.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.324+0000 7ffaeb200640 1 -- 172.21.15.31:0/3862133587 >> 172.21.15.31:0/3862133587 conn(0x7ffae40a9750 msgr2=0x7ffae414a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:07.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.327+0000 7ffaeb200640 1 -- 172.21.15.31:0/3862133587 shutdown_connections 2024-09-11T00:52:07.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:07.327+0000 7ffaeb200640 1 -- 172.21.15.31:0/3862133587 wait complete. 2024-09-11T00:52:07.357 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:52:09.358 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.619+0000 7fcbcce00640 1 -- 172.21.15.31:0/4224500642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc80c5de0 msgr2=0x7fcbc8155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.619+0000 7fcbcce00640 1 --2- 172.21.15.31:0/4224500642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc80c5de0 0x7fcbc8155560 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcbc806de70 tx=0x7fcbc002a8c0 comp rx=0 tx=0).stop 2024-09-11T00:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.619+0000 7fcbcce00640 1 -- 172.21.15.31:0/4224500642 shutdown_connections 2024-09-11T00:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.619+0000 7fcbcce00640 1 --2- 172.21.15.31:0/4224500642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc80c5de0 0x7fcbc8155560 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.619+0000 7fcbcce00640 1 --2- 172.21.15.31:0/4224500642 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcbc80c2ff0 0x7fcbc80c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.619+0000 7fcbcce00640 1 --2- 172.21.15.31:0/4224500642 >> v2:172.21.15.31:3300/0 conn(0x7fcbc806e480 0x7fcbc806e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.619+0000 7fcbcce00640 1 -- 172.21.15.31:0/4224500642 >> 172.21.15.31:0/4224500642 conn(0x7fcbc80a9750 msgr2=0x7fcbc80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:09.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.620+0000 7fcbcce00640 1 -- 172.21.15.31:0/4224500642 shutdown_connections 2024-09-11T00:52:09.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.620+0000 7fcbcce00640 1 -- 172.21.15.31:0/4224500642 wait complete. 2024-09-11T00:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.622+0000 7fcbcce00640 1 Processor -- start 2024-09-11T00:52:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.623+0000 7fcbcce00640 1 -- start start 2024-09-11T00:52:09.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.624+0000 7fcbcce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc80c2ff0 0x7fcbc815f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:09.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.624+0000 7fcbcce00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcbc80c5de0 0x7fcbc80746d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.624+0000 7fcbcce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbc8142e90 0x7fcbc8074c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.624+0000 7fcbcce00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc814fb20 con 0x7fcbc8142e90 2024-09-11T00:52:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.624+0000 7fcbcce00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc806ef90 con 0x7fcbc80c2ff0 2024-09-11T00:52:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.624+0000 7fcbcce00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc80c7690 con 0x7fcbc80c5de0 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.624+0000 7fcbc6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbc8142e90 0x7fcbc8074c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.624+0000 7fcbc6800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbc8142e90 0x7fcbc8074c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33690/0 (socket says 172.21.15.31:33690) 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.624+0000 7fcbc6800640 1 -- 172.21.15.31:0/758504843 learned_addr learned my addr 172.21.15.31:0/758504843 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.624+0000 7fcbc5400640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcbc80c5de0 0x7fcbc80746d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.625+0000 7fcbc5e00640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc80c2ff0 0x7fcbc815f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.625+0000 7fcbc6800640 1 -- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcbc80c5de0 msgr2=0x7fcbc80746d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.625+0000 7fcbc6800640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcbc80c5de0 0x7fcbc80746d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.625+0000 7fcbc6800640 1 -- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc80c2ff0 msgr2=0x7fcbc815f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.625+0000 7fcbc6800640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc80c2ff0 0x7fcbc815f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.625+0000 7fcbc6800640 1 -- 172.21.15.31:0/758504843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc8075150 con 0x7fcbc8142e90 2024-09-11T00:52:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.625+0000 7fcbc5e00640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc80c2ff0 0x7fcbc815f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:52:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.626+0000 7fcbc6800640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbc8142e90 0x7fcbc8074c10 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009ca0 tx=0x7fcbc0003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.626+0000 7fcbbe000640 1 -- 172.21.15.31:0/758504843 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0031ba0 con 0x7fcbc8142e90 2024-09-11T00:52:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.626+0000 7fcbcce00640 1 -- 172.21.15.31:0/758504843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc814b830 con 0x7fcbc8142e90 2024-09-11T00:52:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.626+0000 7fcbcce00640 1 -- 172.21.15.31:0/758504843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc81480c0 con 0x7fcbc8142e90 2024-09-11T00:52:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.627+0000 7fcbbe000640 1 -- 172.21.15.31:0/758504843 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbc0031d40 con 0x7fcbc8142e90 2024-09-11T00:52:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.627+0000 7fcbbe000640 1 -- 172.21.15.31:0/758504843 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00332d0 con 0x7fcbc8142e90 2024-09-11T00:52:09.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.630+0000 7fcbbe000640 1 -- 172.21.15.31:0/758504843 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fcbc004b030 con 0x7fcbc8142e90 2024-09-11T00:52:09.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.631+0000 7fcbcce00640 1 -- 172.21.15.31:0/758504843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc814f740 con 0x7fcbc8142e90 2024-09-11T00:52:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.631+0000 7fcbbe000640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcba8077f70 0x7fcba807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.632+0000 7fcbbe000640 1 -- 172.21.15.31:0/758504843 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7fcbc00bbaa0 con 0x7fcbc8142e90 2024-09-11T00:52:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.632+0000 7fcbc5e00640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcba8077f70 0x7fcba807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.633+0000 7fcbc5e00640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcba8077f70 0x7fcba807a430 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80c1d30 tx=0x7fcbb8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:09.645+0000 7fcbbe000640 1 -- 172.21.15.31:0/758504843 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcbc003f030 con 0x7fcbc8142e90 2024-09-11T00:52:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.085+0000 7fcbcce00640 1 -- 172.21.15.31:0/758504843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbc80c6bb0 con 0x7fcbc8142e90 2024-09-11T00:52:10.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.087+0000 7fcbbe000640 1 -- 172.21.15.31:0/758504843 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6544 (secure 0 0 0) 0x7fcbc0088e20 con 0x7fcbc8142e90 2024-09-11T00:52:10.089 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:10.090 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":152,"btime":"2024-09-11T00:50:32:153473+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:52:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 152 2024-09-11T00:52:10.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.092+0000 7fcbcce00640 1 -- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcba8077f70 msgr2=0x7fcba807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:10.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.092+0000 7fcbcce00640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcba8077f70 0x7fcba807a430 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80c1d30 tx=0x7fcbb8008040 comp rx=0 tx=0).stop 2024-09-11T00:52:10.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.092+0000 7fcbcce00640 1 -- 172.21.15.31:0/758504843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbc8142e90 msgr2=0x7fcbc8074c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.092+0000 7fcbcce00640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbc8142e90 0x7fcbc8074c10 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009ca0 tx=0x7fcbc0003ba0 comp rx=0 tx=0).stop 2024-09-11T00:52:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.092+0000 7fcbcce00640 1 -- 172.21.15.31:0/758504843 shutdown_connections 2024-09-11T00:52:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.092+0000 7fcbcce00640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcba8077f70 0x7fcba807a430 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.092+0000 7fcbcce00640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbc8142e90 0x7fcbc8074c10 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.092+0000 7fcbcce00640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcbc80c5de0 0x7fcbc80746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.092+0000 7fcbcce00640 1 --2- 172.21.15.31:0/758504843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbc80c2ff0 0x7fcbc815f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.092+0000 7fcbcce00640 1 -- 172.21.15.31:0/758504843 >> 172.21.15.31:0/758504843 conn(0x7fcbc80a9750 msgr2=0x7fcbc814e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:10.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.093+0000 7fcbcce00640 1 -- 172.21.15.31:0/758504843 shutdown_connections 2024-09-11T00:52:10.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:10.093+0000 7fcbcce00640 1 -- 172.21.15.31:0/758504843 wait complete. 2024-09-11T00:52:10.124 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:52:12.125 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:12.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.389+0000 7fe815e00640 1 -- 172.21.15.31:0/711711412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8100c7330 msgr2=0x7fe810161b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:12.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.389+0000 7fe815e00640 1 --2- 172.21.15.31:0/711711412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8100c7330 0x7fe810161b70 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe808005fd0 tx=0x7fe80802a820 comp rx=0 tx=0).stop 2024-09-11T00:52:12.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.389+0000 7fe815e00640 1 -- 172.21.15.31:0/711711412 shutdown_connections 2024-09-11T00:52:12.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.389+0000 7fe815e00640 1 --2- 172.21.15.31:0/711711412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8100c7330 0x7fe810161b70 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:12.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.389+0000 7fe815e00640 1 -- 172.21.15.31:0/711711412 >> 172.21.15.31:0/711711412 conn(0x7fe8100c8cc0 msgr2=0x7fe8100ad700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:12.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.389+0000 7fe815e00640 1 -- 172.21.15.31:0/711711412 shutdown_connections 2024-09-11T00:52:12.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.389+0000 7fe815e00640 1 -- 172.21.15.31:0/711711412 wait complete. 2024-09-11T00:52:12.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.392+0000 7fe815e00640 1 Processor -- start 2024-09-11T00:52:12.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.392+0000 7fe815e00640 1 -- start start 2024-09-11T00:52:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.394+0000 7fe815e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe81006cf40 0x7fe810159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.394+0000 7fe815e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8100c7330 0x7fe8101ef170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.394+0000 7fe815e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe810153c10 0x7fe8101ef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.394+0000 7fe815e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe810159f40 con 0x7fe810153c10 2024-09-11T00:52:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.394+0000 7fe815e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe8100c6590 con 0x7fe81006cf40 2024-09-11T00:52:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.394+0000 7fe815e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe8100ae520 con 0x7fe8100c7330 2024-09-11T00:52:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.394+0000 7fe80f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe810153c10 0x7fe8101ef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.394+0000 7fe80f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe810153c10 0x7fe8101ef6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33706/0 (socket says 172.21.15.31:33706) 2024-09-11T00:52:12.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.394+0000 7fe80f400640 1 -- 172.21.15.31:0/3737522871 learned_addr learned my addr 172.21.15.31:0/3737522871 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.394+0000 7fe80ea00640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe81006cf40 0x7fe810159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.395+0000 7fe80e000640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8100c7330 0x7fe8101ef170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.395+0000 7fe80f400640 1 -- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8100c7330 msgr2=0x7fe8101ef170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.395+0000 7fe80f400640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8100c7330 0x7fe8101ef170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.395+0000 7fe80f400640 1 -- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe81006cf40 msgr2=0x7fe810159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.395+0000 7fe80f400640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe81006cf40 0x7fe810159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:12.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.395+0000 7fe80f400640 1 -- 172.21.15.31:0/3737522871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8100ad930 con 0x7fe810153c10 2024-09-11T00:52:12.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.396+0000 7fe80f400640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe810153c10 0x7fe8101ef6b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fe808006010 tx=0x7fe808004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.398+0000 7fe807400640 1 -- 172.21.15.31:0/3737522871 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe808037040 con 0x7fe810153c10 2024-09-11T00:52:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.398+0000 7fe815e00640 1 -- 172.21.15.31:0/3737522871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8101ec850 con 0x7fe810153c10 2024-09-11T00:52:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.398+0000 7fe815e00640 1 -- 172.21.15.31:0/3737522871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe810159770 con 0x7fe810153c10 2024-09-11T00:52:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.399+0000 7fe807400640 1 -- 172.21.15.31:0/3737522871 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe808042070 con 0x7fe810153c10 2024-09-11T00:52:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.400+0000 7fe807400640 1 -- 172.21.15.31:0/3737522871 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe808037040 con 0x7fe810153c10 2024-09-11T00:52:12.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.400+0000 7fe815e00640 1 -- 172.21.15.31:0/3737522871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e8005290 con 0x7fe810153c10 2024-09-11T00:52:12.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.403+0000 7fe807400640 1 -- 172.21.15.31:0/3737522871 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe80803a030 con 0x7fe810153c10 2024-09-11T00:52:12.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.405+0000 7fe807400640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe7f0078170 0x7fe7f007a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:12.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.405+0000 7fe807400640 1 -- 172.21.15.31:0/3737522871 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7fe8080bbfc0 con 0x7fe810153c10 2024-09-11T00:52:12.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.405+0000 7fe80ea00640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe7f0078170 0x7fe7f007a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.406+0000 7fe80ea00640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe7f0078170 0x7fe7f007a630 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe800000f60 tx=0x7fe800009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:12.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.411+0000 7fe807400640 1 -- 172.21.15.31:0/3737522871 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe808003e20 con 0x7fe810153c10 2024-09-11T00:52:12.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.850+0000 7fe815e00640 1 -- 172.21.15.31:0/3737522871 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7e8005580 con 0x7fe810153c10 2024-09-11T00:52:12.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.851+0000 7fe807400640 1 -- 172.21.15.31:0/3737522871 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6544 (secure 0 0 0) 0x7fe808083ce0 con 0x7fe810153c10 2024-09-11T00:52:12.854 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:12.854 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":152,"btime":"2024-09-11T00:50:32:153473+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:52:12.855 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 152 2024-09-11T00:52:12.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.854+0000 7fe815e00640 1 -- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe7f0078170 msgr2=0x7fe7f007a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:12.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.854+0000 7fe815e00640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe7f0078170 0x7fe7f007a630 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe800000f60 tx=0x7fe800009040 comp rx=0 tx=0).stop 2024-09-11T00:52:12.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.854+0000 7fe815e00640 1 -- 172.21.15.31:0/3737522871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe810153c10 msgr2=0x7fe8101ef6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:12.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.854+0000 7fe815e00640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe810153c10 0x7fe8101ef6b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fe808006010 tx=0x7fe808004270 comp rx=0 tx=0).stop 2024-09-11T00:52:12.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.856+0000 7fe815e00640 1 -- 172.21.15.31:0/3737522871 shutdown_connections 2024-09-11T00:52:12.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.856+0000 7fe815e00640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe7f0078170 0x7fe7f007a630 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:12.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.856+0000 7fe815e00640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe810153c10 0x7fe8101ef6b0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:12.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.856+0000 7fe815e00640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8100c7330 0x7fe8101ef170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:12.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.856+0000 7fe815e00640 1 --2- 172.21.15.31:0/3737522871 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe81006cf40 0x7fe810159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:12.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.856+0000 7fe815e00640 1 -- 172.21.15.31:0/3737522871 >> 172.21.15.31:0/3737522871 conn(0x7fe8100c8cc0 msgr2=0x7fe810151f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:12.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.856+0000 7fe815e00640 1 -- 172.21.15.31:0/3737522871 shutdown_connections 2024-09-11T00:52:12.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:12.857+0000 7fe815e00640 1 -- 172.21.15.31:0/3737522871 wait complete. 2024-09-11T00:52:12.886 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:52:14.887 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:15.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.161+0000 7fee6ee00640 1 -- 172.21.15.31:0/1382312204 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee6806f8c0 msgr2=0x7fee6815af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.161+0000 7fee6ee00640 1 --2- 172.21.15.31:0/1382312204 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee6806f8c0 0x7fee6815af80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fee68071e40 tx=0x7fee5802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:52:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.162+0000 7fee6ee00640 1 -- 172.21.15.31:0/1382312204 shutdown_connections 2024-09-11T00:52:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.162+0000 7fee6ee00640 1 --2- 172.21.15.31:0/1382312204 >> v2:172.21.15.31:3300/0 conn(0x7fee680af410 0x7fee680af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.162+0000 7fee6ee00640 1 --2- 172.21.15.31:0/1382312204 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee6806f8c0 0x7fee6815af80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.162+0000 7fee6ee00640 1 -- 172.21.15.31:0/1382312204 >> 172.21.15.31:0/1382312204 conn(0x7fee680d02b0 msgr2=0x7fee680d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:15.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.162+0000 7fee6ee00640 1 -- 172.21.15.31:0/1382312204 shutdown_connections 2024-09-11T00:52:15.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.163+0000 7fee6ee00640 1 -- 172.21.15.31:0/1382312204 wait complete. 2024-09-11T00:52:15.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.165+0000 7fee6ee00640 1 Processor -- start 2024-09-11T00:52:15.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.167+0000 7fee6ee00640 1 -- start start 2024-09-11T00:52:15.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.168+0000 7fee6ee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee6806f8c0 0x7fee6814fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:15.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.168+0000 7fee6ee00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee680af410 0x7fee68150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:15.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.168+0000 7fee6ee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee680cfd80 0x7fee6815d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:15.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.168+0000 7fee6ee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fee6806fe20 con 0x7fee6806f8c0 2024-09-11T00:52:15.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.168+0000 7fee6ee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fee6806fca0 con 0x7fee680cfd80 2024-09-11T00:52:15.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.169+0000 7fee6ee00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fee680cacb0 con 0x7fee680af410 2024-09-11T00:52:15.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.169+0000 7fee5f400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee680af410 0x7fee68150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:15.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.169+0000 7fee6ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee680cfd80 0x7fee6815d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:15.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.169+0000 7fee5f400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee680af410 0x7fee68150250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:55060/0 (socket says 172.21.15.31:55060) 2024-09-11T00:52:15.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.169+0000 7fee6ce00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee680cfd80 0x7fee6815d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:48828/0 (socket says 172.21.15.31:48828) 2024-09-11T00:52:15.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.169+0000 7fee5f400640 1 -- 172.21.15.31:0/23043646 learned_addr learned my addr 172.21.15.31:0/23043646 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:15.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.169+0000 7fee67e00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee6806f8c0 0x7fee6814fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:15.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.169+0000 7fee6ce00640 1 -- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee680af410 msgr2=0x7fee68150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:15.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.169+0000 7fee6ce00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee680af410 0x7fee68150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:15.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.169+0000 7fee6ce00640 1 -- 172.21.15.31:0/23043646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee6806f8c0 msgr2=0x7fee6814fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:15.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.170+0000 7fee6ce00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee6806f8c0 0x7fee6814fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:15.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.170+0000 7fee6ce00640 1 -- 172.21.15.31:0/23043646 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee680724d0 con 0x7fee680cfd80 2024-09-11T00:52:15.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.170+0000 7fee67e00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee6806f8c0 0x7fee6814fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:52:15.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.171+0000 7fee6ce00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee680cfd80 0x7fee6815d630 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fee6000ec50 tx=0x7fee6000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:15.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.172+0000 7fee64c00640 1 -- 172.21.15.31:0/23043646 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee60003b70 con 0x7fee680cfd80 2024-09-11T00:52:15.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.172+0000 7fee64c00640 1 -- 172.21.15.31:0/23043646 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee60003d10 con 0x7fee680cfd80 2024-09-11T00:52:15.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.172+0000 7fee64c00640 1 -- 172.21.15.31:0/23043646 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee60004280 con 0x7fee680cfd80 2024-09-11T00:52:15.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.172+0000 7fee6ee00640 1 -- 172.21.15.31:0/23043646 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee68072790 con 0x7fee680cfd80 2024-09-11T00:52:15.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.172+0000 7fee6ee00640 1 -- 172.21.15.31:0/23043646 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee68072fc0 con 0x7fee680cfd80 2024-09-11T00:52:15.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.177+0000 7fee6ee00640 1 -- 172.21.15.31:0/23043646 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee6815d300 con 0x7fee680cfd80 2024-09-11T00:52:15.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.178+0000 7fee64c00640 1 -- 172.21.15.31:0/23043646 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fee60004420 con 0x7fee680cfd80 2024-09-11T00:52:15.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.179+0000 7fee64c00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee44077f70 0x7fee4407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:15.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.179+0000 7fee64c00640 1 -- 172.21.15.31:0/23043646 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7fee6009b700 con 0x7fee680cfd80 2024-09-11T00:52:15.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.180+0000 7fee67e00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee44077f70 0x7fee4407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:15.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.181+0000 7fee67e00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee44077f70 0x7fee4407a430 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fee58033000 tx=0x7fee58004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:15.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.183+0000 7fee64c00640 1 -- 172.21.15.31:0/23043646 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fee60063420 con 0x7fee680cfd80 2024-09-11T00:52:15.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.625+0000 7fee6ee00640 1 -- 172.21.15.31:0/23043646 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee680af860 con 0x7fee680cfd80 2024-09-11T00:52:15.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.626+0000 7fee64c00640 1 -- 172.21.15.31:0/23043646 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6544 (secure 0 0 0) 0x7fee60062d20 con 0x7fee680cfd80 2024-09-11T00:52:15.629 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:15.630 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":152,"btime":"2024-09-11T00:50:32:153473+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:52:15.630 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 152 2024-09-11T00:52:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.630+0000 7fee6ee00640 1 -- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee44077f70 msgr2=0x7fee4407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.630+0000 7fee6ee00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee44077f70 0x7fee4407a430 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fee58033000 tx=0x7fee58004730 comp rx=0 tx=0).stop 2024-09-11T00:52:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.630+0000 7fee6ee00640 1 -- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee680cfd80 msgr2=0x7fee6815d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.630+0000 7fee6ee00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee680cfd80 0x7fee6815d630 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fee6000ec50 tx=0x7fee6000c6a0 comp rx=0 tx=0).stop 2024-09-11T00:52:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.630+0000 7fee6ee00640 1 -- 172.21.15.31:0/23043646 shutdown_connections 2024-09-11T00:52:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.630+0000 7fee6ee00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fee44077f70 0x7fee4407a430 secure :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fee58033000 tx=0x7fee58004730 comp rx=0 tx=0).stop 2024-09-11T00:52:15.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.630+0000 7fee6ee00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee680cfd80 0x7fee6815d630 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:15.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.630+0000 7fee6ee00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fee680af410 0x7fee68150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:15.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.630+0000 7fee6ee00640 1 --2- 172.21.15.31:0/23043646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee6806f8c0 0x7fee6814fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:15.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.630+0000 7fee6ee00640 1 -- 172.21.15.31:0/23043646 >> 172.21.15.31:0/23043646 conn(0x7fee680d02b0 msgr2=0x7fee680b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:15.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.631+0000 7fee6ee00640 1 -- 172.21.15.31:0/23043646 shutdown_connections 2024-09-11T00:52:15.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:15.631+0000 7fee6ee00640 1 -- 172.21.15.31:0/23043646 wait complete. 2024-09-11T00:52:15.664 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:52:17.665 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:17.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.918+0000 7f56c0200640 1 -- 172.21.15.31:0/3864888795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56b80c2ff0 msgr2=0x7f56b80c58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:17.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.918+0000 7f56c0200640 1 --2- 172.21.15.31:0/3864888795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56b80c2ff0 0x7f56b80c58a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f56b80c2060 tx=0x7f56ac02b250 comp rx=0 tx=0).stop 2024-09-11T00:52:17.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.918+0000 7f56c0200640 1 -- 172.21.15.31:0/3864888795 shutdown_connections 2024-09-11T00:52:17.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.918+0000 7f56c0200640 1 --2- 172.21.15.31:0/3864888795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56b80c2ff0 0x7f56b80c58a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:17.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.918+0000 7f56c0200640 1 --2- 172.21.15.31:0/3864888795 >> v2:172.21.15.31:3300/0 conn(0x7f56b806e480 0x7f56b806e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:17.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.918+0000 7f56c0200640 1 -- 172.21.15.31:0/3864888795 >> 172.21.15.31:0/3864888795 conn(0x7f56b80a9750 msgr2=0x7f56b80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:17.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.919+0000 7f56c0200640 1 -- 172.21.15.31:0/3864888795 shutdown_connections 2024-09-11T00:52:17.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.919+0000 7f56c0200640 1 -- 172.21.15.31:0/3864888795 wait complete. 2024-09-11T00:52:17.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.921+0000 7f56c0200640 1 Processor -- start 2024-09-11T00:52:17.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.921+0000 7f56c0200640 1 -- start start 2024-09-11T00:52:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.922+0000 7f56c0200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56b806e480 0x7f56b8073f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.922+0000 7f56c0200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b80c2ff0 0x7f56b8149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.922+0000 7f56c0200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56b80c5de0 0x7f56b814df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56c0200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f56b814fb20 con 0x7f56b80c2ff0 2024-09-11T00:52:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56c0200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f56b806ef90 con 0x7f56b80c5de0 2024-09-11T00:52:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56c0200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f56b80c7690 con 0x7f56b806e480 2024-09-11T00:52:17.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56bd800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56b806e480 0x7f56b8073f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:17.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56bd800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56b806e480 0x7f56b8073f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:60118/0 (socket says 172.21.15.31:60118) 2024-09-11T00:52:17.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56bd800640 1 -- 172.21.15.31:0/2467686665 learned_addr learned my addr 172.21.15.31:0/2467686665 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:17.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56be200640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56b80c5de0 0x7f56b814df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:17.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56bce00640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b80c2ff0 0x7f56b8149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:17.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56bd800640 1 -- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56b80c5de0 msgr2=0x7f56b814df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:17.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56bd800640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56b80c5de0 0x7f56b814df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:17.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56bd800640 1 -- 172.21.15.31:0/2467686665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b80c2ff0 msgr2=0x7f56b8149120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:17.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56bd800640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b80c2ff0 0x7f56b8149120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:17.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.923+0000 7f56bd800640 1 -- 172.21.15.31:0/2467686665 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b80ad110 con 0x7f56b806e480 2024-09-11T00:52:17.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.924+0000 7f56bce00640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b80c2ff0 0x7f56b8149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:52:17.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.924+0000 7f56bd800640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56b806e480 0x7f56b8073f20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f56a400e9c0 tx=0x7f56a400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:17.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.926+0000 7f56b5600640 1 -- 172.21.15.31:0/2467686665 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a400ccc0 con 0x7f56b806e480 2024-09-11T00:52:17.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.926+0000 7f56b5600640 1 -- 172.21.15.31:0/2467686665 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56a4005020 con 0x7f56b806e480 2024-09-11T00:52:17.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.926+0000 7f56b5600640 1 -- 172.21.15.31:0/2467686665 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a400ce60 con 0x7f56b806e480 2024-09-11T00:52:17.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.926+0000 7f56c0200640 1 -- 172.21.15.31:0/2467686665 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b80acb50 con 0x7f56b806e480 2024-09-11T00:52:17.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.927+0000 7f56c0200640 1 -- 172.21.15.31:0/2467686665 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b81e1ee0 con 0x7f56b806e480 2024-09-11T00:52:17.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.929+0000 7f56c0200640 1 -- 172.21.15.31:0/2467686665 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b814f740 con 0x7f56b806e480 2024-09-11T00:52:17.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.929+0000 7f56b5600640 1 -- 172.21.15.31:0/2467686665 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f56a4005450 con 0x7f56b806e480 2024-09-11T00:52:17.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.931+0000 7f56b5600640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5698077fc0 0x7f569807a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:17.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.931+0000 7f56b5600640 1 -- 172.21.15.31:0/2467686665 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 9074+0+0 (secure 0 0 0) 0x7f56a409b340 con 0x7f56b806e480 2024-09-11T00:52:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.932+0000 7f56bce00640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5698077fc0 0x7f569807a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.933+0000 7f56bce00640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5698077fc0 0x7f569807a480 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f56ac034000 tx=0x7f56ac004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:17.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:17.936+0000 7f56b5600640 1 -- 172.21.15.31:0/2467686665 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f56a4063270 con 0x7f56b806e480 2024-09-11T00:52:18.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.321+0000 7f56c0200640 1 -- 172.21.15.31:0/2467686665 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56b814bc00 con 0x7f56b806e480 2024-09-11T00:52:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.321+0000 7f56b5600640 1 -- 172.21.15.31:0/2467686665 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6544 (secure 0 0 0) 0x7f56a4062b70 con 0x7f56b806e480 2024-09-11T00:52:18.324 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:18.324 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":152,"btime":"2024-09-11T00:50:32:153473+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:50:27.135965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6350},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6350":{"gid":6350,"name":"b","rank":0,"incarnation":148,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/2630216714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2630216714},{"type":"v1","addr":"172.21.15.204:6837","nonce":2630216714}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6350,"qdb_cluster":[6350]},"id":2}]} 2024-09-11T00:52:18.325 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 152 2024-09-11T00:52:18.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.325+0000 7f56c0200640 1 -- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5698077fc0 msgr2=0x7f569807a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.325+0000 7f56c0200640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5698077fc0 0x7f569807a480 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f56ac034000 tx=0x7f56ac004730 comp rx=0 tx=0).stop 2024-09-11T00:52:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.325+0000 7f56c0200640 1 -- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56b806e480 msgr2=0x7f56b8073f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.325+0000 7f56c0200640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56b806e480 0x7f56b8073f20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f56a400e9c0 tx=0x7f56a400ee80 comp rx=0 tx=0).stop 2024-09-11T00:52:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.327+0000 7f56c0200640 1 -- 172.21.15.31:0/2467686665 shutdown_connections 2024-09-11T00:52:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.327+0000 7f56c0200640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5698077fc0 0x7f569807a480 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.327+0000 7f56c0200640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56b80c5de0 0x7f56b814df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.327+0000 7f56c0200640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56b80c2ff0 0x7f56b8149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.327+0000 7f56c0200640 1 --2- 172.21.15.31:0/2467686665 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f56b806e480 0x7f56b8073f20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.327+0000 7f56c0200640 1 -- 172.21.15.31:0/2467686665 >> 172.21.15.31:0/2467686665 conn(0x7f56b80a9750 msgr2=0x7f56b80c6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.328+0000 7f56c0200640 1 -- 172.21.15.31:0/2467686665 shutdown_connections 2024-09-11T00:52:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:18.328+0000 7f56c0200640 1 -- 172.21.15.31:0/2467686665 wait complete. 2024-09-11T00:52:18.358 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:52:20.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:20.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:20.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.484+0000 7f3cd0a00640 1 -- 172.21.15.31:0/1870227027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ccc08c9b0 msgr2=0x7f3ccc06dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:20.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.484+0000 7f3cd0a00640 1 --2- 172.21.15.31:0/1870227027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ccc08c9b0 0x7f3ccc06dc60 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4005fd0 tx=0x7f3cb402a4b0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.488+0000 7f3cd0a00640 1 -- 172.21.15.31:0/1870227027 shutdown_connections 2024-09-11T00:52:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.488+0000 7f3cd0a00640 1 --2- 172.21.15.31:0/1870227027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ccc08c9b0 0x7f3ccc06dc60 secure :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4005fd0 tx=0x7f3cb402a4b0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.488+0000 7f3cd0a00640 1 -- 172.21.15.31:0/1870227027 >> 172.21.15.31:0/1870227027 conn(0x7f3ccc06caf0 msgr2=0x7f3ccc06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.492+0000 7f3cd0a00640 1 -- 172.21.15.31:0/1870227027 shutdown_connections 2024-09-11T00:52:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.492+0000 7f3cd0a00640 1 -- 172.21.15.31:0/1870227027 wait complete. 2024-09-11T00:52:20.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.494+0000 7f3cd0a00640 1 Processor -- start 2024-09-11T00:52:20.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.495+0000 7f3cd0a00640 1 -- start start 2024-09-11T00:52:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.497+0000 7f3cd0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ccc0c43c0 0x7f3ccc089640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.497+0000 7f3cd0a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ccc0c47a0 0x7f3ccc0a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.497+0000 7f3cd0a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ccc0ac460 0x7f3ccc0a9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:20.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.497+0000 7f3cd0a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc08c690 con 0x7f3ccc0c43c0 2024-09-11T00:52:20.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.497+0000 7f3cd0a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc0c7550 con 0x7f3ccc0c47a0 2024-09-11T00:52:20.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.497+0000 7f3cd0a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3ccc162d80 con 0x7f3ccc0ac460 2024-09-11T00:52:20.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.498+0000 7f3cca800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ccc0c43c0 0x7f3ccc089640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:20.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.498+0000 7f3ccb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ccc0ac460 0x7f3ccc0a9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:20.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.498+0000 7f3ccb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ccc0ac460 0x7f3ccc0a9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:60126/0 (socket says 172.21.15.31:60126) 2024-09-11T00:52:20.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.498+0000 7f3ccb200640 1 -- 172.21.15.31:0/2201078080 learned_addr learned my addr 172.21.15.31:0/2201078080 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:20.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.498+0000 7f3cc3e00640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ccc0c47a0 0x7f3ccc0a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.498+0000 7f3ccb200640 1 -- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ccc0c47a0 msgr2=0x7f3ccc0a8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.498+0000 7f3ccb200640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ccc0c47a0 0x7f3ccc0a8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.498+0000 7f3ccb200640 1 -- 172.21.15.31:0/2201078080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ccc0c43c0 msgr2=0x7f3ccc089640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.499+0000 7f3ccb200640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ccc0c43c0 0x7f3ccc089640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.499+0000 7f3ccb200640 1 -- 172.21.15.31:0/2201078080 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ccc0a7820 con 0x7f3ccc0ac460 2024-09-11T00:52:20.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.499+0000 7f3cca800640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ccc0c43c0 0x7f3ccc089640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:52:20.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.499+0000 7f3ccb200640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ccc0ac460 0x7f3ccc0a9350 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3cc400c6b0 tx=0x7f3cc400cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:20.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.500+0000 7f3cc2a00640 1 -- 172.21.15.31:0/2201078080 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4003b70 con 0x7f3ccc0ac460 2024-09-11T00:52:20.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.501+0000 7f3cc2a00640 1 -- 172.21.15.31:0/2201078080 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cc4003d10 con 0x7f3ccc0ac460 2024-09-11T00:52:20.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.501+0000 7f3cd0a00640 1 -- 172.21.15.31:0/2201078080 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ccc1fd0d0 con 0x7f3ccc0ac460 2024-09-11T00:52:20.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.501+0000 7f3cc2a00640 1 -- 172.21.15.31:0/2201078080 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4005330 con 0x7f3ccc0ac460 2024-09-11T00:52:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.502+0000 7f3cd0a00640 1 -- 172.21.15.31:0/2201078080 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ccc0ab8d0 con 0x7f3ccc0ac460 2024-09-11T00:52:20.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.507+0000 7f3cc2a00640 1 -- 172.21.15.31:0/2201078080 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3cc40054d0 con 0x7f3ccc0ac460 2024-09-11T00:52:20.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.507+0000 7f3cd0a00640 1 -- 172.21.15.31:0/2201078080 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ccc0c4fa0 con 0x7f3ccc0ac460 2024-09-11T00:52:20.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.512+0000 7f3cc2a00640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ca8077f70 0x7f3ca807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:20.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.512+0000 7f3cc2a00640 1 -- 172.21.15.31:0/2201078080 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7f3cc409a230 con 0x7f3ccc0ac460 2024-09-11T00:52:20.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.513+0000 7f3cca800640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ca8077f70 0x7f3ca807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:20.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.513+0000 7f3cc2a00640 1 -- 172.21.15.31:0/2201078080 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3ccc0c4fa0 con 0x7f3ccc0ac460 2024-09-11T00:52:20.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.515+0000 7f3cca800640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ca8077f70 0x7f3ca807a430 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc003eb0 tx=0x7f3cbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.554+0000 7f6af7000640 1 -- 172.21.15.31:0/550506693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6af00c8ed0 msgr2=0x7f6af00c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.554+0000 7f6af7000640 1 --2- 172.21.15.31:0/550506693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6af00c8ed0 0x7f6af00c9360 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f6af00cab30 tx=0x7f6ae802a8b0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.555+0000 7f6af7000640 1 -- 172.21.15.31:0/550506693 shutdown_connections 2024-09-11T00:52:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.555+0000 7f6af7000640 1 --2- 172.21.15.31:0/550506693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6af00c8ed0 0x7f6af00c9360 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.555+0000 7f6af7000640 1 --2- 172.21.15.31:0/550506693 >> v2:172.21.15.204:3300/0 conn(0x7f6af00c71c0 0x7f6af00c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.555+0000 7f6af7000640 1 -- 172.21.15.31:0/550506693 >> 172.21.15.31:0/550506693 conn(0x7f6af00688e0 msgr2=0x7f6af0068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.555+0000 7f6af7000640 1 -- 172.21.15.31:0/550506693 shutdown_connections 2024-09-11T00:52:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.555+0000 7f6af7000640 1 -- 172.21.15.31:0/550506693 wait complete. 2024-09-11T00:52:20.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.557+0000 7f6af7000640 1 Processor -- start 2024-09-11T00:52:20.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.557+0000 7f6af7000640 1 -- start start 2024-09-11T00:52:20.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.559+0000 7f6af7000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6af00c71c0 0x7f6af00a9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:20.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.559+0000 7f6af7000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6af00c8ed0 0x7f6af006f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:20.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.559+0000 7f6af7000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6af0158260 0x7f6af01df710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:20.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.559+0000 7f6af7000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6af0158640 con 0x7f6af00c8ed0 2024-09-11T00:52:20.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.559+0000 7f6af7000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6af00c44c0 con 0x7f6af0158260 2024-09-11T00:52:20.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.559+0000 7f6af7000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6af00c98a0 con 0x7f6af00c71c0 2024-09-11T00:52:20.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.559+0000 7f6aef400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6af00c8ed0 0x7f6af006f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:20.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.559+0000 7f6aef400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6af00c8ed0 0x7f6af006f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36250/0 (socket says 172.21.15.31:36250) 2024-09-11T00:52:20.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.559+0000 7f6aef400640 1 -- 172.21.15.31:0/3486604198 learned_addr learned my addr 172.21.15.31:0/3486604198 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.562+0000 7f6aef400640 1 -- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6af00c71c0 msgr2=0x7f6af00a9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:20.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.562+0000 7f6aefe00640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6af00c71c0 0x7f6af00a9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.562+0000 7f6aef400640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6af00c71c0 0x7f6af00a9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.562+0000 7f6aef400640 1 -- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6af0158260 msgr2=0x7f6af01df710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.562+0000 7f6aef400640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6af0158260 0x7f6af01df710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.562+0000 7f6aef400640 1 -- 172.21.15.31:0/3486604198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af00a9f50 con 0x7f6af00c8ed0 2024-09-11T00:52:20.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.564+0000 7f6aefe00640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6af00c71c0 0x7f6af00a9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:52:20.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.564+0000 7f6aef400640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6af00c8ed0 0x7f6af006f140 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00ed60 tx=0x7f6adc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:20.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.566+0000 7f6ae7e00640 1 -- 172.21.15.31:0/3486604198 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc009e10 con 0x7f6af00c8ed0 2024-09-11T00:52:20.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.567+0000 7f6ae7e00640 1 -- 172.21.15.31:0/3486604198 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6adc010430 con 0x7f6af00c8ed0 2024-09-11T00:52:20.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.567+0000 7f6af7000640 1 -- 172.21.15.31:0/3486604198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af01deaa0 con 0x7f6af00c8ed0 2024-09-11T00:52:20.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.567+0000 7f6ae7e00640 1 -- 172.21.15.31:0/3486604198 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc010860 con 0x7f6af00c8ed0 2024-09-11T00:52:20.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.567+0000 7f6af7000640 1 -- 172.21.15.31:0/3486604198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af01ded40 con 0x7f6af00c8ed0 2024-09-11T00:52:20.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.569+0000 7f6af7000640 1 -- 172.21.15.31:0/3486604198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6af00c9360 con 0x7f6af00c8ed0 2024-09-11T00:52:20.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.574+0000 7f6ae7e00640 1 -- 172.21.15.31:0/3486604198 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6adc029080 con 0x7f6af00c8ed0 2024-09-11T00:52:20.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.582+0000 7f6ae7e00640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ad0077f70 0x7f6ad007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:20.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.582+0000 7f6ae7e00640 1 -- 172.21.15.31:0/3486604198 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7f6adc09bcc0 con 0x7f6af00c8ed0 2024-09-11T00:52:20.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.583+0000 7f6ae7e00640 1 -- 172.21.15.31:0/3486604198 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6adc063aa0 con 0x7f6af00c8ed0 2024-09-11T00:52:20.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.583+0000 7f6aefe00640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ad0077f70 0x7f6ad007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:20.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.584+0000 7f6aefe00640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ad0077f70 0x7f6ad007a430 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0000f60 tx=0x7f6ae0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.964+0000 7f3cd0a00640 1 -- 172.21.15.31:0/2201078080 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ccc1fd2a0 con 0x7f3ccc0ac460 2024-09-11T00:52:20.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.966+0000 7f3cc2a00640 1 -- 172.21.15.31:0/2201078080 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5712 (secure 0 0 0) 0x7f3cc40057f0 con 0x7f3ccc0ac460 2024-09-11T00:52:20.969 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:20.969 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":154,"btime":"2024-09-11T00:52:20:838310+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:20.668747+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:52:20.969 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 154 2024-09-11T00:52:20.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.970+0000 7f3cd0a00640 1 -- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ca8077f70 msgr2=0x7f3ca807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:20.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.970+0000 7f3cd0a00640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ca8077f70 0x7f3ca807a430 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc003eb0 tx=0x7f3cbc009040 comp rx=0 tx=0).stop 2024-09-11T00:52:20.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.970+0000 7f3cd0a00640 1 -- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ccc0ac460 msgr2=0x7f3ccc0a9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:20.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.970+0000 7f3cd0a00640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ccc0ac460 0x7f3ccc0a9350 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3cc400c6b0 tx=0x7f3cc400cb70 comp rx=0 tx=0).stop 2024-09-11T00:52:20.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.970+0000 7f3cd0a00640 1 -- 172.21.15.31:0/2201078080 shutdown_connections 2024-09-11T00:52:20.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.970+0000 7f3cd0a00640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3ca8077f70 0x7f3ca807a430 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.970+0000 7f3cd0a00640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3ccc0ac460 0x7f3ccc0a9350 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.970+0000 7f3cd0a00640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3ccc0c47a0 0x7f3ccc0a8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.970+0000 7f3cd0a00640 1 --2- 172.21.15.31:0/2201078080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ccc0c43c0 0x7f3ccc089640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:20.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.970+0000 7f3cd0a00640 1 -- 172.21.15.31:0/2201078080 >> 172.21.15.31:0/2201078080 conn(0x7f3ccc06caf0 msgr2=0x7f3ccc0c2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:20.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.971+0000 7f3cd0a00640 1 -- 172.21.15.31:0/2201078080 shutdown_connections 2024-09-11T00:52:20.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:20.971+0000 7f3cd0a00640 1 -- 172.21.15.31:0/2201078080 wait complete. 2024-09-11T00:52:21.006 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-11T00:52:21.006 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:52:21.006 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:52:21.029 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:52:21.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:21.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.051+0000 7f6af7000640 1 -- 172.21.15.31:0/3486604198 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6af009fc40 con 0x7f6af00c8ed0 2024-09-11T00:52:21.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.052+0000 7f6ae7e00640 1 -- 172.21.15.31:0/3486604198 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5712 (secure 0 0 0) 0x7f6adc0633a0 con 0x7f6af00c8ed0 2024-09-11T00:52:21.055 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:21.055 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":154,"btime":"2024-09-11T00:52:20:838310+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:20.668747+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:52:21.055 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 154 2024-09-11T00:52:21.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.055+0000 7f6af7000640 1 -- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ad0077f70 msgr2=0x7f6ad007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.055+0000 7f6af7000640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ad0077f70 0x7f6ad007a430 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0000f60 tx=0x7f6ae0009040 comp rx=0 tx=0).stop 2024-09-11T00:52:21.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.055+0000 7f6af7000640 1 -- 172.21.15.31:0/3486604198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6af00c8ed0 msgr2=0x7f6af006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.055+0000 7f6af7000640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6af00c8ed0 0x7f6af006f140 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00ed60 tx=0x7f6adc00c6a0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.056+0000 7f6af7000640 1 -- 172.21.15.31:0/3486604198 shutdown_connections 2024-09-11T00:52:21.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.056+0000 7f6af7000640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6ad0077f70 0x7f6ad007a430 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.056+0000 7f6af7000640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6af0158260 0x7f6af01df710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.056+0000 7f6af7000640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6af00c8ed0 0x7f6af006f140 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.056+0000 7f6af7000640 1 --2- 172.21.15.31:0/3486604198 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6af00c71c0 0x7f6af00a9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.056+0000 7f6af7000640 1 -- 172.21.15.31:0/3486604198 >> 172.21.15.31:0/3486604198 conn(0x7f6af00688e0 msgr2=0x7f6af00c5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:21.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.057+0000 7f6af7000640 1 -- 172.21.15.31:0/3486604198 shutdown_connections 2024-09-11T00:52:21.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.057+0000 7f6af7000640 1 -- 172.21.15.31:0/3486604198 wait complete. 2024-09-11T00:52:21.088 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-11T00:52:21.088 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:52:21.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:21.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.320+0000 7fb64a000640 1 -- 172.21.15.31:0/3491297041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6440c64b0 msgr2=0x7fb64406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.320+0000 7fb64a000640 1 --2- 172.21.15.31:0/3491297041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6440c64b0 0x7fb64406dc60 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb64406f6f0 tx=0x7fb63002a4a0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.320+0000 7fb64a000640 1 -- 172.21.15.31:0/3491297041 shutdown_connections 2024-09-11T00:52:21.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.320+0000 7fb64a000640 1 --2- 172.21.15.31:0/3491297041 >> v2:172.21.15.31:3300/0 conn(0x7fb644070410 0x7fb64415c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.320+0000 7fb64a000640 1 --2- 172.21.15.31:0/3491297041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6440c64b0 0x7fb64406dc60 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.321+0000 7fb64a000640 1 -- 172.21.15.31:0/3491297041 >> 172.21.15.31:0/3491297041 conn(0x7fb64406caf0 msgr2=0x7fb64406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:21.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.322+0000 7fb64a000640 1 -- 172.21.15.31:0/3491297041 shutdown_connections 2024-09-11T00:52:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.322+0000 7fb64a000640 1 -- 172.21.15.31:0/3491297041 wait complete. 2024-09-11T00:52:21.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.325+0000 7fb64a000640 1 Processor -- start 2024-09-11T00:52:21.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.325+0000 7fb64a000640 1 -- start start 2024-09-11T00:52:21.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.326+0000 7fb64a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb644070410 0x7fb644074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:21.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.326+0000 7fb64a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6440c64b0 0x7fb64414f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.326+0000 7fb64a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6440c7240 0x7fb64414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.327+0000 7fb64a000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb644155b90 con 0x7fb644070410 2024-09-11T00:52:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.327+0000 7fb64a000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb6440700f0 con 0x7fb6440c64b0 2024-09-11T00:52:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.327+0000 7fb64a000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb6440c7620 con 0x7fb6440c7240 2024-09-11T00:52:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.327+0000 7fb648c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb644070410 0x7fb644074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.327+0000 7fb648c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb644070410 0x7fb644074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36256/0 (socket says 172.21.15.31:36256) 2024-09-11T00:52:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.327+0000 7fb648c00640 1 -- 172.21.15.31:0/2216547776 learned_addr learned my addr 172.21.15.31:0/2216547776 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:21.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.327+0000 7fb643e00640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6440c64b0 0x7fb64414f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:21.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.327+0000 7fb649600640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6440c7240 0x7fb64414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:21.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.328+0000 7fb648c00640 1 -- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6440c7240 msgr2=0x7fb64414fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.328+0000 7fb648c00640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6440c7240 0x7fb64414fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.328+0000 7fb648c00640 1 -- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6440c64b0 msgr2=0x7fb64414f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.328+0000 7fb648c00640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6440c64b0 0x7fb64414f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.328+0000 7fb648c00640 1 -- 172.21.15.31:0/2216547776 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6441507c0 con 0x7fb644070410 2024-09-11T00:52:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.328+0000 7fb649600640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6440c7240 0x7fb64414fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:52:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.329+0000 7fb648c00640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb644070410 0x7fb644074160 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb63800c3a0 tx=0x7fb63800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:21.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.329+0000 7fb640c00640 1 -- 172.21.15.31:0/2216547776 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638017070 con 0x7fb644070410 2024-09-11T00:52:21.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.329+0000 7fb640c00640 1 -- 172.21.15.31:0/2216547776 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6380035f0 con 0x7fb644070410 2024-09-11T00:52:21.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.329+0000 7fb640c00640 1 -- 172.21.15.31:0/2216547776 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638003a40 con 0x7fb644070410 2024-09-11T00:52:21.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.329+0000 7fb64a000640 1 -- 172.21.15.31:0/2216547776 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6440aa460 con 0x7fb644070410 2024-09-11T00:52:21.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.330+0000 7fb64a000640 1 -- 172.21.15.31:0/2216547776 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6440ac600 con 0x7fb644070410 2024-09-11T00:52:21.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.332+0000 7fb64a000640 1 -- 172.21.15.31:0/2216547776 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6440acaf0 con 0x7fb644070410 2024-09-11T00:52:21.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.334+0000 7fb640c00640 1 -- 172.21.15.31:0/2216547776 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb638003be0 con 0x7fb644070410 2024-09-11T00:52:21.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.336+0000 7fb640c00640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb624078040 0x7fb62407a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:21.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.336+0000 7fb640c00640 1 -- 172.21.15.31:0/2216547776 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7fb63809a070 con 0x7fb644070410 2024-09-11T00:52:21.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.336+0000 7fb643e00640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb624078040 0x7fb62407a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:21.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.338+0000 7fb643e00640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb624078040 0x7fb62407a500 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb630034000 tx=0x7fb630004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:21.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.342+0000 7fb640c00640 1 -- 172.21.15.31:0/2216547776 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb64406e520 con 0x7fb644070410 2024-09-11T00:52:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.401+0000 7fdc6fe00640 1 -- 172.21.15.31:0/3742650765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc7006e850 msgr2=0x7fdc70160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.401+0000 7fdc6fe00640 1 --2- 172.21.15.31:0/3742650765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc7006e850 0x7fdc70160860 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdc70153d20 tx=0x7fdc6802a8c0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.401+0000 7fdc6fe00640 1 -- 172.21.15.31:0/3742650765 shutdown_connections 2024-09-11T00:52:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.401+0000 7fdc6fe00640 1 --2- 172.21.15.31:0/3742650765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc7006e850 0x7fdc70160860 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.401+0000 7fdc6fe00640 1 --2- 172.21.15.31:0/3742650765 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc70150020 0x7fdc700ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.401+0000 7fdc6fe00640 1 -- 172.21.15.31:0/3742650765 >> 172.21.15.31:0/3742650765 conn(0x7fdc70068f10 msgr2=0x7fdc700cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.401+0000 7fdc6fe00640 1 -- 172.21.15.31:0/3742650765 shutdown_connections 2024-09-11T00:52:21.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.401+0000 7fdc6fe00640 1 -- 172.21.15.31:0/3742650765 wait complete. 2024-09-11T00:52:21.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.406+0000 7fdc6fe00640 1 Processor -- start 2024-09-11T00:52:21.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.406+0000 7fdc6fe00640 1 -- start start 2024-09-11T00:52:21.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.410+0000 7fdc6fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7006e850 0x7fdc700be680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.411+0000 7fdc6fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc700ada20 0x7fdc700acc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.411+0000 7fdc6fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc70150020 0x7fdc700c4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.411+0000 7fdc6fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdc70157550 con 0x7fdc7006e850 2024-09-11T00:52:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.411+0000 7fdc6fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc701573d0 con 0x7fdc700ada20 2024-09-11T00:52:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.411+0000 7fdc6fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdc70156960 con 0x7fdc70150020 2024-09-11T00:52:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.412+0000 7fdc6e000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc700ada20 0x7fdc700acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.412+0000 7fdc6ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7006e850 0x7fdc700be680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:21.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.412+0000 7fdc6f400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc70150020 0x7fdc700c4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.413+0000 7fdc6f400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc70150020 0x7fdc700c4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:60176/0 (socket says 172.21.15.31:60176) 2024-09-11T00:52:21.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.413+0000 7fdc6f400640 1 -- 172.21.15.31:0/228642605 learned_addr learned my addr 172.21.15.31:0/228642605 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:21.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.415+0000 7fdc6ea00640 1 -- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc70150020 msgr2=0x7fdc700c4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.415+0000 7fdc6ea00640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc70150020 0x7fdc700c4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.415+0000 7fdc6ea00640 1 -- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc700ada20 msgr2=0x7fdc700acc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.415+0000 7fdc6ea00640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc700ada20 0x7fdc700acc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.415+0000 7fdc6ea00640 1 -- 172.21.15.31:0/228642605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc701f4170 con 0x7fdc7006e850 2024-09-11T00:52:21.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.418+0000 7fdc6ea00640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7006e850 0x7fdc700be680 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000ca80 tx=0x7fdc6000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:21.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.419+0000 7fdc66a00640 1 -- 172.21.15.31:0/228642605 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc60007b20 con 0x7fdc7006e850 2024-09-11T00:52:21.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.419+0000 7fdc6fe00640 1 -- 172.21.15.31:0/228642605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc700be3a0 con 0x7fdc7006e850 2024-09-11T00:52:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.420+0000 7fdc6fe00640 1 -- 172.21.15.31:0/228642605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc700c5900 con 0x7fdc7006e850 2024-09-11T00:52:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.421+0000 7fdc66a00640 1 -- 172.21.15.31:0/228642605 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc60007db0 con 0x7fdc7006e850 2024-09-11T00:52:21.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.421+0000 7fdc66a00640 1 -- 172.21.15.31:0/228642605 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc60007b20 con 0x7fdc7006e850 2024-09-11T00:52:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.426+0000 7fdc6fe00640 1 -- 172.21.15.31:0/228642605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc48005290 con 0x7fdc7006e850 2024-09-11T00:52:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.427+0000 7fdc66a00640 1 -- 172.21.15.31:0/228642605 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdc6000f450 con 0x7fdc7006e850 2024-09-11T00:52:21.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.431+0000 7fdc66a00640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc50078370 0x7fdc5007a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:21.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.432+0000 7fdc66a00640 1 -- 172.21.15.31:0/228642605 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7fdc60007920 con 0x7fdc7006e850 2024-09-11T00:52:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.433+0000 7fdc6e000640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc50078370 0x7fdc5007a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.434+0000 7fdc6e000640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc50078370 0x7fdc5007a830 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c007fd0 tx=0x7fdc5c001410 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:21.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.440+0000 7fdc66a00640 1 -- 172.21.15.31:0/228642605 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fdc60014030 con 0x7fdc7006e850 2024-09-11T00:52:21.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.771+0000 7fb64a000640 1 -- 172.21.15.31:0/2216547776 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb644155770 con 0x7fb644070410 2024-09-11T00:52:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.774+0000 7fb640c00640 1 -- 172.21.15.31:0/2216547776 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5712 (secure 0 0 0) 0x7fb638061800 con 0x7fb644070410 2024-09-11T00:52:21.777 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:21.777 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":154,"btime":"2024-09-11T00:52:20:838310+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:20.668747+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:52:21.777 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 154 2024-09-11T00:52:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.778+0000 7fb64a000640 1 -- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb624078040 msgr2=0x7fb62407a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.778+0000 7fb64a000640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb624078040 0x7fb62407a500 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb630034000 tx=0x7fb630004730 comp rx=0 tx=0).stop 2024-09-11T00:52:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.778+0000 7fb64a000640 1 -- 172.21.15.31:0/2216547776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb644070410 msgr2=0x7fb644074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.778+0000 7fb64a000640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb644070410 0x7fb644074160 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb63800c3a0 tx=0x7fb63800c860 comp rx=0 tx=0).stop 2024-09-11T00:52:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.778+0000 7fb64a000640 1 -- 172.21.15.31:0/2216547776 shutdown_connections 2024-09-11T00:52:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.778+0000 7fb64a000640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb624078040 0x7fb62407a500 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.778+0000 7fb64a000640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6440c7240 0x7fb64414fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.778+0000 7fb64a000640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6440c64b0 0x7fb64414f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.778+0000 7fb64a000640 1 --2- 172.21.15.31:0/2216547776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb644070410 0x7fb644074160 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.778+0000 7fb64a000640 1 -- 172.21.15.31:0/2216547776 >> 172.21.15.31:0/2216547776 conn(0x7fb64406caf0 msgr2=0x7fb6440c43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:21.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.779+0000 7fb64a000640 1 -- 172.21.15.31:0/2216547776 shutdown_connections 2024-09-11T00:52:21.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.779+0000 7fb64a000640 1 -- 172.21.15.31:0/2216547776 wait complete. 2024-09-11T00:52:21.808 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:52:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.848+0000 7fdc6fe00640 1 -- 172.21.15.31:0/228642605 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc48005580 con 0x7fdc7006e850 2024-09-11T00:52:21.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.857+0000 7fdc66a00640 1 -- 172.21.15.31:0/228642605 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5712 (secure 0 0 0) 0x7fdc60062230 con 0x7fdc7006e850 2024-09-11T00:52:21.863 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:21.863 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":154,"btime":"2024-09-11T00:52:20:838310+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:20.668747+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:52:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 154 2024-09-11T00:52:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 -- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc50078370 msgr2=0x7fdc5007a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc50078370 0x7fdc5007a830 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c007fd0 tx=0x7fdc5c001410 comp rx=0 tx=0).stop 2024-09-11T00:52:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 -- 172.21.15.31:0/228642605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7006e850 msgr2=0x7fdc700be680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7006e850 0x7fdc700be680 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000ca80 tx=0x7fdc6000cf40 comp rx=0 tx=0).stop 2024-09-11T00:52:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 -- 172.21.15.31:0/228642605 shutdown_connections 2024-09-11T00:52:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc50078370 0x7fdc5007a830 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdc70150020 0x7fdc700c4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc700ada20 0x7fdc700acc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 --2- 172.21.15.31:0/228642605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc7006e850 0x7fdc700be680 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:21.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 -- 172.21.15.31:0/228642605 >> 172.21.15.31:0/228642605 conn(0x7fdc70068f10 msgr2=0x7fdc701576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:21.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 -- 172.21.15.31:0/228642605 shutdown_connections 2024-09-11T00:52:21.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:21.864+0000 7fdc6fe00640 1 -- 172.21.15.31:0/228642605 wait complete. 2024-09-11T00:52:21.896 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:52:20:838310+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 146, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7143": { "addr": "172.21.15.204:6835/1981755707", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 1981755707, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 1981755707, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7143, "incarnation": 143, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:50:17.486435+0000", "qdb_cluster": [ 7143 ], "qdb_leader": 7143, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7143 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7255": { "addr": "172.21.15.31:6835/1855324855", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 1855324855, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 1855324855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7255, "incarnation": 153, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:52:20.668747+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7255 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:52:22.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:52:22.787 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:52:23.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:23.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:24.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.109+0000 7fe892a00640 1 -- 172.21.15.31:0/4151920245 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c009bd0 con 0x7fe878004270 2024-09-11T00:52:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.109+0000 7fe89b000640 1 -- 172.21.15.31:0/4151920245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe878004270 msgr2=0x7fe87800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.109+0000 7fe89b000640 1 --2- 172.21.15.31:0/4151920245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe878004270 0x7fe87800b9c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe88c00d360 tx=0x7fe878005830 comp rx=0 tx=0).stop 2024-09-11T00:52:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.110+0000 7fe89b000640 1 -- 172.21.15.31:0/4151920245 shutdown_connections 2024-09-11T00:52:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.110+0000 7fe89b000640 1 --2- 172.21.15.31:0/4151920245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe878004270 0x7fe87800b9c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.110+0000 7fe89b000640 1 -- 172.21.15.31:0/4151920245 >> 172.21.15.31:0/4151920245 conn(0x7fe894068f10 msgr2=0x7fe8940cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.110+0000 7fe89b000640 1 -- 172.21.15.31:0/4151920245 shutdown_connections 2024-09-11T00:52:24.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.111+0000 7fe89b000640 1 -- 172.21.15.31:0/4151920245 wait complete. 2024-09-11T00:52:24.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.114+0000 7fe89b000640 1 Processor -- start 2024-09-11T00:52:24.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.115+0000 7fe89b000640 1 -- start start 2024-09-11T00:52:24.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.117+0000 7fe89b000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe878004270 0x7fe8940bec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:24.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.118+0000 7fe893e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe878004270 0x7fe8940bec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:24.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.118+0000 7fe893e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe878004270 0x7fe8940bec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36292/0 (socket says 172.21.15.31:36292) 2024-09-11T00:52:24.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.118+0000 7fe893e00640 1 -- 172.21.15.31:0/2447374909 learned_addr learned my addr 172.21.15.31:0/2447374909 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:24.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.118+0000 7fe89b000640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8940ae170 0x7fe8940c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:24.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.119+0000 7fe89b000640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8941eb300 0x7fe894146ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:24.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.119+0000 7fe893400640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8940ae170 0x7fe8940c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:24.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.120+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe8940bf390 con 0x7fe878004270 2024-09-11T00:52:24.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.120+0000 7fe899000640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8941eb300 0x7fe894146ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:24.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.120+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe894156540 con 0x7fe8941eb300 2024-09-11T00:52:24.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.120+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe8940bf510 con 0x7fe8940ae170 2024-09-11T00:52:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.125+0000 7fe893e00640 1 -- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8940ae170 msgr2=0x7fe8940c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.126+0000 7fe893e00640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8940ae170 0x7fe8940c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.126+0000 7fe893e00640 1 -- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8941eb300 msgr2=0x7fe894146ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.126+0000 7fe893e00640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8941eb300 0x7fe894146ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.126+0000 7fe893e00640 1 -- 172.21.15.31:0/2447374909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8941ebcf0 con 0x7fe878004270 2024-09-11T00:52:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.126+0000 7fe893400640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8940ae170 0x7fe8940c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:52:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.126+0000 7fe899000640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8941eb300 0x7fe894146ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:52:24.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.127+0000 7fe893e00640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe878004270 0x7fe8940bec80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fe884001a10 tx=0x7fe88400ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:24.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.128+0000 7fe88be00640 1 -- 172.21.15.31:0/2447374909 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884015070 con 0x7fe878004270 2024-09-11T00:52:24.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.128+0000 7fe88be00640 1 -- 172.21.15.31:0/2447374909 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe884010980 con 0x7fe878004270 2024-09-11T00:52:24.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.128+0000 7fe88be00640 1 -- 172.21.15.31:0/2447374909 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884010dd0 con 0x7fe878004270 2024-09-11T00:52:24.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.128+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8940c3be0 con 0x7fe878004270 2024-09-11T00:52:24.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.132+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8940cb260 con 0x7fe878004270 2024-09-11T00:52:24.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.132+0000 7fe88be00640 1 -- 172.21.15.31:0/2447374909 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe884011460 con 0x7fe878004270 2024-09-11T00:52:24.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.135+0000 7fe88be00640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe874077ff0 0x7fe87407a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:24.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.135+0000 7fe88be00640 1 -- 172.21.15.31:0/2447374909 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7fe88409a210 con 0x7fe878004270 2024-09-11T00:52:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.138+0000 7fe893400640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe874077ff0 0x7fe87407a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.138+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8940766d0 con 0x7fe878004270 2024-09-11T00:52:24.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.140+0000 7fe893400640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe874077ff0 0x7fe87407a4b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe89406a630 tx=0x7fe880006b30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.161+0000 7fe88be00640 1 -- 172.21.15.31:0/2447374909 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe88409f060 con 0x7fe878004270 2024-09-11T00:52:24.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.210+0000 7fc6cca00640 1 -- 172.21.15.31:0/3111980419 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00b06a0 con 0x7fc6c8049a50 2024-09-11T00:52:24.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.212+0000 7fc6d5000640 1 -- 172.21.15.31:0/3111980419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc6c8049a50 msgr2=0x7fc6c80d6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:24.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.212+0000 7fc6d5000640 1 --2- 172.21.15.31:0/3111980419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc6c8049a50 0x7fc6c80d6270 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0052430 tx=0x7fc6d00a7d60 comp rx=0 tx=0).stop 2024-09-11T00:52:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.212+0000 7fc6d5000640 1 -- 172.21.15.31:0/3111980419 shutdown_connections 2024-09-11T00:52:24.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.212+0000 7fc6d5000640 1 --2- 172.21.15.31:0/3111980419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc6c8049a50 0x7fc6c80d6270 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.212+0000 7fc6d5000640 1 --2- 172.21.15.31:0/3111980419 >> v2:172.21.15.31:3300/0 conn(0x7fc6c8046560 0x7fc6c80d5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.212+0000 7fc6d5000640 1 -- 172.21.15.31:0/3111980419 >> 172.21.15.31:0/3111980419 conn(0x7fc6c8047c50 msgr2=0x7fc6c8048030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:24.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.213+0000 7fc6d5000640 1 -- 172.21.15.31:0/3111980419 shutdown_connections 2024-09-11T00:52:24.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.213+0000 7fc6d5000640 1 -- 172.21.15.31:0/3111980419 wait complete. 2024-09-11T00:52:24.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.220+0000 7fc6d5000640 1 Processor -- start 2024-09-11T00:52:24.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.221+0000 7fc6d5000640 1 -- start start 2024-09-11T00:52:24.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.222+0000 7fc6d5000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc6c8046560 0x7fc6c80c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:24.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.222+0000 7fc6d5000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc6c8049a50 0x7fc6c80c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:24.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.223+0000 7fc6d5000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6c80d34b0 0x7fc6c80cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:24.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.223+0000 7fc6d5000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c8049370 con 0x7fc6c80d34b0 2024-09-11T00:52:24.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.223+0000 7fc6d5000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c8046110 con 0x7fc6c8046560 2024-09-11T00:52:24.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.223+0000 7fc6d5000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c8049650 con 0x7fc6c8049a50 2024-09-11T00:52:24.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.225+0000 7fc6ce800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6c80d34b0 0x7fc6c80cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:24.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.226+0000 7fc6ce800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6c80d34b0 0x7fc6c80cbbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36298/0 (socket says 172.21.15.31:36298) 2024-09-11T00:52:24.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.226+0000 7fc6ce800640 1 -- 172.21.15.31:0/1660394602 learned_addr learned my addr 172.21.15.31:0/1660394602 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:24.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.226+0000 7fc6cd400640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc6c8049a50 0x7fc6c80c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:24.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.225+0000 7fc6cde00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc6c8046560 0x7fc6c80c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:24.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.226+0000 7fc6cde00640 1 -- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc6c8049a50 msgr2=0x7fc6c80c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:24.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.226+0000 7fc6cde00640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc6c8049a50 0x7fc6c80c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.226+0000 7fc6cde00640 1 -- 172.21.15.31:0/1660394602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6c80d34b0 msgr2=0x7fc6c80cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:24.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.227+0000 7fc6cde00640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6c80d34b0 0x7fc6c80cbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.227+0000 7fc6cde00640 1 -- 172.21.15.31:0/1660394602 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c80d1f40 con 0x7fc6c8046560 2024-09-11T00:52:24.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.228+0000 7fc6cde00640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc6c8046560 0x7fc6c80c6550 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000b410 tx=0x7fc6c000b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:24.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.229+0000 7fc6c6000640 1 -- 172.21.15.31:0/1660394602 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0015020 con 0x7fc6c8046560 2024-09-11T00:52:24.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.229+0000 7fc6c6000640 1 -- 172.21.15.31:0/1660394602 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6c0011920 con 0x7fc6c8046560 2024-09-11T00:52:24.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.229+0000 7fc6ce800640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6c80d34b0 0x7fc6c80cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:52:24.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.229+0000 7fc6c6000640 1 -- 172.21.15.31:0/1660394602 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0011d90 con 0x7fc6c8046560 2024-09-11T00:52:24.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.230+0000 7fc6d5000640 1 -- 172.21.15.31:0/1660394602 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c80d21c0 con 0x7fc6c8046560 2024-09-11T00:52:24.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.232+0000 7fc6d5000640 1 -- 172.21.15.31:0/1660394602 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c80d2780 con 0x7fc6c8046560 2024-09-11T00:52:24.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.234+0000 7fc6bb400640 1 -- 172.21.15.31:0/1660394602 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c800a230 con 0x7fc6c8046560 2024-09-11T00:52:24.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.235+0000 7fc6c6000640 1 -- 172.21.15.31:0/1660394602 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc6c0022060 con 0x7fc6c8046560 2024-09-11T00:52:24.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.236+0000 7fc6c6000640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc6b0077f70 0x7fc6b007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:24.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.237+0000 7fc6c6000640 1 -- 172.21.15.31:0/1660394602 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7fc6c009a6f0 con 0x7fc6c8046560 2024-09-11T00:52:24.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.239+0000 7fc6cd400640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc6b0077f70 0x7fc6b007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:24.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.242+0000 7fc6cd400640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc6b0077f70 0x7fc6b007a430 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc005fd0 tx=0x7fc6bc005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:24.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.250+0000 7fc6c6000640 1 -- 172.21.15.31:0/1660394602 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc6c80d1f40 con 0x7fc6c8046560 2024-09-11T00:52:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.738+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8941530b0 con 0x7fe878004270 2024-09-11T00:52:24.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.739+0000 7fe88be00640 1 -- 172.21.15.31:0/2447374909 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+5716 (secure 0 0 0) 0x7fe8940b1610 con 0x7fe878004270 2024-09-11T00:52:24.745 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:24.745 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":156,"btime":"2024-09-11T00:52:22:861020+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:24.745 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 156 2024-09-11T00:52:24.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.744+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe874077ff0 msgr2=0x7fe87407a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:24.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.744+0000 7fe89b000640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe874077ff0 0x7fe87407a4b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe89406a630 tx=0x7fe880006b30 comp rx=0 tx=0).stop 2024-09-11T00:52:24.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.744+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe878004270 msgr2=0x7fe8940bec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:24.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.744+0000 7fe89b000640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe878004270 0x7fe8940bec80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fe884001a10 tx=0x7fe88400ea20 comp rx=0 tx=0).stop 2024-09-11T00:52:24.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.745+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 shutdown_connections 2024-09-11T00:52:24.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.745+0000 7fe89b000640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe874077ff0 0x7fe87407a4b0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.745+0000 7fe89b000640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8941eb300 0x7fe894146ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.745+0000 7fe89b000640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe8940ae170 0x7fe8940c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.746+0000 7fe89b000640 1 --2- 172.21.15.31:0/2447374909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe878004270 0x7fe8940bec80 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.746+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 >> 172.21.15.31:0/2447374909 conn(0x7fe894068f10 msgr2=0x7fe89414dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:24.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.747+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 shutdown_connections 2024-09-11T00:52:24.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.747+0000 7fe89b000640 1 -- 172.21.15.31:0/2447374909 wait complete. 2024-09-11T00:52:24.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.748+0000 7fc6bb400640 1 -- 172.21.15.31:0/1660394602 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6c80d5640 con 0x7fc6c8046560 2024-09-11T00:52:24.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.750+0000 7fc6c6000640 1 -- 172.21.15.31:0/1660394602 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+5716 (secure 0 0 0) 0x7fc6c00624b0 con 0x7fc6c8046560 2024-09-11T00:52:24.753 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:24.753 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":156,"btime":"2024-09-11T00:52:22:861020+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:24.753 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 156 2024-09-11T00:52:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.754+0000 7fc6bb400640 1 -- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc6b0077f70 msgr2=0x7fc6b007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.754+0000 7fc6bb400640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc6b0077f70 0x7fc6b007a430 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc005fd0 tx=0x7fc6bc005850 comp rx=0 tx=0).stop 2024-09-11T00:52:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.754+0000 7fc6bb400640 1 -- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc6c8046560 msgr2=0x7fc6c80c6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.754+0000 7fc6bb400640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc6c8046560 0x7fc6c80c6550 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000b410 tx=0x7fc6c000b8d0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.755+0000 7fc6bb400640 1 -- 172.21.15.31:0/1660394602 shutdown_connections 2024-09-11T00:52:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.755+0000 7fc6bb400640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc6b0077f70 0x7fc6b007a430 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.755+0000 7fc6bb400640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6c80d34b0 0x7fc6c80cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.755+0000 7fc6bb400640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc6c8049a50 0x7fc6c80c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.755+0000 7fc6bb400640 1 --2- 172.21.15.31:0/1660394602 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc6c8046560 0x7fc6c80c6550 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.755+0000 7fc6bb400640 1 -- 172.21.15.31:0/1660394602 >> 172.21.15.31:0/1660394602 conn(0x7fc6c8047c50 msgr2=0x7fc6c8006630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.757+0000 7fc6bb400640 1 -- 172.21.15.31:0/1660394602 shutdown_connections 2024-09-11T00:52:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:24.757+0000 7fc6bb400640 1 -- 172.21.15.31:0/1660394602 wait complete. 2024-09-11T00:52:24.783 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:52:24.788 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=6350 2024-09-11T00:52:24.788 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.b 2024-09-11T00:52:26.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:27.177 INFO:tasks.ceph.mds.c.smithi031.stderr:2024-09-11T00:52:27.174+0000 7ff310800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:52:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.745+0000 7f3e68400640 1 -- 172.21.15.31:0/45140783 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e600749c0 msgr2=0x7f3e60073320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.745+0000 7f3e68400640 1 --2- 172.21.15.31:0/45140783 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e600749c0 0x7f3e60073320 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3e600ab900 tx=0x7f3e5802a6f0 comp rx=0 tx=0).stop 2024-09-11T00:52:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.745+0000 7f3e68400640 1 -- 172.21.15.31:0/45140783 shutdown_connections 2024-09-11T00:52:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.745+0000 7f3e68400640 1 --2- 172.21.15.31:0/45140783 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e600749c0 0x7f3e60073320 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.745+0000 7f3e68400640 1 --2- 172.21.15.31:0/45140783 >> v2:172.21.15.31:3300/0 conn(0x7f3e600718b0 0x7f3e60074480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.745+0000 7f3e68400640 1 --2- 172.21.15.31:0/45140783 >> v2:172.21.15.204:3300/0 conn(0x7f3e600c8cb0 0x7f3e60070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.745+0000 7f3e68400640 1 -- 172.21.15.31:0/45140783 >> 172.21.15.31:0/45140783 conn(0x7f3e6006dc00 msgr2=0x7f3e6006e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.745+0000 7f3e68400640 1 -- 172.21.15.31:0/45140783 shutdown_connections 2024-09-11T00:52:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.745+0000 7f3e68400640 1 -- 172.21.15.31:0/45140783 wait complete. 2024-09-11T00:52:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.748+0000 7f3e68400640 1 Processor -- start 2024-09-11T00:52:28.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.748+0000 7f3e68400640 1 -- start start 2024-09-11T00:52:28.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.750+0000 7f3e68400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e600718b0 0x7f3e600c24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:28.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.750+0000 7f3e68400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e600749c0 0x7f3e60156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:28.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.750+0000 7f3e68400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e600c8cb0 0x7f3e601e8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:28.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.750+0000 7f3e68400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60074df0 con 0x7f3e600749c0 2024-09-11T00:52:28.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.750+0000 7f3e68400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3e60153980 con 0x7f3e600718b0 2024-09-11T00:52:28.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.750+0000 7f3e68400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3e600c9090 con 0x7f3e600c8cb0 2024-09-11T00:52:28.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.750+0000 7f3e66400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e600c8cb0 0x7f3e601e8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:28.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.750+0000 7f3e66400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e600c8cb0 0x7f3e601e8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:33610/0 (socket says 172.21.15.31:33610) 2024-09-11T00:52:28.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.750+0000 7f3e66400640 1 -- 172.21.15.31:0/2849276711 learned_addr learned my addr 172.21.15.31:0/2849276711 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:28.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.750+0000 7f3e65000640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e600749c0 0x7f3e60156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:28.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.751+0000 7f3e65a00640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e600718b0 0x7f3e600c24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:28.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.751+0000 7f3e65000640 1 -- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e600c8cb0 msgr2=0x7f3e601e8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.751+0000 7f3e65000640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e600c8cb0 0x7f3e601e8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.751+0000 7f3e65000640 1 -- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e600718b0 msgr2=0x7f3e600c24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.751+0000 7f3e65000640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e600718b0 0x7f3e600c24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.751+0000 7f3e65000640 1 -- 172.21.15.31:0/2849276711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e601ef620 con 0x7f3e600749c0 2024-09-11T00:52:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.752+0000 7f3e66400640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e600c8cb0 0x7f3e601e8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:52:28.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.752+0000 7f3e65000640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e600749c0 0x7f3e60156180 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00c970 tx=0x7f3e4c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:28.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.753+0000 7f3e5d600640 1 -- 172.21.15.31:0/2849276711 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c007bb0 con 0x7f3e600749c0 2024-09-11T00:52:28.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.753+0000 7f3e5d600640 1 -- 172.21.15.31:0/2849276711 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e4c007e40 con 0x7f3e600749c0 2024-09-11T00:52:28.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.753+0000 7f3e5d600640 1 -- 172.21.15.31:0/2849276711 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c00f6b0 con 0x7f3e600749c0 2024-09-11T00:52:28.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.753+0000 7f3e68400640 1 -- 172.21.15.31:0/2849276711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e601ef900 con 0x7f3e600749c0 2024-09-11T00:52:28.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.753+0000 7f3e68400640 1 -- 172.21.15.31:0/2849276711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e601efec0 con 0x7f3e600749c0 2024-09-11T00:52:28.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.755+0000 7f3e68400640 1 -- 172.21.15.31:0/2849276711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e3c005290 con 0x7f3e600749c0 2024-09-11T00:52:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.758+0000 7f3e5d600640 1 -- 172.21.15.31:0/2849276711 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3e4c005020 con 0x7f3e600749c0 2024-09-11T00:52:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.759+0000 7f3e5d600640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e40078070 0x7f3e4007a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.760+0000 7f3e5d600640 1 -- 172.21.15.31:0/2849276711 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7f3e4c09b480 con 0x7f3e600749c0 2024-09-11T00:52:28.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.762+0000 7f3e65a00640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e40078070 0x7f3e4007a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.765+0000 7f3e5d600640 1 -- 172.21.15.31:0/2849276711 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3e4c014030 con 0x7f3e600749c0 2024-09-11T00:52:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:28.766+0000 7f3e65a00640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e40078070 0x7f3e4007a530 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3e54005fd0 tx=0x7f3e54009260 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:29.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.338+0000 7f3e68400640 1 -- 172.21.15.31:0/2849276711 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e3c005580 con 0x7f3e600749c0 2024-09-11T00:52:29.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.340+0000 7f3e5d600640 1 -- 172.21.15.31:0/2849276711 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+5716 (secure 0 0 0) 0x7f3e4c0631d0 con 0x7f3e600749c0 2024-09-11T00:52:29.345 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 156 2024-09-11T00:52:29.346 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:29.346 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":156,"btime":"2024-09-11T00:52:22:861020+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:29.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.344+0000 7f3e68400640 1 -- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e40078070 msgr2=0x7f3e4007a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:29.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.344+0000 7f3e68400640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e40078070 0x7f3e4007a530 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3e54005fd0 tx=0x7f3e54009260 comp rx=0 tx=0).stop 2024-09-11T00:52:29.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.345+0000 7f3e68400640 1 -- 172.21.15.31:0/2849276711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e600749c0 msgr2=0x7f3e60156180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:29.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.345+0000 7f3e68400640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e600749c0 0x7f3e60156180 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00c970 tx=0x7f3e4c00ce30 comp rx=0 tx=0).stop 2024-09-11T00:52:29.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.349+0000 7f3e68400640 1 -- 172.21.15.31:0/2849276711 shutdown_connections 2024-09-11T00:52:29.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.349+0000 7f3e68400640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3e40078070 0x7f3e4007a530 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:29.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.349+0000 7f3e68400640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3e600c8cb0 0x7f3e601e8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:29.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.349+0000 7f3e68400640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e600749c0 0x7f3e60156180 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:29.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.349+0000 7f3e68400640 1 --2- 172.21.15.31:0/2849276711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e600718b0 0x7f3e600c24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:29.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.349+0000 7f3e68400640 1 -- 172.21.15.31:0/2849276711 >> 172.21.15.31:0/2849276711 conn(0x7f3e6006dc00 msgr2=0x7f3e60154210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.349+0000 7f3e68400640 1 -- 172.21.15.31:0/2849276711 shutdown_connections 2024-09-11T00:52:29.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:29.351+0000 7f3e68400640 1 -- 172.21.15.31:0/2849276711 wait complete. 2024-09-11T00:52:31.284 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:52:32.789 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-11T00:52:32.789 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-11T00:52:32.789 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-11T00:52:32.794 INFO:tasks.ceph.mds.b:Started 2024-09-11T00:52:32.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:32.997 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:52:33.285 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:35.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.217+0000 7f0effa00640 1 -- 172.21.15.31:0/1058562246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef80c2ff0 msgr2=0x7f0ef80c58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.217+0000 7f0effa00640 1 --2- 172.21.15.31:0/1058562246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef80c2ff0 0x7f0ef80c58a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8076530 tx=0x7f0ef002aac0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.218+0000 7f0effa00640 1 -- 172.21.15.31:0/1058562246 shutdown_connections 2024-09-11T00:52:35.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.218+0000 7f0effa00640 1 --2- 172.21.15.31:0/1058562246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef80c2ff0 0x7f0ef80c58a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.218+0000 7f0effa00640 1 --2- 172.21.15.31:0/1058562246 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ef8070570 0x7f0ef8155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.218+0000 7f0effa00640 1 --2- 172.21.15.31:0/1058562246 >> v2:172.21.15.204:3300/0 conn(0x7f0ef806e480 0x7f0ef806e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.218+0000 7f0effa00640 1 -- 172.21.15.31:0/1058562246 >> 172.21.15.31:0/1058562246 conn(0x7f0ef80a9750 msgr2=0x7f0ef80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:35.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.219+0000 7f0effa00640 1 -- 172.21.15.31:0/1058562246 shutdown_connections 2024-09-11T00:52:35.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.219+0000 7f0effa00640 1 -- 172.21.15.31:0/1058562246 wait complete. 2024-09-11T00:52:35.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.221+0000 7f1c38c00640 1 -- 172.21.15.31:0/186102884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c180037a0 msgr2=0x7f1c18003c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.221+0000 7f1c38c00640 1 --2- 172.21.15.31:0/186102884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c180037a0 0x7f1c18003c00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1c20006d80 tx=0x7f1c18005830 comp rx=0 tx=0).stop 2024-09-11T00:52:35.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.221+0000 7f1c38c00640 1 -- 172.21.15.31:0/186102884 shutdown_connections 2024-09-11T00:52:35.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.221+0000 7f1c38c00640 1 --2- 172.21.15.31:0/186102884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c180037a0 0x7f1c18003c00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.221+0000 7f1c38c00640 1 -- 172.21.15.31:0/186102884 >> 172.21.15.31:0/186102884 conn(0x7f1c34068f10 msgr2=0x7f1c340cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:35.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.221+0000 7f0effa00640 1 Processor -- start 2024-09-11T00:52:35.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.221+0000 7f1c38c00640 1 -- 172.21.15.31:0/186102884 shutdown_connections 2024-09-11T00:52:35.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.221+0000 7f1c38c00640 1 -- 172.21.15.31:0/186102884 wait complete. 2024-09-11T00:52:35.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.221+0000 7f0effa00640 1 -- start start 2024-09-11T00:52:35.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0effa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef806e480 0x7f0ef80ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0effa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ef8070570 0x7f0ef8148710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0effa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ef80c2ff0 0x7f0ef8073930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0effa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef814fb20 con 0x7f0ef806e480 2024-09-11T00:52:35.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0effa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef806ef90 con 0x7f0ef80c2ff0 2024-09-11T00:52:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0effa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef80c7690 con 0x7f0ef8070570 2024-09-11T00:52:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0efd000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef806e480 0x7f0ef80ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0ef7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ef8070570 0x7f0ef8148710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0efd000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef806e480 0x7f0ef80ac910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51432/0 (socket says 172.21.15.31:51432) 2024-09-11T00:52:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0efd000640 1 -- 172.21.15.31:0/802844309 learned_addr learned my addr 172.21.15.31:0/802844309 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0efda00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ef80c2ff0 0x7f0ef8073930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0ef7e00640 1 -- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ef80c2ff0 msgr2=0x7f0ef8073930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.223+0000 7f0ef7e00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ef80c2ff0 0x7f0ef8073930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.224+0000 7f0ef7e00640 1 -- 172.21.15.31:0/802844309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef806e480 msgr2=0x7f0ef80ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.224+0000 7f1c38c00640 1 Processor -- start 2024-09-11T00:52:35.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.224+0000 7f0ef7e00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef806e480 0x7f0ef80ac910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.224+0000 7f0ef7e00640 1 -- 172.21.15.31:0/802844309 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef8074fb0 con 0x7f0ef8070570 2024-09-11T00:52:35.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.224+0000 7f0efd000640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef806e480 0x7f0ef80ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:52:35.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.224+0000 7f0ef7e00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ef8070570 0x7f0ef8148710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4016f50 tx=0x7f0ee4014550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:35.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.224+0000 7f1c38c00640 1 -- start start 2024-09-11T00:52:35.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.226+0000 7f1c38c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c180037a0 0x7f1c340b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.226+0000 7f1c38c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c340c2d10 0x7f1c340ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c38c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c340c34e0 0x7f1c3414ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c38c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1c340c6cb0 con 0x7f1c180037a0 2024-09-11T00:52:35.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c38c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34156f30 con 0x7f1c340c2d10 2024-09-11T00:52:35.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c38c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1c341538d0 con 0x7f1c340c34e0 2024-09-11T00:52:35.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c31e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c180037a0 0x7f1c340b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c31e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c180037a0 0x7f1c340b0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51436/0 (socket says 172.21.15.31:51436) 2024-09-11T00:52:35.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c31e00640 1 -- 172.21.15.31:0/198884824 learned_addr learned my addr 172.21.15.31:0/198884824 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:35.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.226+0000 7f0ef4c00640 1 -- 172.21.15.31:0/802844309 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4004230 con 0x7f0ef8070570 2024-09-11T00:52:35.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c31400640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c340c2d10 0x7f1c340ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c32800640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c340c34e0 0x7f1c3414ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c31e00640 1 -- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c340c34e0 msgr2=0x7f1c3414ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c31e00640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c340c34e0 0x7f1c3414ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f1c31e00640 1 -- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c340c2d10 msgr2=0x7f1c340ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f0ef4c00640 1 -- 172.21.15.31:0/802844309 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ee40043f0 con 0x7f0ef8070570 2024-09-11T00:52:35.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f0ef4c00640 1 -- 172.21.15.31:0/802844309 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee400fc30 con 0x7f0ef8070570 2024-09-11T00:52:35.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f0effa00640 1 -- 172.21.15.31:0/802844309 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef81e88f0 con 0x7f0ef8070570 2024-09-11T00:52:35.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.227+0000 7f0effa00640 1 -- 172.21.15.31:0/802844309 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef8153c00 con 0x7f0ef8070570 2024-09-11T00:52:35.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.228+0000 7f1c31e00640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c340c2d10 0x7f1c340ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.228+0000 7f1c31e00640 1 -- 172.21.15.31:0/198884824 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c341e7c60 con 0x7f1c180037a0 2024-09-11T00:52:35.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.228+0000 7f1c31e00640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c180037a0 0x7f1c340b0ca0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400c730 tx=0x7f1c2400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:35.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.228+0000 7f0effa00640 1 -- 172.21.15.31:0/802844309 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ed4005290 con 0x7f0ef8070570 2024-09-11T00:52:35.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.229+0000 7f1c2a000640 1 -- 172.21.15.31:0/198884824 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24016020 con 0x7f1c180037a0 2024-09-11T00:52:35.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.229+0000 7f1c2a000640 1 -- 172.21.15.31:0/198884824 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c2400a930 con 0x7f1c180037a0 2024-09-11T00:52:35.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.229+0000 7f1c2a000640 1 -- 172.21.15.31:0/198884824 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400ad60 con 0x7f1c180037a0 2024-09-11T00:52:35.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.229+0000 7f1c38c00640 1 -- 172.21.15.31:0/198884824 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c340c3990 con 0x7f1c180037a0 2024-09-11T00:52:35.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.229+0000 7f1c38c00640 1 -- 172.21.15.31:0/198884824 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c340c5ca0 con 0x7f1c180037a0 2024-09-11T00:52:35.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.231+0000 7f1c38c00640 1 -- 172.21.15.31:0/198884824 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c340cc350 con 0x7f1c180037a0 2024-09-11T00:52:35.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.231+0000 7f0ef4c00640 1 -- 172.21.15.31:0/802844309 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0ee4011090 con 0x7f0ef8070570 2024-09-11T00:52:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.232+0000 7f0ef4c00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0ed8077fc0 0x7f0ed807a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.233+0000 7f0ef4c00640 1 -- 172.21.15.31:0/802844309 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7f0ee40a35c0 con 0x7f0ef8070570 2024-09-11T00:52:35.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.233+0000 7f0efd000640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0ed8077fc0 0x7f0ed807a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.233+0000 7f1c2a000640 1 -- 172.21.15.31:0/198884824 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1c2401b400 con 0x7f1c180037a0 2024-09-11T00:52:35.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.235+0000 7f0efd000640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0ed8077fc0 0x7f0ed807a480 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0eec005fd0 tx=0x7f0eec00b440 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.235+0000 7f1c2a000640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c14077fc0 0x7f1c1407a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.235+0000 7f1c2a000640 1 -- 172.21.15.31:0/198884824 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7f1c2409ae90 con 0x7f1c180037a0 2024-09-11T00:52:35.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.236+0000 7f1c31400640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c14077fc0 0x7f1c1407a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.237+0000 7f1c31400640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c14077fc0 0x7f1c1407a480 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1c20006d50 tx=0x7f1c340c8280 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:35.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.241+0000 7f1c2a000640 1 -- 172.21.15.31:0/198884824 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1c24020020 con 0x7f1c180037a0 2024-09-11T00:52:35.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.242+0000 7f0ef4c00640 1 -- 172.21.15.31:0/802844309 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0ee4018040 con 0x7f0ef8070570 2024-09-11T00:52:35.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.589+0000 7f1c38c00640 1 -- 172.21.15.31:0/198884824 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c34153e90 con 0x7f1c180037a0 2024-09-11T00:52:35.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.590+0000 7f1c2a000640 1 -- 172.21.15.31:0/198884824 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6545 (secure 0 0 0) 0x7f1c2409c090 con 0x7f1c180037a0 2024-09-11T00:52:35.592 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:35.592 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":157,"btime":"2024-09-11T00:52:33:898858+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:35.592 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 157 2024-09-11T00:52:35.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.593+0000 7f1c38c00640 1 -- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c14077fc0 msgr2=0x7f1c1407a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.593+0000 7f1c38c00640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c14077fc0 0x7f1c1407a480 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1c20006d50 tx=0x7f1c340c8280 comp rx=0 tx=0).stop 2024-09-11T00:52:35.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.593+0000 7f1c38c00640 1 -- 172.21.15.31:0/198884824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c180037a0 msgr2=0x7f1c340b0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.593+0000 7f1c38c00640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c180037a0 0x7f1c340b0ca0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400c730 tx=0x7f1c2400cbf0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.593+0000 7f1c38c00640 1 -- 172.21.15.31:0/198884824 shutdown_connections 2024-09-11T00:52:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.593+0000 7f1c38c00640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1c14077fc0 0x7f1c1407a480 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.593+0000 7f1c38c00640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1c340c34e0 0x7f1c3414ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.593+0000 7f1c38c00640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1c340c2d10 0x7f1c340ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.594+0000 7f1c38c00640 1 --2- 172.21.15.31:0/198884824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1c180037a0 0x7f1c340b0ca0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.594+0000 7f1c38c00640 1 -- 172.21.15.31:0/198884824 >> 172.21.15.31:0/198884824 conn(0x7f1c34068f10 msgr2=0x7f1c34150b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.594+0000 7f1c38c00640 1 -- 172.21.15.31:0/198884824 shutdown_connections 2024-09-11T00:52:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.594+0000 7f1c38c00640 1 -- 172.21.15.31:0/198884824 wait complete. 2024-09-11T00:52:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.607+0000 7f0effa00640 1 -- 172.21.15.31:0/802844309 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ed4005580 con 0x7f0ef8070570 2024-09-11T00:52:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.609+0000 7f0ef4c00640 1 -- 172.21.15.31:0/802844309 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6545 (secure 0 0 0) 0x7f0ee406ad50 con 0x7f0ef8070570 2024-09-11T00:52:35.611 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:35.611 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":157,"btime":"2024-09-11T00:52:33:898858+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:35.611 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 157 2024-09-11T00:52:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 -- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0ed8077fc0 msgr2=0x7f0ed807a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0ed8077fc0 0x7f0ed807a480 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0eec005fd0 tx=0x7f0eec00b440 comp rx=0 tx=0).stop 2024-09-11T00:52:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 -- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ef8070570 msgr2=0x7f0ef8148710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ef8070570 0x7f0ef8148710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4016f50 tx=0x7f0ee4014550 comp rx=0 tx=0).stop 2024-09-11T00:52:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 -- 172.21.15.31:0/802844309 shutdown_connections 2024-09-11T00:52:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0ed8077fc0 0x7f0ed807a480 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ef80c2ff0 0x7f0ef8073930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0ef8070570 0x7f0ef8148710 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 --2- 172.21.15.31:0/802844309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ef806e480 0x7f0ef80ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 -- 172.21.15.31:0/802844309 >> 172.21.15.31:0/802844309 conn(0x7f0ef80a9750 msgr2=0x7f0ef814b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:35.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 -- 172.21.15.31:0/802844309 shutdown_connections 2024-09-11T00:52:35.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.612+0000 7f0effa00640 1 -- 172.21.15.31:0/802844309 wait complete. 2024-09-11T00:52:35.666 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:52:35.667 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-11T00:52:35.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:35.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.891+0000 7f68c5800640 1 -- 172.21.15.31:0/3772826923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68c0070360 msgr2=0x7f68c0074050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.891+0000 7f68c5800640 1 --2- 172.21.15.31:0/3772826923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68c0070360 0x7f68c0074050 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009860 tx=0x7f68a802a540 comp rx=0 tx=0).stop 2024-09-11T00:52:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.891+0000 7f68c5800640 1 -- 172.21.15.31:0/3772826923 shutdown_connections 2024-09-11T00:52:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.892+0000 7f68c5800640 1 --2- 172.21.15.31:0/3772826923 >> v2:172.21.15.204:3300/0 conn(0x7f68c006d440 0x7f68c0074590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.892+0000 7f68c5800640 1 --2- 172.21.15.31:0/3772826923 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68c0070360 0x7f68c0074050 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.892+0000 7f68c5800640 1 -- 172.21.15.31:0/3772826923 >> 172.21.15.31:0/3772826923 conn(0x7f68c006dc00 msgr2=0x7f68c006e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.892+0000 7f68c5800640 1 -- 172.21.15.31:0/3772826923 shutdown_connections 2024-09-11T00:52:35.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.892+0000 7f68c5800640 1 -- 172.21.15.31:0/3772826923 wait complete. 2024-09-11T00:52:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.894+0000 7f68c5800640 1 Processor -- start 2024-09-11T00:52:35.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.894+0000 7f68c5800640 1 -- start start 2024-09-11T00:52:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68c5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68c006d440 0x7f68c0146ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68c5800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68c0070360 0x7f68c01e2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68c5800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68c0153b30 0x7f68c0145f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68c5800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f68c00710d0 con 0x7f68c006d440 2024-09-11T00:52:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68c5800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f68c0070f50 con 0x7f68c0153b30 2024-09-11T00:52:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68c5800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f68c0074ad0 con 0x7f68c0070360 2024-09-11T00:52:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68bea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68c006d440 0x7f68c0146ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68b7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68c0070360 0x7f68c01e2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68bea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68c006d440 0x7f68c0146ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51452/0 (socket says 172.21.15.31:51452) 2024-09-11T00:52:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68bea00640 1 -- 172.21.15.31:0/3066289415 learned_addr learned my addr 172.21.15.31:0/3066289415 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68bf400640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68c0153b30 0x7f68c0145f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68b7e00640 1 -- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68c0153b30 msgr2=0x7f68c0145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68b7e00640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68c0153b30 0x7f68c0145f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68b7e00640 1 -- 172.21.15.31:0/3066289415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68c006d440 msgr2=0x7f68c0146ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68b7e00640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68c006d440 0x7f68c0146ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:35.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.895+0000 7f68b7e00640 1 -- 172.21.15.31:0/3066289415 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c01576b0 con 0x7f68c0070360 2024-09-11T00:52:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.896+0000 7f68b7e00640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68c0070360 0x7f68c01e2400 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f68a8031000 tx=0x7f68a8032c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.896+0000 7f68b6a00640 1 -- 172.21.15.31:0/3066289415 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a8040070 con 0x7f68c0070360 2024-09-11T00:52:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.896+0000 7f68c5800640 1 -- 172.21.15.31:0/3066289415 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c0150860 con 0x7f68c0070360 2024-09-11T00:52:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.896+0000 7f68c5800640 1 -- 172.21.15.31:0/3066289415 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c0157990 con 0x7f68c0070360 2024-09-11T00:52:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.896+0000 7f68b6a00640 1 -- 172.21.15.31:0/3066289415 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68a8005220 con 0x7f68c0070360 2024-09-11T00:52:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.896+0000 7f68b6a00640 1 -- 172.21.15.31:0/3066289415 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a8005c90 con 0x7f68c0070360 2024-09-11T00:52:35.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.897+0000 7f68c5800640 1 -- 172.21.15.31:0/3066289415 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68c0070840 con 0x7f68c0070360 2024-09-11T00:52:35.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.899+0000 7f68b6a00640 1 -- 172.21.15.31:0/3066289415 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f68a804c030 con 0x7f68c0070360 2024-09-11T00:52:35.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.900+0000 7f68b6a00640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f689c077f70 0x7f689c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:35.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.900+0000 7f68b6a00640 1 -- 172.21.15.31:0/3066289415 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7f68a80bbae0 con 0x7f68c0070360 2024-09-11T00:52:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.900+0000 7f68bea00640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f689c077f70 0x7f689c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:35.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.901+0000 7f68bea00640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f689c077f70 0x7f689c07a430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f68b0007fd0 tx=0x7f68b0001470 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:35.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:35.905+0000 7f68b6a00640 1 -- 172.21.15.31:0/3066289415 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f68a80838c0 con 0x7f68c0070360 2024-09-11T00:52:36.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.223+0000 7f68c5800640 1 -- 172.21.15.31:0/3066289415 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68c00a8420 con 0x7f68c0070360 2024-09-11T00:52:36.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.225+0000 7f68b6a00640 1 -- 172.21.15.31:0/3066289415 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6545 (secure 0 0 0) 0x7f68a80831c0 con 0x7f68c0070360 2024-09-11T00:52:36.227 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:36.227 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":157,"btime":"2024-09-11T00:52:33:898858+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:36.227 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 157 2024-09-11T00:52:36.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.228+0000 7f68c5800640 1 -- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f689c077f70 msgr2=0x7f689c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:36.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.228+0000 7f68c5800640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f689c077f70 0x7f689c07a430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f68b0007fd0 tx=0x7f68b0001470 comp rx=0 tx=0).stop 2024-09-11T00:52:36.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.228+0000 7f68c5800640 1 -- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68c0070360 msgr2=0x7f68c01e2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:36.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.228+0000 7f68c5800640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68c0070360 0x7f68c01e2400 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f68a8031000 tx=0x7f68a8032c10 comp rx=0 tx=0).stop 2024-09-11T00:52:36.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.228+0000 7f68c5800640 1 -- 172.21.15.31:0/3066289415 shutdown_connections 2024-09-11T00:52:36.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.228+0000 7f68c5800640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f689c077f70 0x7f689c07a430 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.228+0000 7f68c5800640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68c0153b30 0x7f68c0145f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.228+0000 7f68c5800640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f68c0070360 0x7f68c01e2400 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.228+0000 7f68c5800640 1 --2- 172.21.15.31:0/3066289415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68c006d440 0x7f68c0146ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.228+0000 7f68c5800640 1 -- 172.21.15.31:0/3066289415 >> 172.21.15.31:0/3066289415 conn(0x7f68c006dc00 msgr2=0x7f68c00c68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:36.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.230+0000 7f68c5800640 1 -- 172.21.15.31:0/3066289415 shutdown_connections 2024-09-11T00:52:36.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.230+0000 7f68c5800640 1 -- 172.21.15.31:0/3066289415 wait complete. 2024-09-11T00:52:36.252 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-11T00:52:36.252 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-11T00:52:36.252 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:52:36.272 INFO:tasks.ceph.mds.c:Stopped 2024-09-11T00:52:36.273 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:36.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.524+0000 7f814b200640 1 -- 172.21.15.31:0/1851124707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f814c156960 msgr2=0x7f814c06e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.524+0000 7f814b200640 1 --2- 172.21.15.31:0/1851124707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f814c156960 0x7f814c06e140 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f81340097b0 tx=0x7f813402a4f0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.525+0000 7f814b200640 1 -- 172.21.15.31:0/1851124707 shutdown_connections 2024-09-11T00:52:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.525+0000 7f814b200640 1 --2- 172.21.15.31:0/1851124707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f814c156960 0x7f814c06e140 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.525+0000 7f814b200640 1 -- 172.21.15.31:0/1851124707 >> 172.21.15.31:0/1851124707 conn(0x7f814c068f10 msgr2=0x7f814c0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.525+0000 7f814b200640 1 -- 172.21.15.31:0/1851124707 shutdown_connections 2024-09-11T00:52:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.525+0000 7f814b200640 1 -- 172.21.15.31:0/1851124707 wait complete. 2024-09-11T00:52:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.527+0000 7f814b200640 1 Processor -- start 2024-09-11T00:52:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.527+0000 7f814b200640 1 -- start start 2024-09-11T00:52:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f814b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f814c0cc220 0x7f814c14b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:36.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f814b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f814c150020 0x7f814c14ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:36.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f814b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f814c156960 0x7f814c0c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:36.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f814b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f814c0c6ca0 con 0x7f814c156960 2024-09-11T00:52:36.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f814b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f814c0c6b20 con 0x7f814c150020 2024-09-11T00:52:36.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f814b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f814c1573d0 con 0x7f814c0cc220 2024-09-11T00:52:36.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f8143e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f814c150020 0x7f814c14ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:36.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f814a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f814c156960 0x7f814c0c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:36.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f8149e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f814c0cc220 0x7f814c14b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:36.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f8143e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f814c150020 0x7f814c14ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:51286/0 (socket says 172.21.15.31:51286) 2024-09-11T00:52:36.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.528+0000 7f8143e00640 1 -- 172.21.15.31:0/262614811 learned_addr learned my addr 172.21.15.31:0/262614811 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:36.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.529+0000 7f814a800640 1 -- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f814c0cc220 msgr2=0x7f814c14b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:36.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.529+0000 7f814a800640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f814c0cc220 0x7f814c14b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.529+0000 7f814a800640 1 -- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f814c150020 msgr2=0x7f814c14ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.529+0000 7f814a800640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f814c150020 0x7f814c14ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.529+0000 7f814a800640 1 -- 172.21.15.31:0/262614811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814c14c260 con 0x7f814c156960 2024-09-11T00:52:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.529+0000 7f814a800640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f814c156960 0x7f814c0c3de0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f814400c790 tx=0x7f814400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.529+0000 7f8142000640 1 -- 172.21.15.31:0/262614811 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8144017070 con 0x7f814c156960 2024-09-11T00:52:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.529+0000 7f8142000640 1 -- 172.21.15.31:0/262614811 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8144005070 con 0x7f814c156960 2024-09-11T00:52:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.529+0000 7f8142000640 1 -- 172.21.15.31:0/262614811 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81440054a0 con 0x7f814c156960 2024-09-11T00:52:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.530+0000 7f814b200640 1 -- 172.21.15.31:0/262614811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814c0c35e0 con 0x7f814c156960 2024-09-11T00:52:36.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.530+0000 7f814b200640 1 -- 172.21.15.31:0/262614811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f814c153770 con 0x7f814c156960 2024-09-11T00:52:36.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.532+0000 7f814b200640 1 -- 172.21.15.31:0/262614811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f814c0cc8e0 con 0x7f814c156960 2024-09-11T00:52:36.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.535+0000 7f8142000640 1 -- 172.21.15.31:0/262614811 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8144005640 con 0x7f814c156960 2024-09-11T00:52:36.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.536+0000 7f8142000640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8128078040 0x7f812807a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:36.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.536+0000 7f8142000640 1 -- 172.21.15.31:0/262614811 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7f814409a3f0 con 0x7f814c156960 2024-09-11T00:52:36.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.538+0000 7f8149e00640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8128078040 0x7f812807a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:36.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.541+0000 7f8149e00640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8128078040 0x7f812807a500 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f813c007fd0 tx=0x7f813c001400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.544+0000 7f8142000640 1 -- 172.21.15.31:0/262614811 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8144061b80 con 0x7f814c156960 2024-09-11T00:52:36.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.878+0000 7f814b200640 1 -- 172.21.15.31:0/262614811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f814c074c30 con 0x7f814c156960 2024-09-11T00:52:36.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.878+0000 7f8142000640 1 -- 172.21.15.31:0/262614811 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6545 (secure 0 0 0) 0x7f81440665b0 con 0x7f814c156960 2024-09-11T00:52:36.880 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:36.880 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":157,"btime":"2024-09-11T00:52:33:898858+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:36.880 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 157 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 -- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8128078040 msgr2=0x7f812807a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8128078040 0x7f812807a500 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f813c007fd0 tx=0x7f813c001400 comp rx=0 tx=0).stop 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 -- 172.21.15.31:0/262614811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f814c156960 msgr2=0x7f814c0c3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f814c156960 0x7f814c0c3de0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f814400c790 tx=0x7f814400cc50 comp rx=0 tx=0).stop 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 -- 172.21.15.31:0/262614811 shutdown_connections 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8128078040 0x7f812807a500 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f814c156960 0x7f814c0c3de0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f814c150020 0x7f814c14ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 --2- 172.21.15.31:0/262614811 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f814c0cc220 0x7f814c14b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 -- 172.21.15.31:0/262614811 >> 172.21.15.31:0/262614811 conn(0x7f814c068f10 msgr2=0x7f814c154a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:36.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 -- 172.21.15.31:0/262614811 shutdown_connections 2024-09-11T00:52:36.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:36.882+0000 7f814b200640 1 -- 172.21.15.31:0/262614811 wait complete. 2024-09-11T00:52:36.906 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:52:37.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.883+0000 7f2856c00640 1 -- 172.21.15.31:0/3528543110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2850153c10 msgr2=0x7f2850159830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.883+0000 7f2856c00640 1 --2- 172.21.15.31:0/3528543110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2850153c10 0x7f2850159830 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2848005fd0 tx=0x7f284802a910 comp rx=0 tx=0).stop 2024-09-11T00:52:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.883+0000 7f2856c00640 1 -- 172.21.15.31:0/3528543110 shutdown_connections 2024-09-11T00:52:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.883+0000 7f2856c00640 1 --2- 172.21.15.31:0/3528543110 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f28500c7330 0x7f2850160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.883+0000 7f2856c00640 1 --2- 172.21.15.31:0/3528543110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2850153c10 0x7f2850159830 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.883+0000 7f2856c00640 1 -- 172.21.15.31:0/3528543110 >> 172.21.15.31:0/3528543110 conn(0x7f28500c8cc0 msgr2=0x7f28500ad700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.884+0000 7f2856c00640 1 -- 172.21.15.31:0/3528543110 shutdown_connections 2024-09-11T00:52:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.884+0000 7f2856c00640 1 -- 172.21.15.31:0/3528543110 wait complete. 2024-09-11T00:52:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.885+0000 7f2856c00640 1 Processor -- start 2024-09-11T00:52:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.885+0000 7f2856c00640 1 -- start start 2024-09-11T00:52:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.886+0000 7f2856c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f285006cf40 0x7f28501e1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.886+0000 7f2856c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28500c7330 0x7f28500c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.886+0000 7f2856c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2850153c10 0x7f28501ef4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.886+0000 7f2856c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28500ae520 con 0x7f28500c7330 2024-09-11T00:52:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.886+0000 7f2856c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28500bf3c0 con 0x7f285006cf40 2024-09-11T00:52:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.886+0000 7f2856c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f28501e2470 con 0x7f2850153c10 2024-09-11T00:52:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.887+0000 7f284f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28500c7330 0x7f28500c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.887+0000 7f284f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28500c7330 0x7f28500c43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56996/0 (socket says 172.21.15.31:56996) 2024-09-11T00:52:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.887+0000 7f284f400640 1 -- 172.21.15.31:0/2615722786 learned_addr learned my addr 172.21.15.31:0/2615722786 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.887+0000 7f284fe00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f285006cf40 0x7f28501e1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.887+0000 7f2854c00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2850153c10 0x7f28501ef4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.888+0000 7f284f400640 1 -- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2850153c10 msgr2=0x7f28501ef4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.888+0000 7f284f400640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2850153c10 0x7f28501ef4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.888+0000 7f284f400640 1 -- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f285006cf40 msgr2=0x7f28501e1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.888+0000 7f284f400640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f285006cf40 0x7f28501e1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.888+0000 7f284f400640 1 -- 172.21.15.31:0/2615722786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2850154410 con 0x7f28500c7330 2024-09-11T00:52:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.888+0000 7f2854c00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2850153c10 0x7f28501ef4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:52:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.888+0000 7f284fe00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f285006cf40 0x7f28501e1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:52:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.888+0000 7f284f400640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28500c7330 0x7f28500c43a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f283800b450 tx=0x7f283800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.890+0000 7f2847e00640 1 -- 172.21.15.31:0/2615722786 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2838015020 con 0x7f28500c7330 2024-09-11T00:52:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.890+0000 7f2847e00640 1 -- 172.21.15.31:0/2615722786 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28380119a0 con 0x7f28500c7330 2024-09-11T00:52:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.890+0000 7f2847e00640 1 -- 172.21.15.31:0/2615722786 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2838011df0 con 0x7f28500c7330 2024-09-11T00:52:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.890+0000 7f2856c00640 1 -- 172.21.15.31:0/2615722786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28500c56b0 con 0x7f28500c7330 2024-09-11T00:52:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.890+0000 7f2856c00640 1 -- 172.21.15.31:0/2615722786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2850156d10 con 0x7f28500c7330 2024-09-11T00:52:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.892+0000 7f2856c00640 1 -- 172.21.15.31:0/2615722786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28500c4f80 con 0x7f28500c7330 2024-09-11T00:52:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.894+0000 7f2847e00640 1 -- 172.21.15.31:0/2615722786 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f283801a400 con 0x7f28500c7330 2024-09-11T00:52:37.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.895+0000 7f2847e00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f282c077fa0 0x7f282c07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:37.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.895+0000 7f2847e00640 1 -- 172.21.15.31:0/2615722786 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7f283809af60 con 0x7f28500c7330 2024-09-11T00:52:37.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.896+0000 7f284fe00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f282c077fa0 0x7f282c07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.898+0000 7f284fe00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f282c077fa0 0x7f282c07a460 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2848035000 tx=0x7f2848004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:37.901+0000 7f2847e00640 1 -- 172.21.15.31:0/2615722786 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f283801f020 con 0x7f28500c7330 2024-09-11T00:52:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.235+0000 7f2856c00640 1 -- 172.21.15.31:0/2615722786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28500c79f0 con 0x7f28500c7330 2024-09-11T00:52:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.236+0000 7f2847e00640 1 -- 172.21.15.31:0/2615722786 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6545 (secure 0 0 0) 0x7f283800d040 con 0x7f28500c7330 2024-09-11T00:52:38.238 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:38.238 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":157,"btime":"2024-09-11T00:52:33:898858+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:38.238 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 157 2024-09-11T00:52:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.238+0000 7f2856c00640 1 -- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f282c077fa0 msgr2=0x7f282c07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.238+0000 7f2856c00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f282c077fa0 0x7f282c07a460 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2848035000 tx=0x7f2848004ab0 comp rx=0 tx=0).stop 2024-09-11T00:52:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.238+0000 7f2856c00640 1 -- 172.21.15.31:0/2615722786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28500c7330 msgr2=0x7f28500c43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.238+0000 7f2856c00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28500c7330 0x7f28500c43a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f283800b450 tx=0x7f283800b910 comp rx=0 tx=0).stop 2024-09-11T00:52:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.239+0000 7f2856c00640 1 -- 172.21.15.31:0/2615722786 shutdown_connections 2024-09-11T00:52:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.239+0000 7f2856c00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f282c077fa0 0x7f282c07a460 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.239+0000 7f2856c00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2850153c10 0x7f28501ef4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.239+0000 7f2856c00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28500c7330 0x7f28500c43a0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.239+0000 7f2856c00640 1 --2- 172.21.15.31:0/2615722786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f285006cf40 0x7f28501e1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.239+0000 7f2856c00640 1 -- 172.21.15.31:0/2615722786 >> 172.21.15.31:0/2615722786 conn(0x7f28500c8cc0 msgr2=0x7f28500ca3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.239+0000 7f2856c00640 1 -- 172.21.15.31:0/2615722786 shutdown_connections 2024-09-11T00:52:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:38.239+0000 7f2856c00640 1 -- 172.21.15.31:0/2615722786 wait complete. 2024-09-11T00:52:38.264 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:52:38.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.157+0000 7fe0f3c00640 1 -- 172.21.15.31:0/1763767888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0ec0b1230 msgr2=0x7fe0ec1596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.157+0000 7fe0f3c00640 1 --2- 172.21.15.31:0/1763767888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0ec0b1230 0x7fe0ec1596b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0005fd0 tx=0x7fe0e002a400 comp rx=0 tx=0).stop 2024-09-11T00:52:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.157+0000 7fe0f3c00640 1 -- 172.21.15.31:0/1763767888 shutdown_connections 2024-09-11T00:52:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.157+0000 7fe0f3c00640 1 --2- 172.21.15.31:0/1763767888 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0ec0c9fe0 0x7fe0ec0cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.157+0000 7fe0f3c00640 1 --2- 172.21.15.31:0/1763767888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0ec0b1230 0x7fe0ec1596b0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.157+0000 7fe0f3c00640 1 -- 172.21.15.31:0/1763767888 >> 172.21.15.31:0/1763767888 conn(0x7fe0ec068f10 msgr2=0x7fe0ec0cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.157+0000 7fe0f3c00640 1 -- 172.21.15.31:0/1763767888 shutdown_connections 2024-09-11T00:52:39.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.157+0000 7fe0f3c00640 1 -- 172.21.15.31:0/1763767888 wait complete. 2024-09-11T00:52:39.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.159+0000 7fe0f3c00640 1 Processor -- start 2024-09-11T00:52:39.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.159+0000 7fe0f3c00640 1 -- start start 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0f3c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0ec0b1230 0x7fe0ec1e05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0f3c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0ec0c9b00 0x7fe0ec1df7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0f3c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0ec0c9fe0 0x7fe0ec149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0f3c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ec156540 con 0x7fe0ec0b1230 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0f3c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ec156260 con 0x7fe0ec0c9fe0 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0f3c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe0ec0bf390 con 0x7fe0ec0c9b00 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0f1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0ec0b1230 0x7fe0ec1e05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0f1200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0ec0b1230 0x7fe0ec1e05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57014/0 (socket says 172.21.15.31:57014) 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0f1200640 1 -- 172.21.15.31:0/2196182815 learned_addr learned my addr 172.21.15.31:0/2196182815 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0f1c00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0ec0c9fe0 0x7fe0ec149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.160+0000 7fe0ebe00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0ec0c9b00 0x7fe0ec1df7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.161+0000 7fe0f1200640 1 -- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0ec0c9b00 msgr2=0x7fe0ec1df7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.161+0000 7fe0f1200640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0ec0c9b00 0x7fe0ec1df7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.161+0000 7fe0f1200640 1 -- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0ec0c9fe0 msgr2=0x7fe0ec149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.161+0000 7fe0f1200640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0ec0c9fe0 0x7fe0ec149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.161+0000 7fe0f1200640 1 -- 172.21.15.31:0/2196182815 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ec1e2260 con 0x7fe0ec0b1230 2024-09-11T00:52:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.161+0000 7fe0f1200640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0ec0b1230 0x7fe0ec1e05a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0009850 tx=0x7fe0e00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.163+0000 7fe0e8c00640 1 -- 172.21.15.31:0/2196182815 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e003f070 con 0x7fe0ec0b1230 2024-09-11T00:52:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.163+0000 7fe0e8c00640 1 -- 172.21.15.31:0/2196182815 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0e00047a0 con 0x7fe0ec0b1230 2024-09-11T00:52:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.163+0000 7fe0e8c00640 1 -- 172.21.15.31:0/2196182815 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0035030 con 0x7fe0ec0b1230 2024-09-11T00:52:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.163+0000 7fe0f3c00640 1 -- 172.21.15.31:0/2196182815 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0ec1e2540 con 0x7fe0ec0b1230 2024-09-11T00:52:39.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.163+0000 7fe0f3c00640 1 -- 172.21.15.31:0/2196182815 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0ec0cba30 con 0x7fe0ec0b1230 2024-09-11T00:52:39.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.165+0000 7fe0f3c00640 1 -- 172.21.15.31:0/2196182815 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0ec0766d0 con 0x7fe0ec0b1230 2024-09-11T00:52:39.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.167+0000 7fe0e8c00640 1 -- 172.21.15.31:0/2196182815 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe0e0004bf0 con 0x7fe0ec0b1230 2024-09-11T00:52:39.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.168+0000 7fe0e8c00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe0cc077ff0 0x7fe0cc07a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:39.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.168+0000 7fe0e8c00640 1 -- 172.21.15.31:0/2196182815 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9160+0+0 (secure 0 0 0) 0x7fe0e00bbf10 con 0x7fe0ec0b1230 2024-09-11T00:52:39.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.169+0000 7fe0ebe00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe0cc077ff0 0x7fe0cc07a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:39.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.169+0000 7fe0ebe00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe0cc077ff0 0x7fe0cc07a4b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec159080 tx=0x7fe0d8007e30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:39.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.173+0000 7fe0e8c00640 1 -- 172.21.15.31:0/2196182815 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe0e0083610 con 0x7fe0ec0b1230 2024-09-11T00:52:39.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.509+0000 7fe0f3c00640 1 -- 172.21.15.31:0/2196182815 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0ec1596b0 con 0x7fe0ec0b1230 2024-09-11T00:52:39.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.510+0000 7fe0e8c00640 1 -- 172.21.15.31:0/2196182815 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6545 (secure 0 0 0) 0x7fe0e008c380 con 0x7fe0ec0b1230 2024-09-11T00:52:39.512 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:39.512 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":157,"btime":"2024-09-11T00:52:33:898858+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96},{"gid":7357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:50:17.486435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7143":{"gid":7143,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":11,"addr":"172.21.15.204:6835/1981755707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1981755707},{"type":"v1","addr":"172.21.15.204:6835","nonce":1981755707}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7143,"qdb_cluster":[7143]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:39.513 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 157 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 -- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe0cc077ff0 msgr2=0x7fe0cc07a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe0cc077ff0 0x7fe0cc07a4b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec159080 tx=0x7fe0d8007e30 comp rx=0 tx=0).stop 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 -- 172.21.15.31:0/2196182815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0ec0b1230 msgr2=0x7fe0ec1e05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0ec0b1230 0x7fe0ec1e05a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0009850 tx=0x7fe0e00038a0 comp rx=0 tx=0).stop 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 -- 172.21.15.31:0/2196182815 shutdown_connections 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe0cc077ff0 0x7fe0cc07a4b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe0ec0c9fe0 0x7fe0ec149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe0ec0c9b00 0x7fe0ec1df7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 --2- 172.21.15.31:0/2196182815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0ec0b1230 0x7fe0ec1e05a0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 -- 172.21.15.31:0/2196182815 >> 172.21.15.31:0/2196182815 conn(0x7fe0ec068f10 msgr2=0x7fe0ec155500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:39.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 -- 172.21.15.31:0/2196182815 shutdown_connections 2024-09-11T00:52:39.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:39.513+0000 7fe0f3c00640 1 -- 172.21.15.31:0/2196182815 wait complete. 2024-09-11T00:52:39.536 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:52:40.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:40.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.476+0000 7fde81200640 1 -- 172.21.15.31:0/1525074454 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde7c0c5b10 msgr2=0x7fde7c156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:40.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.476+0000 7fde81200640 1 --2- 172.21.15.31:0/1525074454 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde7c0c5b10 0x7fde7c156ad0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009860 tx=0x7fde6c02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:52:40.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.477+0000 7fde81200640 1 -- 172.21.15.31:0/1525074454 shutdown_connections 2024-09-11T00:52:40.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.477+0000 7fde81200640 1 --2- 172.21.15.31:0/1525074454 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde7c0c5b10 0x7fde7c156ad0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:40.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.477+0000 7fde81200640 1 -- 172.21.15.31:0/1525074454 >> 172.21.15.31:0/1525074454 conn(0x7fde7c0a9750 msgr2=0x7fde7c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:40.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.477+0000 7fde81200640 1 -- 172.21.15.31:0/1525074454 shutdown_connections 2024-09-11T00:52:40.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.477+0000 7fde81200640 1 -- 172.21.15.31:0/1525074454 wait complete. 2024-09-11T00:52:40.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.478+0000 7fde81200640 1 Processor -- start 2024-09-11T00:52:40.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.478+0000 7fde81200640 1 -- start start 2024-09-11T00:52:40.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.479+0000 7fde81200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde7c0c5b10 0x7fde7c074090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:40.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.479+0000 7fde81200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde7c0c5ef0 0x7fde7c155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.479+0000 7fde81200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde7c0c6310 0x7fde7c072110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.479+0000 7fde81200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fde7c153b90 con 0x7fde7c0c5b10 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.479+0000 7fde81200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fde7c0c7320 con 0x7fde7c0c5ef0 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.479+0000 7fde81200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fde7c153d10 con 0x7fde7c0c6310 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.479+0000 7fde79e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde7c0c5b10 0x7fde7c074090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.479+0000 7fde79e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde7c0c5b10 0x7fde7c074090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57020/0 (socket says 172.21.15.31:57020) 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.480+0000 7fde79e00640 1 -- 172.21.15.31:0/1304832270 learned_addr learned my addr 172.21.15.31:0/1304832270 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.480+0000 7fde7a800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde7c0c6310 0x7fde7c072110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.480+0000 7fde71400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde7c0c5ef0 0x7fde7c155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.480+0000 7fde79e00640 1 -- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde7c0c6310 msgr2=0x7fde7c072110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.480+0000 7fde79e00640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde7c0c6310 0x7fde7c072110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.480+0000 7fde79e00640 1 -- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde7c0c5ef0 msgr2=0x7fde7c155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:40.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.480+0000 7fde79e00640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde7c0c5ef0 0x7fde7c155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:40.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.480+0000 7fde79e00640 1 -- 172.21.15.31:0/1304832270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde7c1ec970 con 0x7fde7c0c5b10 2024-09-11T00:52:40.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.480+0000 7fde79e00640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde7c0c5b10 0x7fde7c074090 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009980 tx=0x7fde6c032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:40.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.481+0000 7fde72a00640 1 -- 172.21.15.31:0/1304832270 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c040070 con 0x7fde7c0c5b10 2024-09-11T00:52:40.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.481+0000 7fde72a00640 1 -- 172.21.15.31:0/1304832270 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde6c005310 con 0x7fde7c0c5b10 2024-09-11T00:52:40.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.481+0000 7fde81200640 1 -- 172.21.15.31:0/1304832270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde7c1ecc50 con 0x7fde7c0c5b10 2024-09-11T00:52:40.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.481+0000 7fde72a00640 1 -- 172.21.15.31:0/1304832270 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c0047a0 con 0x7fde7c0c5b10 2024-09-11T00:52:40.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.481+0000 7fde81200640 1 -- 172.21.15.31:0/1304832270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde7c074770 con 0x7fde7c0c5b10 2024-09-11T00:52:40.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.482+0000 7fde81200640 1 -- 172.21.15.31:0/1304832270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde7c156ad0 con 0x7fde7c0c5b10 2024-09-11T00:52:40.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.483+0000 7fde72a00640 1 -- 172.21.15.31:0/1304832270 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fde6c004b50 con 0x7fde7c0c5b10 2024-09-11T00:52:40.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.484+0000 7fde72a00640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde58077f70 0x7fde5807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.484+0000 7fde72a00640 1 -- 172.21.15.31:0/1304832270 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9246+0+0 (secure 0 0 0) 0x7fde6c0bbfe0 con 0x7fde7c0c5b10 2024-09-11T00:52:40.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.484+0000 7fde71400640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde58077f70 0x7fde5807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:40.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.486+0000 7fde71400640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde58077f70 0x7fde5807a430 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0a7e40 tx=0x7fde64007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:40.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.488+0000 7fde72a00640 1 -- 172.21.15.31:0/1304832270 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fde6c083640 con 0x7fde7c0c5b10 2024-09-11T00:52:40.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.804+0000 7fde81200640 1 -- 172.21.15.31:0/1304832270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde7c150540 con 0x7fde7c0c5b10 2024-09-11T00:52:40.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.805+0000 7fde72a00640 1 -- 172.21.15.31:0/1304832270 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) ==== 95+0+5708 (secure 0 0 0) 0x7fde6c088070 con 0x7fde7c0c5b10 2024-09-11T00:52:40.809 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:40.811 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":158,"btime":"2024-09-11T00:52:39:518290+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:39.518287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 158 2024-09-11T00:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 -- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde58077f70 msgr2=0x7fde5807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde58077f70 0x7fde5807a430 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0a7e40 tx=0x7fde64007e90 comp rx=0 tx=0).stop 2024-09-11T00:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 -- 172.21.15.31:0/1304832270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde7c0c5b10 msgr2=0x7fde7c074090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde7c0c5b10 0x7fde7c074090 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009980 tx=0x7fde6c032c10 comp rx=0 tx=0).stop 2024-09-11T00:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 -- 172.21.15.31:0/1304832270 shutdown_connections 2024-09-11T00:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fde58077f70 0x7fde5807a430 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde7c0c6310 0x7fde7c072110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde7c0c5ef0 0x7fde7c155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 --2- 172.21.15.31:0/1304832270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde7c0c5b10 0x7fde7c074090 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 -- 172.21.15.31:0/1304832270 >> 172.21.15.31:0/1304832270 conn(0x7fde7c0a9750 msgr2=0x7fde7c14f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 -- 172.21.15.31:0/1304832270 shutdown_connections 2024-09-11T00:52:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:40.808+0000 7fde81200640 1 -- 172.21.15.31:0/1304832270 wait complete. 2024-09-11T00:52:40.834 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-11T00:52:40.834 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:52:40.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.044+0000 7f9786a00640 1 -- 172.21.15.31:0/1501885361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97800c5aa0 msgr2=0x7f97800c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.044+0000 7f9786a00640 1 --2- 172.21.15.31:0/1501885361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97800c5aa0 0x7f97800c7150 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f9774009830 tx=0x7f977402a510 comp rx=0 tx=0).stop 2024-09-11T00:52:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.044+0000 7f9786a00640 1 -- 172.21.15.31:0/1501885361 shutdown_connections 2024-09-11T00:52:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.044+0000 7f9786a00640 1 --2- 172.21.15.31:0/1501885361 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9780156350 0x7f97801567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.044+0000 7f9786a00640 1 --2- 172.21.15.31:0/1501885361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97800c5aa0 0x7f97800c7150 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.044+0000 7f9786a00640 1 -- 172.21.15.31:0/1501885361 >> 172.21.15.31:0/1501885361 conn(0x7f978006caf0 msgr2=0x7f978006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.044+0000 7f9786a00640 1 -- 172.21.15.31:0/1501885361 shutdown_connections 2024-09-11T00:52:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.044+0000 7f9786a00640 1 -- 172.21.15.31:0/1501885361 wait complete. 2024-09-11T00:52:41.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.047+0000 7f9786a00640 1 Processor -- start 2024-09-11T00:52:41.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.047+0000 7f9786a00640 1 -- start start 2024-09-11T00:52:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.048+0000 7f9786a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97800c5aa0 0x7f97801e1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.048+0000 7f9786a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97800c6420 0x7f97801ec430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.048+0000 7f9786a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9780156350 0x7f97801ec970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.048+0000 7f9786a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f97800c2e80 con 0x7f9780156350 2024-09-11T00:52:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.048+0000 7f9786a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f97800c2ba0 con 0x7f97800c5aa0 2024-09-11T00:52:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.048+0000 7f9786a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9780156d20 con 0x7f97800c6420 2024-09-11T00:52:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.049+0000 7f9786000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9780156350 0x7f97801ec970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.049+0000 7f9786000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9780156350 0x7f97801ec970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57036/0 (socket says 172.21.15.31:57036) 2024-09-11T00:52:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.049+0000 7f9785600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97800c5aa0 0x7f97801e1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.049+0000 7f9786000640 1 -- 172.21.15.31:0/1737682650 learned_addr learned my addr 172.21.15.31:0/1737682650 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.049+0000 7f9784c00640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97800c6420 0x7f97801ec430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.049+0000 7f9786000640 1 -- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97800c6420 msgr2=0x7f97801ec430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.049+0000 7f9786000640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97800c6420 0x7f97801ec430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.049+0000 7f9786000640 1 -- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97800c5aa0 msgr2=0x7f97801e1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.049+0000 7f9786000640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97800c5aa0 0x7f97801e1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.049+0000 7f9786000640 1 -- 172.21.15.31:0/1737682650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9780155830 con 0x7f9780156350 2024-09-11T00:52:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.050+0000 7f9786000640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9780156350 0x7f97801ec970 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f977800ea80 tx=0x7f977800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.050+0000 7f977d600640 1 -- 172.21.15.31:0/1737682650 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977800ce40 con 0x7f9780156350 2024-09-11T00:52:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.050+0000 7f977d600640 1 -- 172.21.15.31:0/1737682650 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9778004b20 con 0x7f9780156350 2024-09-11T00:52:41.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.050+0000 7f9786a00640 1 -- 172.21.15.31:0/1737682650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97801ebd70 con 0x7f9780156350 2024-09-11T00:52:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.050+0000 7f977d600640 1 -- 172.21.15.31:0/1737682650 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778005020 con 0x7f9780156350 2024-09-11T00:52:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.050+0000 7f9786a00640 1 -- 172.21.15.31:0/1737682650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97800ab770 con 0x7f9780156350 2024-09-11T00:52:41.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.052+0000 7f9786a00640 1 -- 172.21.15.31:0/1737682650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97800ab580 con 0x7f9780156350 2024-09-11T00:52:41.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.053+0000 7f977d600640 1 -- 172.21.15.31:0/1737682650 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f97780051c0 con 0x7f9780156350 2024-09-11T00:52:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.054+0000 7f977d600640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9760077f70 0x7f976007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.054+0000 7f977d600640 1 -- 172.21.15.31:0/1737682650 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9246+0+0 (secure 0 0 0) 0x7f977809b6f0 con 0x7f9780156350 2024-09-11T00:52:41.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.055+0000 7f9785600640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9760077f70 0x7f976007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:41.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.056+0000 7f9785600640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9760077f70 0x7f976007a430 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9774034000 tx=0x7f9774004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:41.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.060+0000 7f977d600640 1 -- 172.21.15.31:0/1737682650 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9778062db0 con 0x7f9780156350 2024-09-11T00:52:41.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.383+0000 7f9786a00640 1 -- 172.21.15.31:0/1737682650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97800c5e80 con 0x7f9780156350 2024-09-11T00:52:41.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.383+0000 7f977d600640 1 -- 172.21.15.31:0/1737682650 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) ==== 95+0+5711 (secure 0 0 0) 0x7f97780677e0 con 0x7f9780156350 2024-09-11T00:52:41.385 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:41.385 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":159,"btime":"2024-09-11T00:52:40:944836+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:40.550853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:41.385 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 159 2024-09-11T00:52:41.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.386+0000 7f9786a00640 1 -- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9760077f70 msgr2=0x7f976007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:41.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.386+0000 7f9786a00640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9760077f70 0x7f976007a430 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9774034000 tx=0x7f9774004730 comp rx=0 tx=0).stop 2024-09-11T00:52:41.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.386+0000 7f9786a00640 1 -- 172.21.15.31:0/1737682650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9780156350 msgr2=0x7f97801ec970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.386+0000 7f9786a00640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9780156350 0x7f97801ec970 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f977800ea80 tx=0x7f977800ef40 comp rx=0 tx=0).stop 2024-09-11T00:52:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.387+0000 7f9786a00640 1 -- 172.21.15.31:0/1737682650 shutdown_connections 2024-09-11T00:52:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.387+0000 7f9786a00640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9760077f70 0x7f976007a430 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.387+0000 7f9786a00640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9780156350 0x7f97801ec970 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.387+0000 7f9786a00640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f97800c6420 0x7f97801ec430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.387+0000 7f9786a00640 1 --2- 172.21.15.31:0/1737682650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97800c5aa0 0x7f97801e1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.387+0000 7f9786a00640 1 -- 172.21.15.31:0/1737682650 >> 172.21.15.31:0/1737682650 conn(0x7f978006caf0 msgr2=0x7f978014edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.387+0000 7f9786a00640 1 -- 172.21.15.31:0/1737682650 shutdown_connections 2024-09-11T00:52:41.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.387+0000 7f9786a00640 1 -- 172.21.15.31:0/1737682650 wait complete. 2024-09-11T00:52:41.412 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:52:40:944836+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 159, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 159, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7357": { "addr": "172.21.15.204:6835/4094068967", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 4094068967, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 4094068967, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7357, "incarnation": 158, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:52:40.550853+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7357 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 156, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7255": { "addr": "172.21.15.31:6835/1855324855", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 1855324855, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 1855324855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7255, "incarnation": 153, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:52:22.861015+0000", "qdb_cluster": [ 7255 ], "qdb_leader": 7255, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7255 } } } ], "standbys": [ { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:52:41.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.767+0000 7f6dab200640 1 -- 172.21.15.31:0/2452421418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d90004800 msgr2=0x7f6d90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.767+0000 7f6dab200640 1 --2- 172.21.15.31:0/2452421418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d90004800 0x7f6d90004c80 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f6d980099a0 tx=0x7f6d90005830 comp rx=0 tx=0).stop 2024-09-11T00:52:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.767+0000 7f6daa800640 1 -- 172.21.15.31:0/2452421418 reap_dead start 2024-09-11T00:52:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.767+0000 7f6dab200640 1 -- 172.21.15.31:0/2452421418 shutdown_connections 2024-09-11T00:52:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.767+0000 7f6dab200640 1 -- 172.21.15.31:0/2452421418 >> 172.21.15.31:0/2452421418 conn(0x7f6dac068ee0 msgr2=0x7f6dac0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.767+0000 7f6dab200640 1 -- 172.21.15.31:0/2452421418 shutdown_connections 2024-09-11T00:52:41.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.767+0000 7f6dab200640 1 -- 172.21.15.31:0/2452421418 wait complete. 2024-09-11T00:52:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.769+0000 7f6dab200640 1 Processor -- start 2024-09-11T00:52:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.769+0000 7f6dab200640 1 -- start start 2024-09-11T00:52:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6dab200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d90004800 0x7f6dac14c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6dab200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dac0bfb90 0x7f6dac0c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6dab200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dac0c1e40 0x7f6dac0c53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6dab200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac0c6c90 con 0x7f6dac0c1e40 2024-09-11T00:52:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6dab200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac0c6490 con 0x7f6dac0bfb90 2024-09-11T00:52:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6dab200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6dac0adad0 con 0x7f6d90004800 2024-09-11T00:52:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6daa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dac0c1e40 0x7f6dac0c53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6daa800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dac0c1e40 0x7f6dac0c53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57052/0 (socket says 172.21.15.31:57052) 2024-09-11T00:52:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6daa800640 1 -- 172.21.15.31:0/3806973585 learned_addr learned my addr 172.21.15.31:0/3806973585 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6da9400640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dac0bfb90 0x7f6dac0c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6daa800640 1 -- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d90004800 msgr2=0x7f6dac14c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6daa800640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d90004800 0x7f6dac14c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6daa800640 1 -- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dac0bfb90 msgr2=0x7f6dac0c4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6daa800640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dac0bfb90 0x7f6dac0c4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.770+0000 7f6daa800640 1 -- 172.21.15.31:0/3806973585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dac14c700 con 0x7f6dac0c1e40 2024-09-11T00:52:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.771+0000 7f6daa800640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dac0c1e40 0x7f6dac0c53c0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0c8450 tx=0x7f6da400ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.771+0000 7f6da2000640 1 -- 172.21.15.31:0/3806973585 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4015020 con 0x7f6dac0c1e40 2024-09-11T00:52:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.771+0000 7f6da2000640 1 -- 172.21.15.31:0/3806973585 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6da400cd10 con 0x7f6dac0c1e40 2024-09-11T00:52:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.771+0000 7f6dab200640 1 -- 172.21.15.31:0/3806973585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dac0c4610 con 0x7f6dac0c1e40 2024-09-11T00:52:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.771+0000 7f6da2000640 1 -- 172.21.15.31:0/3806973585 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da40116c0 con 0x7f6dac0c1e40 2024-09-11T00:52:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.771+0000 7f6dab200640 1 -- 172.21.15.31:0/3806973585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dac0bf390 con 0x7f6dac0c1e40 2024-09-11T00:52:41.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.773+0000 7f6dab200640 1 -- 172.21.15.31:0/3806973585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dac0ad890 con 0x7f6dac0c1e40 2024-09-11T00:52:41.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.774+0000 7f6da2000640 1 -- 172.21.15.31:0/3806973585 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6da4011860 con 0x7f6dac0c1e40 2024-09-11T00:52:41.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.776+0000 7f6da2000640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6d8c077f70 0x7f6d8c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:41.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.776+0000 7f6da2000640 1 -- 172.21.15.31:0/3806973585 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9246+0+0 (secure 0 0 0) 0x7f6da409b1f0 con 0x7f6dac0c1e40 2024-09-11T00:52:41.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.776+0000 7f6da9e00640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6d8c077f70 0x7f6d8c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:41.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.777+0000 7f6da9e00640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6d8c077f70 0x7f6d8c07a430 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0beb10 tx=0x7f6d9c005ba0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:41.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:41.779+0000 7f6da2000640 1 -- 172.21.15.31:0/3806973585 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6dac0ad890 con 0x7f6dac0c1e40 2024-09-11T00:52:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.013+0000 7f6dab200640 1 -- 172.21.15.31:0/3806973585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dac0c4c60 con 0x7f6dac0c1e40 2024-09-11T00:52:42.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.014+0000 7f6da2000640 1 -- 172.21.15.31:0/3806973585 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+5708 (secure 0 0 0) 0x7f6da401f030 con 0x7f6dac0c1e40 2024-09-11T00:52:42.016 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:42.016 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":160,"btime":"2024-09-11T00:52:41:952326+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:41.321352+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 160 2024-09-11T00:52:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 -- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6d8c077f70 msgr2=0x7f6d8c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6d8c077f70 0x7f6d8c07a430 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0beb10 tx=0x7f6d9c005ba0 comp rx=0 tx=0).stop 2024-09-11T00:52:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 -- 172.21.15.31:0/3806973585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dac0c1e40 msgr2=0x7f6dac0c53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dac0c1e40 0x7f6dac0c53c0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0c8450 tx=0x7f6da400ed90 comp rx=0 tx=0).stop 2024-09-11T00:52:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 -- 172.21.15.31:0/3806973585 shutdown_connections 2024-09-11T00:52:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6d8c077f70 0x7f6d8c07a430 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dac0c1e40 0x7f6dac0c53c0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:42.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dac0bfb90 0x7f6dac0c4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:42.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 --2- 172.21.15.31:0/3806973585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6d90004800 0x7f6dac14c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:42.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 -- 172.21.15.31:0/3806973585 >> 172.21.15.31:0/3806973585 conn(0x7f6dac068ee0 msgr2=0x7f6dac159480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:42.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 -- 172.21.15.31:0/3806973585 shutdown_connections 2024-09-11T00:52:42.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:42.016+0000 7f6dab200640 1 -- 172.21.15.31:0/3806973585 wait complete. 2024-09-11T00:52:42.034 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:52:43.413 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:43.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.672+0000 7f68a2e00640 1 -- 172.21.15.31:0/4197215636 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f689c0c23a0 msgr2=0x7f689c0c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:43.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.672+0000 7f68a2e00640 1 --2- 172.21.15.31:0/4197215636 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f689c0c23a0 0x7f689c0c30c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6888009860 tx=0x7f688802a5d0 comp rx=0 tx=0).stop 2024-09-11T00:52:43.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.673+0000 7f68a2e00640 1 -- 172.21.15.31:0/4197215636 shutdown_connections 2024-09-11T00:52:43.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.673+0000 7f68a2e00640 1 --2- 172.21.15.31:0/4197215636 >> v2:172.21.15.31:3300/0 conn(0x7f689c1518b0 0x7f689c0c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:43.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.673+0000 7f68a2e00640 1 --2- 172.21.15.31:0/4197215636 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f689c0c23a0 0x7f689c0c30c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:43.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.673+0000 7f68a2e00640 1 -- 172.21.15.31:0/4197215636 >> 172.21.15.31:0/4197215636 conn(0x7f689c0a9750 msgr2=0x7f689c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:43.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.675+0000 7f68a2e00640 1 -- 172.21.15.31:0/4197215636 shutdown_connections 2024-09-11T00:52:43.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.675+0000 7f68a2e00640 1 -- 172.21.15.31:0/4197215636 wait complete. 2024-09-11T00:52:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.678+0000 7f68a2e00640 1 Processor -- start 2024-09-11T00:52:43.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.678+0000 7f68a2e00640 1 -- start start 2024-09-11T00:52:43.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.680+0000 7f68a2e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f689c072480 0x7f689c149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:43.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.680+0000 7f68a2e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689c0c23a0 0x7f689c152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:43.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.680+0000 7f68a2e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f689c1518b0 0x7f689c153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:43.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.680+0000 7f68a2e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f689c072860 con 0x7f689c0c23a0 2024-09-11T00:52:43.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.680+0000 7f68a2e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f689c074290 con 0x7f689c072480 2024-09-11T00:52:43.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.680+0000 7f68a2e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f689c0729e0 con 0x7f689c1518b0 2024-09-11T00:52:43.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.680+0000 7f689be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f689c072480 0x7f689c149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:43.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.680+0000 7f689b400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689c0c23a0 0x7f689c152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:43.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.680+0000 7f689be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f689c072480 0x7f689c149120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:43462/0 (socket says 172.21.15.31:43462) 2024-09-11T00:52:43.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.680+0000 7f689be00640 1 -- 172.21.15.31:0/1969175344 learned_addr learned my addr 172.21.15.31:0/1969175344 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:43.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.681+0000 7f68a0e00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f689c1518b0 0x7f689c153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.681+0000 7f689be00640 1 -- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f689c1518b0 msgr2=0x7f689c153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.681+0000 7f689be00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f689c1518b0 0x7f689c153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.681+0000 7f689be00640 1 -- 172.21.15.31:0/1969175344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689c0c23a0 msgr2=0x7f689c152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.681+0000 7f689be00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689c0c23a0 0x7f689c152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.681+0000 7f689be00640 1 -- 172.21.15.31:0/1969175344 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689c1e8320 con 0x7f689c072480 2024-09-11T00:52:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.681+0000 7f689b400640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689c0c23a0 0x7f689c152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:52:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.681+0000 7f689be00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f689c072480 0x7f689c149120 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f688c00c4c0 tx=0x7f688c00c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.682+0000 7f6893e00640 1 -- 172.21.15.31:0/1969175344 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c016020 con 0x7f689c072480 2024-09-11T00:52:43.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.682+0000 7f6893e00640 1 -- 172.21.15.31:0/1969175344 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f688c012920 con 0x7f689c072480 2024-09-11T00:52:43.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.682+0000 7f6893e00640 1 -- 172.21.15.31:0/1969175344 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c012d70 con 0x7f689c072480 2024-09-11T00:52:43.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.682+0000 7f68a2e00640 1 -- 172.21.15.31:0/1969175344 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f689c1e8800 con 0x7f689c072480 2024-09-11T00:52:43.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.682+0000 7f68a2e00640 1 -- 172.21.15.31:0/1969175344 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f689c149dc0 con 0x7f689c072480 2024-09-11T00:52:43.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.686+0000 7f68a2e00640 1 -- 172.21.15.31:0/1969175344 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6874005290 con 0x7f689c072480 2024-09-11T00:52:43.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.688+0000 7f6893e00640 1 -- 172.21.15.31:0/1969175344 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f688c01a3b0 con 0x7f689c072480 2024-09-11T00:52:43.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.689+0000 7f6893e00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f687c078040 0x7f687c07a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:43.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.689+0000 7f6893e00640 1 -- 172.21.15.31:0/1969175344 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9246+0+0 (secure 0 0 0) 0x7f688c09a8a0 con 0x7f689c072480 2024-09-11T00:52:43.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.690+0000 7f689b400640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f687c078040 0x7f687c07a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:43.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.691+0000 7f689b400640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f687c078040 0x7f687c07a500 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6888034000 tx=0x7f6888004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:43.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:43.694+0000 7f6893e00640 1 -- 172.21.15.31:0/1969175344 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f688c020030 con 0x7f689c072480 2024-09-11T00:52:43.925 INFO:tasks.ceph.mds.b.smithi204.stderr:2024-09-11T00:52:43.922+0000 7f4572800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:52:44.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.111+0000 7f68a2e00640 1 -- 172.21.15.31:0/1969175344 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6874005580 con 0x7f689c072480 2024-09-11T00:52:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.113+0000 7f6893e00640 1 -- 172.21.15.31:0/1969175344 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+5715 (secure 0 0 0) 0x7f688c062630 con 0x7f689c072480 2024-09-11T00:52:44.114 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:44.115 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":161,"btime":"2024-09-11T00:52:42:957881+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:44.115 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 161 2024-09-11T00:52:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.117+0000 7f68a2e00640 1 -- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f687c078040 msgr2=0x7f687c07a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.117+0000 7f68a2e00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f687c078040 0x7f687c07a500 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6888034000 tx=0x7f6888004730 comp rx=0 tx=0).stop 2024-09-11T00:52:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.117+0000 7f68a2e00640 1 -- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f689c072480 msgr2=0x7f689c149120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.117+0000 7f68a2e00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f689c072480 0x7f689c149120 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f688c00c4c0 tx=0x7f688c00c980 comp rx=0 tx=0).stop 2024-09-11T00:52:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.118+0000 7f68a2e00640 1 -- 172.21.15.31:0/1969175344 shutdown_connections 2024-09-11T00:52:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.118+0000 7f68a2e00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f687c078040 0x7f687c07a500 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.118+0000 7f68a2e00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f689c1518b0 0x7f689c153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.118+0000 7f68a2e00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f689c0c23a0 0x7f689c152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.118+0000 7f68a2e00640 1 --2- 172.21.15.31:0/1969175344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f689c072480 0x7f689c149120 secure :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f688c00c4c0 tx=0x7f688c00c980 comp rx=0 tx=0).stop 2024-09-11T00:52:44.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.118+0000 7f68a2e00640 1 -- 172.21.15.31:0/1969175344 >> 172.21.15.31:0/1969175344 conn(0x7f689c0a9750 msgr2=0x7f689c14d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:44.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.120+0000 7f68a2e00640 1 -- 172.21.15.31:0/1969175344 shutdown_connections 2024-09-11T00:52:44.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.120+0000 7f68a2e00640 1 -- 172.21.15.31:0/1969175344 wait complete. 2024-09-11T00:52:44.150 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=7143 2024-09-11T00:52:44.150 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.d 2024-09-11T00:52:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.309+0000 7f356b200640 1 -- 172.21.15.31:0/539997524 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f356c0c68b0 msgr2=0x7f356c1535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.309+0000 7f356b200640 1 --2- 172.21.15.31:0/539997524 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f356c0c68b0 0x7f356c1535a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f356c153f70 tx=0x7f355802a4d0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.310+0000 7f356b200640 1 -- 172.21.15.31:0/539997524 shutdown_connections 2024-09-11T00:52:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.310+0000 7f356b200640 1 --2- 172.21.15.31:0/539997524 >> v2:172.21.15.204:3300/0 conn(0x7f356c153ae0 0x7f356c156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.310+0000 7f356b200640 1 --2- 172.21.15.31:0/539997524 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f356c0c68b0 0x7f356c1535a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.310+0000 7f356b200640 1 --2- 172.21.15.31:0/539997524 >> v2:172.21.15.31:3300/0 conn(0x7f356c06cd60 0x7f356c0a7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.310+0000 7f356b200640 1 -- 172.21.15.31:0/539997524 >> 172.21.15.31:0/539997524 conn(0x7f356c0aa760 msgr2=0x7f356c0aab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:44.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.310+0000 7f356b200640 1 -- 172.21.15.31:0/539997524 shutdown_connections 2024-09-11T00:52:44.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.311+0000 7f356b200640 1 -- 172.21.15.31:0/539997524 wait complete. 2024-09-11T00:52:44.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.313+0000 7f356b200640 1 Processor -- start 2024-09-11T00:52:44.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.313+0000 7f356b200640 1 -- start start 2024-09-11T00:52:44.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.314+0000 7f356b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f356c06cd60 0x7f356c1ece40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.314+0000 7f356b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f356c0c68b0 0x7f356c1e1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.314+0000 7f356b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356c153ae0 0x7f356c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.315+0000 7f3569400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f356c0c68b0 0x7f356c1e1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.315+0000 7f356b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f356c06d140 con 0x7f356c153ae0 2024-09-11T00:52:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.315+0000 7f356b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f356c0a8260 con 0x7f356c0c68b0 2024-09-11T00:52:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.315+0000 7f356b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f356c06d2c0 con 0x7f356c06cd60 2024-09-11T00:52:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.315+0000 7f3569400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f356c0c68b0 0x7f356c1e1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:43474/0 (socket says 172.21.15.31:43474) 2024-09-11T00:52:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.315+0000 7f3569400640 1 -- 172.21.15.31:0/3462066250 learned_addr learned my addr 172.21.15.31:0/3462066250 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.315+0000 7f3569e00640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f356c06cd60 0x7f356c1ece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:44.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.316+0000 7f356a800640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356c153ae0 0x7f356c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:44.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.316+0000 7f356a800640 1 -- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f356c06cd60 msgr2=0x7f356c1ece40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:44.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.316+0000 7f356a800640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f356c06cd60 0x7f356c1ece40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.316+0000 7f356a800640 1 -- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f356c0c68b0 msgr2=0x7f356c1e1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:44.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.316+0000 7f356a800640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f356c0c68b0 0x7f356c1e1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.317+0000 7f356a800640 1 -- 172.21.15.31:0/3462066250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356c1ed380 con 0x7f356c153ae0 2024-09-11T00:52:44.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.317+0000 7f3569e00640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f356c06cd60 0x7f356c1ece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:52:44.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.317+0000 7f356a800640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356c153ae0 0x7f356c06ed70 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f356400be20 tx=0x7f356400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:44.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.318+0000 7f3562000640 1 -- 172.21.15.31:0/3462066250 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3564019070 con 0x7f356c153ae0 2024-09-11T00:52:44.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.318+0000 7f3562000640 1 -- 172.21.15.31:0/3462066250 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3564014440 con 0x7f356c153ae0 2024-09-11T00:52:44.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.318+0000 7f3562000640 1 -- 172.21.15.31:0/3462066250 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35640148d0 con 0x7f356c153ae0 2024-09-11T00:52:44.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.318+0000 7f356b200640 1 -- 172.21.15.31:0/3462066250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f356c14bfc0 con 0x7f356c153ae0 2024-09-11T00:52:44.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.319+0000 7f356b200640 1 -- 172.21.15.31:0/3462066250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f356c070090 con 0x7f356c153ae0 2024-09-11T00:52:44.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.321+0000 7f356b200640 1 -- 172.21.15.31:0/3462066250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f356c0c87b0 con 0x7f356c153ae0 2024-09-11T00:52:44.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.326+0000 7f3562000640 1 -- 172.21.15.31:0/3462066250 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f356401d3b0 con 0x7f356c153ae0 2024-09-11T00:52:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.329+0000 7f3562000640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f354c077f70 0x7f354c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:44.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.329+0000 7f3562000640 1 -- 172.21.15.31:0/3462066250 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9246+0+0 (secure 0 0 0) 0x7f356409ad90 con 0x7f356c153ae0 2024-09-11T00:52:44.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.334+0000 7f3569e00640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f354c077f70 0x7f354c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:44.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.335+0000 7f3562000640 1 -- 172.21.15.31:0/3462066250 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f356c0c87b0 con 0x7f356c153ae0 2024-09-11T00:52:44.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.335+0000 7f3569e00640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f354c077f70 0x7f354c07a430 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f355c00bfd0 tx=0x7f355c00b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:44.776 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":161,"btime":"2024-09-11T00:52:42:957881+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:44.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.773+0000 7f356b200640 1 -- 172.21.15.31:0/3462066250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f356c0ae960 con 0x7f356c153ae0 2024-09-11T00:52:44.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.774+0000 7f3562000640 1 -- 172.21.15.31:0/3462066250 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+5715 (secure 0 0 0) 0x7f356c0ae960 con 0x7f356c153ae0 2024-09-11T00:52:44.777 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 161 2024-09-11T00:52:44.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.777+0000 7f356b200640 1 -- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f354c077f70 msgr2=0x7f354c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:44.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.777+0000 7f356b200640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f354c077f70 0x7f354c07a430 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f355c00bfd0 tx=0x7f355c00b390 comp rx=0 tx=0).stop 2024-09-11T00:52:44.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.777+0000 7f356b200640 1 -- 172.21.15.31:0/3462066250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356c153ae0 msgr2=0x7f356c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:44.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.777+0000 7f356b200640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356c153ae0 0x7f356c06ed70 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f356400be20 tx=0x7f356400bf10 comp rx=0 tx=0).stop 2024-09-11T00:52:44.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.778+0000 7f356b200640 1 -- 172.21.15.31:0/3462066250 shutdown_connections 2024-09-11T00:52:44.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.778+0000 7f356b200640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f354c077f70 0x7f354c07a430 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.778+0000 7f356b200640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356c153ae0 0x7f356c06ed70 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.778+0000 7f356b200640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f356c0c68b0 0x7f356c1e1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.778+0000 7f356b200640 1 --2- 172.21.15.31:0/3462066250 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f356c06cd60 0x7f356c1ece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:44.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.778+0000 7f356b200640 1 -- 172.21.15.31:0/3462066250 >> 172.21.15.31:0/3462066250 conn(0x7f356c0aa760 msgr2=0x7f356c152580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:44.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.778+0000 7f356b200640 1 -- 172.21.15.31:0/3462066250 shutdown_connections 2024-09-11T00:52:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:44.780+0000 7f356b200640 1 -- 172.21.15.31:0/3462066250 wait complete. 2024-09-11T00:52:44.806 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:52:46.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:47.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.055+0000 7fa711000640 1 -- 172.21.15.31:0/3069075172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa70c070570 msgr2=0x7fa70c14f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.055+0000 7fa711000640 1 --2- 172.21.15.31:0/3069075172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa70c070570 0x7fa70c14f4d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009830 tx=0x7fa6fc02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:52:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.056+0000 7fa711000640 1 -- 172.21.15.31:0/3069075172 shutdown_connections 2024-09-11T00:52:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.056+0000 7fa711000640 1 --2- 172.21.15.31:0/3069075172 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa70c0c2ff0 0x7fa70c0c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.056+0000 7fa711000640 1 --2- 172.21.15.31:0/3069075172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa70c070570 0x7fa70c14f4d0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:47.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.056+0000 7fa711000640 1 -- 172.21.15.31:0/3069075172 >> 172.21.15.31:0/3069075172 conn(0x7fa70c0a9750 msgr2=0x7fa70c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:47.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.056+0000 7fa711000640 1 -- 172.21.15.31:0/3069075172 shutdown_connections 2024-09-11T00:52:47.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.056+0000 7fa711000640 1 -- 172.21.15.31:0/3069075172 wait complete. 2024-09-11T00:52:47.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.058+0000 7fa711000640 1 Processor -- start 2024-09-11T00:52:47.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.058+0000 7fa711000640 1 -- start start 2024-09-11T00:52:47.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.059+0000 7fa711000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa70c070570 0x7fa70c149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:47.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.059+0000 7fa711000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa70c0c2ff0 0x7fa70c1f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:47.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.059+0000 7fa711000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa70c0c5de0 0x7fa70c1f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.060+0000 7fa711000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c06eb90 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.060+0000 7fa711000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c14fb20 con 0x7fa70c0c5de0 2024-09-11T00:52:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.060+0000 7fa711000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa70c06ed10 con 0x7fa70c070570 2024-09-11T00:52:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.060+0000 7fa709400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa70c0c2ff0 0x7fa70c1f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.060+0000 7fa70a800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa70c0c5de0 0x7fa70c1f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.060+0000 7fa709400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa70c0c2ff0 0x7fa70c1f38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57126/0 (socket says 172.21.15.31:57126) 2024-09-11T00:52:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.060+0000 7fa709400640 1 -- 172.21.15.31:0/3583347869 learned_addr learned my addr 172.21.15.31:0/3583347869 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.060+0000 7fa709e00640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa70c070570 0x7fa70c149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.061+0000 7fa709400640 1 -- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa70c070570 msgr2=0x7fa70c149070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.061+0000 7fa709400640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa70c070570 0x7fa70c149070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.061+0000 7fa709400640 1 -- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa70c0c5de0 msgr2=0x7fa70c1f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.061+0000 7fa709400640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa70c0c5de0 0x7fa70c1f3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.061+0000 7fa709400640 1 -- 172.21.15.31:0/3583347869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70c1495b0 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.061+0000 7fa709400640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa70c0c2ff0 0x7fa70c1f38d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800c950 tx=0x7fa6f800ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:47.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.061+0000 7fa70a800640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa70c0c5de0 0x7fa70c1f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:52:47.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.062+0000 7fa702000640 1 -- 172.21.15.31:0/3583347869 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8007c70 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.062+0000 7fa702000640 1 -- 172.21.15.31:0/3583347869 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa6f800f450 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.062+0000 7fa702000640 1 -- 172.21.15.31:0/3583347869 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800f880 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.062+0000 7fa711000640 1 -- 172.21.15.31:0/3583347869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa70c1509d0 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.062+0000 7fa711000640 1 -- 172.21.15.31:0/3583347869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa70c150f90 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.064+0000 7fa711000640 1 -- 172.21.15.31:0/3583347869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa70c0c7730 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.066+0000 7fa702000640 1 -- 172.21.15.31:0/3583347869 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa6f800fa20 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.068+0000 7fa702000640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa6ec077f70 0x7fa6ec07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:47.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.069+0000 7fa709e00640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa6ec077f70 0x7fa6ec07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:47.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.070+0000 7fa709e00640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa6ec077f70 0x7fa6ec07a430 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc034000 tx=0x7fa6fc004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:47.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.076+0000 7fa702000640 1 -- 172.21.15.31:0/3583347869 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9246+0+0 (secure 0 0 0) 0x7fa6f8098b30 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.077+0000 7fa702000640 1 -- 172.21.15.31:0/3583347869 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa70c1495b0 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.489+0000 7fa711000640 1 -- 172.21.15.31:0/3583347869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa70c0c1e40 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.491+0000 7fa702000640 1 -- 172.21.15.31:0/3583347869 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+5715 (secure 0 0 0) 0x7fa6f8014030 con 0x7fa70c0c2ff0 2024-09-11T00:52:47.493 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:47.493 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":161,"btime":"2024-09-11T00:52:42:957881+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:22.861015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7255":{"gid":7255,"name":"c","rank":0,"incarnation":153,"state":"up:active","state_seq":31,"addr":"172.21.15.31:6835/1855324855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":1855324855},{"type":"v1","addr":"172.21.15.31:6835","nonce":1855324855}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7255,"qdb_cluster":[7255]},"id":2}]} 2024-09-11T00:52:47.494 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 161 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 -- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa6ec077f70 msgr2=0x7fa6ec07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa6ec077f70 0x7fa6ec07a430 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc034000 tx=0x7fa6fc004730 comp rx=0 tx=0).stop 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 -- 172.21.15.31:0/3583347869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa70c0c2ff0 msgr2=0x7fa70c1f38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa70c0c2ff0 0x7fa70c1f38d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800c950 tx=0x7fa6f800ce10 comp rx=0 tx=0).stop 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 -- 172.21.15.31:0/3583347869 shutdown_connections 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa6ec077f70 0x7fa6ec07a430 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa70c0c5de0 0x7fa70c1f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa70c0c2ff0 0x7fa70c1f38d0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 --2- 172.21.15.31:0/3583347869 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa70c070570 0x7fa70c149070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 -- 172.21.15.31:0/3583347869 >> 172.21.15.31:0/3583347869 conn(0x7fa70c0a9750 msgr2=0x7fa70c14bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:47.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 -- 172.21.15.31:0/3583347869 shutdown_connections 2024-09-11T00:52:47.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:47.495+0000 7fa711000640 1 -- 172.21.15.31:0/3583347869 wait complete. 2024-09-11T00:52:47.523 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-11T00:52:49.524 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:49.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.817+0000 7f5b73a00640 1 -- 172.21.15.31:0/3687834752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b6c0a6f90 msgr2=0x7f5b6c0c33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:49.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.817+0000 7f5b73a00640 1 --2- 172.21.15.31:0/3687834752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b6c0a6f90 0x7f5b6c0c33c0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0c2c40 tx=0x7f5b6402a980 comp rx=0 tx=0).stop 2024-09-11T00:52:49.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.817+0000 7f5b73a00640 1 -- 172.21.15.31:0/3687834752 shutdown_connections 2024-09-11T00:52:49.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.817+0000 7f5b73a00640 1 --2- 172.21.15.31:0/3687834752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b6c0a6f90 0x7f5b6c0c33c0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:49.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.817+0000 7f5b73a00640 1 --2- 172.21.15.31:0/3687834752 >> v2:172.21.15.204:3300/0 conn(0x7f5b6c157f40 0x7f5b6c0c2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:49.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.817+0000 7f5b73a00640 1 -- 172.21.15.31:0/3687834752 >> 172.21.15.31:0/3687834752 conn(0x7f5b6c0a9750 msgr2=0x7f5b6c0a9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.817+0000 7f5b73a00640 1 -- 172.21.15.31:0/3687834752 shutdown_connections 2024-09-11T00:52:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.818+0000 7f5b73a00640 1 -- 172.21.15.31:0/3687834752 wait complete. 2024-09-11T00:52:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.820+0000 7f5b73a00640 1 Processor -- start 2024-09-11T00:52:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.820+0000 7f5b73a00640 1 -- start start 2024-09-11T00:52:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.821+0000 7f5b73a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b6c0a6f90 0x7f5b6c06f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.823+0000 7f5b73a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c151860 0x7f5b6c0c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.823+0000 7f5b73a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b6c157f40 0x7f5b6c14bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.823+0000 7f5b73a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6c1546a0 con 0x7f5b6c157f40 2024-09-11T00:52:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.823+0000 7f5b73a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6c1543c0 con 0x7f5b6c151860 2024-09-11T00:52:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.823+0000 7f5b73a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5b6c0c25c0 con 0x7f5b6c0a6f90 2024-09-11T00:52:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.824+0000 7f5b71000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b6c0a6f90 0x7f5b6c06f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.824+0000 7f5b71000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b6c0a6f90 0x7f5b6c06f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:42524/0 (socket says 172.21.15.31:42524) 2024-09-11T00:52:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.824+0000 7f5b71000640 1 -- 172.21.15.31:0/2137159823 learned_addr learned my addr 172.21.15.31:0/2137159823 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.826+0000 7f5b6be00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c151860 0x7f5b6c0c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.826+0000 7f5b71a00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b6c157f40 0x7f5b6c14bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.827+0000 7f5b71a00640 1 -- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b6c0a6f90 msgr2=0x7f5b6c06f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.827+0000 7f5b71a00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b6c0a6f90 0x7f5b6c06f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.827+0000 7f5b71a00640 1 -- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c151860 msgr2=0x7f5b6c0c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.827+0000 7f5b71a00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c151860 0x7f5b6c0c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.827+0000 7f5b71a00640 1 -- 172.21.15.31:0/2137159823 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6c14c980 con 0x7f5b6c157f40 2024-09-11T00:52:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.828+0000 7f5b71a00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b6c157f40 0x7f5b6c14bb00 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f5b64032000 tx=0x7f5b64003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.828+0000 7f5b68c00640 1 -- 172.21.15.31:0/2137159823 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b64034050 con 0x7f5b6c157f40 2024-09-11T00:52:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.828+0000 7f5b73a00640 1 -- 172.21.15.31:0/2137159823 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b6c157580 con 0x7f5b6c157f40 2024-09-11T00:52:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.828+0000 7f5b73a00640 1 -- 172.21.15.31:0/2137159823 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b6c1ecd60 con 0x7f5b6c157f40 2024-09-11T00:52:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.828+0000 7f5b68c00640 1 -- 172.21.15.31:0/2137159823 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b64004450 con 0x7f5b6c157f40 2024-09-11T00:52:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.828+0000 7f5b68c00640 1 -- 172.21.15.31:0/2137159823 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b64034050 con 0x7f5b6c157f40 2024-09-11T00:52:49.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.831+0000 7f5b73a00640 1 -- 172.21.15.31:0/2137159823 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b6c072210 con 0x7f5b6c157f40 2024-09-11T00:52:49.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.833+0000 7f5b68c00640 1 -- 172.21.15.31:0/2137159823 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5b64041070 con 0x7f5b6c157f40 2024-09-11T00:52:49.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.835+0000 7f5b68c00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b4c078170 0x7f5b4c07a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:49.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.835+0000 7f5b68c00640 1 -- 172.21.15.31:0/2137159823 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7f5b640bbc70 con 0x7f5b6c157f40 2024-09-11T00:52:49.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.836+0000 7f5b71000640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b4c078170 0x7f5b4c07a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:49.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.841+0000 7f5b71000640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b4c078170 0x7f5b4c07a630 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5b60004ee0 tx=0x7f5b60009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:49.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:49.846+0000 7f5b68c00640 1 -- 172.21.15.31:0/2137159823 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5b640838a0 con 0x7f5b6c157f40 2024-09-11T00:52:50.151 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-11T00:52:50.151 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:52:50.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:52:50.157 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:52:50.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:50.305 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:52:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.354+0000 7f5b73a00640 1 -- 172.21.15.31:0/2137159823 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b6c072210 con 0x7f5b6c157f40 2024-09-11T00:52:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.358+0000 7f5b68c00640 1 -- 172.21.15.31:0/2137159823 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+4883 (secure 0 0 0) 0x7f5b640831a0 con 0x7f5b6c157f40 2024-09-11T00:52:50.359 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:50.361 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":162,"btime":"2024-09-11T00:52:49:521437+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:49.521434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:52:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 162 2024-09-11T00:52:50.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 -- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b4c078170 msgr2=0x7f5b4c07a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:50.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b4c078170 0x7f5b4c07a630 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5b60004ee0 tx=0x7f5b60009040 comp rx=0 tx=0).stop 2024-09-11T00:52:50.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 -- 172.21.15.31:0/2137159823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b6c157f40 msgr2=0x7f5b6c14bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b6c157f40 0x7f5b6c14bb00 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f5b64032000 tx=0x7f5b64003f30 comp rx=0 tx=0).stop 2024-09-11T00:52:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 -- 172.21.15.31:0/2137159823 shutdown_connections 2024-09-11T00:52:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b4c078170 0x7f5b4c07a630 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b6c157f40 0x7f5b6c14bb00 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b6c151860 0x7f5b6c0c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 --2- 172.21.15.31:0/2137159823 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b6c0a6f90 0x7f5b6c06f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 -- 172.21.15.31:0/2137159823 >> 172.21.15.31:0/2137159823 conn(0x7f5b6c0a9750 msgr2=0x7f5b6c14dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.364+0000 7f5b73a00640 1 -- 172.21.15.31:0/2137159823 shutdown_connections 2024-09-11T00:52:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.365+0000 7f5b73a00640 1 -- 172.21.15.31:0/2137159823 wait complete. 2024-09-11T00:52:50.413 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-11T00:52:50.414 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:52:50.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:50.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.442+0000 7f980c600640 1 -- 172.21.15.31:0/1630596580 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98040c4fe0 msgr2=0x7f9804157530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:50.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.442+0000 7f980c600640 1 --2- 172.21.15.31:0/1630596580 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98040c4fe0 0x7f9804157530 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f9804064820 tx=0x7f97f002a410 comp rx=0 tx=0).stop 2024-09-11T00:52:50.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.442+0000 7f980c600640 1 -- 172.21.15.31:0/1630596580 shutdown_connections 2024-09-11T00:52:50.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.442+0000 7f980c600640 1 --2- 172.21.15.31:0/1630596580 >> v2:172.21.15.31:3300/0 conn(0x7f9804157a70 0x7f980415e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.442+0000 7f980c600640 1 --2- 172.21.15.31:0/1630596580 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98040c4fe0 0x7f9804157530 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.442+0000 7f980c600640 1 -- 172.21.15.31:0/1630596580 >> 172.21.15.31:0/1630596580 conn(0x7f98040ab8a0 msgr2=0x7f98040abcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:50.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.442+0000 7f980c600640 1 -- 172.21.15.31:0/1630596580 shutdown_connections 2024-09-11T00:52:50.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.442+0000 7f980c600640 1 -- 172.21.15.31:0/1630596580 wait complete. 2024-09-11T00:52:50.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.445+0000 7f980c600640 1 Processor -- start 2024-09-11T00:52:50.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.445+0000 7f980c600640 1 -- start start 2024-09-11T00:52:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.446+0000 7f980c600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98040c44f0 0x7f9804156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.446+0000 7f980c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9804157a70 0x7f980414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f9809c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98040c44f0 0x7f9804156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f9809c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98040c44f0 0x7f9804156af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:60010/0 (socket says 172.21.15.31:60010) 2024-09-11T00:52:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f9809c00640 1 -- 172.21.15.31:0/1909887629 learned_addr learned my addr 172.21.15.31:0/1909887629 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f980c600640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98040c8550 0x7f98040aa890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f98040701b0 con 0x7f9804157a70 2024-09-11T00:52:50.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9804151af0 con 0x7f98040c44f0 2024-09-11T00:52:50.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f98040a9380 con 0x7f98040c8550 2024-09-11T00:52:50.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f9809200640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9804157a70 0x7f980414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f980a600640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98040c8550 0x7f98040aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f9809c00640 1 -- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98040c8550 msgr2=0x7f98040aa890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f9809c00640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98040c8550 0x7f98040aa890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.447+0000 7f9809c00640 1 -- 172.21.15.31:0/1909887629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9804157a70 msgr2=0x7f980414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.448+0000 7f9809c00640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9804157a70 0x7f980414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.448+0000 7f9809c00640 1 -- 172.21.15.31:0/1909887629 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980414e8f0 con 0x7f98040c44f0 2024-09-11T00:52:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.448+0000 7f9809200640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9804157a70 0x7f980414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:52:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.449+0000 7f9809c00640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98040c44f0 0x7f9804156af0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f97f800c500 tx=0x7f97f800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.449+0000 7f9801600640 1 -- 172.21.15.31:0/1909887629 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f8017070 con 0x7f98040c44f0 2024-09-11T00:52:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.449+0000 7f9801600640 1 -- 172.21.15.31:0/1909887629 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97f80036e0 con 0x7f98040c44f0 2024-09-11T00:52:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.449+0000 7f9801600640 1 -- 172.21.15.31:0/1909887629 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f8003b70 con 0x7f98040c44f0 2024-09-11T00:52:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.449+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980414c6f0 con 0x7f98040c44f0 2024-09-11T00:52:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.449+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98040af990 con 0x7f98040c44f0 2024-09-11T00:52:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.451+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98040c9800 con 0x7f98040c44f0 2024-09-11T00:52:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.452+0000 7f9801600640 1 -- 172.21.15.31:0/1909887629 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f97f801b3b0 con 0x7f98040c44f0 2024-09-11T00:52:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.454+0000 7f9801600640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97e4077fa0 0x7f97e407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:50.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.454+0000 7f9801600640 1 -- 172.21.15.31:0/1909887629 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7f97f809a320 con 0x7f98040c44f0 2024-09-11T00:52:50.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.455+0000 7f9809200640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97e4077fa0 0x7f97e407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:50.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.456+0000 7f9809200640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97e4077fa0 0x7f97e407a460 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f97f0034000 tx=0x7f97f000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:50.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.458+0000 7f9801600640 1 -- 172.21.15.31:0/1909887629 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f980414c6f0 con 0x7f98040c44f0 2024-09-11T00:52:50.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.689+0000 7f05b2000640 1 -- 172.21.15.31:0/1292686871 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05a404b260 msgr2=0x7f05a404db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:50.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.689+0000 7f05b2000640 1 --2- 172.21.15.31:0/1292686871 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05a404b260 0x7f05a404db60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f05ac075400 tx=0x7f059802a270 comp rx=0 tx=0).stop 2024-09-11T00:52:50.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.689+0000 7f05b2000640 1 -- 172.21.15.31:0/1292686871 shutdown_connections 2024-09-11T00:52:50.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.689+0000 7f05b2000640 1 --2- 172.21.15.31:0/1292686871 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05a404b260 0x7f05a404db60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.689+0000 7f05b2000640 1 -- 172.21.15.31:0/1292686871 >> 172.21.15.31:0/1292686871 conn(0x7f05a4009d30 msgr2=0x7f05a400a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:50.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.690+0000 7f05b2000640 1 -- 172.21.15.31:0/1292686871 shutdown_connections 2024-09-11T00:52:50.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.690+0000 7f05b2000640 1 -- 172.21.15.31:0/1292686871 wait complete. 2024-09-11T00:52:50.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.693+0000 7f05b2000640 1 Processor -- start 2024-09-11T00:52:50.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.695+0000 7f05b2000640 1 -- start start 2024-09-11T00:52:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.696+0000 7f05b2000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05a400cf20 0x7f05a40cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.696+0000 7f05b2000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05a404b260 0x7f05a4008850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.696+0000 7f05b2000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05a404d780 0x7f05a416fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.696+0000 7f05b2000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f05a404b770 con 0x7f05a400cf20 2024-09-11T00:52:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.696+0000 7f05b2000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f05a40d3fc0 con 0x7f05a404b260 2024-09-11T00:52:50.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.696+0000 7f05b2000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f05a4173c70 con 0x7f05a404d780 2024-09-11T00:52:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.697+0000 7f05b0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05a400cf20 0x7f05a40cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.697+0000 7f05b0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05a400cf20 0x7f05a40cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40018/0 (socket says 172.21.15.31:40018) 2024-09-11T00:52:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.697+0000 7f05b0c00640 1 -- 172.21.15.31:0/2320935572 learned_addr learned my addr 172.21.15.31:0/2320935572 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.697+0000 7f05b0c00640 1 -- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05a404d780 msgr2=0x7f05a416fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:50.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.697+0000 7f05b0c00640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05a404d780 0x7f05a416fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.697+0000 7f05b0c00640 1 -- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05a404b260 msgr2=0x7f05a4008850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:52:50.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.697+0000 7f05b0c00640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05a404b260 0x7f05a4008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.697+0000 7f05b0c00640 1 -- 172.21.15.31:0/2320935572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a4173f50 con 0x7f05a400cf20 2024-09-11T00:52:50.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.698+0000 7f05b0c00640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05a400cf20 0x7f05a40cdba0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c390 tx=0x7f05a000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:50.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.698+0000 7f05a8c00640 1 -- 172.21.15.31:0/2320935572 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0011790 con 0x7f05a400cf20 2024-09-11T00:52:50.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.698+0000 7f05a8c00640 1 -- 172.21.15.31:0/2320935572 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05a0017070 con 0x7f05a400cf20 2024-09-11T00:52:50.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.698+0000 7f05a8c00640 1 -- 172.21.15.31:0/2320935572 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0011ce0 con 0x7f05a400cf20 2024-09-11T00:52:50.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.700+0000 7f05b2000640 1 -- 172.21.15.31:0/2320935572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a4046c80 con 0x7f05a400cf20 2024-09-11T00:52:50.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.700+0000 7f05b2000640 1 -- 172.21.15.31:0/2320935572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a40dc1b0 con 0x7f05a400cf20 2024-09-11T00:52:50.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.702+0000 7f05b2000640 1 -- 172.21.15.31:0/2320935572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a404db60 con 0x7f05a400cf20 2024-09-11T00:52:50.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.706+0000 7f05a8c00640 1 -- 172.21.15.31:0/2320935572 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f05a001c460 con 0x7f05a400cf20 2024-09-11T00:52:50.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.711+0000 7f05a8c00640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f058c077f70 0x7f058c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:50.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.712+0000 7f05a8c00640 1 -- 172.21.15.31:0/2320935572 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7f05a009b4c0 con 0x7f05a400cf20 2024-09-11T00:52:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.713+0000 7f05abe00640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f058c077f70 0x7f058c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.715+0000 7f05abe00640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f058c077f70 0x7f058c07a430 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0598034000 tx=0x7f0598004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:50.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.719+0000 7f05a8c00640 1 -- 172.21.15.31:0/2320935572 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f05a0063170 con 0x7f05a400cf20 2024-09-11T00:52:50.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.879+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98040765f0 con 0x7f98040c44f0 2024-09-11T00:52:50.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.883+0000 7f9801600640 1 -- 172.21.15.31:0/1909887629 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+4883 (secure 0 0 0) 0x7f97f8062010 con 0x7f98040c44f0 2024-09-11T00:52:50.886 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:50.886 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":162,"btime":"2024-09-11T00:52:49:521437+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:49.521434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:52:50.886 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 162 2024-09-11T00:52:50.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97e4077fa0 msgr2=0x7f97e407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:50.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97e4077fa0 0x7f97e407a460 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f97f0034000 tx=0x7f97f000b040 comp rx=0 tx=0).stop 2024-09-11T00:52:50.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98040c44f0 msgr2=0x7f9804156af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:50.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98040c44f0 0x7f9804156af0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f97f800c500 tx=0x7f97f800c9c0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 shutdown_connections 2024-09-11T00:52:50.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f97e4077fa0 0x7f97e407a460 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f98040c8550 0x7f98040aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9804157a70 0x7f980414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 --2- 172.21.15.31:0/1909887629 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98040c44f0 0x7f9804156af0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 >> 172.21.15.31:0/1909887629 conn(0x7f98040ab8a0 msgr2=0x7f9804150f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.888+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 shutdown_connections 2024-09-11T00:52:50.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:50.889+0000 7f980c600640 1 -- 172.21.15.31:0/1909887629 wait complete. 2024-09-11T00:52:50.922 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:52:51.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.159+0000 7f05b2000640 1 -- 172.21.15.31:0/2320935572 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05a40332e0 con 0x7f05a400cf20 2024-09-11T00:52:51.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.160+0000 7f05a8c00640 1 -- 172.21.15.31:0/2320935572 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) ==== 95+0+5716 (secure 0 0 0) 0x7f05a0062a70 con 0x7f05a400cf20 2024-09-11T00:52:51.162 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:51.162 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":163,"btime":"2024-09-11T00:52:51:001704+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:50.426732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:reconnect","state_seq":143,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:52:51.162 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 163 2024-09-11T00:52:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.163+0000 7f05b2000640 1 -- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f058c077f70 msgr2=0x7f058c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.163+0000 7f05b2000640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f058c077f70 0x7f058c07a430 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0598034000 tx=0x7f0598004730 comp rx=0 tx=0).stop 2024-09-11T00:52:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.163+0000 7f05b2000640 1 -- 172.21.15.31:0/2320935572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05a400cf20 msgr2=0x7f05a40cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.163+0000 7f05b2000640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05a400cf20 0x7f05a40cdba0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c390 tx=0x7f05a000c850 comp rx=0 tx=0).stop 2024-09-11T00:52:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.164+0000 7f05b2000640 1 -- 172.21.15.31:0/2320935572 shutdown_connections 2024-09-11T00:52:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.164+0000 7f05b2000640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f058c077f70 0x7f058c07a430 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:51.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.164+0000 7f05b2000640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f05a404d780 0x7f05a416fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.164+0000 7f05b2000640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05a404b260 0x7f05a4008850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.164+0000 7f05b2000640 1 --2- 172.21.15.31:0/2320935572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05a400cf20 0x7f05a40cdba0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.164+0000 7f05b2000640 1 -- 172.21.15.31:0/2320935572 >> 172.21.15.31:0/2320935572 conn(0x7f05a4009d30 msgr2=0x7f05a404ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.164+0000 7f05b2000640 1 -- 172.21.15.31:0/2320935572 shutdown_connections 2024-09-11T00:52:51.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:51.164+0000 7f05b2000640 1 -- 172.21.15.31:0/2320935572 wait complete. 2024-09-11T00:52:51.190 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:52:51:001704+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 163, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 161, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7357": { "addr": "172.21.15.204:6835/4094068967", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 4094068967, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 4094068967, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7357, "incarnation": 158, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:52:42.957872+0000", "qdb_cluster": [ 7357 ], "qdb_leader": 7357, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7357 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 163, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5693": { "addr": "172.21.15.31:6833/2956367495", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 2956367495, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 2956367495, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5693, "incarnation": 162, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 143 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:52:50.426732+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5693 } } } ], "standbys": [ { "addr": "172.21.15.204:6837/2689217598", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 2689217598, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 2689217598, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 163, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7429, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:52:52.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:52:52.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:52.926 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:52:53.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:52:53.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.207+0000 7f6558a00640 1 -- 172.21.15.31:0/2649205256 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c004640 con 0x7f6540003a50 2024-09-11T00:52:53.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.208+0000 7f655b200640 1 -- 172.21.15.31:0/2649205256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6540003a50 msgr2=0x7f6540003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:53.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.208+0000 7f655b200640 1 --2- 172.21.15.31:0/2649205256 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6540003a50 0x7f6540003e70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f654c006f20 tx=0x7f6540005830 comp rx=0 tx=0).stop 2024-09-11T00:52:53.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.209+0000 7f655a800640 1 -- 172.21.15.31:0/2649205256 reap_dead start 2024-09-11T00:52:53.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.209+0000 7f655b200640 1 -- 172.21.15.31:0/2649205256 shutdown_connections 2024-09-11T00:52:53.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.209+0000 7f655b200640 1 -- 172.21.15.31:0/2649205256 >> 172.21.15.31:0/2649205256 conn(0x7f655c0a9600 msgr2=0x7f655c0a99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:53.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.209+0000 7f655b200640 1 -- 172.21.15.31:0/2649205256 shutdown_connections 2024-09-11T00:52:53.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.210+0000 7f655b200640 1 -- 172.21.15.31:0/2649205256 wait complete. 2024-09-11T00:52:53.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.212+0000 7f655b200640 1 Processor -- start 2024-09-11T00:52:53.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.212+0000 7f655b200640 1 -- start start 2024-09-11T00:52:53.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.214+0000 7f655b200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f655c14e620 0x7f655c0bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:53.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.214+0000 7f655b200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f655c152950 0x7f655c0c0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:53.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.214+0000 7f655b200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f655c14ea00 0x7f655c150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:53.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.214+0000 7f6559400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f655c152950 0x7f655c0c0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:53.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.215+0000 7f6559400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f655c152950 0x7f655c0c0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:42578/0 (socket says 172.21.15.31:42578) 2024-09-11T00:52:53.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.215+0000 7f655a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f655c14ea00 0x7f655c150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.215+0000 7f655a800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f655c14ea00 0x7f655c150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40034/0 (socket says 172.21.15.31:40034) 2024-09-11T00:52:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.215+0000 7f655b200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f655c0a6ff0 con 0x7f655c14ea00 2024-09-11T00:52:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.215+0000 7f655b200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f655c0a6d10 con 0x7f655c14e620 2024-09-11T00:52:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.215+0000 7f655b200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f655c1562c0 con 0x7f655c152950 2024-09-11T00:52:53.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.215+0000 7f6559400640 1 -- 172.21.15.31:0/1474409994 learned_addr learned my addr 172.21.15.31:0/1474409994 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:53.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.216+0000 7f655a800640 1 -- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f655c152950 msgr2=0x7f655c0c0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:53.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.216+0000 7f655a800640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f655c152950 0x7f655c0c0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:53.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.216+0000 7f655a800640 1 -- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f655c14e620 msgr2=0x7f655c0bfde0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:52:53.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.216+0000 7f655a800640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f655c14e620 0x7f655c0bfde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:53.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.216+0000 7f655a800640 1 -- 172.21.15.31:0/1474409994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c14ef30 con 0x7f655c14ea00 2024-09-11T00:52:53.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.216+0000 7f6559400640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f655c152950 0x7f655c0c0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-11T00:52:53.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.217+0000 7f655a800640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f655c14ea00 0x7f655c150df0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f655c0aa310 tx=0x7f6554009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:53.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.217+0000 7f6552000640 1 -- 172.21.15.31:0/1474409994 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554015070 con 0x7f655c14ea00 2024-09-11T00:52:53.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.218+0000 7f655b200640 1 -- 172.21.15.31:0/1474409994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655c1f0a60 con 0x7f655c14ea00 2024-09-11T00:52:53.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.218+0000 7f655b200640 1 -- 172.21.15.31:0/1474409994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f655c14f210 con 0x7f655c14ea00 2024-09-11T00:52:53.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.218+0000 7f6552000640 1 -- 172.21.15.31:0/1474409994 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65540108a0 con 0x7f655c14ea00 2024-09-11T00:52:53.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.218+0000 7f6552000640 1 -- 172.21.15.31:0/1474409994 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554015070 con 0x7f655c14ea00 2024-09-11T00:52:53.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.220+0000 7f6547400640 1 -- 172.21.15.31:0/1474409994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6534005290 con 0x7f655c14ea00 2024-09-11T00:52:53.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.224+0000 7f6552000640 1 -- 172.21.15.31:0/1474409994 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f655400f040 con 0x7f655c14ea00 2024-09-11T00:52:53.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.225+0000 7f6552000640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f653c078170 0x7f653c07a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:53.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.225+0000 7f6552000640 1 -- 172.21.15.31:0/1474409994 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7f655409ae90 con 0x7f655c14ea00 2024-09-11T00:52:53.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.226+0000 7f6559e00640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f653c078170 0x7f653c07a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:53.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.228+0000 7f6559e00640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f653c078170 0x7f653c07a630 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f654c033000 tx=0x7f654c02d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:53.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.236+0000 7f6552000640 1 -- 172.21.15.31:0/1474409994 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6554062ba0 con 0x7f655c14ea00 2024-09-11T00:52:53.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.474+0000 7f2d91800640 1 -- 172.21.15.31:0/949583597 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d8c0c9b30 msgr2=0x7f2d8c06deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:53.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.474+0000 7f2d91800640 1 --2- 172.21.15.31:0/949583597 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d8c0c9b30 0x7f2d8c06deb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009830 tx=0x7f2d7802a4a0 comp rx=0 tx=0).stop 2024-09-11T00:52:53.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.475+0000 7f2d91800640 1 -- 172.21.15.31:0/949583597 shutdown_connections 2024-09-11T00:52:53.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.475+0000 7f2d91800640 1 --2- 172.21.15.31:0/949583597 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d8c0c9b30 0x7f2d8c06deb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:53.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.475+0000 7f2d91800640 1 --2- 172.21.15.31:0/949583597 >> v2:172.21.15.204:3300/0 conn(0x7f2d8c0b1160 0x7f2d8c15a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:53.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.475+0000 7f2d91800640 1 -- 172.21.15.31:0/949583597 >> 172.21.15.31:0/949583597 conn(0x7f2d8c0cc260 msgr2=0x7f2d8c0ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:53.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.477+0000 7f2d91800640 1 -- 172.21.15.31:0/949583597 shutdown_connections 2024-09-11T00:52:53.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.477+0000 7f2d91800640 1 -- 172.21.15.31:0/949583597 wait complete. 2024-09-11T00:52:53.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.482+0000 7f2d91800640 1 Processor -- start 2024-09-11T00:52:53.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.482+0000 7f2d91800640 1 -- start start 2024-09-11T00:52:53.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.483+0000 7f2d91800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c0b1160 0x7f2d8c14ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:53.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.483+0000 7f2d91800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d8c0bf2c0 0x7f2d8c158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:53.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.484+0000 7f2d91800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d8c0c9b30 0x7f2d8c1e17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:53.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.484+0000 7f2d91800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c0ae540 con 0x7f2d8c0b1160 2024-09-11T00:52:53.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.484+0000 7f2d91800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c0b1540 con 0x7f2d8c0c9b30 2024-09-11T00:52:53.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.484+0000 7f2d91800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2d8c0ca960 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.484+0000 7f2d8a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d8c0bf2c0 0x7f2d8c158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:53.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.484+0000 7f2d8a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d8c0bf2c0 0x7f2d8c158a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:42596/0 (socket says 172.21.15.31:42596) 2024-09-11T00:52:53.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.484+0000 7f2d8a000640 1 -- 172.21.15.31:0/14384425 learned_addr learned my addr 172.21.15.31:0/14384425 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:52:53.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.486+0000 7f2d8a000640 1 -- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d8c0c9b30 msgr2=0x7f2d8c1e17e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:52:53.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.486+0000 7f2d8a000640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d8c0c9b30 0x7f2d8c1e17e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:53.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.486+0000 7f2d8a000640 1 -- 172.21.15.31:0/14384425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c0b1160 msgr2=0x7f2d8c14ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:53.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.486+0000 7f2d8a000640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c0b1160 0x7f2d8c14ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:53.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.486+0000 7f2d8a000640 1 -- 172.21.15.31:0/14384425 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8c154050 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.487+0000 7f2d8a000640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d8c0bf2c0 0x7f2d8c158a50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009950 tx=0x7f2d78033ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:53.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.488+0000 7f2d82a00640 1 -- 172.21.15.31:0/14384425 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78031d60 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.488+0000 7f2d91800640 1 -- 172.21.15.31:0/14384425 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8c14f190 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.488+0000 7f2d91800640 1 -- 172.21.15.31:0/14384425 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d8c1e0fe0 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.488+0000 7f2d82a00640 1 -- 172.21.15.31:0/14384425 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d78033e50 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.488+0000 7f2d82a00640 1 -- 172.21.15.31:0/14384425 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78005260 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.493+0000 7f2d91800640 1 -- 172.21.15.31:0/14384425 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d64005290 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.493+0000 7f2d82a00640 1 -- 172.21.15.31:0/14384425 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2d78005400 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.495+0000 7f2d82a00640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2d6c077f70 0x7f2d6c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:52:53.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.495+0000 7f2d82a00640 1 -- 172.21.15.31:0/14384425 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7f2d780bba40 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.496+0000 7f2d8aa00640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2d6c077f70 0x7f2d6c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:52:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.504+0000 7f2d8aa00640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2d6c077f70 0x7f2d6c07a430 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0058c0 tx=0x7f2d7c005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:52:53.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.505+0000 7f2d82a00640 1 -- 172.21.15.31:0/14384425 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2d780bde20 con 0x7f2d8c0bf2c0 2024-09-11T00:52:53.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.683+0000 7f6547400640 1 -- 172.21.15.31:0/1474409994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6534005580 con 0x7f655c14ea00 2024-09-11T00:52:53.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.685+0000 7f6552000640 1 -- 172.21.15.31:0/1474409994 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+5720 (secure 0 0 0) 0x7f655406b2c0 con 0x7f655c14ea00 2024-09-11T00:52:53.687 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:53.687 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":165,"btime":"2024-09-11T00:52:53:021087+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:53.021082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":145,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5693,"qdb_cluster":[5693]},"id":2}]} 2024-09-11T00:52:53.687 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 165 2024-09-11T00:52:53.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.689+0000 7f6547400640 1 -- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f653c078170 msgr2=0x7f653c07a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.689+0000 7f6547400640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f653c078170 0x7f653c07a630 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f654c033000 tx=0x7f654c02d000 comp rx=0 tx=0).stop 2024-09-11T00:52:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.689+0000 7f6547400640 1 -- 172.21.15.31:0/1474409994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f655c14ea00 msgr2=0x7f655c150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.689+0000 7f6547400640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f655c14ea00 0x7f655c150df0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f655c0aa310 tx=0x7f6554009b00 comp rx=0 tx=0).stop 2024-09-11T00:52:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.689+0000 7f6547400640 1 -- 172.21.15.31:0/1474409994 shutdown_connections 2024-09-11T00:52:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.689+0000 7f6547400640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f653c078170 0x7f653c07a630 secure :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f654c033000 tx=0x7f654c02d000 comp rx=0 tx=0).stop 2024-09-11T00:52:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.689+0000 7f6547400640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f655c14ea00 0x7f655c150df0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.689+0000 7f6547400640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f655c152950 0x7f655c0c0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.689+0000 7f6547400640 1 --2- 172.21.15.31:0/1474409994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f655c14e620 0x7f655c0bfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:53.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.689+0000 7f6547400640 1 -- 172.21.15.31:0/1474409994 >> 172.21.15.31:0/1474409994 conn(0x7f655c0a9600 msgr2=0x7f655c0a7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.695+0000 7f6547400640 1 -- 172.21.15.31:0/1474409994 shutdown_connections 2024-09-11T00:52:53.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:53.695+0000 7f6547400640 1 -- 172.21.15.31:0/1474409994 wait complete. 2024-09-11T00:52:53.733 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-11T00:52:53.734 INFO:tasks.mds_thrash.fs.[a]:waiting for 114 secs before thrashing 2024-09-11T00:52:54.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.083+0000 7f2d91800640 1 -- 172.21.15.31:0/14384425 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d64005580 con 0x7f2d8c0bf2c0 2024-09-11T00:52:54.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.084+0000 7f2d82a00640 1 -- 172.21.15.31:0/14384425 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+5720 (secure 0 0 0) 0x7f2d7803f030 con 0x7f2d8c0bf2c0 2024-09-11T00:52:54.088 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:52:54.088 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":165,"btime":"2024-09-11T00:52:53:021087+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:53.021082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":145,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5693,"qdb_cluster":[5693]},"id":2}]} 2024-09-11T00:52:54.089 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 165 2024-09-11T00:52:54.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.087+0000 7f2d91800640 1 -- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2d6c077f70 msgr2=0x7f2d6c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:54.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.087+0000 7f2d91800640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2d6c077f70 0x7f2d6c07a430 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0058c0 tx=0x7f2d7c005850 comp rx=0 tx=0).stop 2024-09-11T00:52:54.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.087+0000 7f2d91800640 1 -- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d8c0bf2c0 msgr2=0x7f2d8c158a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:52:54.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.087+0000 7f2d91800640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d8c0bf2c0 0x7f2d8c158a50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009950 tx=0x7f2d78033ba0 comp rx=0 tx=0).stop 2024-09-11T00:52:54.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.087+0000 7f2d91800640 1 -- 172.21.15.31:0/14384425 shutdown_connections 2024-09-11T00:52:54.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.087+0000 7f2d91800640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2d6c077f70 0x7f2d6c07a430 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:54.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.087+0000 7f2d91800640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d8c0c9b30 0x7f2d8c1e17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:54.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.087+0000 7f2d91800640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2d8c0bf2c0 0x7f2d8c158a50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:54.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.087+0000 7f2d91800640 1 --2- 172.21.15.31:0/14384425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c0b1160 0x7f2d8c14ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:52:54.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.087+0000 7f2d91800640 1 -- 172.21.15.31:0/14384425 >> 172.21.15.31:0/14384425 conn(0x7f2d8c0cc260 msgr2=0x7f2d8c0adbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:52:54.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.089+0000 7f2d91800640 1 -- 172.21.15.31:0/14384425 shutdown_connections 2024-09-11T00:52:54.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:52:54.089+0000 7f2d91800640 1 -- 172.21.15.31:0/14384425 wait complete. 2024-09-11T00:52:54.120 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=7255 2024-09-11T00:52:54.120 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.c 2024-09-11T00:52:56.790 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-11T00:52:56.788+0000 7f249e200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:53:01.121 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-11T00:53:01.121 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-11T00:53:01.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-11T00:53:01.128 INFO:tasks.ceph.mds.c:Started 2024-09-11T00:53:01.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:53:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.446+0000 7f1bc5a00640 1 -- 172.21.15.31:0/2158697749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bc00c7460 msgr2=0x7f1bc0159fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:53:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.446+0000 7f1bc5a00640 1 --2- 172.21.15.31:0/2158697749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bc00c7460 0x7f1bc0159fe0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0159790 tx=0x7f1bb002a530 comp rx=0 tx=0).stop 2024-09-11T00:53:01.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.448+0000 7f1bc5a00640 1 -- 172.21.15.31:0/2158697749 shutdown_connections 2024-09-11T00:53:01.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.448+0000 7f1bc5a00640 1 --2- 172.21.15.31:0/2158697749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bc00c7460 0x7f1bc0159fe0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:01.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.448+0000 7f1bc5a00640 1 --2- 172.21.15.31:0/2158697749 >> v2:172.21.15.204:3300/0 conn(0x7f1bc00c2d40 0x7f1bc0159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:01.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.448+0000 7f1bc5a00640 1 -- 172.21.15.31:0/2158697749 >> 172.21.15.31:0/2158697749 conn(0x7f1bc00a9750 msgr2=0x7f1bc00a9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:53:01.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.449+0000 7f1bc5a00640 1 -- 172.21.15.31:0/2158697749 shutdown_connections 2024-09-11T00:53:01.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.451+0000 7f1bc5a00640 1 -- 172.21.15.31:0/2158697749 wait complete. 2024-09-11T00:53:01.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.454+0000 7f1bc5a00640 1 Processor -- start 2024-09-11T00:53:01.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.454+0000 7f1bc5a00640 1 -- start start 2024-09-11T00:53:01.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.461+0000 7f1bc5a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bc00c2d40 0x7f1bc0149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:53:01.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.461+0000 7f1bc5a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bc00c7460 0x7f1bc014c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:53:01.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.461+0000 7f1bc5a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bc01537c0 0x7f1bc0159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:53:01.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.461+0000 7f1bc5a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc00c6c60 con 0x7f1bc01537c0 2024-09-11T00:53:01.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.461+0000 7f1bc5a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc00c2960 con 0x7f1bc00c7460 2024-09-11T00:53:01.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.461+0000 7f1bc5a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1bc01560a0 con 0x7f1bc00c2d40 2024-09-11T00:53:01.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.462+0000 7f1bbe000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bc00c7460 0x7f1bc014c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:53:01.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.462+0000 7f1bbe000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bc00c7460 0x7f1bc014c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:38474/0 (socket says 172.21.15.31:38474) 2024-09-11T00:53:01.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.462+0000 7f1bbe000640 1 -- 172.21.15.31:0/1382179155 learned_addr learned my addr 172.21.15.31:0/1382179155 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:53:01.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.464+0000 7f1bbe000640 1 -- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bc00c2d40 msgr2=0x7f1bc0149980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:53:01.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.464+0000 7f1bbe000640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bc00c2d40 0x7f1bc0149980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:01.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.464+0000 7f1bbe000640 1 -- 172.21.15.31:0/1382179155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bc01537c0 msgr2=0x7f1bc0159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:53:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.465+0000 7f1bbf400640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bc01537c0 0x7f1bc0159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:53:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.468+0000 7f1bbe000640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bc01537c0 0x7f1bc0159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:01.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.468+0000 7f1bbe000640 1 -- 172.21.15.31:0/1382179155 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc0148fb0 con 0x7f1bc00c7460 2024-09-11T00:53:01.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.468+0000 7f1bbf400640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bc01537c0 0x7f1bc0159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:53:01.472 INFO:tasks.ceph.mds.c.smithi031.stdout:starting mds.c at 2024-09-11T00:53:01.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.472+0000 7f1bbe000640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bc00c7460 0x7f1bc014c1f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1bb800eb60 tx=0x7f1bb800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:53:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.473+0000 7f1bb6a00640 1 -- 172.21.15.31:0/1382179155 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8010630 con 0x7f1bc00c7460 2024-09-11T00:53:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.473+0000 7f1bb6a00640 1 -- 172.21.15.31:0/1382179155 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bb80108c0 con 0x7f1bc00c7460 2024-09-11T00:53:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.473+0000 7f1bb6a00640 1 -- 172.21.15.31:0/1382179155 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8010d10 con 0x7f1bc00c7460 2024-09-11T00:53:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.473+0000 7f1bc5a00640 1 -- 172.21.15.31:0/1382179155 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc0149290 con 0x7f1bc00c7460 2024-09-11T00:53:01.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.473+0000 7f1bc5a00640 1 -- 172.21.15.31:0/1382179155 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc008cbe0 con 0x7f1bc00c7460 2024-09-11T00:53:01.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.475+0000 7f1bc5a00640 1 -- 172.21.15.31:0/1382179155 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bc0153ba0 con 0x7f1bc00c7460 2024-09-11T00:53:01.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.482+0000 7f1bb6a00640 1 -- 172.21.15.31:0/1382179155 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1bb8018420 con 0x7f1bc00c7460 2024-09-11T00:53:01.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.484+0000 7f1bb6a00640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ba0077f70 0x7f1ba007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:53:01.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.484+0000 7f1bb6a00640 1 -- 172.21.15.31:0/1382179155 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7f1bb809b8e0 con 0x7f1bc00c7460 2024-09-11T00:53:01.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.484+0000 7f1bbea00640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ba0077f70 0x7f1ba007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:53:01.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.485+0000 7f1bbea00640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ba0077f70 0x7f1ba007a430 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1bac0015a0 tx=0x7f1bac001210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:53:01.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.492+0000 7f1bb6a00640 1 -- 172.21.15.31:0/1382179155 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1bb800ee60 con 0x7f1bc00c7460 2024-09-11T00:53:01.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.987+0000 7f1bc5a00640 1 -- 172.21.15.31:0/1382179155 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bc00c68b0 con 0x7f1bc00c7460 2024-09-11T00:53:01.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.991+0000 7f1bb6a00640 1 -- 172.21.15.31:0/1382179155 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+5720 (secure 0 0 0) 0x7f1bb80635f0 con 0x7f1bc00c7460 2024-09-11T00:53:01.993 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:53:01.993 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":165,"btime":"2024-09-11T00:52:53:021087+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:53.021082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":145,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5693,"qdb_cluster":[5693]},"id":2}]} 2024-09-11T00:53:01.995 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 165 2024-09-11T00:53:02.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:01.999+0000 7f1bc5a00640 1 -- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ba0077f70 msgr2=0x7f1ba007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:53:02.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.000+0000 7f1bc5a00640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ba0077f70 0x7f1ba007a430 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1bac0015a0 tx=0x7f1bac001210 comp rx=0 tx=0).stop 2024-09-11T00:53:02.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.000+0000 7f1bc5a00640 1 -- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bc00c7460 msgr2=0x7f1bc014c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:53:02.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.000+0000 7f1bc5a00640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bc00c7460 0x7f1bc014c1f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1bb800eb60 tx=0x7f1bb800c6a0 comp rx=0 tx=0).stop 2024-09-11T00:53:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.004+0000 7f1bc5a00640 1 -- 172.21.15.31:0/1382179155 shutdown_connections 2024-09-11T00:53:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.005+0000 7f1bc5a00640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ba0077f70 0x7f1ba007a430 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.005+0000 7f1bc5a00640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1bc01537c0 0x7f1bc0159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.005+0000 7f1bc5a00640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1bc00c7460 0x7f1bc014c1f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:02.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.005+0000 7f1bc5a00640 1 --2- 172.21.15.31:0/1382179155 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1bc00c2d40 0x7f1bc0149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:02.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.005+0000 7f1bc5a00640 1 -- 172.21.15.31:0/1382179155 >> 172.21.15.31:0/1382179155 conn(0x7f1bc00a9750 msgr2=0x7f1bc00c57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:53:02.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.005+0000 7f1bc5a00640 1 -- 172.21.15.31:0/1382179155 shutdown_connections 2024-09-11T00:53:02.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:02.005+0000 7f1bc5a00640 1 -- 172.21.15.31:0/1382179155 wait complete. 2024-09-11T00:53:02.036 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-11T00:53:04.037 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:53:04.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.337+0000 7fba6f000640 1 -- 172.21.15.31:0/2897060722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba60046530 msgr2=0x7fba6004a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:53:04.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.338+0000 7fba67e00640 1 -- 172.21.15.31:0/2897060722 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c02aa40 con 0x7fba60046530 2024-09-11T00:53:04.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.338+0000 7fba6f000640 1 --2- 172.21.15.31:0/2897060722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba60046530 0x7fba6004a740 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fba6004c0e0 tx=0x7fba5c00ac00 comp rx=0 tx=0).stop 2024-09-11T00:53:04.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.340+0000 7fba6f000640 1 -- 172.21.15.31:0/2897060722 shutdown_connections 2024-09-11T00:53:04.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.340+0000 7fba6f000640 1 --2- 172.21.15.31:0/2897060722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba60046530 0x7fba6004a740 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:04.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.340+0000 7fba6f000640 1 --2- 172.21.15.31:0/2897060722 >> v2:172.21.15.204:3300/0 conn(0x7fba600d3480 0x7fba60045b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:04.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.340+0000 7fba6f000640 1 -- 172.21.15.31:0/2897060722 >> 172.21.15.31:0/2897060722 conn(0x7fba60047e00 msgr2=0x7fba60048210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:53:04.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.340+0000 7fba6f000640 1 -- 172.21.15.31:0/2897060722 shutdown_connections 2024-09-11T00:53:04.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.340+0000 7fba6f000640 1 -- 172.21.15.31:0/2897060722 wait complete. 2024-09-11T00:53:04.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.344+0000 7fba6f000640 1 Processor -- start 2024-09-11T00:53:04.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.345+0000 7fba6f000640 1 -- start start 2024-09-11T00:53:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.347+0000 7fba6f000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba60046530 0x7fba600c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:53:04.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.348+0000 7fba6f000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fba6004ac80 0x7fba6002fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:53:04.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.348+0000 7fba6f000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba600d3480 0x7fba60173870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:53:04.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.348+0000 7fba6f000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fba60049710 con 0x7fba60046530 2024-09-11T00:53:04.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.348+0000 7fba6f000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fba60049430 con 0x7fba600d3480 2024-09-11T00:53:04.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.348+0000 7fba6f000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fba600499f0 con 0x7fba6004ac80 2024-09-11T00:53:04.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.349+0000 7fba6e600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba600d3480 0x7fba60173870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:53:04.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.351+0000 7fba6e600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba600d3480 0x7fba60173870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:38506/0 (socket says 172.21.15.31:38506) 2024-09-11T00:53:04.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.351+0000 7fba6e600640 1 -- 172.21.15.31:0/3718912657 learned_addr learned my addr 172.21.15.31:0/3718912657 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:53:04.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.351+0000 7fba6dc00640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba60046530 0x7fba600c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:53:04.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.351+0000 7fba6dc00640 1 -- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fba6004ac80 msgr2=0x7fba6002fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:53:04.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.351+0000 7fba6dc00640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fba6004ac80 0x7fba6002fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:04.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.352+0000 7fba6dc00640 1 -- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba600d3480 msgr2=0x7fba60173870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:53:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.352+0000 7fba6dc00640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba600d3480 0x7fba60173870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.352+0000 7fba6dc00640 1 -- 172.21.15.31:0/3718912657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba600c8b70 con 0x7fba60046530 2024-09-11T00:53:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.352+0000 7fba6dc00640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba60046530 0x7fba600c8630 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fba6806b230 tx=0x7fba6806a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:53:04.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.353+0000 7fba65600640 1 -- 172.21.15.31:0/3718912657 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68099070 con 0x7fba60046530 2024-09-11T00:53:04.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.353+0000 7fba65600640 1 -- 172.21.15.31:0/3718912657 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba680737d0 con 0x7fba60046530 2024-09-11T00:53:04.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.353+0000 7fba65600640 1 -- 172.21.15.31:0/3718912657 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6806ce60 con 0x7fba60046530 2024-09-11T00:53:04.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.353+0000 7fba6f000640 1 -- 172.21.15.31:0/3718912657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba600c8390 con 0x7fba60046530 2024-09-11T00:53:04.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.353+0000 7fba6f000640 1 -- 172.21.15.31:0/3718912657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba601771b0 con 0x7fba60046530 2024-09-11T00:53:04.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.358+0000 7fba6f000640 1 -- 172.21.15.31:0/3718912657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba600c8d90 con 0x7fba60046530 2024-09-11T00:53:04.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.362+0000 7fba65600640 1 -- 172.21.15.31:0/3718912657 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fba68073ba0 con 0x7fba60046530 2024-09-11T00:53:04.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.363+0000 7fba65600640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba48077fa0 0x7fba4807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:53:04.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.363+0000 7fba65600640 1 -- 172.21.15.31:0/3718912657 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7fba68116c90 con 0x7fba60046530 2024-09-11T00:53:04.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.366+0000 7fba6d200640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba48077fa0 0x7fba4807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:53:04.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.367+0000 7fba6d200640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba48077fa0 0x7fba4807a460 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fba5c031000 tx=0x7fba5c00c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:53:04.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.369+0000 7fba65600640 1 -- 172.21.15.31:0/3718912657 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fba600c8b70 con 0x7fba60046530 2024-09-11T00:53:04.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.912+0000 7fba6f000640 1 -- 172.21.15.31:0/3718912657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba60030180 con 0x7fba60046530 2024-09-11T00:53:04.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.916+0000 7fba65600640 1 -- 172.21.15.31:0/3718912657 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6546 (secure 0 0 0) 0x7fba680de940 con 0x7fba60046530 2024-09-11T00:53:04.918 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:53:04.918 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":166,"btime":"2024-09-11T00:53:02:065604+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:53.021082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":145,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5693,"qdb_cluster":[5693]},"id":2}]} 2024-09-11T00:53:04.918 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 166 2024-09-11T00:53:04.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.919+0000 7fba6f000640 1 -- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba48077fa0 msgr2=0x7fba4807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:53:04.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.920+0000 7fba6f000640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba48077fa0 0x7fba4807a460 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fba5c031000 tx=0x7fba5c00c000 comp rx=0 tx=0).stop 2024-09-11T00:53:04.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.920+0000 7fba6f000640 1 -- 172.21.15.31:0/3718912657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba60046530 msgr2=0x7fba600c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:53:04.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.920+0000 7fba6f000640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba60046530 0x7fba600c8630 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fba6806b230 tx=0x7fba6806a540 comp rx=0 tx=0).stop 2024-09-11T00:53:04.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.920+0000 7fba6f000640 1 -- 172.21.15.31:0/3718912657 shutdown_connections 2024-09-11T00:53:04.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.921+0000 7fba6f000640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fba48077fa0 0x7fba4807a460 secure :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fba5c031000 tx=0x7fba5c00c000 comp rx=0 tx=0).stop 2024-09-11T00:53:04.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.921+0000 7fba6f000640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba600d3480 0x7fba60173870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:04.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.921+0000 7fba6f000640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fba6004ac80 0x7fba6002fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:04.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.921+0000 7fba6f000640 1 --2- 172.21.15.31:0/3718912657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba60046530 0x7fba600c8630 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:53:04.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.921+0000 7fba6f000640 1 -- 172.21.15.31:0/3718912657 >> 172.21.15.31:0/3718912657 conn(0x7fba60047e00 msgr2=0x7fba6002ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:53:04.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.925+0000 7fba6f000640 1 -- 172.21.15.31:0/3718912657 shutdown_connections 2024-09-11T00:53:04.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:53:04.925+0000 7fba6f000640 1 -- 172.21.15.31:0/3718912657 wait complete. 2024-09-11T00:53:04.960 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-11T00:53:04.960 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2024-09-11T00:53:23.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:53:23.063 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:53:53.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:53:53.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:54:06.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.356+0000 7f9be7a00640 1 -- 172.21.15.31:0/1056658940 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9be00c22c0 msgr2=0x7f9be00c3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.356+0000 7f9be7a00640 1 --2- 172.21.15.31:0/1056658940 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9be00c22c0 0x7f9be00c3920 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800bcb0 tx=0x7f9bd802b5e0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.357+0000 7f9be7a00640 1 -- 172.21.15.31:0/1056658940 shutdown_connections 2024-09-11T00:54:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.357+0000 7f9be7a00640 1 --2- 172.21.15.31:0/1056658940 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9be00c22c0 0x7f9be00c3920 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.357+0000 7f9be7a00640 1 --2- 172.21.15.31:0/1056658940 >> v2:172.21.15.31:3300/0 conn(0x7f9be006e170 0x7f9be00a6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.357+0000 7f9be7a00640 1 -- 172.21.15.31:0/1056658940 >> 172.21.15.31:0/1056658940 conn(0x7f9be00a9670 msgr2=0x7f9be00a9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:07.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.359+0000 7f9be7a00640 1 -- 172.21.15.31:0/1056658940 shutdown_connections 2024-09-11T00:54:07.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.359+0000 7f9be7a00640 1 -- 172.21.15.31:0/1056658940 wait complete. 2024-09-11T00:54:07.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.367+0000 7f9be7a00640 1 Processor -- start 2024-09-11T00:54:07.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.368+0000 7f9be7a00640 1 -- start start 2024-09-11T00:54:07.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.372+0000 7f9be7a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9be006e170 0x7f9be0153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:07.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.373+0000 7f9be7a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9be00c22c0 0x7f9be00abb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:07.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.373+0000 7f9be7a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9be014f720 0x7f9be014b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:07.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.373+0000 7f9be7a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9be0152320 con 0x7f9be014f720 2024-09-11T00:54:07.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.373+0000 7f9be7a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9be0152040 con 0x7f9be006e170 2024-09-11T00:54:07.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.373+0000 7f9be7a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9be014fb00 con 0x7f9be00c22c0 2024-09-11T00:54:07.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.374+0000 7f9bdfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9be006e170 0x7f9be0153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.374+0000 7f9bdfe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9be006e170 0x7f9be0153320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:49186/0 (socket says 172.21.15.31:49186) 2024-09-11T00:54:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.374+0000 7f9bdfe00640 1 -- 172.21.15.31:0/4062888203 learned_addr learned my addr 172.21.15.31:0/4062888203 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.374+0000 7f9be5000640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9be014f720 0x7f9be014b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:07.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.375+0000 7f9bdfe00640 1 -- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9be00c22c0 msgr2=0x7f9be00abb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:07.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.375+0000 7f9bdfe00640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9be00c22c0 0x7f9be00abb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.375+0000 7f9bdfe00640 1 -- 172.21.15.31:0/4062888203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9be014f720 msgr2=0x7f9be014b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:07.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.375+0000 7f9bdfe00640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9be014f720 0x7f9be014b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.375+0000 7f9bdfe00640 1 -- 172.21.15.31:0/4062888203 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be01f0f90 con 0x7f9be006e170 2024-09-11T00:54:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.375+0000 7f9be5000640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9be014f720 0x7f9be014b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:54:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.376+0000 7f9bdfe00640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9be006e170 0x7f9be0153320 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400c530 tx=0x7f9bd400c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.377+0000 7f9bdcc00640 1 -- 172.21.15.31:0/4062888203 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4016020 con 0x7f9be006e170 2024-09-11T00:54:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.377+0000 7f9bdcc00640 1 -- 172.21.15.31:0/4062888203 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9bd4012920 con 0x7f9be006e170 2024-09-11T00:54:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.377+0000 7f9bdcc00640 1 -- 172.21.15.31:0/4062888203 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4012d50 con 0x7f9be006e170 2024-09-11T00:54:07.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.379+0000 7f9be7a00640 1 -- 172.21.15.31:0/4062888203 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be014bdf0 con 0x7f9be006e170 2024-09-11T00:54:07.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.379+0000 7f9be7a00640 1 -- 172.21.15.31:0/4062888203 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be00ac040 con 0x7f9be006e170 2024-09-11T00:54:07.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.381+0000 7f9bd2000640 1 -- 172.21.15.31:0/4062888203 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bb4005290 con 0x7f9be006e170 2024-09-11T00:54:07.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.384+0000 7f9bdcc00640 1 -- 172.21.15.31:0/4062888203 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9bd4023060 con 0x7f9be006e170 2024-09-11T00:54:07.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.386+0000 7f9bdcc00640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9bc0078040 0x7f9bc007a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:07.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.387+0000 7f9bdf400640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9bc0078040 0x7f9bc007a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:07.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.388+0000 7f9bdcc00640 1 -- 172.21.15.31:0/4062888203 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7f9bd409adc0 con 0x7f9be006e170 2024-09-11T00:54:07.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.389+0000 7f9bdf400640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9bc0078040 0x7f9bc007a500 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9be00a6d50 tx=0x7f9bcc005ce0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:07.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.390+0000 7f9bdcc00640 1 -- 172.21.15.31:0/4062888203 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9bd4062c00 con 0x7f9be006e170 2024-09-11T00:54:07.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.940+0000 7f9bd2000640 1 -- 172.21.15.31:0/4062888203 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bb4005580 con 0x7f9be006e170 2024-09-11T00:54:07.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.943+0000 7f9bdcc00640 1 -- 172.21.15.31:0/4062888203 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6546 (secure 0 0 0) 0x7f9bd4062500 con 0x7f9be006e170 2024-09-11T00:54:07.945 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:07.946 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":166,"btime":"2024-09-11T00:53:02:065604+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:53.021082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":145,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5693,"qdb_cluster":[5693]},"id":2}]} 2024-09-11T00:54:07.946 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 166 2024-09-11T00:54:07.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.956+0000 7f9bd2000640 1 -- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9bc0078040 msgr2=0x7f9bc007a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:07.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.956+0000 7f9bd2000640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9bc0078040 0x7f9bc007a500 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9be00a6d50 tx=0x7f9bcc005ce0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.956+0000 7f9bd2000640 1 -- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9be006e170 msgr2=0x7f9be0153320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:07.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.956+0000 7f9bd2000640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9be006e170 0x7f9be0153320 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400c530 tx=0x7f9bd400c9f0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.957+0000 7f9bd2000640 1 -- 172.21.15.31:0/4062888203 shutdown_connections 2024-09-11T00:54:07.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.957+0000 7f9bd2000640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9bc0078040 0x7f9bc007a500 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.957+0000 7f9bd2000640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9be014f720 0x7f9be014b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.957+0000 7f9bd2000640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9be00c22c0 0x7f9be00abb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.957+0000 7f9bd2000640 1 --2- 172.21.15.31:0/4062888203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9be006e170 0x7f9be0153320 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:07.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.957+0000 7f9bd2000640 1 -- 172.21.15.31:0/4062888203 >> 172.21.15.31:0/4062888203 conn(0x7f9be00a9670 msgr2=0x7f9be014ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:07.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.958+0000 7f9bd2000640 1 -- 172.21.15.31:0/4062888203 shutdown_connections 2024-09-11T00:54:07.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:07.959+0000 7f9bd2000640 1 -- 172.21.15.31:0/4062888203 wait complete. 2024-09-11T00:54:08.001 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-11T00:54:08.001 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-11T00:54:08.001 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:54:08.038 INFO:tasks.ceph.mds.a:Stopped 2024-09-11T00:54:08.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:08.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.383+0000 7f9d17a00640 1 -- 172.21.15.31:0/2030956165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d10074b90 msgr2=0x7f9d10070970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:08.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.383+0000 7f9d17a00640 1 --2- 172.21.15.31:0/2030956165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d10074b90 0x7f9d10070970 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9d00005fd0 tx=0x7f9d0002a4b0 comp rx=0 tx=0).stop 2024-09-11T00:54:08.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.384+0000 7f9d17a00640 1 -- 172.21.15.31:0/2030956165 shutdown_connections 2024-09-11T00:54:08.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.384+0000 7f9d17a00640 1 --2- 172.21.15.31:0/2030956165 >> v2:172.21.15.204:3300/0 conn(0x7f9d100718b0 0x7f9d10073320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:08.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.384+0000 7f9d17a00640 1 --2- 172.21.15.31:0/2030956165 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d10074b90 0x7f9d10070970 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:08.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.384+0000 7f9d17a00640 1 -- 172.21.15.31:0/2030956165 >> 172.21.15.31:0/2030956165 conn(0x7f9d1006dc00 msgr2=0x7f9d1006e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:08.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.384+0000 7f9d17a00640 1 -- 172.21.15.31:0/2030956165 shutdown_connections 2024-09-11T00:54:08.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.384+0000 7f9d17a00640 1 -- 172.21.15.31:0/2030956165 wait complete. 2024-09-11T00:54:08.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.390+0000 7f9d17a00640 1 Processor -- start 2024-09-11T00:54:08.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.390+0000 7f9d17a00640 1 -- start start 2024-09-11T00:54:08.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.391+0000 7f9d17a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d100718b0 0x7f9d1014f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:08.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.391+0000 7f9d17a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d10074b90 0x7f9d10155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.393+0000 7f9d17a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d100c8cb0 0x7f9d10152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.393+0000 7f9d17a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9d100c9090 con 0x7f9d100718b0 2024-09-11T00:54:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.393+0000 7f9d17a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d10153980 con 0x7f9d10074b90 2024-09-11T00:54:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.393+0000 7f9d17a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f9d100c9210 con 0x7f9d100c8cb0 2024-09-11T00:54:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.393+0000 7f9d15000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d100718b0 0x7f9d1014f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.393+0000 7f9d15000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d100718b0 0x7f9d1014f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38534/0 (socket says 172.21.15.31:38534) 2024-09-11T00:54:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.393+0000 7f9d15000640 1 -- 172.21.15.31:0/4048397808 learned_addr learned my addr 172.21.15.31:0/4048397808 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:08.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.393+0000 7f9d0fe00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d10074b90 0x7f9d10155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.394+0000 7f9d15a00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d100c8cb0 0x7f9d10152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.394+0000 7f9d15000640 1 -- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d100c8cb0 msgr2=0x7f9d10152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.394+0000 7f9d15000640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d100c8cb0 0x7f9d10152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.394+0000 7f9d15000640 1 -- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d10074b90 msgr2=0x7f9d10155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.394+0000 7f9d15000640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d10074b90 0x7f9d10155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:08.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.394+0000 7f9d15000640 1 -- 172.21.15.31:0/4048397808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d100aa4f0 con 0x7f9d100718b0 2024-09-11T00:54:08.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.395+0000 7f9d15000640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d100718b0 0x7f9d1014f3f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800c770 tx=0x7f9d0800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:08.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.396+0000 7f9d0cc00640 1 -- 172.21.15.31:0/4048397808 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d08003b70 con 0x7f9d100718b0 2024-09-11T00:54:08.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.396+0000 7f9d0cc00640 1 -- 172.21.15.31:0/4048397808 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d08003d30 con 0x7f9d100718b0 2024-09-11T00:54:08.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.396+0000 7f9d0cc00640 1 -- 172.21.15.31:0/4048397808 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0800aab0 con 0x7f9d100718b0 2024-09-11T00:54:08.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.396+0000 7f9d0fe00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d10074b90 0x7f9d10155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:54:08.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.397+0000 7f9d17a00640 1 -- 172.21.15.31:0/4048397808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d10154a20 con 0x7f9d100718b0 2024-09-11T00:54:08.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.397+0000 7f9d17a00640 1 -- 172.21.15.31:0/4048397808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d10156150 con 0x7f9d100718b0 2024-09-11T00:54:08.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.402+0000 7f9d17a00640 1 -- 172.21.15.31:0/4048397808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ce8005290 con 0x7f9d100718b0 2024-09-11T00:54:08.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.402+0000 7f9d0cc00640 1 -- 172.21.15.31:0/4048397808 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f9d0800ac50 con 0x7f9d100718b0 2024-09-11T00:54:08.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.405+0000 7f9d0cc00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9cf0078040 0x7f9cf007a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:08.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.405+0000 7f9d0cc00640 1 -- 172.21.15.31:0/4048397808 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7f9d0809a840 con 0x7f9d100718b0 2024-09-11T00:54:08.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.406+0000 7f9d0fe00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9cf0078040 0x7f9cf007a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:08.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.411+0000 7f9d0fe00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9cf0078040 0x7f9cf007a500 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9d00033000 tx=0x7f9d00004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:08.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.419+0000 7f9d0cc00640 1 -- 172.21.15.31:0/4048397808 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f9d08062570 con 0x7f9d100718b0 2024-09-11T00:54:08.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.920+0000 7f9d17a00640 1 -- 172.21.15.31:0/4048397808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ce8005580 con 0x7f9d100718b0 2024-09-11T00:54:08.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.922+0000 7f9d0cc00640 1 -- 172.21.15.31:0/4048397808 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6546 (secure 0 0 0) 0x7f9d08061e70 con 0x7f9d100718b0 2024-09-11T00:54:08.925 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:08.925 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":166,"btime":"2024-09-11T00:53:02:065604+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:53.021082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":145,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5693,"qdb_cluster":[5693]},"id":2}]} 2024-09-11T00:54:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 166 2024-09-11T00:54:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.927+0000 7f9d17a00640 1 -- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9cf0078040 msgr2=0x7f9cf007a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.927+0000 7f9d17a00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9cf0078040 0x7f9cf007a500 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9d00033000 tx=0x7f9d00004730 comp rx=0 tx=0).stop 2024-09-11T00:54:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.927+0000 7f9d17a00640 1 -- 172.21.15.31:0/4048397808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d100718b0 msgr2=0x7f9d1014f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.927+0000 7f9d17a00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d100718b0 0x7f9d1014f3f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800c770 tx=0x7f9d0800cc30 comp rx=0 tx=0).stop 2024-09-11T00:54:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.929+0000 7f9d17a00640 1 -- 172.21.15.31:0/4048397808 shutdown_connections 2024-09-11T00:54:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.929+0000 7f9d17a00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f9cf0078040 0x7f9cf007a500 secure :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9d00033000 tx=0x7f9d00004730 comp rx=0 tx=0).stop 2024-09-11T00:54:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.929+0000 7f9d17a00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f9d100c8cb0 0x7f9d10152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.929+0000 7f9d17a00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d10074b90 0x7f9d10155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.929+0000 7f9d17a00640 1 --2- 172.21.15.31:0/4048397808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d100718b0 0x7f9d1014f3f0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.929+0000 7f9d17a00640 1 -- 172.21.15.31:0/4048397808 >> 172.21.15.31:0/4048397808 conn(0x7f9d1006dc00 msgr2=0x7f9d100a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.931+0000 7f9d17a00640 1 -- 172.21.15.31:0/4048397808 shutdown_connections 2024-09-11T00:54:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:08.932+0000 7f9d17a00640 1 -- 172.21.15.31:0/4048397808 wait complete. 2024-09-11T00:54:08.965 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:54:10.966 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:11.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.263+0000 7f2482000640 1 -- 172.21.15.31:0/1459827064 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478037020 con 0x7f247c0d3490 2024-09-11T00:54:11.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.264+0000 7f2488c00640 1 -- 172.21.15.31:0/1459827064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f247c0d3490 msgr2=0x7f247c045b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:11.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.264+0000 7f2488c00640 1 --2- 172.21.15.31:0/1459827064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f247c0d3490 0x7f247c045b30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2478005fd0 tx=0x7f247802b830 comp rx=0 tx=0).stop 2024-09-11T00:54:11.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.265+0000 7f2488c00640 1 -- 172.21.15.31:0/1459827064 shutdown_connections 2024-09-11T00:54:11.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.265+0000 7f2488c00640 1 --2- 172.21.15.31:0/1459827064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f247c0d3490 0x7f247c045b30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:11.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.265+0000 7f2488c00640 1 -- 172.21.15.31:0/1459827064 >> 172.21.15.31:0/1459827064 conn(0x7f247c047e10 msgr2=0x7f247c048220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:11.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.265+0000 7f2488c00640 1 -- 172.21.15.31:0/1459827064 shutdown_connections 2024-09-11T00:54:11.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.266+0000 7f2488c00640 1 -- 172.21.15.31:0/1459827064 wait complete. 2024-09-11T00:54:11.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.270+0000 7f2488c00640 1 Processor -- start 2024-09-11T00:54:11.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.271+0000 7f2488c00640 1 -- start start 2024-09-11T00:54:11.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.272+0000 7f2488c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f247c046540 0x7f247c0cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:11.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.272+0000 7f2488c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247c0d3490 0x7f247c161ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:11.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.272+0000 7f2488c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f247c0d9370 0x7f247c0d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:11.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.272+0000 7f2488c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f247c049720 con 0x7f247c0d9370 2024-09-11T00:54:11.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.272+0000 7f2488c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f247c049a00 con 0x7f247c046540 2024-09-11T00:54:11.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.272+0000 7f2488c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f247c04ae60 con 0x7f247c0d3490 2024-09-11T00:54:11.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.272+0000 7f2483400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f247c046540 0x7f247c0cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:11.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.272+0000 7f2482a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247c0d3490 0x7f247c161ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:11.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.272+0000 7f2482a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247c0d3490 0x7f247c161ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:55926/0 (socket says 172.21.15.31:55926) 2024-09-11T00:54:11.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.272+0000 7f2482a00640 1 -- 172.21.15.31:0/1666304347 learned_addr learned my addr 172.21.15.31:0/1666304347 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:11.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.273+0000 7f2483e00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f247c0d9370 0x7f247c0d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:11.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.274+0000 7f2483400640 1 -- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247c0d3490 msgr2=0x7f247c161ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:11.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.274+0000 7f2483400640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247c0d3490 0x7f247c161ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:11.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.274+0000 7f2483400640 1 -- 172.21.15.31:0/1666304347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f247c0d9370 msgr2=0x7f247c0d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:11.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.275+0000 7f2483400640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f247c0d9370 0x7f247c0d0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:11.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.275+0000 7f2483400640 1 -- 172.21.15.31:0/1666304347 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247c0cd4e0 con 0x7f247c046540 2024-09-11T00:54:11.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.275+0000 7f2482a00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247c0d3490 0x7f247c161ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:54:11.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.275+0000 7f2483e00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f247c0d9370 0x7f247c0d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:54:11.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.276+0000 7f2483400640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f247c046540 0x7f247c0cdfe0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2478033000 tx=0x7f2478004d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:11.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.280+0000 7f2477400640 1 -- 172.21.15.31:0/1666304347 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478042070 con 0x7f247c046540 2024-09-11T00:54:11.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.280+0000 7f2477400640 1 -- 172.21.15.31:0/1666304347 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f247803e770 con 0x7f247c046540 2024-09-11T00:54:11.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.280+0000 7f2477400640 1 -- 172.21.15.31:0/1666304347 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478037020 con 0x7f247c046540 2024-09-11T00:54:11.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.281+0000 7f2488c00640 1 -- 172.21.15.31:0/1666304347 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247c177a70 con 0x7f247c046540 2024-09-11T00:54:11.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.281+0000 7f2488c00640 1 -- 172.21.15.31:0/1666304347 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f247c15f430 con 0x7f247c046540 2024-09-11T00:54:11.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.283+0000 7f2488c00640 1 -- 172.21.15.31:0/1666304347 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f247c0d9f60 con 0x7f247c046540 2024-09-11T00:54:11.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.286+0000 7f2477400640 1 -- 172.21.15.31:0/1666304347 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f247803c040 con 0x7f247c046540 2024-09-11T00:54:11.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.288+0000 7f2477400640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2464078070 0x7f246407a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:11.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.288+0000 7f2477400640 1 -- 172.21.15.31:0/1666304347 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7f24780bbaa0 con 0x7f247c046540 2024-09-11T00:54:11.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.292+0000 7f2482a00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2464078070 0x7f246407a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:11.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.296+0000 7f2482a00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2464078070 0x7f246407a530 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f247c02bc60 tx=0x7f2470009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:11.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.301+0000 7f2477400640 1 -- 172.21.15.31:0/1666304347 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f24780830f0 con 0x7f247c046540 2024-09-11T00:54:11.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.788+0000 7f2488c00640 1 -- 172.21.15.31:0/1666304347 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f247c162020 con 0x7f247c046540 2024-09-11T00:54:11.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.790+0000 7f2477400640 1 -- 172.21.15.31:0/1666304347 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6546 (secure 0 0 0) 0x7f2478087b20 con 0x7f247c046540 2024-09-11T00:54:11.792 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:11.792 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":166,"btime":"2024-09-11T00:53:02:065604+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:53.021082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":145,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5693,"qdb_cluster":[5693]},"id":2}]} 2024-09-11T00:54:11.792 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 166 2024-09-11T00:54:11.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.794+0000 7f2488c00640 1 -- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2464078070 msgr2=0x7f246407a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:11.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.794+0000 7f2488c00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2464078070 0x7f246407a530 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f247c02bc60 tx=0x7f2470009040 comp rx=0 tx=0).stop 2024-09-11T00:54:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.794+0000 7f2488c00640 1 -- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f247c046540 msgr2=0x7f247c0cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.794+0000 7f2488c00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f247c046540 0x7f247c0cdfe0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2478033000 tx=0x7f2478004d80 comp rx=0 tx=0).stop 2024-09-11T00:54:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.794+0000 7f2488c00640 1 -- 172.21.15.31:0/1666304347 shutdown_connections 2024-09-11T00:54:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.794+0000 7f2488c00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2464078070 0x7f246407a530 secure :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f247c02bc60 tx=0x7f2470009040 comp rx=0 tx=0).stop 2024-09-11T00:54:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.794+0000 7f2488c00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f247c0d9370 0x7f247c0d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.794+0000 7f2488c00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f247c0d3490 0x7f247c161ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.794+0000 7f2488c00640 1 --2- 172.21.15.31:0/1666304347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f247c046540 0x7f247c0cdfe0 secure :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2478033000 tx=0x7f2478004d80 comp rx=0 tx=0).stop 2024-09-11T00:54:11.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.794+0000 7f2488c00640 1 -- 172.21.15.31:0/1666304347 >> 172.21.15.31:0/1666304347 conn(0x7f247c047e10 msgr2=0x7f247c03df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:11.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.801+0000 7f2488c00640 1 -- 172.21.15.31:0/1666304347 shutdown_connections 2024-09-11T00:54:11.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:11.802+0000 7f2488c00640 1 -- 172.21.15.31:0/1666304347 wait complete. 2024-09-11T00:54:11.839 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:54:13.840 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.177+0000 7eff65600640 1 -- 172.21.15.31:0/2298114757 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54006aa0 con 0x7eff4c005910 2024-09-11T00:54:14.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.177+0000 7eff67e00640 1 -- 172.21.15.31:0/2298114757 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff4c005910 msgr2=0x7eff4c005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:14.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.177+0000 7eff67e00640 1 --2- 172.21.15.31:0/2298114757 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff4c005910 0x7eff4c005d90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7eff54006e80 tx=0x7eff4c006940 comp rx=0 tx=0).stop 2024-09-11T00:54:14.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.178+0000 7eff67e00640 1 -- 172.21.15.31:0/2298114757 shutdown_connections 2024-09-11T00:54:14.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.178+0000 7eff67e00640 1 --2- 172.21.15.31:0/2298114757 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff4c005910 0x7eff4c005d90 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:14.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.178+0000 7eff67e00640 1 -- 172.21.15.31:0/2298114757 >> 172.21.15.31:0/2298114757 conn(0x7eff60047e10 msgr2=0x7eff60048220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:14.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.179+0000 7eff67e00640 1 -- 172.21.15.31:0/2298114757 shutdown_connections 2024-09-11T00:54:14.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.179+0000 7eff67e00640 1 -- 172.21.15.31:0/2298114757 wait complete. 2024-09-11T00:54:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.182+0000 7eff67e00640 1 Processor -- start 2024-09-11T00:54:14.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.183+0000 7eff67e00640 1 -- start start 2024-09-11T00:54:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.184+0000 7eff67e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff4c005910 0x7eff600ce470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.184+0000 7eff67e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff6016c1d0 0x7eff600d5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.184+0000 7eff67e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff600421f0 0x7eff600cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.184+0000 7eff67e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7eff60049a00 con 0x7eff600421f0 2024-09-11T00:54:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.184+0000 7eff67e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff60049720 con 0x7eff6016c1d0 2024-09-11T00:54:14.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.184+0000 7eff67e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7eff600493a0 con 0x7eff4c005910 2024-09-11T00:54:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.185+0000 7eff66a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff4c005910 0x7eff600ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.185+0000 7eff66a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff4c005910 0x7eff600ce470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:55944/0 (socket says 172.21.15.31:55944) 2024-09-11T00:54:14.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.185+0000 7eff66a00640 1 -- 172.21.15.31:0/2346097865 learned_addr learned my addr 172.21.15.31:0/2346097865 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.186+0000 7eff66a00640 1 -- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff6016c1d0 msgr2=0x7eff600d5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.186+0000 7eff66a00640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff6016c1d0 0x7eff600d5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.186+0000 7eff66a00640 1 -- 172.21.15.31:0/2346097865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff600421f0 msgr2=0x7eff600cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:14.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.186+0000 7eff66a00640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff600421f0 0x7eff600cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:14.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.186+0000 7eff66a00640 1 -- 172.21.15.31:0/2346097865 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff600ce1b0 con 0x7eff4c005910 2024-09-11T00:54:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.188+0000 7eff66a00640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff4c005910 0x7eff600ce470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c440 tx=0x7eff5800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:14.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.188+0000 7eff5ea00640 1 -- 172.21.15.31:0/2346097865 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58006890 con 0x7eff4c005910 2024-09-11T00:54:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.189+0000 7eff67e00640 1 -- 172.21.15.31:0/2346097865 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6016b6f0 con 0x7eff4c005910 2024-09-11T00:54:14.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.189+0000 7eff67e00640 1 -- 172.21.15.31:0/2346097865 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff600d5840 con 0x7eff4c005910 2024-09-11T00:54:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.190+0000 7eff5ea00640 1 -- 172.21.15.31:0/2346097865 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff58006b40 con 0x7eff4c005910 2024-09-11T00:54:14.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.190+0000 7eff5ea00640 1 -- 172.21.15.31:0/2346097865 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58007020 con 0x7eff4c005910 2024-09-11T00:54:14.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.196+0000 7eff53e00640 1 -- 172.21.15.31:0/2346097865 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff3c005290 con 0x7eff4c005910 2024-09-11T00:54:14.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.211+0000 7eff5ea00640 1 -- 172.21.15.31:0/2346097865 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7eff580071c0 con 0x7eff4c005910 2024-09-11T00:54:14.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.213+0000 7eff5ea00640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff48077fa0 0x7eff4807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:14.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.215+0000 7eff5ea00640 1 -- 172.21.15.31:0/2346097865 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7eff5809a730 con 0x7eff4c005910 2024-09-11T00:54:14.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.216+0000 7eff66000640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff48077fa0 0x7eff4807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:14.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.226+0000 7eff66000640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff48077fa0 0x7eff4807a460 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7eff54033000 tx=0x7eff600d6c40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:14.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.228+0000 7eff5ea00640 1 -- 172.21.15.31:0/2346097865 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7eff58062350 con 0x7eff4c005910 2024-09-11T00:54:14.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.758+0000 7eff53e00640 1 -- 172.21.15.31:0/2346097865 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff3c005580 con 0x7eff4c005910 2024-09-11T00:54:14.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.759+0000 7eff5ea00640 1 -- 172.21.15.31:0/2346097865 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6546 (secure 0 0 0) 0x7eff58061c50 con 0x7eff4c005910 2024-09-11T00:54:14.763 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:14.763 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":166,"btime":"2024-09-11T00:53:02:065604+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:53.021082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":145,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5693,"qdb_cluster":[5693]},"id":2}]} 2024-09-11T00:54:14.763 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 166 2024-09-11T00:54:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.765+0000 7eff53e00640 1 -- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff48077fa0 msgr2=0x7eff4807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.765+0000 7eff53e00640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff48077fa0 0x7eff4807a460 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7eff54033000 tx=0x7eff600d6c40 comp rx=0 tx=0).stop 2024-09-11T00:54:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.765+0000 7eff53e00640 1 -- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff4c005910 msgr2=0x7eff600ce470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.765+0000 7eff53e00640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff4c005910 0x7eff600ce470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c440 tx=0x7eff5800c900 comp rx=0 tx=0).stop 2024-09-11T00:54:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.765+0000 7eff53e00640 1 -- 172.21.15.31:0/2346097865 shutdown_connections 2024-09-11T00:54:14.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.765+0000 7eff53e00640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7eff48077fa0 0x7eff4807a460 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:14.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.765+0000 7eff53e00640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff600421f0 0x7eff600cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:14.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.765+0000 7eff53e00640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff6016c1d0 0x7eff600d5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:14.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.765+0000 7eff53e00640 1 --2- 172.21.15.31:0/2346097865 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7eff4c005910 0x7eff600ce470 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c440 tx=0x7eff5800c900 comp rx=0 tx=0).stop 2024-09-11T00:54:14.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.765+0000 7eff53e00640 1 -- 172.21.15.31:0/2346097865 >> 172.21.15.31:0/2346097865 conn(0x7eff60047e10 msgr2=0x7eff600d9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:14.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.767+0000 7eff53e00640 1 -- 172.21.15.31:0/2346097865 shutdown_connections 2024-09-11T00:54:14.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:14.768+0000 7eff53e00640 1 -- 172.21.15.31:0/2346097865 wait complete. 2024-09-11T00:54:14.801 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:54:16.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:17.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.044+0000 7fa97c600640 1 -- 172.21.15.31:0/2068459024 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9740b1160 msgr2=0x7fa97415a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:17.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.044+0000 7fa97c600640 1 --2- 172.21.15.31:0/2068459024 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9740b1160 0x7fa97415a1a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa9740b1da0 tx=0x7fa96802a4f0 comp rx=0 tx=0).stop 2024-09-11T00:54:17.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.045+0000 7fa97c600640 1 -- 172.21.15.31:0/2068459024 shutdown_connections 2024-09-11T00:54:17.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.045+0000 7fa97c600640 1 --2- 172.21.15.31:0/2068459024 >> v2:172.21.15.31:3300/0 conn(0x7fa9740bf2c0 0x7fa9740ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:17.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.045+0000 7fa97c600640 1 --2- 172.21.15.31:0/2068459024 >> v2:172.21.15.204:3300/0 conn(0x7fa9740c9b30 0x7fa97406deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:17.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.045+0000 7fa97c600640 1 --2- 172.21.15.31:0/2068459024 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9740b1160 0x7fa97415a1a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:17.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.045+0000 7fa97c600640 1 -- 172.21.15.31:0/2068459024 >> 172.21.15.31:0/2068459024 conn(0x7fa9740cc260 msgr2=0x7fa9740ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:17.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.045+0000 7fa97c600640 1 -- 172.21.15.31:0/2068459024 shutdown_connections 2024-09-11T00:54:17.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.045+0000 7fa97c600640 1 -- 172.21.15.31:0/2068459024 wait complete. 2024-09-11T00:54:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.047+0000 7fa97c600640 1 Processor -- start 2024-09-11T00:54:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.048+0000 7fa97c600640 1 -- start start 2024-09-11T00:54:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.049+0000 7fa97c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9740b1160 0x7fa9740ab550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.050+0000 7fa97c600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9740bf2c0 0x7fa9740c5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.050+0000 7fa97c600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9740c9b30 0x7fa9741568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.050+0000 7fa97c600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa97406f300 con 0x7fa9740b1160 2024-09-11T00:54:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.050+0000 7fa97c600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa9740b1540 con 0x7fa9740c9b30 2024-09-11T00:54:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.050+0000 7fa97c600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa97406f480 con 0x7fa9740bf2c0 2024-09-11T00:54:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.050+0000 7fa979200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9740bf2c0 0x7fa9740c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.051+0000 7fa979200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9740bf2c0 0x7fa9740c5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:55964/0 (socket says 172.21.15.31:55964) 2024-09-11T00:54:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.051+0000 7fa979200640 1 -- 172.21.15.31:0/1827975364 learned_addr learned my addr 172.21.15.31:0/1827975364 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.051+0000 7fa97a600640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9740c9b30 0x7fa9741568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.051+0000 7fa979200640 1 -- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9740c9b30 msgr2=0x7fa9741568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.051+0000 7fa979200640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9740c9b30 0x7fa9741568a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.051+0000 7fa979200640 1 -- 172.21.15.31:0/1827975364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9740b1160 msgr2=0x7fa9740ab550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T00:54:17.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.052+0000 7fa979c00640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9740b1160 0x7fa9740ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:17.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.052+0000 7fa979200640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9740b1160 0x7fa9740ab550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:17.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.052+0000 7fa979200640 1 -- 172.21.15.31:0/1827975364 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9741563e0 con 0x7fa9740bf2c0 2024-09-11T00:54:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.053+0000 7fa979200640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9740bf2c0 0x7fa9740c5140 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa96000b440 tx=0x7fa96000b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.053+0000 7fa971600640 1 -- 172.21.15.31:0/1827975364 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960015020 con 0x7fa9740bf2c0 2024-09-11T00:54:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.053+0000 7fa979c00640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9740b1160 0x7fa9740ab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:54:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.053+0000 7fa971600640 1 -- 172.21.15.31:0/1827975364 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9600119a0 con 0x7fa9740bf2c0 2024-09-11T00:54:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.053+0000 7fa971600640 1 -- 172.21.15.31:0/1827975364 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960011df0 con 0x7fa9740bf2c0 2024-09-11T00:54:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.054+0000 7fa97c600640 1 -- 172.21.15.31:0/1827975364 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9740ae670 con 0x7fa9740bf2c0 2024-09-11T00:54:17.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.054+0000 7fa97c600640 1 -- 172.21.15.31:0/1827975364 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9741534a0 con 0x7fa9740bf2c0 2024-09-11T00:54:17.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.055+0000 7fa97c600640 1 -- 172.21.15.31:0/1827975364 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa97415a1a0 con 0x7fa9740bf2c0 2024-09-11T00:54:17.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.058+0000 7fa971600640 1 -- 172.21.15.31:0/1827975364 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa96001a400 con 0x7fa9740bf2c0 2024-09-11T00:54:17.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.059+0000 7fa971600640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa954077fc0 0x7fa95407a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:17.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.059+0000 7fa971600640 1 -- 172.21.15.31:0/1827975364 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9332+0+0 (secure 0 0 0) 0x7fa96009afc0 con 0x7fa9740bf2c0 2024-09-11T00:54:17.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.060+0000 7fa979c00640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa954077fc0 0x7fa95407a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:17.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.062+0000 7fa979c00640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa954077fc0 0x7fa95407a480 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa968032000 tx=0x7fa968004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:17.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.064+0000 7fa971600640 1 -- 172.21.15.31:0/1827975364 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa96001f020 con 0x7fa9740bf2c0 2024-09-11T00:54:17.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.514+0000 7fa97c600640 1 -- 172.21.15.31:0/1827975364 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9740c5e10 con 0x7fa9740bf2c0 2024-09-11T00:54:17.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.515+0000 7fa971600640 1 -- 172.21.15.31:0/1827975364 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6546 (secure 0 0 0) 0x7fa9741563e0 con 0x7fa9740bf2c0 2024-09-11T00:54:17.518 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:17.518 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":166,"btime":"2024-09-11T00:53:02:065604+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7429,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":163}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:52:53.021082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":145,"addr":"172.21.15.31:6833/2956367495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":2956367495},{"type":"v1","addr":"172.21.15.31:6833","nonce":2956367495}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5693,"qdb_cluster":[5693]},"id":2}]} 2024-09-11T00:54:17.519 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 166 2024-09-11T00:54:17.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.518+0000 7fa97c600640 1 -- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa954077fc0 msgr2=0x7fa95407a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:17.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.518+0000 7fa97c600640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa954077fc0 0x7fa95407a480 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa968032000 tx=0x7fa968004730 comp rx=0 tx=0).stop 2024-09-11T00:54:17.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.518+0000 7fa97c600640 1 -- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9740bf2c0 msgr2=0x7fa9740c5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:17.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.518+0000 7fa97c600640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9740bf2c0 0x7fa9740c5140 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa96000b440 tx=0x7fa96000b900 comp rx=0 tx=0).stop 2024-09-11T00:54:17.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.518+0000 7fa97c600640 1 -- 172.21.15.31:0/1827975364 shutdown_connections 2024-09-11T00:54:17.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.518+0000 7fa97c600640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa954077fc0 0x7fa95407a480 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:17.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.518+0000 7fa97c600640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa9740c9b30 0x7fa9741568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:17.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.518+0000 7fa97c600640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa9740bf2c0 0x7fa9740c5140 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:17.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.518+0000 7fa97c600640 1 --2- 172.21.15.31:0/1827975364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9740b1160 0x7fa9740ab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:17.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.518+0000 7fa97c600640 1 -- 172.21.15.31:0/1827975364 >> 172.21.15.31:0/1827975364 conn(0x7fa9740cc260 msgr2=0x7fa974158c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:17.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.520+0000 7fa97c600640 1 -- 172.21.15.31:0/1827975364 shutdown_connections 2024-09-11T00:54:17.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:17.520+0000 7fa97c600640 1 -- 172.21.15.31:0/1827975364 wait complete. 2024-09-11T00:54:17.549 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:54:19.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:19.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.873+0000 7f511be00640 1 -- 172.21.15.31:0/301037928 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110004290 con 0x7f5104004800 2024-09-11T00:54:19.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.874+0000 7f5124000640 1 -- 172.21.15.31:0/301037928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5104004800 msgr2=0x7f5104004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:19.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.874+0000 7f5124000640 1 --2- 172.21.15.31:0/301037928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5104004800 0x7f5104004c80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5110009cf0 tx=0x7f5104005830 comp rx=0 tx=0).stop 2024-09-11T00:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.874+0000 7f5122000640 1 -- 172.21.15.31:0/301037928 reap_dead start 2024-09-11T00:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.874+0000 7f5124000640 1 -- 172.21.15.31:0/301037928 shutdown_connections 2024-09-11T00:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.874+0000 7f5124000640 1 -- 172.21.15.31:0/301037928 >> 172.21.15.31:0/301037928 conn(0x7f511c06dc00 msgr2=0x7f511c06e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:19.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.874+0000 7f5124000640 1 -- 172.21.15.31:0/301037928 shutdown_connections 2024-09-11T00:54:19.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.876+0000 7f5124000640 1 -- 172.21.15.31:0/301037928 wait complete. 2024-09-11T00:54:19.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.881+0000 7f5124000640 1 Processor -- start 2024-09-11T00:54:19.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.881+0000 7f5124000640 1 -- start start 2024-09-11T00:54:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.882+0000 7f5124000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5104004800 0x7f511c14d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.882+0000 7f5124000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f511c14ee00 0x7f511c14acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.882+0000 7f5124000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f511c14b230 0x7f511c1e80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.882+0000 7f5124000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f511c153980 con 0x7f5104004800 2024-09-11T00:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.882+0000 7f5124000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f511c1536a0 con 0x7f511c14b230 2024-09-11T00:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.882+0000 7f5124000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f511c0c9090 con 0x7f511c14ee00 2024-09-11T00:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.883+0000 7f5121600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5104004800 0x7f511c14d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.883+0000 7f5122000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f511c14b230 0x7f511c1e80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.883+0000 7f5121600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5104004800 0x7f511c14d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57650/0 (socket says 172.21.15.31:57650) 2024-09-11T00:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.883+0000 7f5121600640 1 -- 172.21.15.31:0/3868345210 learned_addr learned my addr 172.21.15.31:0/3868345210 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.883+0000 7f5120c00640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f511c14ee00 0x7f511c14acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.884+0000 7f5121600640 1 -- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f511c14ee00 msgr2=0x7f511c14acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.884+0000 7f5121600640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f511c14ee00 0x7f511c14acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.884+0000 7f5121600640 1 -- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f511c14b230 msgr2=0x7f511c1e80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.884+0000 7f5121600640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f511c14b230 0x7f511c1e80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.884+0000 7f5121600640 1 -- 172.21.15.31:0/3868345210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f511c1549a0 con 0x7f5104004800 2024-09-11T00:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.885+0000 7f5121600640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5104004800 0x7f511c14d380 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f510c011a40 tx=0x7f510c011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.885+0000 7f5119600640 1 -- 172.21.15.31:0/3868345210 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c00f940 con 0x7f5104004800 2024-09-11T00:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.886+0000 7f5119600640 1 -- 172.21.15.31:0/3868345210 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f510c00fae0 con 0x7f5104004800 2024-09-11T00:54:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.886+0000 7f5119600640 1 -- 172.21.15.31:0/3868345210 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c005020 con 0x7f5104004800 2024-09-11T00:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.886+0000 7f5124000640 1 -- 172.21.15.31:0/3868345210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f511c154c80 con 0x7f5104004800 2024-09-11T00:54:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.886+0000 7f5124000640 1 -- 172.21.15.31:0/3868345210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f511c0a8430 con 0x7f5104004800 2024-09-11T00:54:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.888+0000 7f5124000640 1 -- 172.21.15.31:0/3868345210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f511c074480 con 0x7f5104004800 2024-09-11T00:54:19.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.891+0000 7f5119600640 1 -- 172.21.15.31:0/3868345210 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f510c0051c0 con 0x7f5104004800 2024-09-11T00:54:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.893+0000 7f5119600640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5100077ff0 0x7f510007a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.893+0000 7f5119600640 1 -- 172.21.15.31:0/3868345210 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7f510c09b6e0 con 0x7f5104004800 2024-09-11T00:54:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.894+0000 7f5120c00640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5100077ff0 0x7f510007a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.895+0000 7f5120c00640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5100077ff0 0x7f510007a4b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5110034000 tx=0x7f51100046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:19.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:19.901+0000 7f5119600640 1 -- 172.21.15.31:0/3868345210 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f510c062d70 con 0x7f5104004800 2024-09-11T00:54:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.349+0000 7f5124000640 1 -- 172.21.15.31:0/3868345210 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f511c0750c0 con 0x7f5104004800 2024-09-11T00:54:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.350+0000 7f5119600640 1 -- 172.21.15.31:0/3868345210 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+5711 (secure 0 0 0) 0x7f510c0677a0 con 0x7f5104004800 2024-09-11T00:54:20.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:20.356 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":167,"btime":"2024-09-11T00:54:19:542586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:19.542577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:54:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 167 2024-09-11T00:54:20.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.354+0000 7f5124000640 1 -- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5100077ff0 msgr2=0x7f510007a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:20.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.354+0000 7f5124000640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5100077ff0 0x7f510007a4b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5110034000 tx=0x7f51100046c0 comp rx=0 tx=0).stop 2024-09-11T00:54:20.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.354+0000 7f5124000640 1 -- 172.21.15.31:0/3868345210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5104004800 msgr2=0x7f511c14d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:20.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.354+0000 7f5124000640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5104004800 0x7f511c14d380 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f510c011a40 tx=0x7f510c011f00 comp rx=0 tx=0).stop 2024-09-11T00:54:20.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.355+0000 7f5124000640 1 -- 172.21.15.31:0/3868345210 shutdown_connections 2024-09-11T00:54:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.355+0000 7f5124000640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5100077ff0 0x7f510007a4b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.355+0000 7f5124000640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f511c14b230 0x7f511c1e80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.355+0000 7f5124000640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f511c14ee00 0x7f511c14acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.355+0000 7f5124000640 1 --2- 172.21.15.31:0/3868345210 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5104004800 0x7f511c14d380 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.355+0000 7f5124000640 1 -- 172.21.15.31:0/3868345210 >> 172.21.15.31:0/3868345210 conn(0x7f511c06dc00 msgr2=0x7f511c0737e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:20.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.355+0000 7f5124000640 1 -- 172.21.15.31:0/3868345210 shutdown_connections 2024-09-11T00:54:20.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.355+0000 7f5124000640 1 -- 172.21.15.31:0/3868345210 wait complete. 2024-09-11T00:54:20.384 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-11T00:54:20.384 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:54:20.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:20.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.673+0000 7fbd93200640 1 -- 172.21.15.31:0/2269244294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd940cc220 msgr2=0x7fbd9406e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.673+0000 7fbd93200640 1 --2- 172.21.15.31:0/2269244294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd940cc220 0x7fbd9406e680 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fbd94153d20 tx=0x7fbd8c02a790 comp rx=0 tx=0).stop 2024-09-11T00:54:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.674+0000 7fbd93200640 1 -- 172.21.15.31:0/2269244294 shutdown_connections 2024-09-11T00:54:20.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.674+0000 7fbd93200640 1 --2- 172.21.15.31:0/2269244294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd940cc220 0x7fbd9406e680 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:20.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.674+0000 7fbd93200640 1 --2- 172.21.15.31:0/2269244294 >> v2:172.21.15.31:3300/0 conn(0x7fbd94156960 0x7fbd9406e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:20.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.674+0000 7fbd93200640 1 -- 172.21.15.31:0/2269244294 >> 172.21.15.31:0/2269244294 conn(0x7fbd94068f10 msgr2=0x7fbd940cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:20.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.674+0000 7fbd93200640 1 -- 172.21.15.31:0/2269244294 shutdown_connections 2024-09-11T00:54:20.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.674+0000 7fbd93200640 1 -- 172.21.15.31:0/2269244294 wait complete. 2024-09-11T00:54:20.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.676+0000 7fbd93200640 1 Processor -- start 2024-09-11T00:54:20.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.677+0000 7fbd93200640 1 -- start start 2024-09-11T00:54:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.678+0000 7fbd93200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd940cc220 0x7fbd9414aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.678+0000 7fbd93200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd94150020 0x7fbd9414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.678+0000 7fbd93200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd94156960 0x7fbd941ef4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.678+0000 7fbd93200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbd940c6ca0 con 0x7fbd94156960 2024-09-11T00:54:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.678+0000 7fbd93200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbd940c6b20 con 0x7fbd940cc220 2024-09-11T00:54:20.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.678+0000 7fbd93200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbd941573d0 con 0x7fbd94150020 2024-09-11T00:54:20.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.679+0000 7fbd92800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd94156960 0x7fbd941ef4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:20.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.679+0000 7fbd92800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd94156960 0x7fbd941ef4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57672/0 (socket says 172.21.15.31:57672) 2024-09-11T00:54:20.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.679+0000 7fbd92800640 1 -- 172.21.15.31:0/3474356781 learned_addr learned my addr 172.21.15.31:0/3474356781 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:20.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.679+0000 7fbd91400640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd94150020 0x7fbd9414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:20.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.679+0000 7fbd91e00640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd940cc220 0x7fbd9414aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:20.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.680+0000 7fbd91400640 1 -- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd940cc220 msgr2=0x7fbd9414aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.680+0000 7fbd91400640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd940cc220 0x7fbd9414aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.680+0000 7fbd91400640 1 -- 172.21.15.31:0/3474356781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd94156960 msgr2=0x7fbd941ef4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.680+0000 7fbd91400640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd94156960 0x7fbd941ef4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:20.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.680+0000 7fbd91400640 1 -- 172.21.15.31:0/3474356781 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd94154ff0 con 0x7fbd94150020 2024-09-11T00:54:20.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.681+0000 7fbd91400640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd94150020 0x7fbd9414aff0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000e710 tx=0x7fbd8000ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:20.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.681+0000 7fbd8a000640 1 -- 172.21.15.31:0/3474356781 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8000cab0 con 0x7fbd94150020 2024-09-11T00:54:20.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.681+0000 7fbd93200640 1 -- 172.21.15.31:0/3474356781 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd94150770 con 0x7fbd94150020 2024-09-11T00:54:20.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.681+0000 7fbd8a000640 1 -- 172.21.15.31:0/3474356781 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd8000cd40 con 0x7fbd94150020 2024-09-11T00:54:20.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.681+0000 7fbd8a000640 1 -- 172.21.15.31:0/3474356781 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8000cab0 con 0x7fbd94150020 2024-09-11T00:54:20.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.681+0000 7fbd93200640 1 -- 172.21.15.31:0/3474356781 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd941e0a80 con 0x7fbd94150020 2024-09-11T00:54:20.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.685+0000 7fbd93200640 1 -- 172.21.15.31:0/3474356781 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd940ad4e0 con 0x7fbd94150020 2024-09-11T00:54:20.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.685+0000 7fbd8a000640 1 -- 172.21.15.31:0/3474356781 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbd80010430 con 0x7fbd94150020 2024-09-11T00:54:20.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.691+0000 7fbd8a000640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbd740781a0 0x7fbd7407a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:20.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.692+0000 7fbd8a000640 1 -- 172.21.15.31:0/3474356781 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7fbd8009b030 con 0x7fbd94150020 2024-09-11T00:54:20.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.693+0000 7fbd91e00640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbd740781a0 0x7fbd7407a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:20.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.693+0000 7fbd8a000640 1 -- 172.21.15.31:0/3474356781 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbd8009d7f0 con 0x7fbd94150020 2024-09-11T00:54:20.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:20.700+0000 7fbd91e00640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbd740781a0 0x7fbd7407a660 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbd84007fd0 tx=0x7fbd84009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:21.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.166+0000 7fbd93200640 1 -- 172.21.15.31:0/3474356781 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd941e0e80 con 0x7fbd94150020 2024-09-11T00:54:21.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.167+0000 7fbd8a000640 1 -- 172.21.15.31:0/3474356781 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+5711 (secure 0 0 0) 0x7fbd80062640 con 0x7fbd94150020 2024-09-11T00:54:21.172 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:21.172 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":167,"btime":"2024-09-11T00:54:19:542586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:19.542577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:54:21.172 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 167 2024-09-11T00:54:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.171+0000 7fbd93200640 1 -- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbd740781a0 msgr2=0x7fbd7407a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.171+0000 7fbd93200640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbd740781a0 0x7fbd7407a660 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbd84007fd0 tx=0x7fbd84009040 comp rx=0 tx=0).stop 2024-09-11T00:54:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.171+0000 7fbd93200640 1 -- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd94150020 msgr2=0x7fbd9414aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.171+0000 7fbd93200640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd94150020 0x7fbd9414aff0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000e710 tx=0x7fbd8000ebd0 comp rx=0 tx=0).stop 2024-09-11T00:54:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.171+0000 7fbd93200640 1 -- 172.21.15.31:0/3474356781 shutdown_connections 2024-09-11T00:54:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.171+0000 7fbd93200640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbd740781a0 0x7fbd7407a660 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.171+0000 7fbd93200640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd94156960 0x7fbd941ef4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.171+0000 7fbd93200640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbd94150020 0x7fbd9414aff0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.171+0000 7fbd93200640 1 --2- 172.21.15.31:0/3474356781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd940cc220 0x7fbd9414aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.171+0000 7fbd93200640 1 -- 172.21.15.31:0/3474356781 >> 172.21.15.31:0/3474356781 conn(0x7fbd94068f10 msgr2=0x7fbd94158530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.172+0000 7fbd93200640 1 -- 172.21.15.31:0/3474356781 shutdown_connections 2024-09-11T00:54:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:21.172+0000 7fbd93200640 1 -- 172.21.15.31:0/3474356781 wait complete. 2024-09-11T00:54:21.201 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:54:19:542586+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 167, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 161, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7357": { "addr": "172.21.15.204:6835/4094068967", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 4094068967, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 4094068967, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7357, "incarnation": 158, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:52:42.957872+0000", "qdb_cluster": [ 7357 ], "qdb_leader": 7357, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7357 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 167, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7429": { "addr": "172.21.15.204:6837/2689217598", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 2689217598, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 2689217598, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7429, "incarnation": 167, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:54:19.542577+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7429 } } } ], "standbys": [ { "addr": "172.21.15.31:6835/2337200751", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 2337200751, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 2337200751, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 166, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6503, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:54:23.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:23.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.487+0000 7f5c72a00640 1 -- 172.21.15.31:0/1556267828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c6c0bf240 msgr2=0x7f5c6c0b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:23.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.487+0000 7f5c72a00640 1 --2- 172.21.15.31:0/1556267828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c6c0bf240 0x7f5c6c0b0be0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009820 tx=0x7f5c6002a530 comp rx=0 tx=0).stop 2024-09-11T00:54:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.489+0000 7f5c72a00640 1 -- 172.21.15.31:0/1556267828 shutdown_connections 2024-09-11T00:54:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.489+0000 7f5c72a00640 1 --2- 172.21.15.31:0/1556267828 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c6c0bf240 0x7f5c6c0b0be0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.489+0000 7f5c72a00640 1 -- 172.21.15.31:0/1556267828 >> 172.21.15.31:0/1556267828 conn(0x7f5c6c068ac0 msgr2=0x7f5c6c068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.490+0000 7f5c72a00640 1 -- 172.21.15.31:0/1556267828 shutdown_connections 2024-09-11T00:54:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.490+0000 7f5c72a00640 1 -- 172.21.15.31:0/1556267828 wait complete. 2024-09-11T00:54:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.493+0000 7f5c72a00640 1 Processor -- start 2024-09-11T00:54:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.493+0000 7f5c72a00640 1 -- start start 2024-09-11T00:54:23.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.495+0000 7f5c72a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c6c0bf240 0x7f5c6c158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:23.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.495+0000 7f5c72a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c6c0c6730 0x7f5c6c1efa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:23.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.495+0000 7f5c72a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c6c156e40 0x7f5c6c14be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:23.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.498+0000 7f5c72a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c159f60 con 0x7f5c6c0c6730 2024-09-11T00:54:23.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.498+0000 7f5c72a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c159de0 con 0x7f5c6c0bf240 2024-09-11T00:54:23.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.498+0000 7f5c72a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6c0c6b10 con 0x7f5c6c156e40 2024-09-11T00:54:23.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.498+0000 7f5c71600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c6c0bf240 0x7f5c6c158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:23.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.498+0000 7f5c71600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c6c0bf240 0x7f5c6c158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:53644/0 (socket says 172.21.15.31:53644) 2024-09-11T00:54:23.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.498+0000 7f5c71600640 1 -- 172.21.15.31:0/2188779850 learned_addr learned my addr 172.21.15.31:0/2188779850 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:23.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.499+0000 7f5c70c00640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c6c0c6730 0x7f5c6c1efa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:23.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.499+0000 7f5c71600640 1 -- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c6c156e40 msgr2=0x7f5c6c14be50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:54:23.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.499+0000 7f5c71600640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c6c156e40 0x7f5c6c14be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:23.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.499+0000 7f5c71600640 1 -- 172.21.15.31:0/2188779850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c6c0c6730 msgr2=0x7f5c6c1efa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:23.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.499+0000 7f5c71600640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c6c0c6730 0x7f5c6c1efa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:23.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.499+0000 7f5c71600640 1 -- 172.21.15.31:0/2188779850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c0ae110 con 0x7f5c6c0bf240 2024-09-11T00:54:23.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.499+0000 7f5c70c00640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c6c0c6730 0x7f5c6c1efa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:54:23.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.500+0000 7f5c71600640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c6c0bf240 0x7f5c6c158d30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800c3a0 tx=0x7f5c5800c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:23.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.500+0000 7f5c69600640 1 -- 172.21.15.31:0/2188779850 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c58011790 con 0x7f5c6c0bf240 2024-09-11T00:54:23.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.500+0000 7f5c69600640 1 -- 172.21.15.31:0/2188779850 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c58017070 con 0x7f5c6c0bf240 2024-09-11T00:54:23.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.500+0000 7f5c69600640 1 -- 172.21.15.31:0/2188779850 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c58011c80 con 0x7f5c6c0bf240 2024-09-11T00:54:23.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.500+0000 7f5c72a00640 1 -- 172.21.15.31:0/2188779850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6c0c8e50 con 0x7f5c6c0bf240 2024-09-11T00:54:23.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.501+0000 7f5c72a00640 1 -- 172.21.15.31:0/2188779850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6c1ef530 con 0x7f5c6c0bf240 2024-09-11T00:54:23.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.503+0000 7f5c72a00640 1 -- 172.21.15.31:0/2188779850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c6c1e1e10 con 0x7f5c6c0bf240 2024-09-11T00:54:23.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.506+0000 7f5c69600640 1 -- 172.21.15.31:0/2188779850 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5c58004030 con 0x7f5c6c0bf240 2024-09-11T00:54:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.508+0000 7f5c69600640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c4c078070 0x7f5c4c07a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.509+0000 7f5c69600640 1 -- 172.21.15.31:0/2188779850 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7f5c5809b090 con 0x7f5c6c0bf240 2024-09-11T00:54:23.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.509+0000 7f5c70c00640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c4c078070 0x7f5c4c07a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:23.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.510+0000 7f5c70c00640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c4c078070 0x7f5c4c07a530 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5c60034000 tx=0x7f5c60004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.511+0000 7f5c69600640 1 -- 172.21.15.31:0/2188779850 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5c6c1560e0 con 0x7f5c6c0bf240 2024-09-11T00:54:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.983+0000 7f5c72a00640 1 -- 172.21.15.31:0/2188779850 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c6c06e8a0 con 0x7f5c6c0bf240 2024-09-11T00:54:23.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.985+0000 7f5c69600640 1 -- 172.21.15.31:0/2188779850 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+5719 (secure 0 0 0) 0x7f5c58062c50 con 0x7f5c6c0bf240 2024-09-11T00:54:23.988 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:23.989 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":170,"btime":"2024-09-11T00:54:23:465899+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:54:23.989 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 170 2024-09-11T00:54:23.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.992+0000 7f5c72a00640 1 -- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c4c078070 msgr2=0x7f5c4c07a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:23.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.992+0000 7f5c72a00640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c4c078070 0x7f5c4c07a530 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5c60034000 tx=0x7f5c60004730 comp rx=0 tx=0).stop 2024-09-11T00:54:23.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.992+0000 7f5c72a00640 1 -- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c6c0bf240 msgr2=0x7f5c6c158d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:23.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.992+0000 7f5c72a00640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c6c0bf240 0x7f5c6c158d30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800c3a0 tx=0x7f5c5800c860 comp rx=0 tx=0).stop 2024-09-11T00:54:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.994+0000 7f5c72a00640 1 -- 172.21.15.31:0/2188779850 shutdown_connections 2024-09-11T00:54:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.994+0000 7f5c72a00640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c4c078070 0x7f5c4c07a530 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.994+0000 7f5c72a00640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c6c156e40 0x7f5c6c14be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.994+0000 7f5c72a00640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c6c0c6730 0x7f5c6c1efa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.994+0000 7f5c72a00640 1 --2- 172.21.15.31:0/2188779850 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c6c0bf240 0x7f5c6c158d30 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.994+0000 7f5c72a00640 1 -- 172.21.15.31:0/2188779850 >> 172.21.15.31:0/2188779850 conn(0x7f5c6c068ac0 msgr2=0x7f5c6c0afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.995+0000 7f5c72a00640 1 -- 172.21.15.31:0/2188779850 shutdown_connections 2024-09-11T00:54:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:23.996+0000 7f5c72a00640 1 -- 172.21.15.31:0/2188779850 wait complete. 2024-09-11T00:54:24.042 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=5693 2024-09-11T00:54:24.042 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.a 2024-09-11T00:54:24.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:54:24.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:54:25.043 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-11T00:54:25.043 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-11T00:54:25.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-11T00:54:25.054 INFO:tasks.ceph.mds.a:Started 2024-09-11T00:54:25.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:25.308 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-11T00:54:25.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.386+0000 7f001de00640 1 -- 172.21.15.31:0/2575136485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f001806cf40 msgr2=0x7f00180ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:25.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.386+0000 7f001de00640 1 --2- 172.21.15.31:0/2575136485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f001806cf40 0x7f00180ade10 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f0010005fd0 tx=0x7f001002a980 comp rx=0 tx=0).stop 2024-09-11T00:54:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.387+0000 7f001de00640 1 -- 172.21.15.31:0/2575136485 shutdown_connections 2024-09-11T00:54:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.387+0000 7f001de00640 1 --2- 172.21.15.31:0/2575136485 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f00180c7330 0x7f0018160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.387+0000 7f001de00640 1 --2- 172.21.15.31:0/2575136485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f001806cf40 0x7f00180ade10 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.387+0000 7f001de00640 1 -- 172.21.15.31:0/2575136485 >> 172.21.15.31:0/2575136485 conn(0x7f00180c8cc0 msgr2=0x7f00180ad700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:25.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.387+0000 7f001de00640 1 -- 172.21.15.31:0/2575136485 shutdown_connections 2024-09-11T00:54:25.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.387+0000 7f001de00640 1 -- 172.21.15.31:0/2575136485 wait complete. 2024-09-11T00:54:25.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.390+0000 7f001de00640 1 Processor -- start 2024-09-11T00:54:25.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.390+0000 7f001de00640 1 -- start start 2024-09-11T00:54:25.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.392+0000 7f001de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f001806cf40 0x7f00181494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:25.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.392+0000 7f001de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00180c7330 0x7f00181477b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:25.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.392+0000 7f001de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0018153c10 0x7f00181e7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:25.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.392+0000 7f001de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0018159f40 con 0x7f00180c7330 2024-09-11T00:54:25.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.392+0000 7f001de00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f00180bf3c0 con 0x7f001806cf40 2024-09-11T00:54:25.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.392+0000 7f001de00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f00180c79f0 con 0x7f0018153c10 2024-09-11T00:54:25.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.393+0000 7f0016a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f001806cf40 0x7f00181494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:25.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.393+0000 7f0016000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00180c7330 0x7f00181477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:25.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.393+0000 7f0016000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00180c7330 0x7f00181477b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57726/0 (socket says 172.21.15.31:57726) 2024-09-11T00:54:25.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.393+0000 7f0016000640 1 -- 172.21.15.31:0/2384846126 learned_addr learned my addr 172.21.15.31:0/2384846126 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:25.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.393+0000 7f0017400640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0018153c10 0x7f00181e7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:25.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.394+0000 7f0016000640 1 -- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0018153c10 msgr2=0x7f00181e7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:25.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.394+0000 7f0016000640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0018153c10 0x7f00181e7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:25.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.394+0000 7f0016000640 1 -- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f001806cf40 msgr2=0x7f00181494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:25.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.394+0000 7f0016000640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f001806cf40 0x7f00181494f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:25.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.395+0000 7f0016000640 1 -- 172.21.15.31:0/2384846126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0018145010 con 0x7f00180c7330 2024-09-11T00:54:25.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.395+0000 7f0016000640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00180c7330 0x7f00181477b0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0010000f80 tx=0x7f0010003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:25.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.396+0000 7f000f400640 1 -- 172.21.15.31:0/2384846126 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0010047070 con 0x7f00180c7330 2024-09-11T00:54:25.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.397+0000 7f000f400640 1 -- 172.21.15.31:0/2384846126 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0010004540 con 0x7f00180c7330 2024-09-11T00:54:25.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.397+0000 7f001de00640 1 -- 172.21.15.31:0/2384846126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0018153060 con 0x7f00180c7330 2024-09-11T00:54:25.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.397+0000 7f000f400640 1 -- 172.21.15.31:0/2384846126 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0010047070 con 0x7f00180c7330 2024-09-11T00:54:25.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.398+0000 7f001de00640 1 -- 172.21.15.31:0/2384846126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0018147cf0 con 0x7f00180c7330 2024-09-11T00:54:25.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.405+0000 7f001de00640 1 -- 172.21.15.31:0/2384846126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff0005290 con 0x7f00180c7330 2024-09-11T00:54:25.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.411+0000 7f000f400640 1 -- 172.21.15.31:0/2384846126 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0010041030 con 0x7f00180c7330 2024-09-11T00:54:25.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.412+0000 7f000f400640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efff8078170 0x7efff807a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:25.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.413+0000 7f000f400640 1 -- 172.21.15.31:0/2384846126 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7f00100bb4e0 con 0x7f00180c7330 2024-09-11T00:54:25.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.414+0000 7f0016a00640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efff8078170 0x7efff807a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:25.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.416+0000 7f0016a00640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efff8078170 0x7efff807a630 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0008001720 tx=0x7f0008009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:25.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.429+0000 7f000f400640 1 -- 172.21.15.31:0/2384846126 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f001003f070 con 0x7f00180c7330 2024-09-11T00:54:25.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.969+0000 7f001de00640 1 -- 172.21.15.31:0/2384846126 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efff0005580 con 0x7f00180c7330 2024-09-11T00:54:25.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.970+0000 7f000f400640 1 -- 172.21.15.31:0/2384846126 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6545 (secure 0 0 0) 0x7f0010082ae0 con 0x7f00180c7330 2024-09-11T00:54:25.974 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:25.974 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":171,"btime":"2024-09-11T00:54:25:481521+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7522,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:54:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 171 2024-09-11T00:54:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.976+0000 7f001de00640 1 -- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efff8078170 msgr2=0x7efff807a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.976+0000 7f001de00640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efff8078170 0x7efff807a630 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f0008001720 tx=0x7f0008009040 comp rx=0 tx=0).stop 2024-09-11T00:54:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.976+0000 7f001de00640 1 -- 172.21.15.31:0/2384846126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00180c7330 msgr2=0x7f00181477b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.976+0000 7f001de00640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00180c7330 0x7f00181477b0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0010000f80 tx=0x7f0010003f90 comp rx=0 tx=0).stop 2024-09-11T00:54:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.977+0000 7f001de00640 1 -- 172.21.15.31:0/2384846126 shutdown_connections 2024-09-11T00:54:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.977+0000 7f001de00640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7efff8078170 0x7efff807a630 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.977+0000 7f001de00640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0018153c10 0x7f00181e7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.977+0000 7f001de00640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00180c7330 0x7f00181477b0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.977+0000 7f001de00640 1 --2- 172.21.15.31:0/2384846126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f001806cf40 0x7f00181494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.977+0000 7f001de00640 1 -- 172.21.15.31:0/2384846126 >> 172.21.15.31:0/2384846126 conn(0x7f00180c8cc0 msgr2=0x7f0018157420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.977+0000 7f001de00640 1 -- 172.21.15.31:0/2384846126 shutdown_connections 2024-09-11T00:54:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:25.977+0000 7f001de00640 1 -- 172.21.15.31:0/2384846126 wait complete. 2024-09-11T00:54:26.011 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-11T00:54:26.011 INFO:tasks.mds_thrash.fs.[b]:waiting for 95 secs before thrashing 2024-09-11T00:54:26.023 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:54:26.021+0000 7f9e81e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:54:47.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:48.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.065+0000 7f5c9be00640 1 -- 172.21.15.31:0/2796414225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c9400b6b0 msgr2=0x7f5c94045b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:48.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.065+0000 7f5c9be00640 1 --2- 172.21.15.31:0/2796414225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c9400b6b0 0x7f5c94045b30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c005fd0 tx=0x7f5c8c02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:54:48.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.066+0000 7f5c9be00640 1 -- 172.21.15.31:0/2796414225 shutdown_connections 2024-09-11T00:54:48.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.066+0000 7f5c9be00640 1 --2- 172.21.15.31:0/2796414225 >> v2:172.21.15.204:3300/0 conn(0x7f5c94046540 0x7f5c940d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:48.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.066+0000 7f5c9be00640 1 --2- 172.21.15.31:0/2796414225 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c9400b6b0 0x7f5c94045b30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:48.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.066+0000 7f5c9be00640 1 -- 172.21.15.31:0/2796414225 >> 172.21.15.31:0/2796414225 conn(0x7f5c94047e10 msgr2=0x7f5c94048220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:48.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.067+0000 7f5c9be00640 1 -- 172.21.15.31:0/2796414225 shutdown_connections 2024-09-11T00:54:48.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.067+0000 7f5c9be00640 1 -- 172.21.15.31:0/2796414225 wait complete. 2024-09-11T00:54:48.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.069+0000 7f5c9be00640 1 Processor -- start 2024-09-11T00:54:48.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.069+0000 7f5c9be00640 1 -- start start 2024-09-11T00:54:48.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.075+0000 7f5c9be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9400b6b0 0x7f5c9416e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:48.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.075+0000 7f5c9be00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c94046540 0x7f5c941645c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:48.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.078+0000 7f5c9be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c9404a750 0x7f5c940cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:48.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.078+0000 7f5c9be00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c940d3760 con 0x7f5c9400b6b0 2024-09-11T00:54:48.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.078+0000 7f5c9be00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94049a00 con 0x7f5c9404a750 2024-09-11T00:54:48.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.078+0000 7f5c9be00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5c940d38e0 con 0x7f5c94046540 2024-09-11T00:54:48.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.079+0000 7f5c9aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9400b6b0 0x7f5c9416e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:48.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.079+0000 7f5c9a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c94046540 0x7f5c941645c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:48.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.079+0000 7f5c9aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9400b6b0 0x7f5c9416e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50272/0 (socket says 172.21.15.31:50272) 2024-09-11T00:54:48.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.079+0000 7f5c9aa00640 1 -- 172.21.15.31:0/2491597723 learned_addr learned my addr 172.21.15.31:0/2491597723 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:48.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.081+0000 7f5c9a000640 1 -- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c9404a750 msgr2=0x7f5c940cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:48.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.081+0000 7f5c9a000640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c9404a750 0x7f5c940cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:48.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.081+0000 7f5c9a000640 1 -- 172.21.15.31:0/2491597723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9400b6b0 msgr2=0x7f5c9416e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:48.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.084+0000 7f5c9a000640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9400b6b0 0x7f5c9416e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:48.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.084+0000 7f5c9a000640 1 -- 172.21.15.31:0/2491597723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c94044a40 con 0x7f5c94046540 2024-09-11T00:54:48.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.086+0000 7f5c9aa00640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9400b6b0 0x7f5c9416e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:54:48.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.088+0000 7f5c9a000640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c94046540 0x7f5c941645c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009fd0 tx=0x7f5c8800b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.093+0000 7f5c92a00640 1 -- 172.21.15.31:0/2491597723 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c88015020 con 0x7f5c94046540 2024-09-11T00:54:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.093+0000 7f5c92a00640 1 -- 172.21.15.31:0/2491597723 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c88011940 con 0x7f5c94046540 2024-09-11T00:54:48.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.093+0000 7f5c92a00640 1 -- 172.21.15.31:0/2491597723 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c88011d90 con 0x7f5c94046540 2024-09-11T00:54:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.102+0000 7f5c9be00640 1 -- 172.21.15.31:0/2491597723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c94044d20 con 0x7f5c94046540 2024-09-11T00:54:48.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.102+0000 7f5c9be00640 1 -- 172.21.15.31:0/2491597723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c9403e630 con 0x7f5c94046540 2024-09-11T00:54:48.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.109+0000 7f5c92a00640 1 -- 172.21.15.31:0/2491597723 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5c88022060 con 0x7f5c94046540 2024-09-11T00:54:48.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.109+0000 7f5c9be00640 1 -- 172.21.15.31:0/2491597723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c9404ae60 con 0x7f5c94046540 2024-09-11T00:54:48.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.111+0000 7f5c92a00640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c7c07c450 0x7f5c7c07e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:48.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.111+0000 7f5c92a00640 1 -- 172.21.15.31:0/2491597723 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7f5c8809a880 con 0x7f5c94046540 2024-09-11T00:54:48.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.113+0000 7f5c9aa00640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c7c07c450 0x7f5c7c07e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:48.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.116+0000 7f5c9aa00640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c7c07c450 0x7f5c7c07e910 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5c9416ba80 tx=0x7f5c8c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:48.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.126+0000 7f5c92a00640 1 -- 172.21.15.31:0/2491597723 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5c88062560 con 0x7f5c94046540 2024-09-11T00:54:48.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.636+0000 7f5c9be00640 1 -- 172.21.15.31:0/2491597723 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c9402ddd0 con 0x7f5c94046540 2024-09-11T00:54:48.644 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:48.644 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":171,"btime":"2024-09-11T00:54:25:481521+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7522,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:54:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.639+0000 7f5c92a00640 1 -- 172.21.15.31:0/2491597723 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6545 (secure 0 0 0) 0x7f5c88061e60 con 0x7f5c94046540 2024-09-11T00:54:48.644 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 171 2024-09-11T00:54:48.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.648+0000 7f5c9be00640 1 -- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c7c07c450 msgr2=0x7f5c7c07e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:48.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.648+0000 7f5c9be00640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c7c07c450 0x7f5c7c07e910 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5c9416ba80 tx=0x7f5c8c004730 comp rx=0 tx=0).stop 2024-09-11T00:54:48.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.648+0000 7f5c9be00640 1 -- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c94046540 msgr2=0x7f5c941645c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:48.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.648+0000 7f5c9be00640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c94046540 0x7f5c941645c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009fd0 tx=0x7f5c8800b870 comp rx=0 tx=0).stop 2024-09-11T00:54:48.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.652+0000 7f5c9be00640 1 -- 172.21.15.31:0/2491597723 shutdown_connections 2024-09-11T00:54:48.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.652+0000 7f5c9be00640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5c7c07c450 0x7f5c7c07e910 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:48.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.652+0000 7f5c9be00640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c9404a750 0x7f5c940cd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:48.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.652+0000 7f5c9be00640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5c94046540 0x7f5c941645c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:48.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.652+0000 7f5c9be00640 1 --2- 172.21.15.31:0/2491597723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c9400b6b0 0x7f5c9416e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:48.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.652+0000 7f5c9be00640 1 -- 172.21.15.31:0/2491597723 >> 172.21.15.31:0/2491597723 conn(0x7f5c94047e10 msgr2=0x7f5c940dae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:48.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.652+0000 7f5c9be00640 1 -- 172.21.15.31:0/2491597723 shutdown_connections 2024-09-11T00:54:48.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:48.652+0000 7f5c9be00640 1 -- 172.21.15.31:0/2491597723 wait complete. 2024-09-11T00:54:48.691 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-09-11T00:54:48.691 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-11T00:54:48.691 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:54:48.720 INFO:tasks.ceph.mds.b:Stopped 2024-09-11T00:54:48.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.044+0000 7faf02000640 1 -- 172.21.15.31:0/2584298559 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec031600 con 0x7faf040acaf0 2024-09-11T00:54:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.044+0000 7faf08c00640 1 -- 172.21.15.31:0/2584298559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf040acaf0 msgr2=0x7faf0406dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.044+0000 7faf08c00640 1 --2- 172.21.15.31:0/2584298559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf040acaf0 0x7faf0406dc60 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7faeec009860 tx=0x7faeec02a4b0 comp rx=0 tx=0).stop 2024-09-11T00:54:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.045+0000 7faf08c00640 1 -- 172.21.15.31:0/2584298559 shutdown_connections 2024-09-11T00:54:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.045+0000 7faf08c00640 1 --2- 172.21.15.31:0/2584298559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf040acaf0 0x7faf0406dc60 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.045+0000 7faf08c00640 1 -- 172.21.15.31:0/2584298559 >> 172.21.15.31:0/2584298559 conn(0x7faf0406caf0 msgr2=0x7faf0406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:49.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.050+0000 7faf08c00640 1 -- 172.21.15.31:0/2584298559 shutdown_connections 2024-09-11T00:54:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.050+0000 7faf08c00640 1 -- 172.21.15.31:0/2584298559 wait complete. 2024-09-11T00:54:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.056+0000 7faf08c00640 1 Processor -- start 2024-09-11T00:54:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.057+0000 7faf08c00640 1 -- start start 2024-09-11T00:54:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.058+0000 7faf08c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf040acaf0 0x7faf0420cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.058+0000 7faf08c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf040c64b0 0x7faf0414ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.058+0000 7faf08c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf040c1ac0 0x7faf0414b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.058+0000 7faf08c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faf040c4ad0 con 0x7faf040c1ac0 2024-09-11T00:54:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.058+0000 7faf08c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faf040700f0 con 0x7faf040c64b0 2024-09-11T00:54:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.058+0000 7faf08c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faf040c4c50 con 0x7faf040acaf0 2024-09-11T00:54:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.060+0000 7faf02a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf040c64b0 0x7faf0414ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.060+0000 7faf02a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf040c64b0 0x7faf0414ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:42198/0 (socket says 172.21.15.31:42198) 2024-09-11T00:54:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.060+0000 7faf02a00640 1 -- 172.21.15.31:0/2216456173 learned_addr learned my addr 172.21.15.31:0/2216456173 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.060+0000 7faf03400640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf040acaf0 0x7faf0420cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.060+0000 7faf03e00640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf040c1ac0 0x7faf0414b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.061+0000 7faf03400640 1 -- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf040c64b0 msgr2=0x7faf0414ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.061+0000 7faf03400640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf040c64b0 0x7faf0414ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.061+0000 7faf03400640 1 -- 172.21.15.31:0/2216456173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf040c1ac0 msgr2=0x7faf0414b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.062+0000 7faf03400640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf040c1ac0 0x7faf0414b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.062+0000 7faf03400640 1 -- 172.21.15.31:0/2216456173 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf040bfbf0 con 0x7faf040acaf0 2024-09-11T00:54:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.062+0000 7faf03e00640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf040c1ac0 0x7faf0414b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:54:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.063+0000 7faf03400640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf040acaf0 0x7faf0420cfd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faf04071a80 tx=0x7faef4009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:49.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.064+0000 7faefb400640 1 -- 172.21.15.31:0/2216456173 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef4019070 con 0x7faf040acaf0 2024-09-11T00:54:49.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.065+0000 7faefb400640 1 -- 172.21.15.31:0/2216456173 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faef4014440 con 0x7faf040acaf0 2024-09-11T00:54:49.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.065+0000 7faefb400640 1 -- 172.21.15.31:0/2216456173 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef4019070 con 0x7faf040acaf0 2024-09-11T00:54:49.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.065+0000 7faf08c00640 1 -- 172.21.15.31:0/2216456173 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0414b600 con 0x7faf040acaf0 2024-09-11T00:54:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.068+0000 7faf08c00640 1 -- 172.21.15.31:0/2216456173 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf040a6eb0 con 0x7faf040acaf0 2024-09-11T00:54:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.072+0000 7faf08c00640 1 -- 172.21.15.31:0/2216456173 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf0406dc60 con 0x7faf040acaf0 2024-09-11T00:54:49.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.073+0000 7faefb400640 1 -- 172.21.15.31:0/2216456173 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7faef4014650 con 0x7faf040acaf0 2024-09-11T00:54:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.075+0000 7faefb400640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faee00783a0 0x7faee007a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.075+0000 7faefb400640 1 -- 172.21.15.31:0/2216456173 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7faef400cc70 con 0x7faf040acaf0 2024-09-11T00:54:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.078+0000 7faf02a00640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faee00783a0 0x7faee007a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.079+0000 7faf02a00640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faee00783a0 0x7faee007a860 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7faf040c4090 tx=0x7faeec004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:49.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.086+0000 7faefb400640 1 -- 172.21.15.31:0/2216456173 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7faef4061e90 con 0x7faf040acaf0 2024-09-11T00:54:49.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.609+0000 7faf08c00640 1 -- 172.21.15.31:0/2216456173 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf040a9650 con 0x7faf040acaf0 2024-09-11T00:54:49.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.611+0000 7faefb400640 1 -- 172.21.15.31:0/2216456173 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6545 (secure 0 0 0) 0x7faef40668c0 con 0x7faf040acaf0 2024-09-11T00:54:49.614 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:49.614 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":171,"btime":"2024-09-11T00:54:25:481521+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7522,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:54:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 171 2024-09-11T00:54:49.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.616+0000 7faf08c00640 1 -- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faee00783a0 msgr2=0x7faee007a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:49.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.616+0000 7faf08c00640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faee00783a0 0x7faee007a860 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7faf040c4090 tx=0x7faeec004730 comp rx=0 tx=0).stop 2024-09-11T00:54:49.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.616+0000 7faf08c00640 1 -- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf040acaf0 msgr2=0x7faf0420cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:49.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.616+0000 7faf08c00640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf040acaf0 0x7faf0420cfd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faf04071a80 tx=0x7faef4009f70 comp rx=0 tx=0).stop 2024-09-11T00:54:49.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.616+0000 7faf08c00640 1 -- 172.21.15.31:0/2216456173 shutdown_connections 2024-09-11T00:54:49.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.616+0000 7faf08c00640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faee00783a0 0x7faee007a860 secure :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7faf040c4090 tx=0x7faeec004730 comp rx=0 tx=0).stop 2024-09-11T00:54:49.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.616+0000 7faf08c00640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf040c1ac0 0x7faf0414b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:49.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.616+0000 7faf08c00640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faf040c64b0 0x7faf0414ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:49.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.616+0000 7faf08c00640 1 --2- 172.21.15.31:0/2216456173 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faf040acaf0 0x7faf0420cfd0 secure :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faf04071a80 tx=0x7faef4009f70 comp rx=0 tx=0).stop 2024-09-11T00:54:49.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.616+0000 7faf08c00640 1 -- 172.21.15.31:0/2216456173 >> 172.21.15.31:0/2216456173 conn(0x7faf0406caf0 msgr2=0x7faf040a8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.619+0000 7faf08c00640 1 -- 172.21.15.31:0/2216456173 shutdown_connections 2024-09-11T00:54:49.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:49.619+0000 7faf08c00640 1 -- 172.21.15.31:0/2216456173 wait complete. 2024-09-11T00:54:49.655 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:54:51.656 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.959+0000 7f0c48000640 1 -- 172.21.15.31:0/120706788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c40158260 msgr2=0x7f0c400c5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.959+0000 7f0c48000640 1 --2- 172.21.15.31:0/120706788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c40158260 0x7f0c400c5210 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009830 tx=0x7f0c3402a420 comp rx=0 tx=0).stop 2024-09-11T00:54:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.960+0000 7f0c48000640 1 -- 172.21.15.31:0/120706788 shutdown_connections 2024-09-11T00:54:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.960+0000 7f0c48000640 1 --2- 172.21.15.31:0/120706788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c40158260 0x7f0c400c5210 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.960+0000 7f0c48000640 1 -- 172.21.15.31:0/120706788 >> 172.21.15.31:0/120706788 conn(0x7f0c400688e0 msgr2=0x7f0c40068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.960+0000 7f0c48000640 1 -- 172.21.15.31:0/120706788 shutdown_connections 2024-09-11T00:54:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.960+0000 7f0c48000640 1 -- 172.21.15.31:0/120706788 wait complete. 2024-09-11T00:54:51.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.965+0000 7f0c48000640 1 Processor -- start 2024-09-11T00:54:51.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.966+0000 7f0c48000640 1 -- start start 2024-09-11T00:54:51.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.968+0000 7f0c48000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c400c5750 0x7f0c400aa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.968+0000 7f0c48000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0c400c71c0 0x7f0c400abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.968+0000 7f0c48000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c40158260 0x7f0c400a96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.969+0000 7f0c44c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0c400c71c0 0x7f0c400abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.969+0000 7f0c48000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0c40158640 con 0x7f0c40158260 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.969+0000 7f0c48000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0c400c44c0 con 0x7f0c400c5750 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.969+0000 7f0c48000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0c400c5b90 con 0x7f0c400c71c0 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.969+0000 7f0c46000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c40158260 0x7f0c400a96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.969+0000 7f0c46000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c40158260 0x7f0c400a96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50310/0 (socket says 172.21.15.31:50310) 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.969+0000 7f0c46000640 1 -- 172.21.15.31:0/1320698978 learned_addr learned my addr 172.21.15.31:0/1320698978 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.969+0000 7f0c45600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c400c5750 0x7f0c400aa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.970+0000 7f0c46000640 1 -- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0c400c71c0 msgr2=0x7f0c400abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.970+0000 7f0c46000640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0c400c71c0 0x7f0c400abd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.970+0000 7f0c46000640 1 -- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c400c5750 msgr2=0x7f0c400aa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.970+0000 7f0c46000640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c400c5750 0x7f0c400aa8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.970+0000 7f0c46000640 1 -- 172.21.15.31:0/1320698978 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c400c2fd0 con 0x7f0c40158260 2024-09-11T00:54:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.970+0000 7f0c46000640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c40158260 0x7f0c400a96e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800c790 tx=0x7f0c3800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.971+0000 7f0c3d600640 1 -- 172.21.15.31:0/1320698978 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c38003970 con 0x7f0c40158260 2024-09-11T00:54:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.971+0000 7f0c3d600640 1 -- 172.21.15.31:0/1320698978 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c38003b10 con 0x7f0c40158260 2024-09-11T00:54:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.971+0000 7f0c3d600640 1 -- 172.21.15.31:0/1320698978 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c38005070 con 0x7f0c40158260 2024-09-11T00:54:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.971+0000 7f0c48000640 1 -- 172.21.15.31:0/1320698978 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c400c1770 con 0x7f0c40158260 2024-09-11T00:54:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.971+0000 7f0c48000640 1 -- 172.21.15.31:0/1320698978 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c400c1d30 con 0x7f0c40158260 2024-09-11T00:54:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.973+0000 7f0c48000640 1 -- 172.21.15.31:0/1320698978 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c400c5210 con 0x7f0c40158260 2024-09-11T00:54:51.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.975+0000 7f0c3d600640 1 -- 172.21.15.31:0/1320698978 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0c38005210 con 0x7f0c40158260 2024-09-11T00:54:51.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.976+0000 7f0c3d600640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0c20077ff0 0x7f0c2007a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:51.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.977+0000 7f0c3d600640 1 -- 172.21.15.31:0/1320698978 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7f0c3809aa00 con 0x7f0c40158260 2024-09-11T00:54:51.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.977+0000 7f0c45600640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0c20077ff0 0x7f0c2007a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:51.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.985+0000 7f0c3d600640 1 -- 172.21.15.31:0/1320698978 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0c38062000 con 0x7f0c40158260 2024-09-11T00:54:51.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:51.986+0000 7f0c45600640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0c20077ff0 0x7f0c2007a4b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0c34034000 tx=0x7f0c34040040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:52.527 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:52.527 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":171,"btime":"2024-09-11T00:54:25:481521+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7522,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:54:52.528 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 171 2024-09-11T00:54:52.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.524+0000 7f0c48000640 1 -- 172.21.15.31:0/1320698978 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c40157150 con 0x7f0c40158260 2024-09-11T00:54:52.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.525+0000 7f0c3d600640 1 -- 172.21.15.31:0/1320698978 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6545 (secure 0 0 0) 0x7f0c38066a30 con 0x7f0c40158260 2024-09-11T00:54:52.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.529+0000 7f0c48000640 1 -- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0c20077ff0 msgr2=0x7f0c2007a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.529+0000 7f0c48000640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0c20077ff0 0x7f0c2007a4b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0c34034000 tx=0x7f0c34040040 comp rx=0 tx=0).stop 2024-09-11T00:54:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.529+0000 7f0c48000640 1 -- 172.21.15.31:0/1320698978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c40158260 msgr2=0x7f0c400a96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.529+0000 7f0c48000640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c40158260 0x7f0c400a96e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f0c3800c790 tx=0x7f0c3800cc50 comp rx=0 tx=0).stop 2024-09-11T00:54:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.530+0000 7f0c48000640 1 -- 172.21.15.31:0/1320698978 shutdown_connections 2024-09-11T00:54:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.530+0000 7f0c48000640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0c20077ff0 0x7f0c2007a4b0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:52.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.530+0000 7f0c48000640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0c40158260 0x7f0c400a96e0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.530+0000 7f0c48000640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0c400c71c0 0x7f0c400abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.530+0000 7f0c48000640 1 --2- 172.21.15.31:0/1320698978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c400c5750 0x7f0c400aa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.530+0000 7f0c48000640 1 -- 172.21.15.31:0/1320698978 >> 172.21.15.31:0/1320698978 conn(0x7f0c400688e0 msgr2=0x7f0c40157500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:52.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.530+0000 7f0c48000640 1 -- 172.21.15.31:0/1320698978 shutdown_connections 2024-09-11T00:54:52.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:52.531+0000 7f0c48000640 1 -- 172.21.15.31:0/1320698978 wait complete. 2024-09-11T00:54:52.573 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:54:54.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:54:54.315 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:54:54.574 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.970+0000 7f7d9de00640 1 -- 172.21.15.31:0/2342684272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d980c2ff0 msgr2=0x7f7d9815a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.970+0000 7f7d9de00640 1 --2- 172.21.15.31:0/2342684272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d980c2ff0 0x7f7d9815a8e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f7d980c2ce0 tx=0x7f7d8802b7b0 comp rx=0 tx=0).stop 2024-09-11T00:54:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.971+0000 7f7d9de00640 1 -- 172.21.15.31:0/2342684272 shutdown_connections 2024-09-11T00:54:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.971+0000 7f7d9de00640 1 --2- 172.21.15.31:0/2342684272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d980c2ff0 0x7f7d9815a8e0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.971+0000 7f7d9de00640 1 -- 172.21.15.31:0/2342684272 >> 172.21.15.31:0/2342684272 conn(0x7f7d980a9750 msgr2=0x7f7d980a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.971+0000 7f7d9de00640 1 -- 172.21.15.31:0/2342684272 shutdown_connections 2024-09-11T00:54:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.971+0000 7f7d9de00640 1 -- 172.21.15.31:0/2342684272 wait complete. 2024-09-11T00:54:54.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.975+0000 7f7d9de00640 1 Processor -- start 2024-09-11T00:54:54.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.975+0000 7f7d9de00640 1 -- start start 2024-09-11T00:54:54.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.978+0000 7f7d9de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d98070570 0x7f7d9815fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:54.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.978+0000 7f7d9de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d980c2ff0 0x7f7d98148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:54.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.978+0000 7f7d9de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d980c3d10 0x7f7d981f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:54.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.978+0000 7f7d9de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9814fb20 con 0x7f7d980c2ff0 2024-09-11T00:54:54.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.978+0000 7f7d9de00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9806ef90 con 0x7f7d98070570 2024-09-11T00:54:54.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.978+0000 7f7d9de00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7d9806eb90 con 0x7f7d980c3d10 2024-09-11T00:54:54.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.979+0000 7f7d96000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d980c2ff0 0x7f7d98148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:54.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.979+0000 7f7d97400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d980c3d10 0x7f7d981f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:54.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.979+0000 7f7d97400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d980c3d10 0x7f7d981f3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:49246/0 (socket says 172.21.15.31:49246) 2024-09-11T00:54:54.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.979+0000 7f7d97400640 1 -- 172.21.15.31:0/3510247166 learned_addr learned my addr 172.21.15.31:0/3510247166 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:54.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.980+0000 7f7d96a00640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d98070570 0x7f7d9815fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:54.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.981+0000 7f7d96a00640 1 -- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d980c3d10 msgr2=0x7f7d981f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:54.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.981+0000 7f7d96a00640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d980c3d10 0x7f7d981f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:54.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.981+0000 7f7d96a00640 1 -- 172.21.15.31:0/3510247166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d980c2ff0 msgr2=0x7f7d98148e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:54.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.981+0000 7f7d96a00640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d980c2ff0 0x7f7d98148e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:54.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.981+0000 7f7d96a00640 1 -- 172.21.15.31:0/3510247166 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d98073990 con 0x7f7d98070570 2024-09-11T00:54:54.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.981+0000 7f7d96000640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d980c2ff0 0x7f7d98148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:54:54.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.984+0000 7f7d96a00640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d98070570 0x7f7d9815fb00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400c440 tx=0x7f7d8400c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.987+0000 7f7d8f400640 1 -- 172.21.15.31:0/3510247166 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d84017070 con 0x7f7d98070570 2024-09-11T00:54:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.987+0000 7f7d8f400640 1 -- 172.21.15.31:0/3510247166 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d84005320 con 0x7f7d98070570 2024-09-11T00:54:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.987+0000 7f7d8f400640 1 -- 172.21.15.31:0/3510247166 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d84004840 con 0x7f7d98070570 2024-09-11T00:54:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.988+0000 7f7d9de00640 1 -- 172.21.15.31:0/3510247166 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d981f0de0 con 0x7f7d98070570 2024-09-11T00:54:54.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.988+0000 7f7d9de00640 1 -- 172.21.15.31:0/3510247166 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d98072240 con 0x7f7d98070570 2024-09-11T00:54:55.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:54.991+0000 7f7d9de00640 1 -- 172.21.15.31:0/3510247166 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d980a9dc0 con 0x7f7d98070570 2024-09-11T00:54:55.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.003+0000 7f7d8f400640 1 -- 172.21.15.31:0/3510247166 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f7d840049e0 con 0x7f7d98070570 2024-09-11T00:54:55.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.005+0000 7f7d8f400640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7d78078040 0x7f7d7807a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:55.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.006+0000 7f7d8f400640 1 -- 172.21.15.31:0/3510247166 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7f7d8409a170 con 0x7f7d98070570 2024-09-11T00:54:55.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.007+0000 7f7d96000640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7d78078040 0x7f7d7807a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:55.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.009+0000 7f7d96000640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7d78078040 0x7f7d7807a500 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7d980721a0 tx=0x7f7d88004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:55.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.013+0000 7f7d8f400640 1 -- 172.21.15.31:0/3510247166 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f7d980a9dc0 con 0x7f7d98070570 2024-09-11T00:54:55.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.511+0000 7f7d9de00640 1 -- 172.21.15.31:0/3510247166 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d980c7690 con 0x7f7d98070570 2024-09-11T00:54:55.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.514+0000 7f7d8f400640 1 -- 172.21.15.31:0/3510247166 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6545 (secure 0 0 0) 0x7f7d84061800 con 0x7f7d98070570 2024-09-11T00:54:55.516 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:55.516 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":171,"btime":"2024-09-11T00:54:25:481521+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7522,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:54:55.516 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 171 2024-09-11T00:54:55.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.518+0000 7f7d9de00640 1 -- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7d78078040 msgr2=0x7f7d7807a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:55.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.518+0000 7f7d9de00640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7d78078040 0x7f7d7807a500 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7d980721a0 tx=0x7f7d88004730 comp rx=0 tx=0).stop 2024-09-11T00:54:55.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.518+0000 7f7d9de00640 1 -- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d98070570 msgr2=0x7f7d9815fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:55.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.518+0000 7f7d9de00640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d98070570 0x7f7d9815fb00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7d8400c440 tx=0x7f7d8400c900 comp rx=0 tx=0).stop 2024-09-11T00:54:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.520+0000 7f7d9de00640 1 -- 172.21.15.31:0/3510247166 shutdown_connections 2024-09-11T00:54:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.520+0000 7f7d9de00640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7d78078040 0x7f7d7807a500 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.520+0000 7f7d9de00640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7d980c3d10 0x7f7d981f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.520+0000 7f7d9de00640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d980c2ff0 0x7f7d98148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.520+0000 7f7d9de00640 1 --2- 172.21.15.31:0/3510247166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7d98070570 0x7f7d9815fb00 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.520+0000 7f7d9de00640 1 -- 172.21.15.31:0/3510247166 >> 172.21.15.31:0/3510247166 conn(0x7f7d980a9750 msgr2=0x7f7d980a8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.520+0000 7f7d9de00640 1 -- 172.21.15.31:0/3510247166 shutdown_connections 2024-09-11T00:54:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:55.520+0000 7f7d9de00640 1 -- 172.21.15.31:0/3510247166 wait complete. 2024-09-11T00:54:55.551 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:54:57.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:54:57.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:57.987+0000 7f2e72e00640 1 -- 172.21.15.31:0/1439929075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e6c0c5de0 msgr2=0x7f2e6c155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:57.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:57.987+0000 7f2e72e00640 1 --2- 172.21.15.31:0/1439929075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e6c0c5de0 0x7f2e6c155560 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c076530 tx=0x7f2e6402aae0 comp rx=0 tx=0).stop 2024-09-11T00:54:57.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:57.992+0000 7f2e72e00640 1 -- 172.21.15.31:0/1439929075 shutdown_connections 2024-09-11T00:54:57.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:57.992+0000 7f2e72e00640 1 --2- 172.21.15.31:0/1439929075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e6c0c5de0 0x7f2e6c155560 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:57.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:57.992+0000 7f2e72e00640 1 --2- 172.21.15.31:0/1439929075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e6c0c2ff0 0x7f2e6c0c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:57.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:57.992+0000 7f2e72e00640 1 --2- 172.21.15.31:0/1439929075 >> v2:172.21.15.204:3300/0 conn(0x7f2e6c06e480 0x7f2e6c06e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:57.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:57.992+0000 7f2e72e00640 1 -- 172.21.15.31:0/1439929075 >> 172.21.15.31:0/1439929075 conn(0x7f2e6c0a9750 msgr2=0x7f2e6c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:58.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:57.994+0000 7f2e72e00640 1 -- 172.21.15.31:0/1439929075 shutdown_connections 2024-09-11T00:54:58.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:57.994+0000 7f2e72e00640 1 -- 172.21.15.31:0/1439929075 wait complete. 2024-09-11T00:54:58.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.008+0000 7f2e72e00640 1 Processor -- start 2024-09-11T00:54:58.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.009+0000 7f2e72e00640 1 -- start start 2024-09-11T00:54:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.010+0000 7f2e72e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e6c06e480 0x7f2e6c146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.010+0000 7f2e72e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e6c0c2ff0 0x7f2e6c0a4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.010+0000 7f2e72e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e6c0c5de0 0x7f2e6c0a4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.010+0000 7f2e72e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c14fb20 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.010+0000 7f2e72e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6c06ef90 con 0x7f2e6c0c2ff0 2024-09-11T00:54:58.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.010+0000 7f2e72e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2e6c0c7690 con 0x7f2e6c06e480 2024-09-11T00:54:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.019+0000 7f2e70e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e6c0c5de0 0x7f2e6c0a4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.019+0000 7f2e70e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e6c0c5de0 0x7f2e6c0a4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54680/0 (socket says 172.21.15.31:54680) 2024-09-11T00:54:58.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.020+0000 7f2e70e00640 1 -- 172.21.15.31:0/2881305926 learned_addr learned my addr 172.21.15.31:0/2881305926 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:54:58.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.020+0000 7f2e6b400640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e6c0c2ff0 0x7f2e6c0a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:58.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.022+0000 7f2e6be00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e6c06e480 0x7f2e6c146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:58.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.024+0000 7f2e70e00640 1 -- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e6c06e480 msgr2=0x7f2e6c146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:58.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.025+0000 7f2e70e00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e6c06e480 0x7f2e6c146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:58.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.025+0000 7f2e70e00640 1 -- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e6c0c2ff0 msgr2=0x7f2e6c0a4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:58.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.025+0000 7f2e70e00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e6c0c2ff0 0x7f2e6c0a4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:58.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.025+0000 7f2e70e00640 1 -- 172.21.15.31:0/2881305926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6c0a5510 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.026+0000 7f2e6be00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e6c06e480 0x7f2e6c146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:54:58.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.027+0000 7f2e6b400640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e6c0c2ff0 0x7f2e6c0a4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:54:58.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.029+0000 7f2e70e00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e6c0c5de0 0x7f2e6c0a4a40 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009d60 tx=0x7f2e6402d970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:58.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.030+0000 7f2e63e00640 1 -- 172.21.15.31:0/2881305926 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64044070 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.030+0000 7f2e63e00640 1 -- 172.21.15.31:0/2881305926 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e6402dc60 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.030+0000 7f2e63e00640 1 -- 172.21.15.31:0/2881305926 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6403f6e0 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.038+0000 7f2e72e00640 1 -- 172.21.15.31:0/2881305926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6c072490 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.038+0000 7f2e72e00640 1 -- 172.21.15.31:0/2881305926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e6c0a4f80 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.039+0000 7f2e72e00640 1 -- 172.21.15.31:0/2881305926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e6c072280 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.054+0000 7f2e63e00640 1 -- 172.21.15.31:0/2881305926 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2e640483f0 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.055+0000 7f2e63e00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2e4c078040 0x7f2e4c07a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:54:58.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.055+0000 7f2e63e00640 1 -- 172.21.15.31:0/2881305926 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7f2e640bbeb0 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.056+0000 7f2e63e00640 1 -- 172.21.15.31:0/2881305926 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2e6404d020 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.056+0000 7f2e6be00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2e4c078040 0x7f2e4c07a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:54:58.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.059+0000 7f2e6be00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2e4c078040 0x7f2e4c07a500 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0058c0 tx=0x7f2e5c005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:54:58.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.539+0000 7f2e72e00640 1 -- 172.21.15.31:0/2881305926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e6c14f740 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.539+0000 7f2e63e00640 1 -- 172.21.15.31:0/2881305926 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6545 (secure 0 0 0) 0x7f2e64032030 con 0x7f2e6c0c5de0 2024-09-11T00:54:58.541 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:54:58.541 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":171,"btime":"2024-09-11T00:54:25:481521+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7522,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:54:58.542 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 171 2024-09-11T00:54:58.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.542+0000 7f2e72e00640 1 -- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2e4c078040 msgr2=0x7f2e4c07a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:58.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.542+0000 7f2e72e00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2e4c078040 0x7f2e4c07a500 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0058c0 tx=0x7f2e5c005850 comp rx=0 tx=0).stop 2024-09-11T00:54:58.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.543+0000 7f2e72e00640 1 -- 172.21.15.31:0/2881305926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e6c0c5de0 msgr2=0x7f2e6c0a4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:54:58.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.543+0000 7f2e72e00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e6c0c5de0 0x7f2e6c0a4a40 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009d60 tx=0x7f2e6402d970 comp rx=0 tx=0).stop 2024-09-11T00:54:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.543+0000 7f2e72e00640 1 -- 172.21.15.31:0/2881305926 shutdown_connections 2024-09-11T00:54:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.543+0000 7f2e72e00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2e4c078040 0x7f2e4c07a500 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.543+0000 7f2e72e00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e6c0c5de0 0x7f2e6c0a4a40 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.543+0000 7f2e72e00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e6c0c2ff0 0x7f2e6c0a4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.543+0000 7f2e72e00640 1 --2- 172.21.15.31:0/2881305926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2e6c06e480 0x7f2e6c146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:54:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.543+0000 7f2e72e00640 1 -- 172.21.15.31:0/2881305926 >> 172.21.15.31:0/2881305926 conn(0x7f2e6c0a9750 msgr2=0x7f2e6c14c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:54:58.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.545+0000 7f2e72e00640 1 -- 172.21.15.31:0/2881305926 shutdown_connections 2024-09-11T00:54:58.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:54:58.545+0000 7f2e72e00640 1 -- 172.21.15.31:0/2881305926 wait complete. 2024-09-11T00:54:58.576 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:55:00.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:55:00.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.803+0000 7ff701800640 1 -- 172.21.15.31:0/2910809541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6fc0c3d10 msgr2=0x7ff6fc15c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:00.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.803+0000 7ff701800640 1 --2- 172.21.15.31:0/2910809541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6fc0c3d10 0x7ff6fc15c8f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400afd0 tx=0x7ff6f400aec0 comp rx=0 tx=0).stop 2024-09-11T00:55:00.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.803+0000 7ff701800640 1 -- 172.21.15.31:0/2910809541 shutdown_connections 2024-09-11T00:55:00.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.803+0000 7ff701800640 1 --2- 172.21.15.31:0/2910809541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6fc0c3d10 0x7ff6fc15c8f0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:00.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.803+0000 7ff701800640 1 --2- 172.21.15.31:0/2910809541 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6fc06e480 0x7ff6fc06e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:00.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.803+0000 7ff701800640 1 -- 172.21.15.31:0/2910809541 >> 172.21.15.31:0/2910809541 conn(0x7ff6fc0a9750 msgr2=0x7ff6fc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:00.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.803+0000 7ff701800640 1 -- 172.21.15.31:0/2910809541 shutdown_connections 2024-09-11T00:55:00.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.804+0000 7ff701800640 1 -- 172.21.15.31:0/2910809541 wait complete. 2024-09-11T00:55:00.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.806+0000 7ff701800640 1 Processor -- start 2024-09-11T00:55:00.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.807+0000 7ff701800640 1 -- start start 2024-09-11T00:55:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff701800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6fc06e480 0x7ff6fc14ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff701800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6fc0c2ff0 0x7ff6fc150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff701800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc0c3d10 0x7ff6fc14afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff701800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc14fb20 con 0x7ff6fc0c2ff0 2024-09-11T00:55:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff701800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc06ef90 con 0x7ff6fc0c3d10 2024-09-11T00:55:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff701800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff6fc0c5fb0 con 0x7ff6fc06e480 2024-09-11T00:55:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff6fa000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6fc0c2ff0 0x7ff6fc150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff6fa000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6fc0c2ff0 0x7ff6fc150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54694/0 (socket says 172.21.15.31:54694) 2024-09-11T00:55:00.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff6fa000640 1 -- 172.21.15.31:0/1592578069 learned_addr learned my addr 172.21.15.31:0/1592578069 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff6fb400640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc0c3d10 0x7ff6fc14afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.808+0000 7ff6faa00640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6fc06e480 0x7ff6fc14ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff6fa000640 1 -- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6fc06e480 msgr2=0x7ff6fc14ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff6fa000640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6fc06e480 0x7ff6fc14ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff6fa000640 1 -- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc0c3d10 msgr2=0x7ff6fc14afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff6fa000640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc0c3d10 0x7ff6fc14afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff6fa000640 1 -- 172.21.15.31:0/1592578069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6fc151610 con 0x7ff6fc0c2ff0 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff6faa00640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6fc06e480 0x7ff6fc14ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff6fa000640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6fc0c2ff0 0x7ff6fc150000 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800c400 tx=0x7ff6e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff6f2a00640 1 -- 172.21.15.31:0/1592578069 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8011790 con 0x7ff6fc0c2ff0 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff6f2a00640 1 -- 172.21.15.31:0/1592578069 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6e8017070 con 0x7ff6fc0c2ff0 2024-09-11T00:55:00.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff6f2a00640 1 -- 172.21.15.31:0/1592578069 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8011d00 con 0x7ff6fc0c2ff0 2024-09-11T00:55:00.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.809+0000 7ff701800640 1 -- 172.21.15.31:0/1592578069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6fc1518f0 con 0x7ff6fc0c2ff0 2024-09-11T00:55:00.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.810+0000 7ff701800640 1 -- 172.21.15.31:0/1592578069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6fc14b4f0 con 0x7ff6fc0c2ff0 2024-09-11T00:55:00.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.811+0000 7ff701800640 1 -- 172.21.15.31:0/1592578069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6fc1f3ab0 con 0x7ff6fc0c2ff0 2024-09-11T00:55:00.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.813+0000 7ff6f2a00640 1 -- 172.21.15.31:0/1592578069 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff6e8026430 con 0x7ff6fc0c2ff0 2024-09-11T00:55:00.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.815+0000 7ff6f2a00640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6dc078040 0x7ff6dc07a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:00.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.815+0000 7ff6f2a00640 1 -- 172.21.15.31:0/1592578069 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7ff6e809b450 con 0x7ff6fc0c2ff0 2024-09-11T00:55:00.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.816+0000 7ff6faa00640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6dc078040 0x7ff6dc07a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:00.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.818+0000 7ff6faa00640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6dc078040 0x7ff6dc07a500 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00b3c0 tx=0x7ff6ec00d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:00.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:00.821+0000 7ff6f2a00640 1 -- 172.21.15.31:0/1592578069 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff6e8063130 con 0x7ff6fc0c2ff0 2024-09-11T00:55:01.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.183+0000 7ff701800640 1 -- 172.21.15.31:0/1592578069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6fc0c7690 con 0x7ff6fc0c2ff0 2024-09-11T00:55:01.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.186+0000 7ff6f2a00640 1 -- 172.21.15.31:0/1592578069 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6545 (secure 0 0 0) 0x7ff6e8062a30 con 0x7ff6fc0c2ff0 2024-09-11T00:55:01.188 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:55:01.189 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":171,"btime":"2024-09-11T00:54:25:481521+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7522,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:55:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 171 2024-09-11T00:55:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.190+0000 7ff701800640 1 -- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6dc078040 msgr2=0x7ff6dc07a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.190+0000 7ff701800640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6dc078040 0x7ff6dc07a500 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00b3c0 tx=0x7ff6ec00d040 comp rx=0 tx=0).stop 2024-09-11T00:55:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.190+0000 7ff701800640 1 -- 172.21.15.31:0/1592578069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6fc0c2ff0 msgr2=0x7ff6fc150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.190+0000 7ff701800640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6fc0c2ff0 0x7ff6fc150000 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800c400 tx=0x7ff6e800c8c0 comp rx=0 tx=0).stop 2024-09-11T00:55:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.191+0000 7ff701800640 1 -- 172.21.15.31:0/1592578069 shutdown_connections 2024-09-11T00:55:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.191+0000 7ff701800640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff6dc078040 0x7ff6dc07a500 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.191+0000 7ff701800640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc0c3d10 0x7ff6fc14afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.191+0000 7ff701800640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6fc0c2ff0 0x7ff6fc150000 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.191+0000 7ff701800640 1 --2- 172.21.15.31:0/1592578069 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff6fc06e480 0x7ff6fc14ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:01.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.191+0000 7ff701800640 1 -- 172.21.15.31:0/1592578069 >> 172.21.15.31:0/1592578069 conn(0x7ff6fc0a9750 msgr2=0x7ff6fc0a8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:01.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.191+0000 7ff701800640 1 -- 172.21.15.31:0/1592578069 shutdown_connections 2024-09-11T00:55:01.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:01.191+0000 7ff701800640 1 -- 172.21.15.31:0/1592578069 wait complete. 2024-09-11T00:55:01.215 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:55:03.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:55:03.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.459+0000 7f13ef000640 1 -- 172.21.15.31:0/285979585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e81539c0 msgr2=0x7f13e8159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:03.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.459+0000 7f13ef000640 1 --2- 172.21.15.31:0/285979585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e81539c0 0x7f13e8159670 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f13e0009830 tx=0x7f13e002a510 comp rx=0 tx=0).stop 2024-09-11T00:55:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.459+0000 7f13ef000640 1 -- 172.21.15.31:0/285979585 shutdown_connections 2024-09-11T00:55:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.459+0000 7f13ef000640 1 --2- 172.21.15.31:0/285979585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e81539c0 0x7f13e8159670 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.459+0000 7f13ef000640 1 --2- 172.21.15.31:0/285979585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e8156ad0 0x7f13e8153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.460+0000 7f13ef000640 1 -- 172.21.15.31:0/285979585 >> 172.21.15.31:0/285979585 conn(0x7f13e80a9750 msgr2=0x7f13e80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.460+0000 7f13ef000640 1 -- 172.21.15.31:0/285979585 shutdown_connections 2024-09-11T00:55:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.460+0000 7f13ef000640 1 -- 172.21.15.31:0/285979585 wait complete. 2024-09-11T00:55:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.463+0000 7f13ef000640 1 Processor -- start 2024-09-11T00:55:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.463+0000 7f13ef000640 1 -- start start 2024-09-11T00:55:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.464+0000 7f13ef000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e80c5b10 0x7f13e81e1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.464+0000 7f13ef000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13e81539c0 0x7f13e8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.464+0000 7f13ef000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e8156ad0 0x7f13e81e2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.464+0000 7f13ef000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f13e8156ef0 con 0x7f13e8156ad0 2024-09-11T00:55:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.464+0000 7f13ef000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13e80c7320 con 0x7f13e81539c0 2024-09-11T00:55:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.464+0000 7f13ef000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f13e80c3950 con 0x7f13e80c5b10 2024-09-11T00:55:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.465+0000 7f13e7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e80c5b10 0x7f13e81e1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.465+0000 7f13e7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e80c5b10 0x7f13e81e1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:53214/0 (socket says 172.21.15.31:53214) 2024-09-11T00:55:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.465+0000 7f13e7e00640 1 -- 172.21.15.31:0/4196003195 learned_addr learned my addr 172.21.15.31:0/4196003195 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:55:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.465+0000 7f13e7400640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13e81539c0 0x7f13e8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.466+0000 7f13e7e00640 1 -- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13e81539c0 msgr2=0x7f13e8158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.466+0000 7f13e7e00640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13e81539c0 0x7f13e8158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.466+0000 7f13e7e00640 1 -- 172.21.15.31:0/4196003195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e8156ad0 msgr2=0x7f13e81e2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.466+0000 7f13ed000640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e8156ad0 0x7f13e81e2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.466+0000 7f13e7e00640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e8156ad0 0x7f13e81e2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.466+0000 7f13e7e00640 1 -- 172.21.15.31:0/4196003195 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e814cb10 con 0x7f13e80c5b10 2024-09-11T00:55:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.466+0000 7f13ed000640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e8156ad0 0x7f13e81e2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:55:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.467+0000 7f13e7e00640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e80c5b10 0x7f13e81e1800 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f13d800c820 tx=0x7f13d800cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.468+0000 7f13dfe00640 1 -- 172.21.15.31:0/4196003195 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8017070 con 0x7f13e80c5b10 2024-09-11T00:55:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.468+0000 7f13dfe00640 1 -- 172.21.15.31:0/4196003195 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13d8005070 con 0x7f13e80c5b10 2024-09-11T00:55:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.468+0000 7f13ef000640 1 -- 172.21.15.31:0/4196003195 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e81e6790 con 0x7f13e80c5b10 2024-09-11T00:55:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.468+0000 7f13ef000640 1 -- 172.21.15.31:0/4196003195 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e81593c0 con 0x7f13e80c5b10 2024-09-11T00:55:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.468+0000 7f13dfe00640 1 -- 172.21.15.31:0/4196003195 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80054c0 con 0x7f13e80c5b10 2024-09-11T00:55:03.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.469+0000 7f13ef000640 1 -- 172.21.15.31:0/4196003195 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c0005290 con 0x7f13e80c5b10 2024-09-11T00:55:03.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.471+0000 7f13dfe00640 1 -- 172.21.15.31:0/4196003195 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f13d8005660 con 0x7f13e80c5b10 2024-09-11T00:55:03.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.472+0000 7f13dfe00640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077fc0 0x7f13c407a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:03.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.472+0000 7f13dfe00640 1 -- 172.21.15.31:0/4196003195 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 9418+0+0 (secure 0 0 0) 0x7f13d809a490 con 0x7f13e80c5b10 2024-09-11T00:55:03.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.473+0000 7f13e7400640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077fc0 0x7f13c407a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:03.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.473+0000 7f13e7400640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077fc0 0x7f13c407a480 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f13e80752f0 tx=0x7f13d0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:03.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.482+0000 7f13dfe00640 1 -- 172.21.15.31:0/4196003195 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f13d8061b20 con 0x7f13e80c5b10 2024-09-11T00:55:03.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.809+0000 7f13ef000640 1 -- 172.21.15.31:0/4196003195 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13c0005580 con 0x7f13e80c5b10 2024-09-11T00:55:03.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.810+0000 7f13dfe00640 1 -- 172.21.15.31:0/4196003195 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6545 (secure 0 0 0) 0x7f13d8066550 con 0x7f13e80c5b10 2024-09-11T00:55:03.812 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:55:03.812 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":171,"btime":"2024-09-11T00:54:25:481521+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7522,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:52:42.957872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7357},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7357":{"gid":7357,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":5,"addr":"172.21.15.204:6835/4094068967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4094068967},{"type":"v1","addr":"172.21.15.204:6835","nonce":4094068967}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7357,"qdb_cluster":[7357]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:55:03.812 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 171 2024-09-11T00:55:03.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 -- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077fc0 msgr2=0x7f13c407a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077fc0 0x7f13c407a480 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f13e80752f0 tx=0x7f13d0005f90 comp rx=0 tx=0).stop 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 -- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e80c5b10 msgr2=0x7f13e81e1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e80c5b10 0x7f13e81e1800 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f13d800c820 tx=0x7f13d800cce0 comp rx=0 tx=0).stop 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 -- 172.21.15.31:0/4196003195 shutdown_connections 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f13c4077fc0 0x7f13c407a480 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13e8156ad0 0x7f13e81e2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13e81539c0 0x7f13e8158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 --2- 172.21.15.31:0/4196003195 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13e80c5b10 0x7f13e81e1800 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 -- 172.21.15.31:0/4196003195 >> 172.21.15.31:0/4196003195 conn(0x7f13e80a9750 msgr2=0x7f13e80a9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 -- 172.21.15.31:0/4196003195 shutdown_connections 2024-09-11T00:55:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:03.813+0000 7f13ef000640 1 -- 172.21.15.31:0/4196003195 wait complete. 2024-09-11T00:55:03.835 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-11T00:55:05.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:55:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.489+0000 7f0a7d000640 1 -- 172.21.15.31:0/823793002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7806e9c0 msgr2=0x7f0a781568c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.489+0000 7f0a7d000640 1 --2- 172.21.15.31:0/823793002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7806e9c0 0x7f0a781568c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0a70005fd0 tx=0x7f0a7002a940 comp rx=0 tx=0).stop 2024-09-11T00:55:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.489+0000 7f0a7d000640 1 -- 172.21.15.31:0/823793002 shutdown_connections 2024-09-11T00:55:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.489+0000 7f0a7d000640 1 --2- 172.21.15.31:0/823793002 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7806e9c0 0x7f0a781568c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:06.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.489+0000 7f0a7d000640 1 --2- 172.21.15.31:0/823793002 >> v2:172.21.15.31:3300/0 conn(0x7f0a780c2ff0 0x7f0a7806e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:06.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.489+0000 7f0a7d000640 1 -- 172.21.15.31:0/823793002 >> 172.21.15.31:0/823793002 conn(0x7f0a780a9750 msgr2=0x7f0a780a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:06.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.489+0000 7f0a7d000640 1 -- 172.21.15.31:0/823793002 shutdown_connections 2024-09-11T00:55:06.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.489+0000 7f0a7d000640 1 -- 172.21.15.31:0/823793002 wait complete. 2024-09-11T00:55:06.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.491+0000 7f0a7d000640 1 Processor -- start 2024-09-11T00:55:06.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.491+0000 7f0a7d000640 1 -- start start 2024-09-11T00:55:06.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.492+0000 7f0a7d000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7806e9c0 0x7f0a781f1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:06.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.492+0000 7f0a7d000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a78070570 0x7f0a78149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:06.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.492+0000 7f0a7d000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a780c2ff0 0x7f0a78073430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:06.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.492+0000 7f0a7d000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7814fb20 con 0x7f0a780c2ff0 2024-09-11T00:55:06.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.492+0000 7f0a7d000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7806ef90 con 0x7f0a78070570 2024-09-11T00:55:06.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.492+0000 7f0a7d000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0a780c5fb0 con 0x7f0a7806e9c0 2024-09-11T00:55:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.493+0000 7f0a76800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a780c2ff0 0x7f0a78073430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.493+0000 7f0a76800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a780c2ff0 0x7f0a78073430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54736/0 (socket says 172.21.15.31:54736) 2024-09-11T00:55:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.493+0000 7f0a76800640 1 -- 172.21.15.31:0/4027293049 learned_addr learned my addr 172.21.15.31:0/4027293049 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:55:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.493+0000 7f0a76800640 1 -- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7806e9c0 msgr2=0x7f0a781f1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.493+0000 7f0a76800640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7806e9c0 0x7f0a781f1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.493+0000 7f0a76800640 1 -- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a78070570 msgr2=0x7f0a78149100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.493+0000 7f0a76800640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a78070570 0x7f0a78149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.493+0000 7f0a76800640 1 -- 172.21.15.31:0/4027293049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a78149640 con 0x7f0a780c2ff0 2024-09-11T00:55:06.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.494+0000 7f0a76800640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a780c2ff0 0x7f0a78073430 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f0a70006010 tx=0x7f0a70031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:06.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.494+0000 7f0a6e000640 1 -- 172.21.15.31:0/4027293049 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70030430 con 0x7f0a780c2ff0 2024-09-11T00:55:06.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.494+0000 7f0a6e000640 1 -- 172.21.15.31:0/4027293049 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a70004240 con 0x7f0a780c2ff0 2024-09-11T00:55:06.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.494+0000 7f0a6e000640 1 -- 172.21.15.31:0/4027293049 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70004670 con 0x7f0a780c2ff0 2024-09-11T00:55:06.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.494+0000 7f0a7d000640 1 -- 172.21.15.31:0/4027293049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a78074cf0 con 0x7f0a780c2ff0 2024-09-11T00:55:06.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.494+0000 7f0a7d000640 1 -- 172.21.15.31:0/4027293049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a780aa470 con 0x7f0a780c2ff0 2024-09-11T00:55:06.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.496+0000 7f0a7d000640 1 -- 172.21.15.31:0/4027293049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a781f1890 con 0x7f0a780c2ff0 2024-09-11T00:55:06.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.497+0000 7f0a6e000640 1 -- 172.21.15.31:0/4027293049 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0a7002f070 con 0x7f0a780c2ff0 2024-09-11T00:55:06.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.499+0000 7f0a6e000640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a58077fa0 0x7f0a5807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:06.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.499+0000 7f0a6e000640 1 -- 172.21.15.31:0/4027293049 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7f0a700bc210 con 0x7f0a780c2ff0 2024-09-11T00:55:06.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.499+0000 7f0a75e00640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a58077fa0 0x7f0a5807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:06.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.501+0000 7f0a75e00640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a58077fa0 0x7f0a5807a460 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0a68007fd0 tx=0x7f0a68007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:06.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.505+0000 7f0a6e000640 1 -- 172.21.15.31:0/4027293049 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0a78149640 con 0x7f0a780c2ff0 2024-09-11T00:55:06.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.840+0000 7f0a7d000640 1 -- 172.21.15.31:0/4027293049 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a780c7690 con 0x7f0a780c2ff0 2024-09-11T00:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.841+0000 7f0a6e000640 1 -- 172.21.15.31:0/4027293049 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) ==== 95+0+5713 (secure 0 0 0) 0x7f0a70083e40 con 0x7f0a780c2ff0 2024-09-11T00:55:06.843 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:55:06.843 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":173,"btime":"2024-09-11T00:55:06:016126+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:05.294044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:55:06.843 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 173 2024-09-11T00:55:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.843+0000 7f0a7d000640 1 -- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a58077fa0 msgr2=0x7f0a5807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:06.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.843+0000 7f0a7d000640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a58077fa0 0x7f0a5807a460 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0a68007fd0 tx=0x7f0a68007900 comp rx=0 tx=0).stop 2024-09-11T00:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.843+0000 7f0a7d000640 1 -- 172.21.15.31:0/4027293049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a780c2ff0 msgr2=0x7f0a78073430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.843+0000 7f0a7d000640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a780c2ff0 0x7f0a78073430 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f0a70006010 tx=0x7f0a70031e20 comp rx=0 tx=0).stop 2024-09-11T00:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.844+0000 7f0a7d000640 1 -- 172.21.15.31:0/4027293049 shutdown_connections 2024-09-11T00:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.844+0000 7f0a7d000640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0a58077fa0 0x7f0a5807a460 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.844+0000 7f0a7d000640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a780c2ff0 0x7f0a78073430 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.844+0000 7f0a7d000640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a78070570 0x7f0a78149100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.844+0000 7f0a7d000640 1 --2- 172.21.15.31:0/4027293049 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0a7806e9c0 0x7f0a781f1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.844+0000 7f0a7d000640 1 -- 172.21.15.31:0/4027293049 >> 172.21.15.31:0/4027293049 conn(0x7f0a780a9750 msgr2=0x7f0a780c6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.844+0000 7f0a7d000640 1 -- 172.21.15.31:0/4027293049 shutdown_connections 2024-09-11T00:55:06.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:06.844+0000 7f0a7d000640 1 -- 172.21.15.31:0/4027293049 wait complete. 2024-09-11T00:55:06.867 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-09-11T00:55:06.867 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:55:06.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:55:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.095+0000 7f74b3200640 1 -- 172.21.15.31:0/3916681363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b406dd70 msgr2=0x7f74b4154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.095+0000 7f74b3200640 1 --2- 172.21.15.31:0/3916681363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b406dd70 0x7f74b4154930 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009830 tx=0x7f74a402a530 comp rx=0 tx=0).stop 2024-09-11T00:55:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.095+0000 7f74b3200640 1 -- 172.21.15.31:0/3916681363 shutdown_connections 2024-09-11T00:55:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.095+0000 7f74b3200640 1 --2- 172.21.15.31:0/3916681363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b406dd70 0x7f74b4154930 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.095+0000 7f74b3200640 1 -- 172.21.15.31:0/3916681363 >> 172.21.15.31:0/3916681363 conn(0x7f74b40c4d20 msgr2=0x7f74b40c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.095+0000 7f74b3200640 1 -- 172.21.15.31:0/3916681363 shutdown_connections 2024-09-11T00:55:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.095+0000 7f74b3200640 1 -- 172.21.15.31:0/3916681363 wait complete. 2024-09-11T00:55:07.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.097+0000 7f74b3200640 1 Processor -- start 2024-09-11T00:55:07.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.097+0000 7f74b3200640 1 -- start start 2024-09-11T00:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.098+0000 7f74b3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b406dd70 0x7f74b414dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.098+0000 7f74b3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b4074a30 0x7f74b41ef530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.098+0000 7f74b3200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b4150d60 0x7f74b41dffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.098+0000 7f74b3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f74b4070c10 con 0x7f74b406dd70 2024-09-11T00:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.098+0000 7f74b3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f74b4071260 con 0x7f74b4074a30 2024-09-11T00:55:07.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.098+0000 7f74b3200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f74b406f470 con 0x7f74b4150d60 2024-09-11T00:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.099+0000 7f74b1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b406dd70 0x7f74b414dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.099+0000 7f74b1e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b406dd70 0x7f74b414dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54746/0 (socket says 172.21.15.31:54746) 2024-09-11T00:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.099+0000 7f74b1e00640 1 -- 172.21.15.31:0/420976998 learned_addr learned my addr 172.21.15.31:0/420976998 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.099+0000 7f74b2800640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b4150d60 0x7f74b41dffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.099+0000 7f74b1400640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b4074a30 0x7f74b41ef530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.100+0000 7f74b1e00640 1 -- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b4150d60 msgr2=0x7f74b41dffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.100+0000 7f74b1e00640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b4150d60 0x7f74b41dffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.100+0000 7f74b1e00640 1 -- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b4074a30 msgr2=0x7f74b41ef530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.100+0000 7f74b1e00640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b4074a30 0x7f74b41ef530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.100+0000 7f74b1e00640 1 -- 172.21.15.31:0/420976998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b414e180 con 0x7f74b406dd70 2024-09-11T00:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.100+0000 7f74b1400640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b4074a30 0x7f74b41ef530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:55:07.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.100+0000 7f74b1e00640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b406dd70 0x7f74b414dc40 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f74a4000c00 tx=0x7f74a4032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:07.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.101+0000 7f74aa000640 1 -- 172.21.15.31:0/420976998 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a4040070 con 0x7f74b406dd70 2024-09-11T00:55:07.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.101+0000 7f74aa000640 1 -- 172.21.15.31:0/420976998 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74a40038e0 con 0x7f74b406dd70 2024-09-11T00:55:07.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.101+0000 7f74aa000640 1 -- 172.21.15.31:0/420976998 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a4003d30 con 0x7f74b406dd70 2024-09-11T00:55:07.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.101+0000 7f74b3200640 1 -- 172.21.15.31:0/420976998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b41502c0 con 0x7f74b406dd70 2024-09-11T00:55:07.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.101+0000 7f74b3200640 1 -- 172.21.15.31:0/420976998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b41e15f0 con 0x7f74b406dd70 2024-09-11T00:55:07.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.103+0000 7f74b3200640 1 -- 172.21.15.31:0/420976998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74b414e400 con 0x7f74b406dd70 2024-09-11T00:55:07.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.105+0000 7f74aa000640 1 -- 172.21.15.31:0/420976998 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f74a4045460 con 0x7f74b406dd70 2024-09-11T00:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.106+0000 7f74aa000640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7494077fa0 0x7f749407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.106+0000 7f74aa000640 1 -- 172.21.15.31:0/420976998 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7f74a40bc450 con 0x7f74b406dd70 2024-09-11T00:55:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.107+0000 7f74b1400640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7494077fa0 0x7f749407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.108+0000 7f74b1400640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7494077fa0 0x7f749407a460 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f74a00046b0 tx=0x7f74a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:07.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.111+0000 7f74aa000640 1 -- 172.21.15.31:0/420976998 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f74b414e180 con 0x7f74b406dd70 2024-09-11T00:55:07.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.454+0000 7f74b3200640 1 -- 172.21.15.31:0/420976998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74b406e150 con 0x7f74b406dd70 2024-09-11T00:55:07.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.454+0000 7f74aa000640 1 -- 172.21.15.31:0/420976998 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) ==== 95+0+5710 (secure 0 0 0) 0x7f74a4084080 con 0x7f74b406dd70 2024-09-11T00:55:07.456 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:55:07.457 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":174,"btime":"2024-09-11T00:55:07:028191+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:06.351088+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:55:07.457 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 174 2024-09-11T00:55:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.460+0000 7f74b3200640 1 -- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7494077fa0 msgr2=0x7f749407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.460+0000 7f74b3200640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7494077fa0 0x7f749407a460 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f74a00046b0 tx=0x7f74a0009040 comp rx=0 tx=0).stop 2024-09-11T00:55:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.460+0000 7f74b3200640 1 -- 172.21.15.31:0/420976998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b406dd70 msgr2=0x7f74b414dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:07.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.460+0000 7f74b3200640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b406dd70 0x7f74b414dc40 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f74a4000c00 tx=0x7f74a4032c10 comp rx=0 tx=0).stop 2024-09-11T00:55:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.461+0000 7f74b3200640 1 -- 172.21.15.31:0/420976998 shutdown_connections 2024-09-11T00:55:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.461+0000 7f74b3200640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7494077fa0 0x7f749407a460 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.461+0000 7f74b3200640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f74b4150d60 0x7f74b41dffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.461+0000 7f74b3200640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f74b4074a30 0x7f74b41ef530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.461+0000 7f74b3200640 1 --2- 172.21.15.31:0/420976998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b406dd70 0x7f74b414dc40 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.461+0000 7f74b3200640 1 -- 172.21.15.31:0/420976998 >> 172.21.15.31:0/420976998 conn(0x7f74b40c4d20 msgr2=0x7f74b40c7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:07.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.461+0000 7f74b3200640 1 -- 172.21.15.31:0/420976998 shutdown_connections 2024-09-11T00:55:07.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:07.461+0000 7f74b3200640 1 -- 172.21.15.31:0/420976998 wait complete. 2024-09-11T00:55:07.487 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:55:07:028191+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 174, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 174, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7522": { "addr": "172.21.15.31:6833/3721481399", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3721481399, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 3721481399, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7522, "incarnation": 172, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:55:06.351088+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7522 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 170, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7429": { "addr": "172.21.15.204:6837/2689217598", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 2689217598, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 2689217598, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7429, "incarnation": 167, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:54:23.465895+0000", "qdb_cluster": [ 7429 ], "qdb_leader": 7429, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7429 } } } ], "standbys": [ { "addr": "172.21.15.31:6835/2337200751", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 2337200751, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 2337200751, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 166, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6503, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:55:09.489 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:55:09.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.804+0000 7f5fedc00640 1 -- 172.21.15.31:0/481551620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe80c2ff0 msgr2=0x7f5fe8155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:09.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.804+0000 7f5fedc00640 1 --2- 172.21.15.31:0/481551620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe80c2ff0 0x7f5fe8155c70 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0005fd0 tx=0x7f5fe002ab20 comp rx=0 tx=0).stop 2024-09-11T00:55:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.805+0000 7f5fedc00640 1 -- 172.21.15.31:0/481551620 shutdown_connections 2024-09-11T00:55:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.805+0000 7f5fedc00640 1 --2- 172.21.15.31:0/481551620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe80c2ff0 0x7f5fe8155c70 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.805+0000 7f5fedc00640 1 --2- 172.21.15.31:0/481551620 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5fe8070570 0x7f5fe806e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.805+0000 7f5fedc00640 1 -- 172.21.15.31:0/481551620 >> 172.21.15.31:0/481551620 conn(0x7f5fe80a9750 msgr2=0x7f5fe80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:09.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.809+0000 7f5fedc00640 1 -- 172.21.15.31:0/481551620 shutdown_connections 2024-09-11T00:55:09.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.811+0000 7f5fedc00640 1 -- 172.21.15.31:0/481551620 wait complete. 2024-09-11T00:55:09.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.816+0000 7f5fedc00640 1 Processor -- start 2024-09-11T00:55:09.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.816+0000 7f5fedc00640 1 -- start start 2024-09-11T00:55:09.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.819+0000 7f5fedc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fe806e9c0 0x7f5fe814ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:09.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.819+0000 7f5fedc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe8070570 0x7f5fe8073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:09.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.819+0000 7f5fe6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fe806e9c0 0x7f5fe814ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:09.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.820+0000 7f5fedc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5fe80c2ff0 0x7f5fe8073d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:09.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.820+0000 7f5fe6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fe806e9c0 0x7f5fe814ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:55958/0 (socket says 172.21.15.31:55958) 2024-09-11T00:55:09.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.820+0000 7f5fe6000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe8070570 0x7f5fe8073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:09.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.820+0000 7f5fedc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe814fb20 con 0x7f5fe8070570 2024-09-11T00:55:09.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.820+0000 7f5fe6a00640 1 -- 172.21.15.31:0/870086713 learned_addr learned my addr 172.21.15.31:0/870086713 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:55:09.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.820+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe806ef90 con 0x7f5fe806e9c0 2024-09-11T00:55:09.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.820+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5fe80c7690 con 0x7f5fe80c2ff0 2024-09-11T00:55:09.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.821+0000 7f5fe7400640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5fe80c2ff0 0x7f5fe8073d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:09.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.821+0000 7f5fe6000640 1 -- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5fe80c2ff0 msgr2=0x7f5fe8073d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:09.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.821+0000 7f5fe6000640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5fe80c2ff0 0x7f5fe8073d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:09.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.821+0000 7f5fe6000640 1 -- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fe806e9c0 msgr2=0x7f5fe814ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:09.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.821+0000 7f5fe6000640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fe806e9c0 0x7f5fe814ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:09.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.821+0000 7f5fe6000640 1 -- 172.21.15.31:0/870086713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe81ed460 con 0x7f5fe8070570 2024-09-11T00:55:09.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.822+0000 7f5fe6000640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe8070570 0x7f5fe8073810 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400c760 tx=0x7f5fd400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:09.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.822+0000 7f5fdea00640 1 -- 172.21.15.31:0/870086713 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd40050d0 con 0x7f5fe8070570 2024-09-11T00:55:09.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.823+0000 7f5fdea00640 1 -- 172.21.15.31:0/870086713 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5fd4005270 con 0x7f5fe8070570 2024-09-11T00:55:09.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.823+0000 7f5fdea00640 1 -- 172.21.15.31:0/870086713 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd400a640 con 0x7f5fe8070570 2024-09-11T00:55:09.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.823+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe8151240 con 0x7f5fe8070570 2024-09-11T00:55:09.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.823+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe81ed180 con 0x7f5fe8070570 2024-09-11T00:55:09.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.827+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fe80a8710 con 0x7f5fe8070570 2024-09-11T00:55:09.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.829+0000 7f5fdea00640 1 -- 172.21.15.31:0/870086713 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5fd400a7e0 con 0x7f5fe8070570 2024-09-11T00:55:09.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.831+0000 7f5fdea00640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5fc8077fc0 0x7f5fc807a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:09.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.831+0000 7f5fdea00640 1 -- 172.21.15.31:0/870086713 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7f5fd409a210 con 0x7f5fe8070570 2024-09-11T00:55:09.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.833+0000 7f5fe6a00640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5fc8077fc0 0x7f5fc807a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:09.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.838+0000 7f5fe6a00640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5fc8077fc0 0x7f5fc807a480 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80058c0 tx=0x7f5fd8005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:09.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:09.843+0000 7f5fdea00640 1 -- 172.21.15.31:0/870086713 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5fe80a8710 con 0x7f5fe8070570 2024-09-11T00:55:10.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.353+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fe814f740 con 0x7f5fe8070570 2024-09-11T00:55:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.355+0000 7f5fdea00640 1 -- 172.21.15.31:0/870086713 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+5717 (secure 0 0 0) 0x7f5fd4020060 con 0x7f5fe8070570 2024-09-11T00:55:10.357 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:55:10.357 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":175,"btime":"2024-09-11T00:55:08:036352+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:55:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 175 2024-09-11T00:55:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.361+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5fc8077fc0 msgr2=0x7f5fc807a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.361+0000 7f5fedc00640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5fc8077fc0 0x7f5fc807a480 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80058c0 tx=0x7f5fd8005f90 comp rx=0 tx=0).stop 2024-09-11T00:55:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.361+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe8070570 msgr2=0x7f5fe8073810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.361+0000 7f5fedc00640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe8070570 0x7f5fe8073810 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400c760 tx=0x7f5fd400cc20 comp rx=0 tx=0).stop 2024-09-11T00:55:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.363+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 shutdown_connections 2024-09-11T00:55:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.363+0000 7f5fedc00640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5fc8077fc0 0x7f5fc807a480 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.363+0000 7f5fedc00640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5fe80c2ff0 0x7f5fe8073d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.363+0000 7f5fedc00640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5fe8070570 0x7f5fe8073810 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.363+0000 7f5fedc00640 1 --2- 172.21.15.31:0/870086713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fe806e9c0 0x7f5fe814ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.363+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 >> 172.21.15.31:0/870086713 conn(0x7f5fe80a9750 msgr2=0x7f5fe80c39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.365+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 shutdown_connections 2024-09-11T00:55:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:10.365+0000 7f5fedc00640 1 -- 172.21.15.31:0/870086713 wait complete. 2024-09-11T00:55:10.409 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=7357 2024-09-11T00:55:10.409 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.b 2024-09-11T00:55:10.524 INFO:tasks.ceph.mds.a.smithi031.stderr:2024-09-11T00:55:10.522+0000 7f9666a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:55:14.411 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-09-11T00:55:14.411 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-11T00:55:14.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-11T00:55:14.420 INFO:tasks.ceph.mds.b:Started 2024-09-11T00:55:14.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:55:14.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.732+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3535055482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7dc0c9b00 msgr2=0x7fb7dc1596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:14.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.732+0000 7fb7e2c00640 1 --2- 172.21.15.31:0/3535055482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7dc0c9b00 0x7fb7dc1596b0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009830 tx=0x7fb7cc02a470 comp rx=0 tx=0).stop 2024-09-11T00:55:14.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.733+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3535055482 shutdown_connections 2024-09-11T00:55:14.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.733+0000 7fb7e2c00640 1 --2- 172.21.15.31:0/3535055482 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7dc0ca010 0x7fb7dc0cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:14.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.733+0000 7fb7e2c00640 1 --2- 172.21.15.31:0/3535055482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7dc0c9b00 0x7fb7dc1596b0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:14.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.733+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3535055482 >> 172.21.15.31:0/3535055482 conn(0x7fb7dc068f10 msgr2=0x7fb7dc0cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:14.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.733+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3535055482 shutdown_connections 2024-09-11T00:55:14.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.733+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3535055482 wait complete. 2024-09-11T00:55:14.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.735+0000 7fb7e2c00640 1 Processor -- start 2024-09-11T00:55:14.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.736+0000 7fb7e2c00640 1 -- start start 2024-09-11T00:55:14.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.737+0000 7fb7e2c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7dc0b1260 0x7fb7dc158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:14.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.737+0000 7fb7e2c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7dc0c9b00 0x7fb7dc147380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:14.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.737+0000 7fb7e2c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7dc0ca010 0x7fb7dc1478c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:14.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.737+0000 7fb7e2c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc06f000 con 0x7fb7dc0b1260 2024-09-11T00:55:14.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.737+0000 7fb7e2c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc1564b0 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.737+0000 7fb7e2c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb7dc0bf420 con 0x7fb7dc0ca010 2024-09-11T00:55:14.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.738+0000 7fb7db400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7dc0c9b00 0x7fb7dc147380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:14.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.738+0000 7fb7db400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7dc0c9b00 0x7fb7dc147380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:55964/0 (socket says 172.21.15.31:55964) 2024-09-11T00:55:14.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.738+0000 7fb7db400640 1 -- 172.21.15.31:0/3074138304 learned_addr learned my addr 172.21.15.31:0/3074138304 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:55:14.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.738+0000 7fb7e0c00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7dc0ca010 0x7fb7dc1478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:14.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.738+0000 7fb7db400640 1 -- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7dc0ca010 msgr2=0x7fb7dc1478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:14.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.738+0000 7fb7db400640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7dc0ca010 0x7fb7dc1478c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:14.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.739+0000 7fb7db400640 1 -- 172.21.15.31:0/3074138304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7dc0b1260 msgr2=0x7fb7dc158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:14.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.739+0000 7fb7dbe00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7dc0b1260 0x7fb7dc158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:14.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.739+0000 7fb7db400640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7dc0b1260 0x7fb7dc158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:14.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.739+0000 7fb7db400640 1 -- 172.21.15.31:0/3074138304 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7dc0b0b40 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.739+0000 7fb7dbe00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7dc0b1260 0x7fb7dc158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:55:14.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.739+0000 7fb7db400640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7dc0c9b00 0x7fb7dc147380 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc000c00 tx=0x7fb7cc032b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:14.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.740+0000 7fb7d3e00640 1 -- 172.21.15.31:0/3074138304 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc040070 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.740+0000 7fb7d3e00640 1 -- 172.21.15.31:0/3074138304 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7cc0038e0 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.740+0000 7fb7d3e00640 1 -- 172.21.15.31:0/3074138304 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc003d10 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.740+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3074138304 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7dc148360 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.740+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3074138304 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7dc14ce10 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.744+0000 7fb7d3e00640 1 -- 172.21.15.31:0/3074138304 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb7cc03c420 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.744+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3074138304 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7dc1596b0 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.745+0000 7fb7d3e00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb7bc077f70 0x7fb7bc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:55:14.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.745+0000 7fb7d3e00640 1 -- 172.21.15.31:0/3074138304 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7fb7cc0bc3a0 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.745+0000 7fb7dbe00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb7bc077f70 0x7fb7bc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:55:14.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.746+0000 7fb7dbe00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb7bc077f70 0x7fb7bc07a430 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc1ee270 tx=0x7fb7c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:55:14.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:14.757+0000 7fb7d3e00640 1 -- 172.21.15.31:0/3074138304 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb7cc083fd0 con 0x7fb7dc0c9b00 2024-09-11T00:55:14.932 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:55:15.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.304+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3074138304 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7dc14ebe0 con 0x7fb7dc0c9b00 2024-09-11T00:55:15.319 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:55:15.320 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":176,"btime":"2024-09-11T00:55:15:098934+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:55:15.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.307+0000 7fb7d3e00640 1 -- 172.21.15.31:0/3074138304 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6546 (secure 0 0 0) 0x7fb7cc0838d0 con 0x7fb7dc0c9b00 2024-09-11T00:55:15.320 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 176 2024-09-11T00:55:15.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.319+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb7bc077f70 msgr2=0x7fb7bc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:15.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.319+0000 7fb7e2c00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb7bc077f70 0x7fb7bc07a430 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc1ee270 tx=0x7fb7c8009040 comp rx=0 tx=0).stop 2024-09-11T00:55:15.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.319+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7dc0c9b00 msgr2=0x7fb7dc147380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:55:15.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.319+0000 7fb7e2c00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7dc0c9b00 0x7fb7dc147380 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc000c00 tx=0x7fb7cc032b90 comp rx=0 tx=0).stop 2024-09-11T00:55:15.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.320+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3074138304 shutdown_connections 2024-09-11T00:55:15.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.320+0000 7fb7e2c00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb7bc077f70 0x7fb7bc07a430 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:15.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.320+0000 7fb7e2c00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb7dc0ca010 0x7fb7dc1478c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:15.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.320+0000 7fb7e2c00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb7dc0c9b00 0x7fb7dc147380 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:15.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.320+0000 7fb7e2c00640 1 --2- 172.21.15.31:0/3074138304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb7dc0b1260 0x7fb7dc158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:55:15.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.320+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3074138304 >> 172.21.15.31:0/3074138304 conn(0x7fb7dc068f10 msgr2=0x7fb7dc0aeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:55:15.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.321+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3074138304 shutdown_connections 2024-09-11T00:55:15.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:55:15.321+0000 7fb7e2c00640 1 -- 172.21.15.31:0/3074138304 wait complete. 2024-09-11T00:55:15.359 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-09-11T00:55:15.359 INFO:tasks.mds_thrash.fs.[a]:waiting for 59 secs before thrashing 2024-09-11T00:55:24.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:55:24.811 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:55:33.765 INFO:tasks.workunit.client.0.smithi031.stdout:Syncing()...12 sec 2024-09-11T00:55:33.765 INFO:tasks.workunit.client.0.smithi031.stdout:FFSB benchmark finished at: Wed Sep 11 00:55:33 2024 2024-09-11T00:55:33.765 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout:Results: 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout:Benchmark took 335.00 sec 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout:Total Results 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout:=============== 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout: ======= ============ ========= ======= =========== ========== 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout: write : 1029120 3072.04 100.000% 100.000% 12MB/sec 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout:- 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout:3072.04 Transactions per Second 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout:Throughput Results 2024-09-11T00:55:33.766 INFO:tasks.workunit.client.0.smithi031.stdout:=================== 2024-09-11T00:55:33.767 INFO:tasks.workunit.client.0.smithi031.stdout:Write Throughput: 12MB/sec 2024-09-11T00:55:33.774 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.774 INFO:tasks.workunit.client.0.smithi031.stdout:System Call Latency statistics in millisecs 2024-09-11T00:55:33.775 INFO:tasks.workunit.client.0.smithi031.stdout:===== 2024-09-11T00:55:33.775 INFO:tasks.workunit.client.0.smithi031.stdout: Min Avg Max Total Calls 2024-09-11T00:55:33.775 INFO:tasks.workunit.client.0.smithi031.stdout: ======== ======== ======== ============ 2024-09-11T00:55:33.775 INFO:tasks.workunit.client.0.smithi031.stdout:[ open] 1.192000 2009.832359 21787.039062 804 2024-09-11T00:55:33.775 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[0] 5000.000000 - 10000.000000 : 1 2024-09-11T00:55:33.775 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-11T00:55:33.775 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-11T00:55:33.775 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[3] 500.000000 - 1000.000000 : 2 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[4] 200.000000 - 500.000000 : 5 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[5] 100.000000 - 200.000000 : 7 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[6] 50.000000 - 100.000000 : 66 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[7] 20.000000 - 50.000000 : 386 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[8] 10.000000 - 20.000000 : 167 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[9] 5.000000 - 10.000000 : 52 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[10] 2.000000 - 5.000000 : 18 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[11] 1.000000 - 2.000000 : 5 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-09-11T00:55:33.776 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout:[ write] 0.172000 8.372146 691.293030 1029120 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[3] 500.000000 - 1000.000000 : 2 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[4] 200.000000 - 500.000000 : 203 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[5] 100.000000 - 200.000000 : 175 2024-09-11T00:55:33.777 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[6] 50.000000 - 100.000000 : 7305 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[7] 20.000000 - 50.000000 : 89926 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[8] 10.000000 - 20.000000 : 197065 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[9] 5.000000 - 10.000000 : 251054 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[10] 2.000000 - 5.000000 : 208361 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[11] 1.000000 - 2.000000 : 66156 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[12] 0.500000 - 1.000000 : 93792 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[13] 0.200000 - 0.500000 : 114583 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[14] 0.100000 - 0.200000 : 498 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.778 INFO:tasks.workunit.client.0.smithi031.stdout:[ lseek] 0.000000 0.001889 2.148000 1029120 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[10] 2.000000 - 5.000000 : 5 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[11] 1.000000 - 2.000000 : 23 2024-09-11T00:55:33.779 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[12] 0.500000 - 1.000000 : 34 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[13] 0.200000 - 0.500000 : 128 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[14] 0.100000 - 0.200000 : 225 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[15] 0.050000 - 0.100000 : 329 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[16] 0.020000 - 0.050000 : 1177 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[17] 0.010000 - 0.020000 : 2161 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[18] 0.000000 - 0.010000 : 1025038 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout:[ close] 0.042000 3.698607 84.357002 804 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-11T00:55:33.780 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[6] 50.000000 - 100.000000 : 4 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[7] 20.000000 - 50.000000 : 27 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[8] 10.000000 - 20.000000 : 56 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[9] 5.000000 - 10.000000 : 68 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[10] 2.000000 - 5.000000 : 135 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[11] 1.000000 - 2.000000 : 65 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[12] 0.500000 - 1.000000 : 45 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[13] 0.200000 - 0.500000 : 80 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[14] 0.100000 - 0.200000 : 120 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[15] 0.050000 - 0.100000 : 194 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[16] 0.020000 - 0.050000 : 10 2024-09-11T00:55:33.781 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-11T00:55:33.782 INFO:tasks.workunit.client.0.smithi031.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-11T00:55:33.782 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.782 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.782 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.782 INFO:tasks.workunit.client.0.smithi031.stdout: 2024-09-11T00:55:33.782 INFO:tasks.workunit.client.0.smithi031.stdout:0.7% User Time 2024-09-11T00:55:33.782 INFO:tasks.workunit.client.0.smithi031.stdout:27.3% System Time 2024-09-11T00:55:33.782 INFO:tasks.workunit.client.0.smithi031.stdout:28.0% CPU Utilization 2024-09-11T00:55:33.782 INFO:tasks.workunit.client.0.smithi031.stderr:+ cd .. 2024-09-11T00:55:33.787 INFO:tasks.workunit.client.0.smithi031.stderr:+ rm -r tmp ffsb 2024-09-11T00:55:43.069 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T00:55:43.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T00:55:43.196 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-09-11T00:55:43.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-11T00:55:44.827 DEBUG:teuthology.parallel:result is None 2024-09-11T00:56:00.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:56:00.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:56:01.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.187+0000 7f49fb200640 1 -- 172.21.15.31:0/709427772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49fc0c64b0 msgr2=0x7f49fc0c7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.187+0000 7f49fb200640 1 --2- 172.21.15.31:0/709427772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49fc0c64b0 0x7f49fc0c7240 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f49f4005fd0 tx=0x7f49f402b8f0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.187+0000 7f49fb200640 1 -- 172.21.15.31:0/709427772 shutdown_connections 2024-09-11T00:56:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.187+0000 7f49fb200640 1 --2- 172.21.15.31:0/709427772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49fc0c64b0 0x7f49fc0c7240 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.187+0000 7f49fb200640 1 --2- 172.21.15.31:0/709427772 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49fc070410 0x7f49fc06dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.187+0000 7f49fb200640 1 -- 172.21.15.31:0/709427772 >> 172.21.15.31:0/709427772 conn(0x7f49fc06caf0 msgr2=0x7f49fc06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.187+0000 7f49fb200640 1 -- 172.21.15.31:0/709427772 shutdown_connections 2024-09-11T00:56:01.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.187+0000 7f49fb200640 1 -- 172.21.15.31:0/709427772 wait complete. 2024-09-11T00:56:01.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.189+0000 7f49fb200640 1 Processor -- start 2024-09-11T00:56:01.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.189+0000 7f49fb200640 1 -- start start 2024-09-11T00:56:01.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.189+0000 7f49fb200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49fc070410 0x7f49fc0a97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:01.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.189+0000 7f49fb200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49fc0acaf0 0x7f49fc155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:01.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49fb200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49fc0c64b0 0x7f49fc0747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:01.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49fb200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc0700f0 con 0x7f49fc070410 2024-09-11T00:56:01.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49fb200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f49fc0c5cb0 con 0x7f49fc0c64b0 2024-09-11T00:56:01.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49fb200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f49fc0c4ad0 con 0x7f49fc0acaf0 2024-09-11T00:56:01.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49fc070410 0x7f49fc0a97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:01.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49fc070410 0x7f49fc0a97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37198/0 (socket says 172.21.15.31:37198) 2024-09-11T00:56:01.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9e00640 1 -- 172.21.15.31:0/4180994585 learned_addr learned my addr 172.21.15.31:0/4180994585 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:01.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9400640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49fc0acaf0 0x7f49fc155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:01.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49fa800640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49fc0c64b0 0x7f49fc0747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:01.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9e00640 1 -- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49fc0acaf0 msgr2=0x7f49fc155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:01.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9e00640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49fc0acaf0 0x7f49fc155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9e00640 1 -- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49fc0c64b0 msgr2=0x7f49fc0747b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:01.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9e00640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49fc0c64b0 0x7f49fc0747b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9e00640 1 -- 172.21.15.31:0/4180994585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49fc0a7df0 con 0x7f49fc070410 2024-09-11T00:56:01.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9400640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49fc0acaf0 0x7f49fc155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:56:01.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49fa800640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49fc0c64b0 0x7f49fc0747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:56:01.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.190+0000 7f49f9e00640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49fc070410 0x7f49fc0a97b0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f49ec00d950 tx=0x7f49ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:01.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.191+0000 7f49f2000640 1 -- 172.21.15.31:0/4180994585 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec017070 con 0x7f49fc070410 2024-09-11T00:56:01.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.191+0000 7f49f2000640 1 -- 172.21.15.31:0/4180994585 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49ec009d20 con 0x7f49fc070410 2024-09-11T00:56:01.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.191+0000 7f49f2000640 1 -- 172.21.15.31:0/4180994585 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0106c0 con 0x7f49fc070410 2024-09-11T00:56:01.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.191+0000 7f49fb200640 1 -- 172.21.15.31:0/4180994585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49fc0a8350 con 0x7f49fc070410 2024-09-11T00:56:01.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.191+0000 7f49fb200640 1 -- 172.21.15.31:0/4180994585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49fc0a8070 con 0x7f49fc070410 2024-09-11T00:56:01.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.192+0000 7f49fb200640 1 -- 172.21.15.31:0/4180994585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49fc06dc60 con 0x7f49fc070410 2024-09-11T00:56:01.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.193+0000 7f49f2000640 1 -- 172.21.15.31:0/4180994585 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f49ec010860 con 0x7f49fc070410 2024-09-11T00:56:01.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.195+0000 7f49f2000640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49dc077fa0 0x7f49dc07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:01.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.195+0000 7f49f2000640 1 -- 172.21.15.31:0/4180994585 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7f49ec09b100 con 0x7f49fc070410 2024-09-11T00:56:01.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.195+0000 7f49f9400640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49dc077fa0 0x7f49dc07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:01.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.196+0000 7f49f9400640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49dc077fa0 0x7f49dc07a460 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0aa6c0 tx=0x7f49e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:01.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.199+0000 7f49f2000640 1 -- 172.21.15.31:0/4180994585 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f49fc0a7df0 con 0x7f49fc070410 2024-09-11T00:56:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.433+0000 7f49fb200640 1 -- 172.21.15.31:0/4180994585 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49fc0a9cf0 con 0x7f49fc070410 2024-09-11T00:56:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.434+0000 7f49f2000640 1 -- 172.21.15.31:0/4180994585 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6546 (secure 0 0 0) 0x7f49ec062d30 con 0x7f49fc070410 2024-09-11T00:56:01.436 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:01.436 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":176,"btime":"2024-09-11T00:55:15:098934+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:56:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 176 2024-09-11T00:56:01.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.436+0000 7f49fb200640 1 -- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49dc077fa0 msgr2=0x7f49dc07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:01.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.436+0000 7f49fb200640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49dc077fa0 0x7f49dc07a460 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0aa6c0 tx=0x7f49e8009040 comp rx=0 tx=0).stop 2024-09-11T00:56:01.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.436+0000 7f49fb200640 1 -- 172.21.15.31:0/4180994585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49fc070410 msgr2=0x7f49fc0a97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:01.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.436+0000 7f49fb200640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49fc070410 0x7f49fc0a97b0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f49ec00d950 tx=0x7f49ec00de10 comp rx=0 tx=0).stop 2024-09-11T00:56:01.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.436+0000 7f49fb200640 1 -- 172.21.15.31:0/4180994585 shutdown_connections 2024-09-11T00:56:01.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.436+0000 7f49fb200640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49dc077fa0 0x7f49dc07a460 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.436+0000 7f49fb200640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49fc0c64b0 0x7f49fc0747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.436+0000 7f49fb200640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49fc0acaf0 0x7f49fc155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.436+0000 7f49fb200640 1 --2- 172.21.15.31:0/4180994585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49fc070410 0x7f49fc0a97b0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.436+0000 7f49fb200640 1 -- 172.21.15.31:0/4180994585 >> 172.21.15.31:0/4180994585 conn(0x7f49fc06caf0 msgr2=0x7f49fc14e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:01.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.437+0000 7f49fb200640 1 -- 172.21.15.31:0/4180994585 shutdown_connections 2024-09-11T00:56:01.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.437+0000 7f49fb200640 1 -- 172.21.15.31:0/4180994585 wait complete. 2024-09-11T00:56:01.456 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-11T00:56:01.457 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:56:01.457 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:56:01.477 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:56:01.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.642+0000 7fc053e00640 1 -- 172.21.15.31:0/3511196596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0380039e0 msgr2=0x7fc038003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.642+0000 7fc053e00640 1 --2- 172.21.15.31:0/3511196596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0380039e0 0x7fc038003dc0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc044001740 tx=0x7fc038005830 comp rx=0 tx=0).stop 2024-09-11T00:56:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.642+0000 7fc053400640 1 -- 172.21.15.31:0/3511196596 reap_dead start 2024-09-11T00:56:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.642+0000 7fc053e00640 1 -- 172.21.15.31:0/3511196596 shutdown_connections 2024-09-11T00:56:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.642+0000 7fc053e00640 1 -- 172.21.15.31:0/3511196596 >> 172.21.15.31:0/3511196596 conn(0x7fc05406caf0 msgr2=0x7fc05406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.642+0000 7fc053e00640 1 -- 172.21.15.31:0/3511196596 shutdown_connections 2024-09-11T00:56:01.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.642+0000 7fc053e00640 1 -- 172.21.15.31:0/3511196596 wait complete. 2024-09-11T00:56:01.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.643+0000 7fc053e00640 1 Processor -- start 2024-09-11T00:56:01.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.643+0000 7fc053e00640 1 -- start start 2024-09-11T00:56:01.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.644+0000 7fc053e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0380039e0 0x7fc0541defb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:01.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.644+0000 7fc053e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc05414f310 0x7fc05414f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.644+0000 7fc053e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05414b6d0 0x7fc05414bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.644+0000 7fc053e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc0541510c0 con 0x7fc05414b6d0 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.644+0000 7fc053e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc0540c5d20 con 0x7fc05414f310 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.644+0000 7fc053e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc054151240 con 0x7fc0380039e0 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc053400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05414b6d0 0x7fc05414bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc053400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05414b6d0 0x7fc05414bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37212/0 (socket says 172.21.15.31:37212) 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc053400640 1 -- 172.21.15.31:0/404290926 learned_addr learned my addr 172.21.15.31:0/404290926 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc052a00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0380039e0 0x7fc0541defb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc04be00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc05414f310 0x7fc05414f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc053400640 1 -- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0380039e0 msgr2=0x7fc0541defb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:01.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc053400640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0380039e0 0x7fc0541defb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc053400640 1 -- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc05414f310 msgr2=0x7fc05414f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:01.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc053400640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc05414f310 0x7fc05414f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc053400640 1 -- 172.21.15.31:0/404290926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0541e01f0 con 0x7fc05414b6d0 2024-09-11T00:56:01.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc04be00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc05414f310 0x7fc05414f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.645+0000 7fc053400640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05414b6d0 0x7fc05414bb50 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc054072020 tx=0x7fc04c0064f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.646+0000 7fc04aa00640 1 -- 172.21.15.31:0/404290926 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c00f760 con 0x7fc05414b6d0 2024-09-11T00:56:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.646+0000 7fc04aa00640 1 -- 172.21.15.31:0/404290926 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc04c016020 con 0x7fc05414b6d0 2024-09-11T00:56:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.646+0000 7fc04aa00640 1 -- 172.21.15.31:0/404290926 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c00fba0 con 0x7fc05414b6d0 2024-09-11T00:56:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.646+0000 7fc053e00640 1 -- 172.21.15.31:0/404290926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0541eb2c0 con 0x7fc05414b6d0 2024-09-11T00:56:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.646+0000 7fc053e00640 1 -- 172.21.15.31:0/404290926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0541eb840 con 0x7fc05414b6d0 2024-09-11T00:56:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.647+0000 7fc053e00640 1 -- 172.21.15.31:0/404290926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0540ac6a0 con 0x7fc05414b6d0 2024-09-11T00:56:01.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.648+0000 7fc04aa00640 1 -- 172.21.15.31:0/404290926 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc04c011430 con 0x7fc05414b6d0 2024-09-11T00:56:01.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.649+0000 7fc04aa00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc030077fa0 0x7fc03007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:01.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.650+0000 7fc04aa00640 1 -- 172.21.15.31:0/404290926 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7fc04c09b770 con 0x7fc05414b6d0 2024-09-11T00:56:01.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.650+0000 7fc052a00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc030077fa0 0x7fc03007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:01.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.651+0000 7fc052a00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc030077fa0 0x7fc03007a460 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc044001710 tx=0x7fc044001270 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:01.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.654+0000 7fc04aa00640 1 -- 172.21.15.31:0/404290926 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc04c020020 con 0x7fc05414b6d0 2024-09-11T00:56:01.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.889+0000 7fc053e00640 1 -- 172.21.15.31:0/404290926 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0541ec040 con 0x7fc05414b6d0 2024-09-11T00:56:01.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.890+0000 7fc04aa00640 1 -- 172.21.15.31:0/404290926 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6546 (secure 0 0 0) 0x7fc04c0633a0 con 0x7fc05414b6d0 2024-09-11T00:56:01.892 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:01.892 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":176,"btime":"2024-09-11T00:55:15:098934+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:56:01.892 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 176 2024-09-11T00:56:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 -- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc030077fa0 msgr2=0x7fc03007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc030077fa0 0x7fc03007a460 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc044001710 tx=0x7fc044001270 comp rx=0 tx=0).stop 2024-09-11T00:56:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 -- 172.21.15.31:0/404290926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05414b6d0 msgr2=0x7fc05414bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05414b6d0 0x7fc05414bb50 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc054072020 tx=0x7fc04c0064f0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 -- 172.21.15.31:0/404290926 shutdown_connections 2024-09-11T00:56:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc030077fa0 0x7fc03007a460 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc05414b6d0 0x7fc05414bb50 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc05414f310 0x7fc05414f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 --2- 172.21.15.31:0/404290926 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc0380039e0 0x7fc0541defb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 -- 172.21.15.31:0/404290926 >> 172.21.15.31:0/404290926 conn(0x7fc05406caf0 msgr2=0x7fc0540c0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 -- 172.21.15.31:0/404290926 shutdown_connections 2024-09-11T00:56:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:01.892+0000 7fc053e00640 1 -- 172.21.15.31:0/404290926 wait complete. 2024-09-11T00:56:01.910 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:56:03.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:04.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.081+0000 7f0141a00640 1 -- 172.21.15.31:0/684378022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f013c0c5a20 msgr2=0x7f013c0c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:04.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.081+0000 7f0141a00640 1 --2- 172.21.15.31:0/684378022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f013c0c5a20 0x7f013c0c8c80 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f0124005fd0 tx=0x7f012402a670 comp rx=0 tx=0).stop 2024-09-11T00:56:04.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.081+0000 7f0141a00640 1 -- 172.21.15.31:0/684378022 shutdown_connections 2024-09-11T00:56:04.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.081+0000 7f0141a00640 1 --2- 172.21.15.31:0/684378022 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f013c0c6b40 0x7f013c157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:04.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.081+0000 7f0141a00640 1 --2- 172.21.15.31:0/684378022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f013c0c5a20 0x7f013c0c8c80 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:04.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.081+0000 7f0141a00640 1 -- 172.21.15.31:0/684378022 >> 172.21.15.31:0/684378022 conn(0x7f013c0694f0 msgr2=0x7f013c0c9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:04.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.082+0000 7f0141a00640 1 -- 172.21.15.31:0/684378022 shutdown_connections 2024-09-11T00:56:04.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.082+0000 7f0141a00640 1 -- 172.21.15.31:0/684378022 wait complete. 2024-09-11T00:56:04.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.083+0000 7f0141a00640 1 Processor -- start 2024-09-11T00:56:04.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.083+0000 7f0141a00640 1 -- start start 2024-09-11T00:56:04.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.084+0000 7f0141a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f013c0c5a20 0x7f013c1ddbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.084+0000 7f0141a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f013c0c6760 0x7f013c06dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.084+0000 7f0141a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f013c0c6b40 0x7f013c06e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.084+0000 7f0141a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f013c0c92e0 con 0x7f013c0c6760 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.084+0000 7f0141a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f013c0c8760 con 0x7f013c0c6b40 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.084+0000 7f0141a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f013c0c9460 con 0x7f013c0c5a20 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.084+0000 7f0133400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f013c0c6760 0x7f013c06dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.084+0000 7f0133400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f013c0c6760 0x7f013c06dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37236/0 (socket says 172.21.15.31:37236) 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.084+0000 7f0133400640 1 -- 172.21.15.31:0/607475033 learned_addr learned my addr 172.21.15.31:0/607475033 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f013be00640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f013c0c5a20 0x7f013c1ddbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0133400640 1 -- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f013c0c5a20 msgr2=0x7f013c1ddbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0141000640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f013c0c6b40 0x7f013c06e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:04.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0133400640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f013c0c5a20 0x7f013c1ddbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:04.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0133400640 1 -- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f013c0c6b40 msgr2=0x7f013c06e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:04.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0133400640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f013c0c6b40 0x7f013c06e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:04.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0133400640 1 -- 172.21.15.31:0/607475033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013c1e3130 con 0x7f013c0c6760 2024-09-11T00:56:04.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0141000640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f013c0c6b40 0x7f013c06e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:56:04.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0133400640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f013c0c6760 0x7f013c06dc00 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f0124034040 tx=0x7f01240038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0138c00640 1 -- 172.21.15.31:0/607475033 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012403e030 con 0x7f013c0c6760 2024-09-11T00:56:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0141a00640 1 -- 172.21.15.31:0/607475033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013c153a00 con 0x7f013c0c6760 2024-09-11T00:56:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0138c00640 1 -- 172.21.15.31:0/607475033 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0124003d90 con 0x7f013c0c6760 2024-09-11T00:56:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.085+0000 7f0138c00640 1 -- 172.21.15.31:0/607475033 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124035040 con 0x7f013c0c6760 2024-09-11T00:56:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.086+0000 7f0141a00640 1 -- 172.21.15.31:0/607475033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013c1e2c20 con 0x7f013c0c6760 2024-09-11T00:56:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.087+0000 7f0141a00640 1 -- 172.21.15.31:0/607475033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f013c0c8c80 con 0x7f013c0c6760 2024-09-11T00:56:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.088+0000 7f0138c00640 1 -- 172.21.15.31:0/607475033 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f01240049e0 con 0x7f013c0c6760 2024-09-11T00:56:04.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.089+0000 7f0138c00640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0118077f70 0x7f011807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:04.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.089+0000 7f0138c00640 1 -- 172.21.15.31:0/607475033 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7f01240bb660 con 0x7f013c0c6760 2024-09-11T00:56:04.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.089+0000 7f013be00640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0118077f70 0x7f011807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:04.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.090+0000 7f013be00640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0118077f70 0x7f011807a430 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f012c001680 tx=0x7f012c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:04.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.093+0000 7f0138c00640 1 -- 172.21.15.31:0/607475033 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0124083210 con 0x7f013c0c6760 2024-09-11T00:56:04.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.324+0000 7f0141a00640 1 -- 172.21.15.31:0/607475033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f013c06fa40 con 0x7f013c0c6760 2024-09-11T00:56:04.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.324+0000 7f0138c00640 1 -- 172.21.15.31:0/607475033 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6546 (secure 0 0 0) 0x7f0124082b10 con 0x7f013c0c6760 2024-09-11T00:56:04.326 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:04.326 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":176,"btime":"2024-09-11T00:55:15:098934+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:56:04.326 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 176 2024-09-11T00:56:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 -- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0118077f70 msgr2=0x7f011807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0118077f70 0x7f011807a430 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f012c001680 tx=0x7f012c009040 comp rx=0 tx=0).stop 2024-09-11T00:56:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 -- 172.21.15.31:0/607475033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f013c0c6760 msgr2=0x7f013c06dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f013c0c6760 0x7f013c06dc00 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f0124034040 tx=0x7f01240038a0 comp rx=0 tx=0).stop 2024-09-11T00:56:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 -- 172.21.15.31:0/607475033 shutdown_connections 2024-09-11T00:56:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0118077f70 0x7f011807a430 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f013c0c6b40 0x7f013c06e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f013c0c6760 0x7f013c06dc00 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 --2- 172.21.15.31:0/607475033 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f013c0c5a20 0x7f013c1ddbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:04.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 -- 172.21.15.31:0/607475033 >> 172.21.15.31:0/607475033 conn(0x7f013c0694f0 msgr2=0x7f013c0c3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:04.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 -- 172.21.15.31:0/607475033 shutdown_connections 2024-09-11T00:56:04.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:04.327+0000 7f0141a00640 1 -- 172.21.15.31:0/607475033 wait complete. 2024-09-11T00:56:04.345 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:56:06.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.517+0000 7f8e9d600640 1 -- 172.21.15.31:0/3174898937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980ada20 msgr2=0x7f8e9806e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.517+0000 7f8e9d600640 1 --2- 172.21.15.31:0/3174898937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980ada20 0x7f8e9806e140 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f8e98153d20 tx=0x7f8e8002a5e0 comp rx=0 tx=0).stop 2024-09-11T00:56:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.518+0000 7f8e9d600640 1 -- 172.21.15.31:0/3174898937 shutdown_connections 2024-09-11T00:56:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.518+0000 7f8e9d600640 1 --2- 172.21.15.31:0/3174898937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980ada20 0x7f8e9806e140 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.518+0000 7f8e9d600640 1 --2- 172.21.15.31:0/3174898937 >> v2:172.21.15.204:3300/0 conn(0x7f8e98150020 0x7f8e980ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.518+0000 7f8e9d600640 1 -- 172.21.15.31:0/3174898937 >> 172.21.15.31:0/3174898937 conn(0x7f8e98068f10 msgr2=0x7f8e980cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:06.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.518+0000 7f8e9d600640 1 -- 172.21.15.31:0/3174898937 shutdown_connections 2024-09-11T00:56:06.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.518+0000 7f8e9d600640 1 -- 172.21.15.31:0/3174898937 wait complete. 2024-09-11T00:56:06.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.519+0000 7f8e9d600640 1 Processor -- start 2024-09-11T00:56:06.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.519+0000 7f8e9d600640 1 -- start start 2024-09-11T00:56:06.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.520+0000 7f8e9d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980ada20 0x7f8e9814e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:06.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.520+0000 7f8e9d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8e98150020 0x7f8e980c0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:06.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.520+0000 7f8e9d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e98156960 0x7f8e980c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.520+0000 7f8e9d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8e98157550 con 0x7f8e980ada20 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.520+0000 7f8e9d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8e981573d0 con 0x7f8e98156960 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.520+0000 7f8e9d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8e980cc220 con 0x7f8e98150020 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.520+0000 7f8e97e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980ada20 0x7f8e9814e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e97e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980ada20 0x7f8e9814e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37252/0 (socket says 172.21.15.31:37252) 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e97e00640 1 -- 172.21.15.31:0/4164649928 learned_addr learned my addr 172.21.15.31:0/4164649928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e8f400640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8e98150020 0x7f8e980c0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e9cc00640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e98156960 0x7f8e980c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e97e00640 1 -- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8e98150020 msgr2=0x7f8e980c0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e97e00640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8e98150020 0x7f8e980c0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:06.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e97e00640 1 -- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e98156960 msgr2=0x7f8e980c4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:06.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e97e00640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e98156960 0x7f8e980c4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:06.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e97e00640 1 -- 172.21.15.31:0/4164649928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e980c01f0 con 0x7f8e980ada20 2024-09-11T00:56:06.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e97e00640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980ada20 0x7f8e9814e450 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800eab0 tx=0x7f8e8800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:06.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.521+0000 7f8e94c00640 1 -- 172.21.15.31:0/4164649928 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8800ce40 con 0x7f8e980ada20 2024-09-11T00:56:06.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.522+0000 7f8e94c00640 1 -- 172.21.15.31:0/4164649928 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e88011560 con 0x7f8e980ada20 2024-09-11T00:56:06.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.522+0000 7f8e94c00640 1 -- 172.21.15.31:0/4164649928 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e88011930 con 0x7f8e980ada20 2024-09-11T00:56:06.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.522+0000 7f8e9d600640 1 -- 172.21.15.31:0/4164649928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e980bff10 con 0x7f8e980ada20 2024-09-11T00:56:06.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.522+0000 7f8e9d600640 1 -- 172.21.15.31:0/4164649928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e98152dc0 con 0x7f8e980ada20 2024-09-11T00:56:06.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.523+0000 7f8e9d600640 1 -- 172.21.15.31:0/4164649928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e980ad4e0 con 0x7f8e980ada20 2024-09-11T00:56:06.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.524+0000 7f8e94c00640 1 -- 172.21.15.31:0/4164649928 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f8e88022030 con 0x7f8e980ada20 2024-09-11T00:56:06.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.525+0000 7f8e94c00640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e74077f70 0x7f8e7407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:06.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.525+0000 7f8e94c00640 1 -- 172.21.15.31:0/4164649928 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7f8e8809b4a0 con 0x7f8e980ada20 2024-09-11T00:56:06.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.525+0000 7f8e8f400640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e74077f70 0x7f8e7407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:06.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.526+0000 7f8e8f400640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e74077f70 0x7f8e7407a430 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8e80033000 tx=0x7f8e80004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:06.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.529+0000 7f8e94c00640 1 -- 172.21.15.31:0/4164649928 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8e88016030 con 0x7f8e980ada20 2024-09-11T00:56:06.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.759+0000 7f8e9d600640 1 -- 172.21.15.31:0/4164649928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e980cc8e0 con 0x7f8e980ada20 2024-09-11T00:56:06.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.760+0000 7f8e94c00640 1 -- 172.21.15.31:0/4164649928 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6546 (secure 0 0 0) 0x7f8e88062a80 con 0x7f8e980ada20 2024-09-11T00:56:06.762 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:06.762 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":176,"btime":"2024-09-11T00:55:15:098934+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:56:06.762 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 176 2024-09-11T00:56:06.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 -- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e74077f70 msgr2=0x7f8e7407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:06.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e74077f70 0x7f8e7407a430 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8e80033000 tx=0x7f8e80004730 comp rx=0 tx=0).stop 2024-09-11T00:56:06.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 -- 172.21.15.31:0/4164649928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980ada20 msgr2=0x7f8e9814e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:06.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980ada20 0x7f8e9814e450 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800eab0 tx=0x7f8e8800ef70 comp rx=0 tx=0).stop 2024-09-11T00:56:06.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 -- 172.21.15.31:0/4164649928 shutdown_connections 2024-09-11T00:56:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8e74077f70 0x7f8e7407a430 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e98156960 0x7f8e980c4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f8e98150020 0x7f8e980c0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 --2- 172.21.15.31:0/4164649928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e980ada20 0x7f8e9814e450 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 -- 172.21.15.31:0/4164649928 >> 172.21.15.31:0/4164649928 conn(0x7f8e98068f10 msgr2=0x7f8e980ac370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 -- 172.21.15.31:0/4164649928 shutdown_connections 2024-09-11T00:56:06.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:06.762+0000 7f8e9d600640 1 -- 172.21.15.31:0/4164649928 wait complete. 2024-09-11T00:56:06.780 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:56:08.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:08.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.953+0000 7fc1c3200640 1 -- 172.21.15.31:0/850998993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1a8004020 msgr2=0x7fc1a8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.953+0000 7fc1c3200640 1 --2- 172.21.15.31:0/850998993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1a8004020 0x7fc1a8004800 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009b50 tx=0x7fc1a8005830 comp rx=0 tx=0).stop 2024-09-11T00:56:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.953+0000 7fc1c2800640 1 -- 172.21.15.31:0/850998993 reap_dead start 2024-09-11T00:56:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.954+0000 7fc1c3200640 1 -- 172.21.15.31:0/850998993 shutdown_connections 2024-09-11T00:56:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.954+0000 7fc1c3200640 1 -- 172.21.15.31:0/850998993 >> 172.21.15.31:0/850998993 conn(0x7fc1c406caf0 msgr2=0x7fc1c406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.954+0000 7fc1c3200640 1 -- 172.21.15.31:0/850998993 shutdown_connections 2024-09-11T00:56:08.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.954+0000 7fc1c3200640 1 -- 172.21.15.31:0/850998993 wait complete. 2024-09-11T00:56:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.955+0000 7fc1c3200640 1 Processor -- start 2024-09-11T00:56:08.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.955+0000 7fc1c3200640 1 -- start start 2024-09-11T00:56:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.956+0000 7fc1c3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1a8004020 0x7fc1c41528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.956+0000 7fc1c3200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1c41eab90 0x7fc1c41eb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.956+0000 7fc1c3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1c41eb550 0x7fc1c41ecc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:08.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.956+0000 7fc1c3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c40c6a40 con 0x7fc1c41eb550 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.956+0000 7fc1c3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c40c68c0 con 0x7fc1a8004020 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.956+0000 7fc1c3200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c40c6bc0 con 0x7fc1c41eab90 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1c2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1c41eb550 0x7fc1c41ecc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1b9400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1c41eab90 0x7fc1c41eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1c2800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1c41eb550 0x7fc1c41ecc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35332/0 (socket says 172.21.15.31:35332) 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1c2800640 1 -- 172.21.15.31:0/4255787075 learned_addr learned my addr 172.21.15.31:0/4255787075 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1c1e00640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1a8004020 0x7fc1c41528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1c2800640 1 -- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1c41eab90 msgr2=0x7fc1c41eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1c2800640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1c41eab90 0x7fc1c41eb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1c2800640 1 -- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1a8004020 msgr2=0x7fc1c41528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:08.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1c2800640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1a8004020 0x7fc1c41528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:08.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1c2800640 1 -- 172.21.15.31:0/4255787075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c40c42c0 con 0x7fc1c41eb550 2024-09-11T00:56:08.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.957+0000 7fc1c2800640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1c41eb550 0x7fc1c41ecc20 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40acfc0 tx=0x7fc1bc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:08.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.958+0000 7fc1baa00640 1 -- 172.21.15.31:0/4255787075 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc015070 con 0x7fc1c41eb550 2024-09-11T00:56:08.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.958+0000 7fc1baa00640 1 -- 172.21.15.31:0/4255787075 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1bc00cac0 con 0x7fc1c41eb550 2024-09-11T00:56:08.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.958+0000 7fc1baa00640 1 -- 172.21.15.31:0/4255787075 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc011460 con 0x7fc1c41eb550 2024-09-11T00:56:08.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.958+0000 7fc1c3200640 1 -- 172.21.15.31:0/4255787075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c40c39b0 con 0x7fc1c41eb550 2024-09-11T00:56:08.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.958+0000 7fc1c3200640 1 -- 172.21.15.31:0/4255787075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c40c2ce0 con 0x7fc1c41eb550 2024-09-11T00:56:08.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.959+0000 7fc1c3200640 1 -- 172.21.15.31:0/4255787075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c40ac6a0 con 0x7fc1c41eb550 2024-09-11T00:56:08.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.960+0000 7fc1baa00640 1 -- 172.21.15.31:0/4255787075 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc1bc011600 con 0x7fc1c41eb550 2024-09-11T00:56:08.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.961+0000 7fc1baa00640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc1a0077f70 0x7fc1a007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:08.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.961+0000 7fc1baa00640 1 -- 172.21.15.31:0/4255787075 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7fc1bc09b200 con 0x7fc1c41eb550 2024-09-11T00:56:08.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.961+0000 7fc1c1e00640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc1a0077f70 0x7fc1a007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:08.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.962+0000 7fc1c1e00640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc1a0077f70 0x7fc1a007a430 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009cd0 tx=0x7fc1b4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:08.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:08.965+0000 7fc1baa00640 1 -- 172.21.15.31:0/4255787075 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc1bc0627e0 con 0x7fc1c41eb550 2024-09-11T00:56:09.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.193+0000 7fc1c3200640 1 -- 172.21.15.31:0/4255787075 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1c41512b0 con 0x7fc1c41eb550 2024-09-11T00:56:09.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.194+0000 7fc1baa00640 1 -- 172.21.15.31:0/4255787075 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6546 (secure 0 0 0) 0x7fc1bc067210 con 0x7fc1c41eb550 2024-09-11T00:56:09.196 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:09.196 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":176,"btime":"2024-09-11T00:55:15:098934+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:56:09.197 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 176 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 -- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc1a0077f70 msgr2=0x7fc1a007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc1a0077f70 0x7fc1a007a430 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009cd0 tx=0x7fc1b4005f90 comp rx=0 tx=0).stop 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 -- 172.21.15.31:0/4255787075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1c41eb550 msgr2=0x7fc1c41ecc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1c41eb550 0x7fc1c41ecc20 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40acfc0 tx=0x7fc1bc00ec50 comp rx=0 tx=0).stop 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 -- 172.21.15.31:0/4255787075 shutdown_connections 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc1a0077f70 0x7fc1a007a430 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1c41eb550 0x7fc1c41ecc20 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc1c41eab90 0x7fc1c41eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 --2- 172.21.15.31:0/4255787075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1a8004020 0x7fc1c41528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 -- 172.21.15.31:0/4255787075 >> 172.21.15.31:0/4255787075 conn(0x7fc1c406caf0 msgr2=0x7fc1c40c7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:09.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 -- 172.21.15.31:0/4255787075 shutdown_connections 2024-09-11T00:56:09.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:09.197+0000 7fc1c3200640 1 -- 172.21.15.31:0/4255787075 wait complete. 2024-09-11T00:56:09.214 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:56:10.523 INFO:tasks.workunit.client.1.smithi204.stdout:Syncing()...13 sec 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout:FFSB benchmark finished at: Wed Sep 11 00:56:10 2024 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout:Results: 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout:Benchmark took 317.33 sec 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout:Total Results 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout:=============== 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout: ======= ============ ========= ======= =========== ========== 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout: write : 1112320 3505.27 100.000% 100.000% 13.7MB/sec 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout:- 2024-09-11T00:56:10.524 INFO:tasks.workunit.client.1.smithi204.stdout:3505.27 Transactions per Second 2024-09-11T00:56:10.525 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.525 INFO:tasks.workunit.client.1.smithi204.stdout:Throughput Results 2024-09-11T00:56:10.525 INFO:tasks.workunit.client.1.smithi204.stdout:=================== 2024-09-11T00:56:10.525 INFO:tasks.workunit.client.1.smithi204.stdout:Write Throughput: 13.7MB/sec 2024-09-11T00:56:10.535 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.535 INFO:tasks.workunit.client.1.smithi204.stdout:System Call Latency statistics in millisecs 2024-09-11T00:56:10.535 INFO:tasks.workunit.client.1.smithi204.stdout:===== 2024-09-11T00:56:10.535 INFO:tasks.workunit.client.1.smithi204.stdout: Min Avg Max Total Calls 2024-09-11T00:56:10.535 INFO:tasks.workunit.client.1.smithi204.stdout: ======== ======== ======== ============ 2024-09-11T00:56:10.535 INFO:tasks.workunit.client.1.smithi204.stdout:[ open] 0.939000 1503.000814 19293.789062 869 2024-09-11T00:56:10.535 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[0] 5000.000000 - 10000.000000 : 16 2024-09-11T00:56:10.535 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[3] 500.000000 - 1000.000000 : 1 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[4] 200.000000 - 500.000000 : 11 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[5] 100.000000 - 200.000000 : 16 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[6] 50.000000 - 100.000000 : 64 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[7] 20.000000 - 50.000000 : 449 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[8] 10.000000 - 20.000000 : 171 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[9] 5.000000 - 10.000000 : 46 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[10] 2.000000 - 5.000000 : 11 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[11] 1.000000 - 2.000000 : 5 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[12] 0.500000 - 1.000000 : 1 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-09-11T00:56:10.536 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout:[ write] 0.165000 7.506033 19568.332031 1112320 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[3] 500.000000 - 1000.000000 : 5 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[4] 200.000000 - 500.000000 : 110 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[5] 100.000000 - 200.000000 : 193 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[6] 50.000000 - 100.000000 : 3560 2024-09-11T00:56:10.537 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[7] 20.000000 - 50.000000 : 75387 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[8] 10.000000 - 20.000000 : 201235 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[9] 5.000000 - 10.000000 : 282633 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[10] 2.000000 - 5.000000 : 253928 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[11] 1.000000 - 2.000000 : 70993 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[12] 0.500000 - 1.000000 : 88686 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[13] 0.200000 - 0.500000 : 134068 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[14] 0.100000 - 0.200000 : 1520 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.538 INFO:tasks.workunit.client.1.smithi204.stdout:[ lseek] 0.000000 0.001791 2.711000 1112320 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[10] 2.000000 - 5.000000 : 3 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[11] 1.000000 - 2.000000 : 24 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[12] 0.500000 - 1.000000 : 36 2024-09-11T00:56:10.539 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[13] 0.200000 - 0.500000 : 138 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[14] 0.100000 - 0.200000 : 227 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[15] 0.050000 - 0.100000 : 325 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[16] 0.020000 - 0.050000 : 1233 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[17] 0.010000 - 0.020000 : 2359 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[18] 0.000000 - 0.010000 : 1107975 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout:[ close] 0.036000 2.844313 36.884998 869 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-11T00:56:10.540 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[7] 20.000000 - 50.000000 : 22 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[8] 10.000000 - 20.000000 : 42 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[9] 5.000000 - 10.000000 : 88 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[10] 2.000000 - 5.000000 : 154 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[11] 1.000000 - 2.000000 : 68 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[12] 0.500000 - 1.000000 : 40 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[13] 0.200000 - 0.500000 : 91 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[14] 0.100000 - 0.200000 : 124 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[15] 0.050000 - 0.100000 : 220 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[16] 0.020000 - 0.050000 : 20 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.541 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.542 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.542 INFO:tasks.workunit.client.1.smithi204.stdout: 2024-09-11T00:56:10.542 INFO:tasks.workunit.client.1.smithi204.stdout:0.7% User Time 2024-09-11T00:56:10.542 INFO:tasks.workunit.client.1.smithi204.stdout:28.0% System Time 2024-09-11T00:56:10.542 INFO:tasks.workunit.client.1.smithi204.stdout:28.7% CPU Utilization 2024-09-11T00:56:10.543 INFO:tasks.workunit.client.1.smithi204.stderr:+ cd .. 2024-09-11T00:56:11.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:11.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.385+0000 7f40d6400640 1 -- 172.21.15.31:0/3246920650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d0150020 msgr2=0x7f40d00ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.385+0000 7f40d6400640 1 --2- 172.21.15.31:0/3246920650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d0150020 0x7f40d00ad4e0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f40d0153d20 tx=0x7f40c402a540 comp rx=0 tx=0).stop 2024-09-11T00:56:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.386+0000 7f40d6400640 1 -- 172.21.15.31:0/3246920650 shutdown_connections 2024-09-11T00:56:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.386+0000 7f40d6400640 1 --2- 172.21.15.31:0/3246920650 >> v2:172.21.15.204:3300/0 conn(0x7f40d00ada20 0x7f40d006e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.386+0000 7f40d6400640 1 --2- 172.21.15.31:0/3246920650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d0150020 0x7f40d00ad4e0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.386+0000 7f40d6400640 1 -- 172.21.15.31:0/3246920650 >> 172.21.15.31:0/3246920650 conn(0x7f40d0068f10 msgr2=0x7f40d00cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.386+0000 7f40d6400640 1 -- 172.21.15.31:0/3246920650 shutdown_connections 2024-09-11T00:56:11.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.386+0000 7f40d6400640 1 -- 172.21.15.31:0/3246920650 wait complete. 2024-09-11T00:56:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.387+0000 7f40d6400640 1 Processor -- start 2024-09-11T00:56:11.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.387+0000 7f40d6400640 1 -- start start 2024-09-11T00:56:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.388+0000 7f40d6400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d00ada20 0x7f40d014be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.388+0000 7f40d6400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40d0150020 0x7f40d014ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.388+0000 7f40d6400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40d0156960 0x7f40d01ec130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.388+0000 7f40d6400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f40d0157550 con 0x7f40d00ada20 2024-09-11T00:56:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.388+0000 7f40d6400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f40d01573d0 con 0x7f40d0156960 2024-09-11T00:56:11.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.388+0000 7f40d6400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f40d00cc220 con 0x7f40d0150020 2024-09-11T00:56:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.388+0000 7f40d5000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d00ada20 0x7f40d014be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40d5000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d00ada20 0x7f40d014be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35358/0 (socket says 172.21.15.31:35358) 2024-09-11T00:56:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40d5000640 1 -- 172.21.15.31:0/1231148451 learned_addr learned my addr 172.21.15.31:0/1231148451 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40cfe00640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40d0150020 0x7f40d014ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:11.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40d5a00640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40d0156960 0x7f40d01ec130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:11.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40d5000640 1 -- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40d0150020 msgr2=0x7f40d014ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:11.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40d5000640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40d0150020 0x7f40d014ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:11.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40d5000640 1 -- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40d0156960 msgr2=0x7f40d01ec130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:11.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40d5000640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40d0156960 0x7f40d01ec130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:11.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40d5000640 1 -- 172.21.15.31:0/1231148451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d014c390 con 0x7f40d00ada20 2024-09-11T00:56:11.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40d5a00640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40d0156960 0x7f40d01ec130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:11.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.389+0000 7f40d5000640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d00ada20 0x7f40d014be50 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f40c4000c00 tx=0x7f40c4003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:11.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.390+0000 7f40ccc00640 1 -- 172.21.15.31:0/1231148451 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c403f070 con 0x7f40d00ada20 2024-09-11T00:56:11.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.390+0000 7f40ccc00640 1 -- 172.21.15.31:0/1231148451 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40c40047a0 con 0x7f40d00ada20 2024-09-11T00:56:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.390+0000 7f40d6400640 1 -- 172.21.15.31:0/1231148451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d0152e60 con 0x7f40d00ada20 2024-09-11T00:56:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.390+0000 7f40ccc00640 1 -- 172.21.15.31:0/1231148451 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4035030 con 0x7f40d00ada20 2024-09-11T00:56:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.390+0000 7f40d6400640 1 -- 172.21.15.31:0/1231148451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d0151580 con 0x7f40d00ada20 2024-09-11T00:56:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.391+0000 7f40d6400640 1 -- 172.21.15.31:0/1231148451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d00cc8e0 con 0x7f40d00ada20 2024-09-11T00:56:11.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.392+0000 7f40ccc00640 1 -- 172.21.15.31:0/1231148451 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f40c4004bb0 con 0x7f40d00ada20 2024-09-11T00:56:11.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.393+0000 7f40ccc00640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40b0077f70 0x7f40b007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:11.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.393+0000 7f40ccc00640 1 -- 172.21.15.31:0/1231148451 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7f40c40bbdd0 con 0x7f40d00ada20 2024-09-11T00:56:11.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.393+0000 7f40cfe00640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40b0077f70 0x7f40b007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:11.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.394+0000 7f40cfe00640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40b0077f70 0x7f40b007a430 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f40d0154560 tx=0x7f40bc00b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:11.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.397+0000 7f40ccc00640 1 -- 172.21.15.31:0/1231148451 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f40c40833b0 con 0x7f40d00ada20 2024-09-11T00:56:11.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.633+0000 7f40d6400640 1 -- 172.21.15.31:0/1231148451 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40d01ebde0 con 0x7f40d00ada20 2024-09-11T00:56:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.633+0000 7f40ccc00640 1 -- 172.21.15.31:0/1231148451 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6546 (secure 0 0 0) 0x7f40c408c120 con 0x7f40d00ada20 2024-09-11T00:56:11.635 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:11.635 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":176,"btime":"2024-09-11T00:55:15:098934+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:56:11.636 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 176 2024-09-11T00:56:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 -- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40b0077f70 msgr2=0x7f40b007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40b0077f70 0x7f40b007a430 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f40d0154560 tx=0x7f40bc00b2e0 comp rx=0 tx=0).stop 2024-09-11T00:56:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 -- 172.21.15.31:0/1231148451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d00ada20 msgr2=0x7f40d014be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d00ada20 0x7f40d014be50 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f40c4000c00 tx=0x7f40c4003910 comp rx=0 tx=0).stop 2024-09-11T00:56:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 -- 172.21.15.31:0/1231148451 shutdown_connections 2024-09-11T00:56:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f40b0077f70 0x7f40b007a430 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40d0156960 0x7f40d01ec130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f40d0150020 0x7f40d014ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 --2- 172.21.15.31:0/1231148451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40d00ada20 0x7f40d014be50 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 -- 172.21.15.31:0/1231148451 >> 172.21.15.31:0/1231148451 conn(0x7f40d0068f10 msgr2=0x7f40d0153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 -- 172.21.15.31:0/1231148451 shutdown_connections 2024-09-11T00:56:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:11.636+0000 7f40d6400640 1 -- 172.21.15.31:0/1231148451 wait complete. 2024-09-11T00:56:11.654 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:56:13.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.824+0000 7fbe58000640 1 -- 172.21.15.31:0/603873847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe500c23a0 msgr2=0x7fbe501516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.824+0000 7fbe58000640 1 --2- 172.21.15.31:0/603873847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe500c23a0 0x7fbe501516e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fbe500c4400 tx=0x7fbe3c02a5f0 comp rx=0 tx=0).stop 2024-09-11T00:56:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.824+0000 7fbe58000640 1 -- 172.21.15.31:0/603873847 shutdown_connections 2024-09-11T00:56:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.824+0000 7fbe58000640 1 --2- 172.21.15.31:0/603873847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe500c23a0 0x7fbe501516e0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.824+0000 7fbe58000640 1 --2- 172.21.15.31:0/603873847 >> v2:172.21.15.204:3300/0 conn(0x7fbe50072480 0x7fbe501511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.824+0000 7fbe58000640 1 -- 172.21.15.31:0/603873847 >> 172.21.15.31:0/603873847 conn(0x7fbe500a9750 msgr2=0x7fbe500a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:13.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.824+0000 7fbe58000640 1 -- 172.21.15.31:0/603873847 shutdown_connections 2024-09-11T00:56:13.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.824+0000 7fbe58000640 1 -- 172.21.15.31:0/603873847 wait complete. 2024-09-11T00:56:13.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.825+0000 7fbe58000640 1 Processor -- start 2024-09-11T00:56:13.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.825+0000 7fbe58000640 1 -- start start 2024-09-11T00:56:13.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.826+0000 7fbe58000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe50072480 0x7fbe500c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:13.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.826+0000 7fbe58000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe500c23a0 0x7fbe5006f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.826+0000 7fbe58000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe50153f20 0x7fbe5006fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.826+0000 7fbe58000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbe50074200 con 0x7fbe50153f20 2024-09-11T00:56:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.826+0000 7fbe58000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbe50074080 con 0x7fbe50072480 2024-09-11T00:56:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.826+0000 7fbe58000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fbe50072860 con 0x7fbe500c23a0 2024-09-11T00:56:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe56000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe50153f20 0x7fbe5006fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe55600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe50072480 0x7fbe500c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe56000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe50153f20 0x7fbe5006fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35374/0 (socket says 172.21.15.31:35374) 2024-09-11T00:56:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe56000640 1 -- 172.21.15.31:0/2515854359 learned_addr learned my addr 172.21.15.31:0/2515854359 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:13.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe54c00640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe500c23a0 0x7fbe5006f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:13.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe56000640 1 -- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe500c23a0 msgr2=0x7fbe5006f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:13.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe56000640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe500c23a0 0x7fbe5006f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:13.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe56000640 1 -- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe50072480 msgr2=0x7fbe500c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:13.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe56000640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe50072480 0x7fbe500c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:13.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe56000640 1 -- 172.21.15.31:0/2515854359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe501ecb20 con 0x7fbe50153f20 2024-09-11T00:56:13.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.827+0000 7fbe56000640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe50153f20 0x7fbe5006fdb0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbe4800b9c0 tx=0x7fbe4800bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:13.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.828+0000 7fbe4d600640 1 -- 172.21.15.31:0/2515854359 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe48019070 con 0x7fbe50153f20 2024-09-11T00:56:13.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.828+0000 7fbe4d600640 1 -- 172.21.15.31:0/2515854359 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe4800cbb0 con 0x7fbe50153f20 2024-09-11T00:56:13.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.828+0000 7fbe4d600640 1 -- 172.21.15.31:0/2515854359 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe48014440 con 0x7fbe50153f20 2024-09-11T00:56:13.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.828+0000 7fbe58000640 1 -- 172.21.15.31:0/2515854359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe501ecdc0 con 0x7fbe50153f20 2024-09-11T00:56:13.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.828+0000 7fbe58000640 1 -- 172.21.15.31:0/2515854359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe500c4710 con 0x7fbe50153f20 2024-09-11T00:56:13.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.829+0000 7fbe58000640 1 -- 172.21.15.31:0/2515854359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe501511a0 con 0x7fbe50153f20 2024-09-11T00:56:13.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.830+0000 7fbe4d600640 1 -- 172.21.15.31:0/2515854359 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fbe4801d3b0 con 0x7fbe50153f20 2024-09-11T00:56:13.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.831+0000 7fbe4d600640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe30077f70 0x7fbe3007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:13.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.831+0000 7fbe4d600640 1 -- 172.21.15.31:0/2515854359 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7fbe4809ac40 con 0x7fbe50153f20 2024-09-11T00:56:13.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.832+0000 7fbe55600640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe30077f70 0x7fbe3007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:13.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.832+0000 7fbe55600640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe30077f70 0x7fbe3007a430 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009990 tx=0x7fbe44009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:13.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:13.835+0000 7fbe4d600640 1 -- 172.21.15.31:0/2515854359 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fbe48062190 con 0x7fbe50153f20 2024-09-11T00:56:14.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.074+0000 7fbe58000640 1 -- 172.21.15.31:0/2515854359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe5014a440 con 0x7fbe50153f20 2024-09-11T00:56:14.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.074+0000 7fbe4d600640 1 -- 172.21.15.31:0/2515854359 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6546 (secure 0 0 0) 0x7fbe480690e0 con 0x7fbe50153f20 2024-09-11T00:56:14.076 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:14.076 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":176,"btime":"2024-09-11T00:55:15:098934+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":6593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:54:23.465895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7429":{"gid":7429,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":26,"addr":"172.21.15.204:6837/2689217598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":2689217598},{"type":"v1","addr":"172.21.15.204:6837","nonce":2689217598}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7429,"qdb_cluster":[7429]},"id":2}]} 2024-09-11T00:56:14.077 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 176 2024-09-11T00:56:14.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 -- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe30077f70 msgr2=0x7fbe3007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe30077f70 0x7fbe3007a430 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009990 tx=0x7fbe44009110 comp rx=0 tx=0).stop 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 -- 172.21.15.31:0/2515854359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe50153f20 msgr2=0x7fbe5006fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe50153f20 0x7fbe5006fdb0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbe4800b9c0 tx=0x7fbe4800bdc0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 -- 172.21.15.31:0/2515854359 shutdown_connections 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fbe30077f70 0x7fbe3007a430 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe50153f20 0x7fbe5006fdb0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fbe500c23a0 0x7fbe5006f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 --2- 172.21.15.31:0/2515854359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe50072480 0x7fbe500c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 -- 172.21.15.31:0/2515854359 >> 172.21.15.31:0/2515854359 conn(0x7fbe500a9750 msgr2=0x7fbe5014cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 -- 172.21.15.31:0/2515854359 shutdown_connections 2024-09-11T00:56:14.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.077+0000 7fbe58000640 1 -- 172.21.15.31:0/2515854359 wait complete. 2024-09-11T00:56:14.094 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:56:14.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:14.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.520+0000 7fc41a000640 1 -- 172.21.15.31:0/2971133426 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4141518b0 msgr2=0x7fc4140c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:14.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.520+0000 7fc41a000640 1 --2- 172.21.15.31:0/2971133426 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4141518b0 0x7fc4140c3600 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009be0 tx=0x7fc40c02a970 comp rx=0 tx=0).stop 2024-09-11T00:56:14.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.520+0000 7fc41a000640 1 -- 172.21.15.31:0/2971133426 shutdown_connections 2024-09-11T00:56:14.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.520+0000 7fc41a000640 1 --2- 172.21.15.31:0/2971133426 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4141518b0 0x7fc4140c3600 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.520+0000 7fc41a000640 1 --2- 172.21.15.31:0/2971133426 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc4140c23a0 0x7fc4140c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.520+0000 7fc41a000640 1 -- 172.21.15.31:0/2971133426 >> 172.21.15.31:0/2971133426 conn(0x7fc4140a9750 msgr2=0x7fc4140a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:14.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.521+0000 7fc41a000640 1 -- 172.21.15.31:0/2971133426 shutdown_connections 2024-09-11T00:56:14.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.521+0000 7fc41a000640 1 -- 172.21.15.31:0/2971133426 wait complete. 2024-09-11T00:56:14.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.522+0000 7fc41a000640 1 Processor -- start 2024-09-11T00:56:14.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.522+0000 7fc41a000640 1 -- start start 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc41a000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc414072480 0x7fc4141ead60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc41a000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4140c23a0 0x7fc41414d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc41a000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4141518b0 0x7fc414146920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc41a000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc414074290 con 0x7fc4140c23a0 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc41a000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc414072860 con 0x7fc4141518b0 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc41a000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fc4140729e0 con 0x7fc414072480 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc412a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4140c23a0 0x7fc41414d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc412a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4140c23a0 0x7fc41414d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35378/0 (socket says 172.21.15.31:35378) 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc412a00640 1 -- 172.21.15.31:0/3486661000 learned_addr learned my addr 172.21.15.31:0/3486661000 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc413400640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc414072480 0x7fc4141ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:14.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.523+0000 7fc413e00640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4141518b0 0x7fc414146920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.524+0000 7fc412a00640 1 -- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc414072480 msgr2=0x7fc4141ead60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.524+0000 7fc412a00640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc414072480 0x7fc4141ead60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.524+0000 7fc412a00640 1 -- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4141518b0 msgr2=0x7fc414146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.524+0000 7fc412a00640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4141518b0 0x7fc414146920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.524+0000 7fc412a00640 1 -- 172.21.15.31:0/3486661000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4141eb2a0 con 0x7fc4140c23a0 2024-09-11T00:56:14.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.524+0000 7fc412a00640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4140c23a0 0x7fc41414d8d0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fc40000e9f0 tx=0x7fc40000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.525+0000 7fc40b400640 1 -- 172.21.15.31:0/3486661000 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40000ccc0 con 0x7fc4140c23a0 2024-09-11T00:56:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.525+0000 7fc41a000640 1 -- 172.21.15.31:0/3486661000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4141eded0 con 0x7fc4140c23a0 2024-09-11T00:56:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.525+0000 7fc40b400640 1 -- 172.21.15.31:0/3486661000 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc40000ce60 con 0x7fc4140c23a0 2024-09-11T00:56:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.525+0000 7fc40b400640 1 -- 172.21.15.31:0/3486661000 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc400004280 con 0x7fc4140c23a0 2024-09-11T00:56:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.525+0000 7fc41a000640 1 -- 172.21.15.31:0/3486661000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc414149be0 con 0x7fc4140c23a0 2024-09-11T00:56:14.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.526+0000 7fc41a000640 1 -- 172.21.15.31:0/3486661000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc414153f20 con 0x7fc4140c23a0 2024-09-11T00:56:14.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.527+0000 7fc40b400640 1 -- 172.21.15.31:0/3486661000 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fc4000035f0 con 0x7fc4140c23a0 2024-09-11T00:56:14.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.528+0000 7fc40b400640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc3f4077f70 0x7fc3f407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:14.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.528+0000 7fc40b400640 1 -- 172.21.15.31:0/3486661000 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9504+0+0 (secure 0 0 0) 0x7fc40009b5e0 con 0x7fc4140c23a0 2024-09-11T00:56:14.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.528+0000 7fc413400640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc3f4077f70 0x7fc3f407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:14.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.529+0000 7fc413400640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc3f4077f70 0x7fc3f407a430 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc4040017c0 tx=0x7fc404009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:14.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.532+0000 7fc40b400640 1 -- 172.21.15.31:0/3486661000 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fc4141eb2a0 con 0x7fc4140c23a0 2024-09-11T00:56:14.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.778+0000 7fc41a000640 1 -- 172.21.15.31:0/3486661000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc41414a720 con 0x7fc4140c23a0 2024-09-11T00:56:14.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.779+0000 7fc40b400640 1 -- 172.21.15.31:0/3486661000 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) ==== 95+0+5709 (secure 0 0 0) 0x7fc400062b30 con 0x7fc4140c23a0 2024-09-11T00:56:14.780 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:14.781 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":177,"btime":"2024-09-11T00:56:14:574814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:14.574808+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:56:14.781 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 177 2024-09-11T00:56:14.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.781+0000 7fc41a000640 1 -- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc3f4077f70 msgr2=0x7fc3f407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:14.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.781+0000 7fc41a000640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc3f4077f70 0x7fc3f407a430 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc4040017c0 tx=0x7fc404009040 comp rx=0 tx=0).stop 2024-09-11T00:56:14.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.781+0000 7fc41a000640 1 -- 172.21.15.31:0/3486661000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4140c23a0 msgr2=0x7fc41414d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:14.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.781+0000 7fc41a000640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4140c23a0 0x7fc41414d8d0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fc40000e9f0 tx=0x7fc40000eeb0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.782+0000 7fc41a000640 1 -- 172.21.15.31:0/3486661000 shutdown_connections 2024-09-11T00:56:14.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.782+0000 7fc41a000640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fc3f4077f70 0x7fc3f407a430 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.782+0000 7fc41a000640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4141518b0 0x7fc414146920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.782+0000 7fc41a000640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4140c23a0 0x7fc41414d8d0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.782+0000 7fc41a000640 1 --2- 172.21.15.31:0/3486661000 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fc414072480 0x7fc4141ead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:14.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.782+0000 7fc41a000640 1 -- 172.21.15.31:0/3486661000 >> 172.21.15.31:0/3486661000 conn(0x7fc4140a9750 msgr2=0x7fc4140a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:14.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.782+0000 7fc41a000640 1 -- 172.21.15.31:0/3486661000 shutdown_connections 2024-09-11T00:56:14.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:14.782+0000 7fc41a000640 1 -- 172.21.15.31:0/3486661000 wait complete. 2024-09-11T00:56:14.800 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-11T00:56:14.800 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-11T00:56:14.800 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:56:14.817 INFO:tasks.ceph.mds.a:Stopped 2024-09-11T00:56:14.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:15.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.017+0000 7feec9200640 1 -- 172.21.15.31:0/3006457657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec4150020 msgr2=0x7feec40ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:15.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.017+0000 7feec9200640 1 --2- 172.21.15.31:0/3006457657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec4150020 0x7feec40ad4e0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7feec408d7a0 tx=0x7feeb402a530 comp rx=0 tx=0).stop 2024-09-11T00:56:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.017+0000 7feec9200640 1 -- 172.21.15.31:0/3006457657 shutdown_connections 2024-09-11T00:56:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.017+0000 7feec9200640 1 --2- 172.21.15.31:0/3006457657 >> v2:172.21.15.204:3300/0 conn(0x7feec406e850 0x7feec4160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.017+0000 7feec9200640 1 --2- 172.21.15.31:0/3006457657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec4150020 0x7feec40ad4e0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.017+0000 7feec9200640 1 -- 172.21.15.31:0/3006457657 >> 172.21.15.31:0/3006457657 conn(0x7feec4068f10 msgr2=0x7feec40cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.018+0000 7feec9200640 1 -- 172.21.15.31:0/3006457657 shutdown_connections 2024-09-11T00:56:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.018+0000 7feec9200640 1 -- 172.21.15.31:0/3006457657 wait complete. 2024-09-11T00:56:15.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.019+0000 7feec9200640 1 Processor -- start 2024-09-11T00:56:15.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.019+0000 7feec9200640 1 -- start start 2024-09-11T00:56:15.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feec9200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feec406e850 0x7feec414caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:15.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feec9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec40ada20 0x7feec41e1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feec9200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feec4150020 0x7feec41efa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feec9200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feec4157550 con 0x7feec40ada20 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feec9200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feec41573d0 con 0x7feec4150020 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feec9200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7feec4156960 con 0x7feec406e850 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feebaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec40ada20 0x7feec41e1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feebaa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec40ada20 0x7feec41e1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35392/0 (socket says 172.21.15.31:35392) 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feebaa00640 1 -- 172.21.15.31:0/106467431 learned_addr learned my addr 172.21.15.31:0/106467431 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feec3400640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feec406e850 0x7feec414caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.020+0000 7feec3e00640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feec4150020 0x7feec41efa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.021+0000 7feebaa00640 1 -- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feec406e850 msgr2=0x7feec414caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.021+0000 7feebaa00640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feec406e850 0x7feec414caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.021+0000 7feebaa00640 1 -- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feec4150020 msgr2=0x7feec41efa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:15.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.021+0000 7feebaa00640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feec4150020 0x7feec41efa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:15.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.021+0000 7feebaa00640 1 -- 172.21.15.31:0/106467431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec414cfe0 con 0x7feec40ada20 2024-09-11T00:56:15.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.021+0000 7feebaa00640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec40ada20 0x7feec41e1e60 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7feeac00c460 tx=0x7feeac00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:15.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.021+0000 7feebbe00640 1 -- 172.21.15.31:0/106467431 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac016020 con 0x7feec40ada20 2024-09-11T00:56:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.021+0000 7feebbe00640 1 -- 172.21.15.31:0/106467431 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feeac012960 con 0x7feec40ada20 2024-09-11T00:56:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.021+0000 7feebbe00640 1 -- 172.21.15.31:0/106467431 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac012db0 con 0x7feec40ada20 2024-09-11T00:56:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.021+0000 7feec9200640 1 -- 172.21.15.31:0/106467431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec41ed030 con 0x7feec40ada20 2024-09-11T00:56:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.022+0000 7feec9200640 1 -- 172.21.15.31:0/106467431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec41ed2d0 con 0x7feec40ada20 2024-09-11T00:56:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.022+0000 7feec9200640 1 -- 172.21.15.31:0/106467431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feec4158b00 con 0x7feec40ada20 2024-09-11T00:56:15.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.024+0000 7feebbe00640 1 -- 172.21.15.31:0/106467431 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7feeac01b400 con 0x7feec40ada20 2024-09-11T00:56:15.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.024+0000 7feebbe00640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feea0077f70 0x7feea007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:15.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.025+0000 7feebbe00640 1 -- 172.21.15.31:0/106467431 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9590+0+0 (secure 0 0 0) 0x7feeac09ae60 con 0x7feec40ada20 2024-09-11T00:56:15.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.025+0000 7feec3400640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feea0077f70 0x7feea007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:15.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.026+0000 7feec3400640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feea0077f70 0x7feea007a430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7feeb4033000 tx=0x7feeb4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:15.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.029+0000 7feebbe00640 1 -- 172.21.15.31:0/106467431 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7feec4158b00 con 0x7feec40ada20 2024-09-11T00:56:15.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.266+0000 7feec9200640 1 -- 172.21.15.31:0/106467431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feec40cc220 con 0x7feec40ada20 2024-09-11T00:56:15.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.266+0000 7feebbe00640 1 -- 172.21.15.31:0/106467431 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) ==== 95+0+5709 (secure 0 0 0) 0x7feeac020020 con 0x7feec40ada20 2024-09-11T00:56:15.268 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:15.268 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":177,"btime":"2024-09-11T00:56:14:574814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:14.574808+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:56:15.268 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 177 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 -- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feea0077f70 msgr2=0x7feea007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feea0077f70 0x7feea007a430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7feeb4033000 tx=0x7feeb4004730 comp rx=0 tx=0).stop 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 -- 172.21.15.31:0/106467431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec40ada20 msgr2=0x7feec41e1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec40ada20 0x7feec41e1e60 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7feeac00c460 tx=0x7feeac00c920 comp rx=0 tx=0).stop 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 -- 172.21.15.31:0/106467431 shutdown_connections 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7feea0077f70 0x7feea007a430 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feec4150020 0x7feec41efa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feec40ada20 0x7feec41e1e60 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 --2- 172.21.15.31:0/106467431 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7feec406e850 0x7feec414caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 -- 172.21.15.31:0/106467431 >> 172.21.15.31:0/106467431 conn(0x7feec4068f10 msgr2=0x7feec414eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:15.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 -- 172.21.15.31:0/106467431 shutdown_connections 2024-09-11T00:56:15.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:15.269+0000 7feec9200640 1 -- 172.21.15.31:0/106467431 wait complete. 2024-09-11T00:56:15.287 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:56:16.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:16.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.263+0000 7f4129c00640 1 -- 172.21.15.31:0/1775799113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41241518b0 msgr2=0x7f41240c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:16.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.263+0000 7f4129c00640 1 --2- 172.21.15.31:0/1775799113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41241518b0 0x7f41240c3600 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f411c009be0 tx=0x7f411c02a960 comp rx=0 tx=0).stop 2024-09-11T00:56:16.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.263+0000 7f4129c00640 1 -- 172.21.15.31:0/1775799113 shutdown_connections 2024-09-11T00:56:16.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.263+0000 7f4129c00640 1 --2- 172.21.15.31:0/1775799113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41241518b0 0x7f41240c3600 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.263+0000 7f4129c00640 1 --2- 172.21.15.31:0/1775799113 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41240c23a0 0x7f41240c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.263+0000 7f4129c00640 1 -- 172.21.15.31:0/1775799113 >> 172.21.15.31:0/1775799113 conn(0x7f41240a9750 msgr2=0x7f41240a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:16.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.263+0000 7f4129c00640 1 -- 172.21.15.31:0/1775799113 shutdown_connections 2024-09-11T00:56:16.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.263+0000 7f4129c00640 1 -- 172.21.15.31:0/1775799113 wait complete. 2024-09-11T00:56:16.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.265+0000 7f4129c00640 1 Processor -- start 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.265+0000 7f4129c00640 1 -- start start 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.266+0000 7f4129c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4124072480 0x7f41240c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.266+0000 7f4129c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41240c23a0 0x7f41240c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.266+0000 7f4129c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41241518b0 0x7f41241eda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.266+0000 7f4129c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4124074290 con 0x7f4124072480 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.266+0000 7f4129c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4124072860 con 0x7f41241518b0 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.266+0000 7f4129c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f41240729e0 con 0x7f41240c23a0 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.266+0000 7f4122a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4124072480 0x7f41240c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.266+0000 7f4122a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4124072480 0x7f41240c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35402/0 (socket says 172.21.15.31:35402) 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.266+0000 7f4122a00640 1 -- 172.21.15.31:0/3575996754 learned_addr learned my addr 172.21.15.31:0/3575996754 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f411a000640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41240c23a0 0x7f41240c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f4123400640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41241518b0 0x7f41241eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:16.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f4122a00640 1 -- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41240c23a0 msgr2=0x7f41240c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f4122a00640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41240c23a0 0x7f41240c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f4122a00640 1 -- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41241518b0 msgr2=0x7f41241eda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f4122a00640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41241518b0 0x7f41241eda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f4122a00640 1 -- 172.21.15.31:0/3575996754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41241eada0 con 0x7f4124072480 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f411a000640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41240c23a0 0x7f41240c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f4122a00640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4124072480 0x7f41240c7370 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f411400c490 tx=0x7f411400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f411b400640 1 -- 172.21.15.31:0/3575996754 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4114016020 con 0x7f4124072480 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f411b400640 1 -- 172.21.15.31:0/3575996754 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4114012940 con 0x7f4124072480 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f411b400640 1 -- 172.21.15.31:0/3575996754 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4114012d90 con 0x7f4124072480 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.267+0000 7f4129c00640 1 -- 172.21.15.31:0/3575996754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412414e480 con 0x7f4124072480 2024-09-11T00:56:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.268+0000 7f4129c00640 1 -- 172.21.15.31:0/3575996754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41240c70d0 con 0x7f4124072480 2024-09-11T00:56:16.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.269+0000 7f4129c00640 1 -- 172.21.15.31:0/3575996754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4124153f20 con 0x7f4124072480 2024-09-11T00:56:16.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.270+0000 7f411b400640 1 -- 172.21.15.31:0/3575996754 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f411401b400 con 0x7f4124072480 2024-09-11T00:56:16.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.271+0000 7f411b400640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4100077f70 0x7f410007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:16.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.271+0000 7f411b400640 1 -- 172.21.15.31:0/3575996754 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9590+0+0 (secure 0 0 0) 0x7f411409aca0 con 0x7f4124072480 2024-09-11T00:56:16.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.271+0000 7f411a000640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4100077f70 0x7f410007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:16.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.272+0000 7f411a000640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4100077f70 0x7f410007a430 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f410c005d00 tx=0x7f410c005c70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:16.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.275+0000 7f411b400640 1 -- 172.21.15.31:0/3575996754 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4114020030 con 0x7f4124072480 2024-09-11T00:56:16.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.505+0000 7f4129c00640 1 -- 172.21.15.31:0/3575996754 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41240c4080 con 0x7f4124072480 2024-09-11T00:56:16.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.506+0000 7f411b400640 1 -- 172.21.15.31:0/3575996754 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) ==== 95+0+5713 (secure 0 0 0) 0x7f4114062850 con 0x7f4124072480 2024-09-11T00:56:16.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:16.508 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":178,"btime":"2024-09-11T00:56:15:945562+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:15.033717+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:56:16.508 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 178 2024-09-11T00:56:16.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.508+0000 7f4129c00640 1 -- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4100077f70 msgr2=0x7f410007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.508+0000 7f4129c00640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4100077f70 0x7f410007a430 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f410c005d00 tx=0x7f410c005c70 comp rx=0 tx=0).stop 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.509+0000 7f4129c00640 1 -- 172.21.15.31:0/3575996754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4124072480 msgr2=0x7f41240c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.509+0000 7f4129c00640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4124072480 0x7f41240c7370 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f411400c490 tx=0x7f411400c950 comp rx=0 tx=0).stop 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.509+0000 7f4129c00640 1 -- 172.21.15.31:0/3575996754 shutdown_connections 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.509+0000 7f4129c00640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4100077f70 0x7f410007a430 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.509+0000 7f4129c00640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41241518b0 0x7f41241eda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.509+0000 7f4129c00640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f41240c23a0 0x7f41240c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.509+0000 7f4129c00640 1 --2- 172.21.15.31:0/3575996754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4124072480 0x7f41240c7370 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.509+0000 7f4129c00640 1 -- 172.21.15.31:0/3575996754 >> 172.21.15.31:0/3575996754 conn(0x7f41240a9750 msgr2=0x7f412414fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.509+0000 7f4129c00640 1 -- 172.21.15.31:0/3575996754 shutdown_connections 2024-09-11T00:56:16.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.509+0000 7f4129c00640 1 -- 172.21.15.31:0/3575996754 wait complete. 2024-09-11T00:56:16.526 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-11T00:56:16.526 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-11T00:56:16.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.685+0000 7f1eef600640 1 -- 172.21.15.31:0/1788253749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee80c3d10 msgr2=0x7f1ee815ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.685+0000 7f1eef600640 1 --2- 172.21.15.31:0/1788253749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee80c3d10 0x7f1ee815ce30 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0005fd0 tx=0x7f1ee002a940 comp rx=0 tx=0).stop 2024-09-11T00:56:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.685+0000 7f1eef600640 1 -- 172.21.15.31:0/1788253749 shutdown_connections 2024-09-11T00:56:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.685+0000 7f1eef600640 1 --2- 172.21.15.31:0/1788253749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee80c3d10 0x7f1ee815ce30 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.685+0000 7f1eef600640 1 --2- 172.21.15.31:0/1788253749 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1ee80c2ff0 0x7f1ee815a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.686+0000 7f1eef600640 1 -- 172.21.15.31:0/1788253749 >> 172.21.15.31:0/1788253749 conn(0x7f1ee80a9750 msgr2=0x7f1ee80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.686+0000 7f1eef600640 1 -- 172.21.15.31:0/1788253749 shutdown_connections 2024-09-11T00:56:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.686+0000 7f1eef600640 1 -- 172.21.15.31:0/1788253749 wait complete. 2024-09-11T00:56:16.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.687+0000 7f1eef600640 1 Processor -- start 2024-09-11T00:56:16.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.687+0000 7f1eef600640 1 -- start start 2024-09-11T00:56:16.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.688+0000 7f1eef600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee8070570 0x7f1ee814a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:16.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.688+0000 7f1eef600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ee80c2ff0 0x7f1ee8150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.688+0000 7f1eef600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1ee80c3d10 0x7f1ee8148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.688+0000 7f1eef600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee814fb20 con 0x7f1ee8070570 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.688+0000 7f1eef600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee806ef90 con 0x7f1ee80c2ff0 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.688+0000 7f1eef600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee806eb90 con 0x7f1ee80c3d10 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.688+0000 7f1eecc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee8070570 0x7f1ee814a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.688+0000 7f1eecc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee8070570 0x7f1ee814a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35416/0 (socket says 172.21.15.31:35416) 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.688+0000 7f1eecc00640 1 -- 172.21.15.31:0/1912084700 learned_addr learned my addr 172.21.15.31:0/1912084700 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1eed600640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1ee80c3d10 0x7f1ee8148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1edfe00640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ee80c2ff0 0x7f1ee8150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1eecc00640 1 -- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1ee80c3d10 msgr2=0x7f1ee8148bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1eecc00640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1ee80c3d10 0x7f1ee8148bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1eecc00640 1 -- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ee80c2ff0 msgr2=0x7f1ee8150370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1eecc00640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ee80c2ff0 0x7f1ee8150370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1eecc00640 1 -- 172.21.15.31:0/1912084700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee81514b0 con 0x7f1ee8070570 2024-09-11T00:56:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1eecc00640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee8070570 0x7f1ee814a2f0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800c500 tx=0x7f1ed800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1ee5600640 1 -- 172.21.15.31:0/1912084700 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8017070 con 0x7f1ee8070570 2024-09-11T00:56:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1ee5600640 1 -- 172.21.15.31:0/1912084700 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ed80035f0 con 0x7f1ee8070570 2024-09-11T00:56:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1ee5600640 1 -- 172.21.15.31:0/1912084700 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8003a40 con 0x7f1ee8070570 2024-09-11T00:56:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1eef600640 1 -- 172.21.15.31:0/1912084700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee8151790 con 0x7f1ee8070570 2024-09-11T00:56:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.689+0000 7f1eef600640 1 -- 172.21.15.31:0/1912084700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee8151d50 con 0x7f1ee8070570 2024-09-11T00:56:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.690+0000 7f1eef600640 1 -- 172.21.15.31:0/1912084700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ee80a9200 con 0x7f1ee8070570 2024-09-11T00:56:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.692+0000 7f1ee5600640 1 -- 172.21.15.31:0/1912084700 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1ed8003be0 con 0x7f1ee8070570 2024-09-11T00:56:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.693+0000 7f1ee5600640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ec4077fa0 0x7f1ec407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.693+0000 7f1ee5600640 1 -- 172.21.15.31:0/1912084700 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9590+0+0 (secure 0 0 0) 0x7f1ed809a430 con 0x7f1ee8070570 2024-09-11T00:56:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.694+0000 7f1edfe00640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ec4077fa0 0x7f1ec407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.695+0000 7f1edfe00640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ec4077fa0 0x7f1ec407a460 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8156610 tx=0x7f1ed0005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:16.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.698+0000 7f1ee5600640 1 -- 172.21.15.31:0/1912084700 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1ee80a9200 con 0x7f1ee8070570 2024-09-11T00:56:16.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.927+0000 7f1eef600640 1 -- 172.21.15.31:0/1912084700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ee8152c20 con 0x7f1ee8070570 2024-09-11T00:56:16.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.928+0000 7f1ee5600640 1 -- 172.21.15.31:0/1912084700 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) ==== 95+0+5713 (secure 0 0 0) 0x7f1ee8152c20 con 0x7f1ee8070570 2024-09-11T00:56:16.930 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:16.930 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":178,"btime":"2024-09-11T00:56:15:945562+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:15.033717+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:56:16.930 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 178 2024-09-11T00:56:16.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.930+0000 7f1eef600640 1 -- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ec4077fa0 msgr2=0x7f1ec407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:16.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.930+0000 7f1eef600640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ec4077fa0 0x7f1ec407a460 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8156610 tx=0x7f1ed0005f90 comp rx=0 tx=0).stop 2024-09-11T00:56:16.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.931+0000 7f1eef600640 1 -- 172.21.15.31:0/1912084700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee8070570 msgr2=0x7f1ee814a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.931+0000 7f1eef600640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee8070570 0x7f1ee814a2f0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800c500 tx=0x7f1ed800c9c0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.931+0000 7f1eef600640 1 -- 172.21.15.31:0/1912084700 shutdown_connections 2024-09-11T00:56:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.931+0000 7f1eef600640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1ec4077fa0 0x7f1ec407a460 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.931+0000 7f1eef600640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1ee80c3d10 0x7f1ee8148bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.931+0000 7f1eef600640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ee80c2ff0 0x7f1ee8150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.931+0000 7f1eef600640 1 --2- 172.21.15.31:0/1912084700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ee8070570 0x7f1ee814a2f0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.931+0000 7f1eef600640 1 -- 172.21.15.31:0/1912084700 >> 172.21.15.31:0/1912084700 conn(0x7f1ee80a9750 msgr2=0x7f1ee814b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.931+0000 7f1eef600640 1 -- 172.21.15.31:0/1912084700 shutdown_connections 2024-09-11T00:56:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:16.931+0000 7f1eef600640 1 -- 172.21.15.31:0/1912084700 wait complete. 2024-09-11T00:56:16.949 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-11T00:56:15:945562+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 178, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 175, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7522": { "addr": "172.21.15.31:6833/3721481399", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 3721481399, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 3721481399, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7522, "incarnation": 172, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:55:08.036343+0000", "qdb_cluster": [ 7522 ], "qdb_leader": 7522, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7522 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 178, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6593": { "addr": "172.21.15.204:6835/3620025884", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 3620025884, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 3620025884, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6593, "incarnation": 177, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:56:15.033717+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6593 } } } ], "standbys": [ { "addr": "172.21.15.31:6835/2337200751", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 2337200751, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 2337200751, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 166, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6503, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:56:17.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:17.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.452+0000 7f4132000640 1 -- 172.21.15.31:0/951902973 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f412c154b60 msgr2=0x7f412c0c2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:17.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.452+0000 7f4132000640 1 --2- 172.21.15.31:0/951902973 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f412c154b60 0x7f412c0c2fa0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4114009830 tx=0x7f411402a490 comp rx=0 tx=0).stop 2024-09-11T00:56:17.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.452+0000 7f4132000640 1 -- 172.21.15.31:0/951902973 shutdown_connections 2024-09-11T00:56:17.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.452+0000 7f4132000640 1 --2- 172.21.15.31:0/951902973 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f412c154b60 0x7f412c0c2fa0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:17.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.452+0000 7f4132000640 1 --2- 172.21.15.31:0/951902973 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f412c072480 0x7f412c074a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:17.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.452+0000 7f4132000640 1 -- 172.21.15.31:0/951902973 >> 172.21.15.31:0/951902973 conn(0x7f412c0a9750 msgr2=0x7f412c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:17.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.452+0000 7f4132000640 1 -- 172.21.15.31:0/951902973 shutdown_connections 2024-09-11T00:56:17.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.452+0000 7f4132000640 1 -- 172.21.15.31:0/951902973 wait complete. 2024-09-11T00:56:17.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.453+0000 7f4132000640 1 Processor -- start 2024-09-11T00:56:17.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.453+0000 7f4132000640 1 -- start start 2024-09-11T00:56:17.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.454+0000 7f4132000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f412c072480 0x7f412c1436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:17.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.454+0000 7f4132000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f412c0c34e0 0x7f412c06f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:17.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.454+0000 7f4132000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f412c154b60 0x7f412c0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:17.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.454+0000 7f4132000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f412c0c23a0 con 0x7f412c0c34e0 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.454+0000 7f4132000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f412c154420 con 0x7f412c072480 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.454+0000 7f4132000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f412c0a7140 con 0x7f412c154b60 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f4122a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f412c0c34e0 0x7f412c06f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f4122a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f412c0c34e0 0x7f412c06f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35428/0 (socket says 172.21.15.31:35428) 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f4122a00640 1 -- 172.21.15.31:0/3890439615 learned_addr learned my addr 172.21.15.31:0/3890439615 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f412b400640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f412c072480 0x7f412c1436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f412be00640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f412c154b60 0x7f412c0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f4122a00640 1 -- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f412c154b60 msgr2=0x7f412c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f4122a00640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f412c154b60 0x7f412c0c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f4122a00640 1 -- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f412c072480 msgr2=0x7f412c1436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f4122a00640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f412c072480 0x7f412c1436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f4122a00640 1 -- 172.21.15.31:0/3890439615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c0a9410 con 0x7f412c0c34e0 2024-09-11T00:56:17.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.455+0000 7f4122a00640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f412c0c34e0 0x7f412c06f810 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4114000c00 tx=0x7f4114033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:17.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.456+0000 7f4123e00640 1 -- 172.21.15.31:0/3890439615 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4114031d60 con 0x7f412c0c34e0 2024-09-11T00:56:17.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.456+0000 7f4123e00640 1 -- 172.21.15.31:0/3890439615 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4114033e10 con 0x7f412c0c34e0 2024-09-11T00:56:17.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.456+0000 7f4123e00640 1 -- 172.21.15.31:0/3890439615 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4114033310 con 0x7f412c0c34e0 2024-09-11T00:56:17.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.456+0000 7f4132000640 1 -- 172.21.15.31:0/3890439615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412c0a8890 con 0x7f412c0c34e0 2024-09-11T00:56:17.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.456+0000 7f4132000640 1 -- 172.21.15.31:0/3890439615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f412c0a8e50 con 0x7f412c0c34e0 2024-09-11T00:56:17.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.457+0000 7f4132000640 1 -- 172.21.15.31:0/3890439615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f412c0728f0 con 0x7f412c0c34e0 2024-09-11T00:56:17.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.458+0000 7f4123e00640 1 -- 172.21.15.31:0/3890439615 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f41140334b0 con 0x7f412c0c34e0 2024-09-11T00:56:17.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.459+0000 7f4123e00640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4108077f70 0x7f410807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:17.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.459+0000 7f4123e00640 1 -- 172.21.15.31:0/3890439615 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9590+0+0 (secure 0 0 0) 0x7f41140bbfe0 con 0x7f412c0c34e0 2024-09-11T00:56:17.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.460+0000 7f412b400640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4108077f70 0x7f410807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:17.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.460+0000 7f412b400640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4108077f70 0x7f410807a430 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f411c009b60 tx=0x7f411c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:17.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.463+0000 7f4123e00640 1 -- 172.21.15.31:0/3890439615 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f411403f030 con 0x7f412c0c34e0 2024-09-11T00:56:17.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.698+0000 7f4132000640 1 -- 172.21.15.31:0/3890439615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f412c0a6f20 con 0x7f412c0c34e0 2024-09-11T00:56:17.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.698+0000 7f4123e00640 1 -- 172.21.15.31:0/3890439615 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) ==== 95+0+5710 (secure 0 0 0) 0x7f4114083b60 con 0x7f412c0c34e0 2024-09-11T00:56:17.702 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:17.703 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":179,"btime":"2024-09-11T00:56:16:955140+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:15.961768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-11T00:56:17.704 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 179 2024-09-11T00:56:17.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 -- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4108077f70 msgr2=0x7f410807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:17.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4108077f70 0x7f410807a430 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f411c009b60 tx=0x7f411c005f90 comp rx=0 tx=0).stop 2024-09-11T00:56:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 -- 172.21.15.31:0/3890439615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f412c0c34e0 msgr2=0x7f412c06f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f412c0c34e0 0x7f412c06f810 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4114000c00 tx=0x7f4114033c20 comp rx=0 tx=0).stop 2024-09-11T00:56:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 -- 172.21.15.31:0/3890439615 shutdown_connections 2024-09-11T00:56:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4108077f70 0x7f410807a430 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f412c154b60 0x7f412c0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f412c0c34e0 0x7f412c06f810 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 --2- 172.21.15.31:0/3890439615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f412c072480 0x7f412c1436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:17.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 -- 172.21.15.31:0/3890439615 >> 172.21.15.31:0/3890439615 conn(0x7f412c0a9750 msgr2=0x7f412c14a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:17.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 -- 172.21.15.31:0/3890439615 shutdown_connections 2024-09-11T00:56:17.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:17.701+0000 7f4132000640 1 -- 172.21.15.31:0/3890439615 wait complete. 2024-09-11T00:56:17.719 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:56:17.994 INFO:tasks.workunit.client.1.smithi204.stderr:+ rm -r tmp ffsb 2024-09-11T00:56:18.950 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:19.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.121+0000 7f1095200640 1 -- 172.21.15.31:0/4013714467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1090072480 msgr2=0x7f10901511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:19.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.121+0000 7f1095200640 1 --2- 172.21.15.31:0/4013714467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1090072480 0x7f10901511a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1080009830 tx=0x7f108002a5a0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.121+0000 7f1095200640 1 -- 172.21.15.31:0/4013714467 shutdown_connections 2024-09-11T00:56:19.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.121+0000 7f1095200640 1 --2- 172.21.15.31:0/4013714467 >> v2:172.21.15.204:3300/0 conn(0x7f1090153f20 0x7f10901516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.121+0000 7f1095200640 1 --2- 172.21.15.31:0/4013714467 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1090072480 0x7f10901511a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.121+0000 7f1095200640 1 -- 172.21.15.31:0/4013714467 >> 172.21.15.31:0/4013714467 conn(0x7f10900a9750 msgr2=0x7f10900a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:19.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.121+0000 7f1095200640 1 -- 172.21.15.31:0/4013714467 shutdown_connections 2024-09-11T00:56:19.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.121+0000 7f1095200640 1 -- 172.21.15.31:0/4013714467 wait complete. 2024-09-11T00:56:19.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.123+0000 7f1095200640 1 Processor -- start 2024-09-11T00:56:19.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.123+0000 7f1095200640 1 -- start start 2024-09-11T00:56:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f1095200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1090072480 0x7f10900c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f1095200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10900c23a0 0x7f1090146de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f1095200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1090153f20 0x7f1090148470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f1095200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1090074200 con 0x7f10900c23a0 2024-09-11T00:56:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f1095200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1090074080 con 0x7f1090072480 2024-09-11T00:56:19.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f1095200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1090072860 con 0x7f1090153f20 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10900c23a0 0x7f1090146de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1090072480 0x7f10900c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10900c23a0 0x7f1090146de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58920/0 (socket says 172.21.15.31:58920) 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108d400640 1 -- 172.21.15.31:0/1321725903 learned_addr learned my addr 172.21.15.31:0/1321725903 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1090072480 0x7f10900c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:40180/0 (socket says 172.21.15.31:40180) 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108e800640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1090153f20 0x7f1090148470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108d400640 1 -- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1090153f20 msgr2=0x7f1090148470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108d400640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1090153f20 0x7f1090148470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108d400640 1 -- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1090072480 msgr2=0x7f10900c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108d400640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1090072480 0x7f10900c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108d400640 1 -- 172.21.15.31:0/1321725903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109014acc0 con 0x7f10900c23a0 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.124+0000 7f108de00640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1090072480 0x7f10900c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:56:19.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.125+0000 7f108d400640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10900c23a0 0x7f1090146de0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f107c00c420 tx=0x7f107c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:19.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.125+0000 7f1086000640 1 -- 172.21.15.31:0/1321725903 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c011790 con 0x7f10900c23a0 2024-09-11T00:56:19.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.125+0000 7f1086000640 1 -- 172.21.15.31:0/1321725903 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f107c017070 con 0x7f10900c23a0 2024-09-11T00:56:19.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.125+0000 7f1095200640 1 -- 172.21.15.31:0/1321725903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10901f01a0 con 0x7f10900c23a0 2024-09-11T00:56:19.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.125+0000 7f1086000640 1 -- 172.21.15.31:0/1321725903 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c011d20 con 0x7f10900c23a0 2024-09-11T00:56:19.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.125+0000 7f1095200640 1 -- 172.21.15.31:0/1321725903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10901e9cb0 con 0x7f10900c23a0 2024-09-11T00:56:19.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.126+0000 7f1095200640 1 -- 172.21.15.31:0/1321725903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10901511a0 con 0x7f10900c23a0 2024-09-11T00:56:19.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.127+0000 7f1086000640 1 -- 172.21.15.31:0/1321725903 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f107c01c460 con 0x7f10900c23a0 2024-09-11T00:56:19.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.129+0000 7f1086000640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 0x7f107007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:19.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.129+0000 7f1086000640 1 -- 172.21.15.31:0/1321725903 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9590+0+0 (secure 0 0 0) 0x7f107c09a870 con 0x7f10900c23a0 2024-09-11T00:56:19.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.129+0000 7f108de00640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 0x7f107007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:19.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.130+0000 7f108de00640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 0x7f107007a430 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1080034000 tx=0x7f1080004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:19.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.133+0000 7f1086000640 1 -- 172.21.15.31:0/1321725903 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f107c061e30 con 0x7f10900c23a0 2024-09-11T00:56:19.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.364+0000 7f1095200640 1 -- 172.21.15.31:0/1321725903 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f109014c8b0 con 0x7f10900c23a0 2024-09-11T00:56:19.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.365+0000 7f1086000640 1 -- 172.21.15.31:0/1321725903 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5717 (secure 0 0 0) 0x7f107c066860 con 0x7f10900c23a0 2024-09-11T00:56:19.367 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:19.368 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":180,"btime":"2024-09-11T00:56:17:960293+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:19.368 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 180 2024-09-11T00:56:19.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 -- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 msgr2=0x7f107007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:19.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 0x7f107007a430 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1080034000 tx=0x7f1080004730 comp rx=0 tx=0).stop 2024-09-11T00:56:19.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 -- 172.21.15.31:0/1321725903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10900c23a0 msgr2=0x7f1090146de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:19.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10900c23a0 0x7f1090146de0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f107c00c420 tx=0x7f107c00c8e0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 -- 172.21.15.31:0/1321725903 shutdown_connections 2024-09-11T00:56:19.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1070077f70 0x7f107007a430 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f1090153f20 0x7f1090148470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10900c23a0 0x7f1090146de0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 --2- 172.21.15.31:0/1321725903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1090072480 0x7f10900c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 -- 172.21.15.31:0/1321725903 >> 172.21.15.31:0/1321725903 conn(0x7f10900a9750 msgr2=0x7f109014b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:19.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 -- 172.21.15.31:0/1321725903 shutdown_connections 2024-09-11T00:56:19.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.367+0000 7f1095200640 1 -- 172.21.15.31:0/1321725903 wait complete. 2024-09-11T00:56:19.385 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=7429 2024-09-11T00:56:19.385 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.d 2024-09-11T00:56:19.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.883+0000 7f2658e00640 1 -- 172.21.15.31:0/2403318784 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2654153f20 msgr2=0x7f26541516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.883+0000 7f2658e00640 1 --2- 172.21.15.31:0/2403318784 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2654153f20 0x7f26541516e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f264c005fd0 tx=0x7f264c02a940 comp rx=0 tx=0).stop 2024-09-11T00:56:19.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.883+0000 7f2658e00640 1 -- 172.21.15.31:0/2403318784 shutdown_connections 2024-09-11T00:56:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.883+0000 7f2658e00640 1 --2- 172.21.15.31:0/2403318784 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2654153f20 0x7f26541516e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.883+0000 7f2658e00640 1 --2- 172.21.15.31:0/2403318784 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2654072480 0x7f26541511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.883+0000 7f2658e00640 1 -- 172.21.15.31:0/2403318784 >> 172.21.15.31:0/2403318784 conn(0x7f26540a9750 msgr2=0x7f26540a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.883+0000 7f2658e00640 1 -- 172.21.15.31:0/2403318784 shutdown_connections 2024-09-11T00:56:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.884+0000 7f2658e00640 1 -- 172.21.15.31:0/2403318784 wait complete. 2024-09-11T00:56:19.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.885+0000 7f2658e00640 1 Processor -- start 2024-09-11T00:56:19.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.885+0000 7f2658e00640 1 -- start start 2024-09-11T00:56:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2658e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2654072480 0x7f26541e00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2658e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26540c23a0 0x7f26541de510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2658e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2654153f20 0x7f26540c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2658e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2654074200 con 0x7f2654153f20 2024-09-11T00:56:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2658e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2654074080 con 0x7f2654072480 2024-09-11T00:56:19.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2658e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2654072860 con 0x7f26540c23a0 2024-09-11T00:56:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2652800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2654153f20 0x7f26540c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2651e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2654072480 0x7f26541e00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2651e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2654072480 0x7f26541e00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:40202/0 (socket says 172.21.15.31:40202) 2024-09-11T00:56:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2652800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2654153f20 0x7f26540c3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58926/0 (socket says 172.21.15.31:58926) 2024-09-11T00:56:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2651e00640 1 -- 172.21.15.31:0/4243106456 learned_addr learned my addr 172.21.15.31:0/4243106456 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:19.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.886+0000 7f2649400640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26540c23a0 0x7f26541de510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f2651e00640 1 -- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26540c23a0 msgr2=0x7f26541de510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f2651e00640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26540c23a0 0x7f26541de510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f2651e00640 1 -- 172.21.15.31:0/4243106456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2654153f20 msgr2=0x7f26540c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f2651e00640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2654153f20 0x7f26540c3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f2651e00640 1 -- 172.21.15.31:0/4243106456 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26541de050 con 0x7f2654072480 2024-09-11T00:56:19.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f2652800640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2654153f20 0x7f26540c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:56:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f2651e00640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2654072480 0x7f26541e00c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f264400ea60 tx=0x7f264400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f264aa00640 1 -- 172.21.15.31:0/4243106456 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264400cde0 con 0x7f2654072480 2024-09-11T00:56:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f264aa00640 1 -- 172.21.15.31:0/4243106456 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2644004020 con 0x7f2654072480 2024-09-11T00:56:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f2658e00640 1 -- 172.21.15.31:0/4243106456 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26541dd620 con 0x7f2654072480 2024-09-11T00:56:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.887+0000 7f264aa00640 1 -- 172.21.15.31:0/4243106456 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2644004450 con 0x7f2654072480 2024-09-11T00:56:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.888+0000 7f2658e00640 1 -- 172.21.15.31:0/4243106456 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26541dea50 con 0x7f2654072480 2024-09-11T00:56:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.889+0000 7f2658e00640 1 -- 172.21.15.31:0/4243106456 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26541dd860 con 0x7f2654072480 2024-09-11T00:56:19.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.890+0000 7f264aa00640 1 -- 172.21.15.31:0/4243106456 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f26440035f0 con 0x7f2654072480 2024-09-11T00:56:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.891+0000 7f264aa00640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2630077f70 0x7f263007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.891+0000 7f264aa00640 1 -- 172.21.15.31:0/4243106456 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9590+0+0 (secure 0 0 0) 0x7f264409b520 con 0x7f2654072480 2024-09-11T00:56:19.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.891+0000 7f2649400640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2630077f70 0x7f263007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:19.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.892+0000 7f2649400640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2630077f70 0x7f263007a430 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f265406dac0 tx=0x7f263c007a00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:19.895+0000 7f264aa00640 1 -- 172.21.15.31:0/4243106456 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f26541dd860 con 0x7f2654072480 2024-09-11T00:56:20.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.133+0000 7f2658e00640 1 -- 172.21.15.31:0/4243106456 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2654077240 con 0x7f2654072480 2024-09-11T00:56:20.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.134+0000 7f264aa00640 1 -- 172.21.15.31:0/4243106456 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5717 (secure 0 0 0) 0x7f2654077240 con 0x7f2654072480 2024-09-11T00:56:20.136 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:20.136 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":180,"btime":"2024-09-11T00:56:17:960293+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:20.136 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 180 2024-09-11T00:56:20.137 INFO:tasks.ceph.mds.b.smithi204.stderr:2024-09-11T00:56:20.134+0000 7f8427400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:56:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 -- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2630077f70 msgr2=0x7f263007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2630077f70 0x7f263007a430 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f265406dac0 tx=0x7f263c007a00 comp rx=0 tx=0).stop 2024-09-11T00:56:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 -- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2654072480 msgr2=0x7f26541e00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2654072480 0x7f26541e00c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f264400ea60 tx=0x7f264400ef20 comp rx=0 tx=0).stop 2024-09-11T00:56:20.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 -- 172.21.15.31:0/4243106456 shutdown_connections 2024-09-11T00:56:20.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2630077f70 0x7f263007a430 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:20.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2654153f20 0x7f26540c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:20.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f26540c23a0 0x7f26541de510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:20.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 --2- 172.21.15.31:0/4243106456 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2654072480 0x7f26541e00c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:20.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 -- 172.21.15.31:0/4243106456 >> 172.21.15.31:0/4243106456 conn(0x7f26540a9750 msgr2=0x7f2654152170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:20.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 -- 172.21.15.31:0/4243106456 shutdown_connections 2024-09-11T00:56:20.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:20.136+0000 7f2658e00640 1 -- 172.21.15.31:0/4243106456 wait complete. 2024-09-11T00:56:20.155 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:56:22.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:22.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.316+0000 7f5b58200640 1 -- 172.21.15.31:0/2227993238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b501518b0 msgr2=0x7f5b500c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:22.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.316+0000 7f5b58200640 1 --2- 172.21.15.31:0/2227993238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b501518b0 0x7f5b500c3600 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f5b500add10 tx=0x7f5b4802ab40 comp rx=0 tx=0).stop 2024-09-11T00:56:22.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.316+0000 7f5b58200640 1 -- 172.21.15.31:0/2227993238 shutdown_connections 2024-09-11T00:56:22.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.316+0000 7f5b58200640 1 --2- 172.21.15.31:0/2227993238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b501518b0 0x7f5b500c3600 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:22.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.316+0000 7f5b58200640 1 --2- 172.21.15.31:0/2227993238 >> v2:172.21.15.204:3300/0 conn(0x7f5b500c23a0 0x7f5b500c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:22.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.316+0000 7f5b58200640 1 -- 172.21.15.31:0/2227993238 >> 172.21.15.31:0/2227993238 conn(0x7f5b500a9750 msgr2=0x7f5b500a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:22.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.316+0000 7f5b58200640 1 -- 172.21.15.31:0/2227993238 shutdown_connections 2024-09-11T00:56:22.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.316+0000 7f5b58200640 1 -- 172.21.15.31:0/2227993238 wait complete. 2024-09-11T00:56:22.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.318+0000 7f5b58200640 1 Processor -- start 2024-09-11T00:56:22.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.318+0000 7f5b58200640 1 -- start start 2024-09-11T00:56:22.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.319+0000 7f5b58200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b50072480 0x7f5b501ebe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:22.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.319+0000 7f5b58200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b500c23a0 0x7f5b501500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:22.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.319+0000 7f5b58200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b501518b0 0x7f5b501531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.319+0000 7f5b58200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5b50074200 con 0x7f5b500c23a0 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.319+0000 7f5b58200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5b50074080 con 0x7f5b501518b0 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.319+0000 7f5b58200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5b50072860 con 0x7f5b50072480 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.319+0000 7f5b54e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b500c23a0 0x7f5b501500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.319+0000 7f5b54e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b500c23a0 0x7f5b501500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58954/0 (socket says 172.21.15.31:58954) 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.319+0000 7f5b54e00640 1 -- 172.21.15.31:0/4013369563 learned_addr learned my addr 172.21.15.31:0/4013369563 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b56200640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b501518b0 0x7f5b501531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b55800640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b50072480 0x7f5b501ebe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b54e00640 1 -- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b50072480 msgr2=0x7f5b501ebe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b54e00640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b50072480 0x7f5b501ebe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b54e00640 1 -- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b501518b0 msgr2=0x7f5b501531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b54e00640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b501518b0 0x7f5b501531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b54e00640 1 -- 172.21.15.31:0/4013369563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b501529f0 con 0x7f5b500c23a0 2024-09-11T00:56:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b56200640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b501518b0 0x7f5b501531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:56:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b55800640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b50072480 0x7f5b501ebe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:56:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b54e00640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b500c23a0 0x7f5b501500f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00e9f0 tx=0x7f5b3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b4d600640 1 -- 172.21.15.31:0/4013369563 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c00cd80 con 0x7f5b500c23a0 2024-09-11T00:56:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b4d600640 1 -- 172.21.15.31:0/4013369563 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b3c005020 con 0x7f5b500c23a0 2024-09-11T00:56:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b4d600640 1 -- 172.21.15.31:0/4013369563 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c005450 con 0x7f5b500c23a0 2024-09-11T00:56:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b58200640 1 -- 172.21.15.31:0/4013369563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b5006fab0 con 0x7f5b500c23a0 2024-09-11T00:56:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.320+0000 7f5b58200640 1 -- 172.21.15.31:0/4013369563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b50070070 con 0x7f5b500c23a0 2024-09-11T00:56:22.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.321+0000 7f5b58200640 1 -- 172.21.15.31:0/4013369563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b50153f20 con 0x7f5b500c23a0 2024-09-11T00:56:22.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.323+0000 7f5b4d600640 1 -- 172.21.15.31:0/4013369563 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5b3c0047a0 con 0x7f5b500c23a0 2024-09-11T00:56:22.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.323+0000 7f5b4d600640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b30077f70 0x7f5b3007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:22.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.324+0000 7f5b4d600640 1 -- 172.21.15.31:0/4013369563 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9590+0+0 (secure 0 0 0) 0x7f5b3c09b780 con 0x7f5b500c23a0 2024-09-11T00:56:22.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.324+0000 7f5b55800640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b30077f70 0x7f5b3007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:22.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.325+0000 7f5b55800640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b30077f70 0x7f5b3007a430 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5b44001740 tx=0x7f5b44009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:22.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.328+0000 7f5b4d600640 1 -- 172.21.15.31:0/4013369563 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5b3c063360 con 0x7f5b500c23a0 2024-09-11T00:56:22.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.711+0000 7f5b58200640 1 -- 172.21.15.31:0/4013369563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b500c3820 con 0x7f5b500c23a0 2024-09-11T00:56:22.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.711+0000 7f5b4d600640 1 -- 172.21.15.31:0/4013369563 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5717 (secure 0 0 0) 0x7f5b3c062c60 con 0x7f5b500c23a0 2024-09-11T00:56:22.717 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:22.717 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":180,"btime":"2024-09-11T00:56:17:960293+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:22.718 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 180 2024-09-11T00:56:22.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 -- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b30077f70 msgr2=0x7f5b3007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:22.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b30077f70 0x7f5b3007a430 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5b44001740 tx=0x7f5b44009040 comp rx=0 tx=0).stop 2024-09-11T00:56:22.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 -- 172.21.15.31:0/4013369563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b500c23a0 msgr2=0x7f5b501500f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:22.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b500c23a0 0x7f5b501500f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00e9f0 tx=0x7f5b3c00eeb0 comp rx=0 tx=0).stop 2024-09-11T00:56:22.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 -- 172.21.15.31:0/4013369563 shutdown_connections 2024-09-11T00:56:22.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5b30077f70 0x7f5b3007a430 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:22.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b501518b0 0x7f5b501531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:22.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b500c23a0 0x7f5b501500f0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:22.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 --2- 172.21.15.31:0/4013369563 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5b50072480 0x7f5b501ebe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:22.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 -- 172.21.15.31:0/4013369563 >> 172.21.15.31:0/4013369563 conn(0x7f5b500a9750 msgr2=0x7f5b5014c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:22.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 -- 172.21.15.31:0/4013369563 shutdown_connections 2024-09-11T00:56:22.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:22.714+0000 7f5b58200640 1 -- 172.21.15.31:0/4013369563 wait complete. 2024-09-11T00:56:22.738 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:56:24.314 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T00:56:24.314 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-09-11T00:56:24.378 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.1... 2024-09-11T00:56:24.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-09-11T00:56:24.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:24.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.963+0000 7f17f8000640 1 -- 172.21.15.31:0/1019846902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17f0072480 msgr2=0x7f17f01511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:24.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.963+0000 7f17f8000640 1 --2- 172.21.15.31:0/1019846902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17f0072480 0x7f17f01511a0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f17e4005fd0 tx=0x7f17e402aaa0 comp rx=0 tx=0).stop 2024-09-11T00:56:24.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.963+0000 7f17f8000640 1 -- 172.21.15.31:0/1019846902 shutdown_connections 2024-09-11T00:56:24.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.963+0000 7f17f8000640 1 --2- 172.21.15.31:0/1019846902 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17f00c23a0 0x7f17f01516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:24.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.963+0000 7f17f8000640 1 --2- 172.21.15.31:0/1019846902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17f0072480 0x7f17f01511a0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:24.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.963+0000 7f17f8000640 1 -- 172.21.15.31:0/1019846902 >> 172.21.15.31:0/1019846902 conn(0x7f17f00a9750 msgr2=0x7f17f00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:24.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.965+0000 7f17f8000640 1 -- 172.21.15.31:0/1019846902 shutdown_connections 2024-09-11T00:56:24.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.965+0000 7f17f8000640 1 -- 172.21.15.31:0/1019846902 wait complete. 2024-09-11T00:56:24.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.967+0000 7f17f8000640 1 Processor -- start 2024-09-11T00:56:24.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.967+0000 7f17f8000640 1 -- start start 2024-09-11T00:56:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.968+0000 7f17f8000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f0072480 0x7f17f01436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.968+0000 7f17f8000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17f00c23a0 0x7f17f0152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.968+0000 7f17f8000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17f0153f20 0x7f17f01ea290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.968+0000 7f17f8000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0074200 con 0x7f17f00c23a0 2024-09-11T00:56:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.968+0000 7f17f8000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0074080 con 0x7f17f0072480 2024-09-11T00:56:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.968+0000 7f17f8000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f17f0072860 con 0x7f17f0153f20 2024-09-11T00:56:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.968+0000 7f17f5600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f0072480 0x7f17f01436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.968+0000 7f17f5600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f0072480 0x7f17f01436c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:40230/0 (socket says 172.21.15.31:40230) 2024-09-11T00:56:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.968+0000 7f17f5600640 1 -- 172.21.15.31:0/2999695291 learned_addr learned my addr 172.21.15.31:0/2999695291 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.969+0000 7f17f4c00640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17f00c23a0 0x7f17f0152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:24.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.969+0000 7f17f6000640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17f0153f20 0x7f17f01ea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:24.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.969+0000 7f17f5600640 1 -- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17f0153f20 msgr2=0x7f17f01ea290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:24.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.969+0000 7f17f5600640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17f0153f20 0x7f17f01ea290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:24.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.969+0000 7f17f5600640 1 -- 172.21.15.31:0/2999695291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17f00c23a0 msgr2=0x7f17f0152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:24.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.969+0000 7f17f5600640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17f00c23a0 0x7f17f0152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:24.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.969+0000 7f17f5600640 1 -- 172.21.15.31:0/2999695291 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f01e9d90 con 0x7f17f0072480 2024-09-11T00:56:24.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.970+0000 7f17f5600640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f0072480 0x7f17f01436c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f17e4000f80 tx=0x7f17e4003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:24.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.970+0000 7f17ed600640 1 -- 172.21.15.31:0/2999695291 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4004450 con 0x7f17f0072480 2024-09-11T00:56:24.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.970+0000 7f17ed600640 1 -- 172.21.15.31:0/2999695291 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17e4044070 con 0x7f17f0072480 2024-09-11T00:56:24.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.970+0000 7f17ed600640 1 -- 172.21.15.31:0/2999695291 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e403f400 con 0x7f17f0072480 2024-09-11T00:56:24.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.971+0000 7f17f8000640 1 -- 172.21.15.31:0/2999695291 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f0144060 con 0x7f17f0072480 2024-09-11T00:56:24.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.971+0000 7f17f8000640 1 -- 172.21.15.31:0/2999695291 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f01539a0 con 0x7f17f0072480 2024-09-11T00:56:24.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.976+0000 7f17f8000640 1 -- 172.21.15.31:0/2999695291 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f01e9f90 con 0x7f17f0072480 2024-09-11T00:56:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.978+0000 7f17ed600640 1 -- 172.21.15.31:0/2999695291 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f17e403f5a0 con 0x7f17f0072480 2024-09-11T00:56:24.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.980+0000 7f17ed600640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17d0078070 0x7f17d007a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:24.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.980+0000 7f17ed600640 1 -- 172.21.15.31:0/2999695291 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9590+0+0 (secure 0 0 0) 0x7f17e40bc340 con 0x7f17f0072480 2024-09-11T00:56:24.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.983+0000 7f17f4c00640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17d0078070 0x7f17d007a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:24.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.986+0000 7f17f4c00640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17d0078070 0x7f17d007a530 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f17dc006fd0 tx=0x7f17dc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:24.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:24.991+0000 7f17ed600640 1 -- 172.21.15.31:0/2999695291 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f17f01e9f90 con 0x7f17f0072480 2024-09-11T00:56:25.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.258+0000 7f17f8000640 1 -- 172.21.15.31:0/2999695291 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17f00c30c0 con 0x7f17f0072480 2024-09-11T00:56:25.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.259+0000 7f17ed600640 1 -- 172.21.15.31:0/2999695291 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5717 (secure 0 0 0) 0x7f17e404e020 con 0x7f17f0072480 2024-09-11T00:56:25.262 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:25.262 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":180,"btime":"2024-09-11T00:56:17:960293+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:25.262 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 180 2024-09-11T00:56:25.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 -- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17d0078070 msgr2=0x7f17d007a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:25.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17d0078070 0x7f17d007a530 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f17dc006fd0 tx=0x7f17dc008040 comp rx=0 tx=0).stop 2024-09-11T00:56:25.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 -- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f0072480 msgr2=0x7f17f01436c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:25.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f0072480 0x7f17f01436c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f17e4000f80 tx=0x7f17e4003f10 comp rx=0 tx=0).stop 2024-09-11T00:56:25.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 -- 172.21.15.31:0/2999695291 shutdown_connections 2024-09-11T00:56:25.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f17d0078070 0x7f17d007a530 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:25.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f17f0153f20 0x7f17f01ea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:25.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17f00c23a0 0x7f17f0152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:25.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 --2- 172.21.15.31:0/2999695291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f17f0072480 0x7f17f01436c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:25.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 -- 172.21.15.31:0/2999695291 >> 172.21.15.31:0/2999695291 conn(0x7f17f00a9750 msgr2=0x7f17f01504d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:25.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 -- 172.21.15.31:0/2999695291 shutdown_connections 2024-09-11T00:56:25.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:25.261+0000 7f17f8000640 1 -- 172.21.15.31:0/2999695291 wait complete. 2024-09-11T00:56:25.281 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:56:25.537 DEBUG:teuthology.parallel:result is None 2024-09-11T00:56:25.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T00:56:27.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:27.386 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-11T00:56:27.386 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:56:27.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:56:27.394 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:56:27.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:27.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.540+0000 7f6143a00640 1 -- 172.21.15.31:0/373146381 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f61240037a0 msgr2=0x7f6124003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:27.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.540+0000 7f6143a00640 1 --2- 172.21.15.31:0/373146381 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f61240037a0 0x7f6124003c20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6128010a40 tx=0x7f6124005830 comp rx=0 tx=0).stop 2024-09-11T00:56:27.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.540+0000 7f6141a00640 1 -- 172.21.15.31:0/373146381 reap_dead start 2024-09-11T00:56:27.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.540+0000 7f6143a00640 1 -- 172.21.15.31:0/373146381 shutdown_connections 2024-09-11T00:56:27.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.540+0000 7f6143a00640 1 -- 172.21.15.31:0/373146381 >> 172.21.15.31:0/373146381 conn(0x7f613c0a9750 msgr2=0x7f613c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:27.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.541+0000 7f6143a00640 1 -- 172.21.15.31:0/373146381 shutdown_connections 2024-09-11T00:56:27.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.541+0000 7f6143a00640 1 -- 172.21.15.31:0/373146381 wait complete. 2024-09-11T00:56:27.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.544+0000 7f6143a00640 1 Processor -- start 2024-09-11T00:56:27.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.544+0000 7f6143a00640 1 -- start start 2024-09-11T00:56:27.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.546+0000 7f6143a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f61240037a0 0x7f613c1470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:27.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.546+0000 7f6143a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f613c06f920 0x7f613c153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:27.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.546+0000 7f6143a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f613c147600 0x7f613c149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:27.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.546+0000 7f6143a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f613c074200 con 0x7f613c147600 2024-09-11T00:56:27.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.546+0000 7f6143a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f613c074080 con 0x7f613c06f920 2024-09-11T00:56:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.546+0000 7f6143a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f613c072860 con 0x7f61240037a0 2024-09-11T00:56:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.546+0000 7f613be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f613c06f920 0x7f613c153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.547+0000 7f6141a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f613c147600 0x7f613c149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.546+0000 7f6141000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f61240037a0 0x7f613c1470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.547+0000 7f6141000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f61240037a0 0x7f613c1470c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:54530/0 (socket says 172.21.15.31:54530) 2024-09-11T00:56:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.547+0000 7f6141000640 1 -- 172.21.15.31:0/4145083638 learned_addr learned my addr 172.21.15.31:0/4145083638 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.547+0000 7f613be00640 1 -- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f61240037a0 msgr2=0x7f613c1470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.547+0000 7f613be00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f61240037a0 0x7f613c1470c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.547+0000 7f613be00640 1 -- 172.21.15.31:0/4145083638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f613c147600 msgr2=0x7f613c149a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:27.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.548+0000 7f613be00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f613c147600 0x7f613c149a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:27.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.548+0000 7f613be00640 1 -- 172.21.15.31:0/4145083638 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c1452a0 con 0x7f613c06f920 2024-09-11T00:56:27.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.548+0000 7f6141a00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f613c147600 0x7f613c149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:56:27.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.548+0000 7f613be00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f613c06f920 0x7f613c153160 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f612400c460 tx=0x7f612802acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:27.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.549+0000 7f6138c00640 1 -- 172.21.15.31:0/4145083638 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128006440 con 0x7f613c06f920 2024-09-11T00:56:27.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.549+0000 7f6138c00640 1 -- 172.21.15.31:0/4145083638 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6128039070 con 0x7f613c06f920 2024-09-11T00:56:27.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.549+0000 7f6138c00640 1 -- 172.21.15.31:0/4145083638 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128003940 con 0x7f613c06f920 2024-09-11T00:56:27.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.549+0000 7f6143a00640 1 -- 172.21.15.31:0/4145083638 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613c14be40 con 0x7f613c06f920 2024-09-11T00:56:27.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.549+0000 7f6143a00640 1 -- 172.21.15.31:0/4145083638 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f613c153bb0 con 0x7f613c06f920 2024-09-11T00:56:27.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.551+0000 7f6143a00640 1 -- 172.21.15.31:0/4145083638 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f613c151280 con 0x7f613c06f920 2024-09-11T00:56:27.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.558+0000 7f6138c00640 1 -- 172.21.15.31:0/4145083638 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f612805ece0 con 0x7f613c06f920 2024-09-11T00:56:27.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.560+0000 7f6138c00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f611c078070 0x7f611c07a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.560+0000 7f6138c00640 1 -- 172.21.15.31:0/4145083638 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9590+0+0 (secure 0 0 0) 0x7f61280b9ff0 con 0x7f613c06f920 2024-09-11T00:56:27.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.564+0000 7f6138c00640 1 -- 172.21.15.31:0/4145083638 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6128081bc0 con 0x7f613c06f920 2024-09-11T00:56:27.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.566+0000 7f6141000640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f611c078070 0x7f611c07a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:27.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.571+0000 7f6141000640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f611c078070 0x7f611c07a530 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6130004040 tx=0x7f6130018040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:27.615 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:56:27.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.698+0000 7f859a400640 1 -- 172.21.15.31:0/3912531946 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85940ad820 msgr2=0x7f8594156a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:27.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.698+0000 7f859a400640 1 --2- 172.21.15.31:0/3912531946 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85940ad820 0x7f8594156a10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8588005fd0 tx=0x7f858802a420 comp rx=0 tx=0).stop 2024-09-11T00:56:27.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.699+0000 7f859a400640 1 -- 172.21.15.31:0/3912531946 shutdown_connections 2024-09-11T00:56:27.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.699+0000 7f859a400640 1 --2- 172.21.15.31:0/3912531946 >> v2:172.21.15.204:3300/0 conn(0x7f859406d910 0x7f859406dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:27.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.699+0000 7f859a400640 1 --2- 172.21.15.31:0/3912531946 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f85940ad820 0x7f8594156a10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:27.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.699+0000 7f859a400640 1 -- 172.21.15.31:0/3912531946 >> 172.21.15.31:0/3912531946 conn(0x7f85940690d0 msgr2=0x7f85940cbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:27.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.702+0000 7f859a400640 1 -- 172.21.15.31:0/3912531946 shutdown_connections 2024-09-11T00:56:27.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.702+0000 7f859a400640 1 -- 172.21.15.31:0/3912531946 wait complete. 2024-09-11T00:56:27.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.705+0000 7f859a400640 1 Processor -- start 2024-09-11T00:56:27.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.705+0000 7f859a400640 1 -- start start 2024-09-11T00:56:27.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.707+0000 7f859a400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f859406d910 0x7f85940c1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:27.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.708+0000 7f859a400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85940ad820 0x7f85940c1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:27.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.708+0000 7f859a400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85940c64b0 0x7f85940afb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:27.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.708+0000 7f859a400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f859406ec00 con 0x7f85940c64b0 2024-09-11T00:56:27.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.709+0000 7f859a400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f859406e920 con 0x7f85940ad820 2024-09-11T00:56:27.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.709+0000 7f859a400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f8594150300 con 0x7f859406d910 2024-09-11T00:56:27.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.710+0000 7f8599a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85940c64b0 0x7f85940afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:27.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.710+0000 7f8599a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85940c64b0 0x7f85940afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59002/0 (socket says 172.21.15.31:59002) 2024-09-11T00:56:27.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.710+0000 7f8599a00640 1 -- 172.21.15.31:0/2313303401 learned_addr learned my addr 172.21.15.31:0/2313303401 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:27.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.711+0000 7f8599a00640 1 -- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f859406d910 msgr2=0x7f85940c1050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T00:56:27.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.712+0000 7f8599a00640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f859406d910 0x7f85940c1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:27.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.712+0000 7f8599a00640 1 -- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85940ad820 msgr2=0x7f85940c1cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T00:56:27.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.712+0000 7f8599a00640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85940ad820 0x7f85940c1cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:27.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.712+0000 7f8599a00640 1 -- 172.21.15.31:0/2313303401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85940c2230 con 0x7f85940c64b0 2024-09-11T00:56:27.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.713+0000 7f8599a00640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85940c64b0 0x7f85940afb40 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f858c00eb10 tx=0x7f858c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:27.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.714+0000 7f8590c00640 1 -- 172.21.15.31:0/2313303401 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858c004b20 con 0x7f85940c64b0 2024-09-11T00:56:27.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.717+0000 7f8590c00640 1 -- 172.21.15.31:0/2313303401 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f858c004cc0 con 0x7f85940c64b0 2024-09-11T00:56:27.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.717+0000 7f8590c00640 1 -- 172.21.15.31:0/2313303401 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858c004b20 con 0x7f85940c64b0 2024-09-11T00:56:27.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.717+0000 7f859a400640 1 -- 172.21.15.31:0/2313303401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85940bffb0 con 0x7f85940c64b0 2024-09-11T00:56:27.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.717+0000 7f859a400640 1 -- 172.21.15.31:0/2313303401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85940c0570 con 0x7f85940c64b0 2024-09-11T00:56:27.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.720+0000 7f859a400640 1 -- 172.21.15.31:0/2313303401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8594069cd0 con 0x7f85940c64b0 2024-09-11T00:56:27.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.726+0000 7f8590c00640 1 -- 172.21.15.31:0/2313303401 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f858c005020 con 0x7f85940c64b0 2024-09-11T00:56:27.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.728+0000 7f8590c00640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8574078170 0x7f857407a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:27.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.728+0000 7f8590c00640 1 -- 172.21.15.31:0/2313303401 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9590+0+0 (secure 0 0 0) 0x7f858c09bb20 con 0x7f85940c64b0 2024-09-11T00:56:27.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.732+0000 7f8599000640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8574078170 0x7f857407a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:27.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.733+0000 7f8590c00640 1 -- 172.21.15.31:0/2313303401 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f8594069cd0 con 0x7f85940c64b0 2024-09-11T00:56:27.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:27.739+0000 7f8599000640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8574078170 0x7f857407a630 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8588033000 tx=0x7f8588004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:28.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.011+0000 7f6143a00640 1 -- 172.21.15.31:0/4145083638 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f613c14c010 con 0x7f613c06f920 2024-09-11T00:56:28.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.013+0000 7f6138c00640 1 -- 172.21.15.31:0/4145083638 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5717 (secure 0 0 0) 0x7f612808a370 con 0x7f613c06f920 2024-09-11T00:56:28.017 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:28.020 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":180,"btime":"2024-09-11T00:56:17:960293+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:55:08.036343+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7522},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7522":{"gid":7522,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.31:6833/3721481399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3721481399},{"type":"v1","addr":"172.21.15.31:6833","nonce":3721481399}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7522,"qdb_cluster":[7522]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:28.020 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 180 2024-09-11T00:56:28.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.019+0000 7f6143a00640 1 -- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f611c078070 msgr2=0x7f611c07a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:28.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.019+0000 7f6143a00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f611c078070 0x7f611c07a530 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6130004040 tx=0x7f6130018040 comp rx=0 tx=0).stop 2024-09-11T00:56:28.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.019+0000 7f6143a00640 1 -- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f613c06f920 msgr2=0x7f613c153160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:28.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.019+0000 7f6143a00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f613c06f920 0x7f613c153160 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f612400c460 tx=0x7f612802acf0 comp rx=0 tx=0).stop 2024-09-11T00:56:28.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.020+0000 7f6143a00640 1 -- 172.21.15.31:0/4145083638 shutdown_connections 2024-09-11T00:56:28.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.020+0000 7f6143a00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f611c078070 0x7f611c07a530 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:28.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.020+0000 7f6143a00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f613c147600 0x7f613c149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:28.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.020+0000 7f6143a00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f613c06f920 0x7f613c153160 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:28.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.020+0000 7f6143a00640 1 --2- 172.21.15.31:0/4145083638 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f61240037a0 0x7f613c1470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:28.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.020+0000 7f6143a00640 1 -- 172.21.15.31:0/4145083638 >> 172.21.15.31:0/4145083638 conn(0x7f613c0a9750 msgr2=0x7f613c0a8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:28.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.020+0000 7f6143a00640 1 -- 172.21.15.31:0/4145083638 shutdown_connections 2024-09-11T00:56:28.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.020+0000 7f6143a00640 1 -- 172.21.15.31:0/4145083638 wait complete. 2024-09-11T00:56:28.055 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-11T00:56:28.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.156+0000 7f859a400640 1 -- 172.21.15.31:0/2313303401 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85940caf30 con 0x7f85940c64b0 2024-09-11T00:56:28.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.157+0000 7f8590c00640 1 -- 172.21.15.31:0/2313303401 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) ==== 95+0+5712 (secure 0 0 0) 0x7f858c0636d0 con 0x7f85940c64b0 2024-09-11T00:56:28.159 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:28.160 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":182,"btime":"2024-09-11T00:56:28:030548+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:28.030544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:replay","state_seq":1,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:28.160 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 182 2024-09-11T00:56:28.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 -- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8574078170 msgr2=0x7f857407a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:28.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8574078170 0x7f857407a630 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8588033000 tx=0x7f8588004730 comp rx=0 tx=0).stop 2024-09-11T00:56:28.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 -- 172.21.15.31:0/2313303401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85940c64b0 msgr2=0x7f85940afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:28.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85940c64b0 0x7f85940afb40 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f858c00eb10 tx=0x7f858c00c2d0 comp rx=0 tx=0).stop 2024-09-11T00:56:28.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 -- 172.21.15.31:0/2313303401 shutdown_connections 2024-09-11T00:56:28.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f8574078170 0x7f857407a630 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:28.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f85940c64b0 0x7f85940afb40 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:28.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85940ad820 0x7f85940c1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:28.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 --2- 172.21.15.31:0/2313303401 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f859406d910 0x7f85940c1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:28.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 -- 172.21.15.31:0/2313303401 >> 172.21.15.31:0/2313303401 conn(0x7f85940690d0 msgr2=0x7f85940ca380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:28.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 -- 172.21.15.31:0/2313303401 shutdown_connections 2024-09-11T00:56:28.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:28.159+0000 7f859a400640 1 -- 172.21.15.31:0/2313303401 wait complete. 2024-09-11T00:56:28.178 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:56:30.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:30.180 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:30.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.227+0000 7fd0ed400640 1 -- 172.21.15.31:0/2283787517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e8074630 msgr2=0x7fd0e8074a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.227+0000 7fd0ed400640 1 --2- 172.21.15.31:0/2283787517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e8074630 0x7fd0e8074a10 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fd0e814d3f0 tx=0x7fd0d802a540 comp rx=0 tx=0).stop 2024-09-11T00:56:30.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.227+0000 7fd0ed400640 1 -- 172.21.15.31:0/2283787517 shutdown_connections 2024-09-11T00:56:30.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.227+0000 7fd0ed400640 1 --2- 172.21.15.31:0/2283787517 >> v2:172.21.15.204:3300/0 conn(0x7fd0e80abde0 0x7fd0e80c37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.227+0000 7fd0ed400640 1 --2- 172.21.15.31:0/2283787517 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e8074630 0x7fd0e8074a10 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.227+0000 7fd0ed400640 1 -- 172.21.15.31:0/2283787517 >> 172.21.15.31:0/2283787517 conn(0x7fd0e80c8320 msgr2=0x7fd0e80c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:30.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.227+0000 7fd0ed400640 1 -- 172.21.15.31:0/2283787517 shutdown_connections 2024-09-11T00:56:30.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.227+0000 7fd0ed400640 1 -- 172.21.15.31:0/2283787517 wait complete. 2024-09-11T00:56:30.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.229+0000 7fd0ed400640 1 Processor -- start 2024-09-11T00:56:30.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.229+0000 7fd0ed400640 1 -- start start 2024-09-11T00:56:30.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0ed400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0e8074630 0x7fd0e80bd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0ed400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e8074f50 0x7fd0e806fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0ed400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0e80abde0 0x7fd0e8145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0ed400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e8153b70 con 0x7fd0e8074f50 2024-09-11T00:56:30.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0ed400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e806e210 con 0x7fd0e80abde0 2024-09-11T00:56:30.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0ed400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e81469f0 con 0x7fd0e8074630 2024-09-11T00:56:30.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0dfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e8074f50 0x7fd0e806fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0dfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e8074f50 0x7fd0e806fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39138/0 (socket says 172.21.15.31:39138) 2024-09-11T00:56:30.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0dfe00640 1 -- 172.21.15.31:0/111082252 learned_addr learned my addr 172.21.15.31:0/111082252 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:30.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0e7200640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0e80abde0 0x7fd0e8145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.230+0000 7fd0e6800640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0e8074630 0x7fd0e80bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0dfe00640 1 -- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0e8074630 msgr2=0x7fd0e80bd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0dfe00640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0e8074630 0x7fd0e80bd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0dfe00640 1 -- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0e80abde0 msgr2=0x7fd0e8145e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0dfe00640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0e80abde0 0x7fd0e8145e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0dfe00640 1 -- 172.21.15.31:0/111082252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e8154710 con 0x7fd0e8074f50 2024-09-11T00:56:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0dfe00640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e8074f50 0x7fd0e806fc00 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000c4d0 tx=0x7fd0d000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0dea00640 1 -- 172.21.15.31:0/111082252 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0016020 con 0x7fd0e8074f50 2024-09-11T00:56:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0dea00640 1 -- 172.21.15.31:0/111082252 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0d0012920 con 0x7fd0e8074f50 2024-09-11T00:56:30.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0ed400640 1 -- 172.21.15.31:0/111082252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e8146c40 con 0x7fd0e8074f50 2024-09-11T00:56:30.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0dea00640 1 -- 172.21.15.31:0/111082252 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0012d70 con 0x7fd0e8074f50 2024-09-11T00:56:30.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.231+0000 7fd0ed400640 1 -- 172.21.15.31:0/111082252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e8154430 con 0x7fd0e8074f50 2024-09-11T00:56:30.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.233+0000 7fd0ed400640 1 -- 172.21.15.31:0/111082252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e80c25c0 con 0x7fd0e8074f50 2024-09-11T00:56:30.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.234+0000 7fd0dea00640 1 -- 172.21.15.31:0/111082252 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd0d001b400 con 0x7fd0e8074f50 2024-09-11T00:56:30.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.235+0000 7fd0dea00640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0c4077f70 0x7fd0c407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.235+0000 7fd0dea00640 1 -- 172.21.15.31:0/111082252 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fd0d009ab00 con 0x7fd0e8074f50 2024-09-11T00:56:30.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.236+0000 7fd0e6800640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0c4077f70 0x7fd0c407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.236+0000 7fd0e6800640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0c4077f70 0x7fd0c407a430 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8034000 tx=0x7fd0d8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:30.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.240+0000 7fd0dea00640 1 -- 172.21.15.31:0/111082252 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd0e80c25c0 con 0x7fd0e8074f50 2024-09-11T00:56:30.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.344+0000 7fd4daa00640 1 -- 172.21.15.31:0/3860662901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b8003a00 msgr2=0x7fd4b8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.344+0000 7fd4daa00640 1 --2- 172.21.15.31:0/3860662901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b8003a00 0x7fd4b8003e20 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fd4d408dc00 tx=0x7fd4b8005830 comp rx=0 tx=0).stop 2024-09-11T00:56:30.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.344+0000 7fd4d8a00640 1 -- 172.21.15.31:0/3860662901 reap_dead start 2024-09-11T00:56:30.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.344+0000 7fd4daa00640 1 -- 172.21.15.31:0/3860662901 shutdown_connections 2024-09-11T00:56:30.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.344+0000 7fd4daa00640 1 -- 172.21.15.31:0/3860662901 >> 172.21.15.31:0/3860662901 conn(0x7fd4d40cc260 msgr2=0x7fd4d40ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:30.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.345+0000 7fd4daa00640 1 -- 172.21.15.31:0/3860662901 shutdown_connections 2024-09-11T00:56:30.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.345+0000 7fd4daa00640 1 -- 172.21.15.31:0/3860662901 wait complete. 2024-09-11T00:56:30.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.346+0000 7fd4daa00640 1 Processor -- start 2024-09-11T00:56:30.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.346+0000 7fd4daa00640 1 -- start start 2024-09-11T00:56:30.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.347+0000 7fd4daa00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4b8003a00 0x7fd4d414a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.347+0000 7fd4daa00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4d414d6a0 0x7fd4d4149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.347+0000 7fd4daa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4d41499a0 0x7fd4d41ebb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.347+0000 7fd4daa00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d40b1540 con 0x7fd4d41499a0 2024-09-11T00:56:30.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.347+0000 7fd4daa00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d406f9a0 con 0x7fd4b8003a00 2024-09-11T00:56:30.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.347+0000 7fd4daa00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d406f300 con 0x7fd4d414d6a0 2024-09-11T00:56:30.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4d41499a0 0x7fd4d41ebb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d8a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4d41499a0 0x7fd4d41ebb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39158/0 (socket says 172.21.15.31:39158) 2024-09-11T00:56:30.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d8a00640 1 -- 172.21.15.31:0/3751910008 learned_addr learned my addr 172.21.15.31:0/3751910008 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:30.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d3e00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4b8003a00 0x7fd4d414a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4cb400640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4d414d6a0 0x7fd4d4149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d8a00640 1 -- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4d414d6a0 msgr2=0x7fd4d4149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d8a00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4d414d6a0 0x7fd4d4149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d8a00640 1 -- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4b8003a00 msgr2=0x7fd4d414a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d8a00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4b8003a00 0x7fd4d414a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d8a00640 1 -- 172.21.15.31:0/3751910008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d414a8f0 con 0x7fd4d41499a0 2024-09-11T00:56:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d3e00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4b8003a00 0x7fd4d414a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.348+0000 7fd4d8a00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4d41499a0 0x7fd4d41ebb70 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40cbbd0 tx=0x7fd4cc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.349+0000 7fd4d0c00640 1 -- 172.21.15.31:0/3751910008 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc003d80 con 0x7fd4d41499a0 2024-09-11T00:56:30.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.349+0000 7fd4d0c00640 1 -- 172.21.15.31:0/3751910008 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4cc004b20 con 0x7fd4d41499a0 2024-09-11T00:56:30.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.349+0000 7fd4d0c00640 1 -- 172.21.15.31:0/3751910008 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc004030 con 0x7fd4d41499a0 2024-09-11T00:56:30.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.349+0000 7fd4daa00640 1 -- 172.21.15.31:0/3751910008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d4149e30 con 0x7fd4d41499a0 2024-09-11T00:56:30.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.349+0000 7fd4daa00640 1 -- 172.21.15.31:0/3751910008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d41ef250 con 0x7fd4d41499a0 2024-09-11T00:56:30.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.350+0000 7fd4daa00640 1 -- 172.21.15.31:0/3751910008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d406dfe0 con 0x7fd4d41499a0 2024-09-11T00:56:30.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.351+0000 7fd4d0c00640 1 -- 172.21.15.31:0/3751910008 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd4cc018030 con 0x7fd4d41499a0 2024-09-11T00:56:30.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.353+0000 7fd4d0c00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4b0077fa0 0x7fd4b007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.353+0000 7fd4d0c00640 1 -- 172.21.15.31:0/3751910008 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fd4cc09b820 con 0x7fd4d41499a0 2024-09-11T00:56:30.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.353+0000 7fd4d3e00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4b0077fa0 0x7fd4b007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.354+0000 7fd4d3e00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4b0077fa0 0x7fd4b007a460 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4035000 tx=0x7fd4c402d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:30.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.357+0000 7fd4d0c00640 1 -- 172.21.15.31:0/3751910008 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd4cc00de20 con 0x7fd4d41499a0 2024-09-11T00:56:30.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.477+0000 7fd0ed400640 1 -- 172.21.15.31:0/111082252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0e80c2e90 con 0x7fd0e8074f50 2024-09-11T00:56:30.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.477+0000 7fd0dea00640 1 -- 172.21.15.31:0/111082252 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+5712 (secure 0 0 0) 0x7fd0d009ae20 con 0x7fd0e8074f50 2024-09-11T00:56:30.480 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:30.480 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":184,"btime":"2024-09-11T00:56:30:045869+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:29.049168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:30.481 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 184 2024-09-11T00:56:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 -- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0c4077f70 msgr2=0x7fd0c407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0c4077f70 0x7fd0c407a430 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8034000 tx=0x7fd0d8004730 comp rx=0 tx=0).stop 2024-09-11T00:56:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 -- 172.21.15.31:0/111082252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e8074f50 msgr2=0x7fd0e806fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e8074f50 0x7fd0e806fc00 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000c4d0 tx=0x7fd0d000c990 comp rx=0 tx=0).stop 2024-09-11T00:56:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 -- 172.21.15.31:0/111082252 shutdown_connections 2024-09-11T00:56:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd0c4077f70 0x7fd0c407a430 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd0e80abde0 0x7fd0e8145e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0e8074f50 0x7fd0e806fc00 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 --2- 172.21.15.31:0/111082252 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd0e8074630 0x7fd0e80bd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 -- 172.21.15.31:0/111082252 >> 172.21.15.31:0/111082252 conn(0x7fd0e80c8320 msgr2=0x7fd0e80c5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 -- 172.21.15.31:0/111082252 shutdown_connections 2024-09-11T00:56:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.480+0000 7fd0ed400640 1 -- 172.21.15.31:0/111082252 wait complete. 2024-09-11T00:56:30.500 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-11T00:56:30.500 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:56:30.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:30.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.595+0000 7fd4daa00640 1 -- 172.21.15.31:0/3751910008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4d415a1a0 con 0x7fd4d41499a0 2024-09-11T00:56:30.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.596+0000 7fd4d0c00640 1 -- 172.21.15.31:0/3751910008 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+5712 (secure 0 0 0) 0x7fd4cc063300 con 0x7fd4d41499a0 2024-09-11T00:56:30.599 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:30.599 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":184,"btime":"2024-09-11T00:56:30:045869+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:29.049168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:30.599 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 184 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 -- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4b0077fa0 msgr2=0x7fd4b007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4b0077fa0 0x7fd4b007a460 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4035000 tx=0x7fd4c402d040 comp rx=0 tx=0).stop 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 -- 172.21.15.31:0/3751910008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4d41499a0 msgr2=0x7fd4d41ebb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4d41499a0 0x7fd4d41ebb70 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40cbbd0 tx=0x7fd4cc00f790 comp rx=0 tx=0).stop 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 -- 172.21.15.31:0/3751910008 shutdown_connections 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd4b0077fa0 0x7fd4b007a460 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4d41499a0 0x7fd4d41ebb70 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4d414d6a0 0x7fd4d4149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 --2- 172.21.15.31:0/3751910008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4b8003a00 0x7fd4d414a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 -- 172.21.15.31:0/3751910008 >> 172.21.15.31:0/3751910008 conn(0x7fd4d40cc260 msgr2=0x7fd4d4150290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 -- 172.21.15.31:0/3751910008 shutdown_connections 2024-09-11T00:56:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.599+0000 7fd4daa00640 1 -- 172.21.15.31:0/3751910008 wait complete. 2024-09-11T00:56:30.617 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:56:30.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.663+0000 7f91fe200640 1 -- 172.21.15.31:0/2026052118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f80c3ee0 msgr2=0x7f91f8156400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.663+0000 7f91fe200640 1 --2- 172.21.15.31:0/2026052118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f80c3ee0 0x7f91f8156400 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f91f0005fd0 tx=0x7f91f002a810 comp rx=0 tx=0).stop 2024-09-11T00:56:30.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.663+0000 7f91fe200640 1 -- 172.21.15.31:0/2026052118 shutdown_connections 2024-09-11T00:56:30.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.663+0000 7f91fe200640 1 --2- 172.21.15.31:0/2026052118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f80c3ee0 0x7f91f8156400 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.663+0000 7f91fe200640 1 --2- 172.21.15.31:0/2026052118 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91f8073b70 0x7f91f80c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.663+0000 7f91fe200640 1 -- 172.21.15.31:0/2026052118 >> 172.21.15.31:0/2026052118 conn(0x7f91f80aa760 msgr2=0x7f91f80aab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:30.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.663+0000 7f91fe200640 1 -- 172.21.15.31:0/2026052118 shutdown_connections 2024-09-11T00:56:30.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.663+0000 7f91fe200640 1 -- 172.21.15.31:0/2026052118 wait complete. 2024-09-11T00:56:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.664+0000 7f91fe200640 1 Processor -- start 2024-09-11T00:56:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.664+0000 7f91fe200640 1 -- start start 2024-09-11T00:56:30.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.665+0000 7f91fe200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f8073b70 0x7f91f81e0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.665+0000 7f91fe200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91f80c3ee0 0x7f91f80adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.665+0000 7f91fe200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91f80c7fe0 0x7f91f81ead80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.665+0000 7f91fe200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f91f8074050 con 0x7f91f8073b70 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.665+0000 7f91fe200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f91f80c7800 con 0x7f91f80c3ee0 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.665+0000 7f91fe200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f91f80c83c0 con 0x7f91f80c7fe0 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91fce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f8073b70 0x7f91f81e0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91fce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f8073b70 0x7f91f81e0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39170/0 (socket says 172.21.15.31:39170) 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91fce00640 1 -- 172.21.15.31:0/1775636063 learned_addr learned my addr 172.21.15.31:0/1775636063 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91fd800640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91f80c7fe0 0x7f91f81ead80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91f7e00640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91f80c3ee0 0x7f91f80adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91fce00640 1 -- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91f80c7fe0 msgr2=0x7f91f81ead80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91fce00640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91f80c7fe0 0x7f91f81ead80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91fce00640 1 -- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91f80c3ee0 msgr2=0x7f91f80adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91fce00640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91f80c3ee0 0x7f91f80adc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91fce00640 1 -- 172.21.15.31:0/1775636063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f81dd350 con 0x7f91f8073b70 2024-09-11T00:56:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.666+0000 7f91fce00640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f8073b70 0x7f91f81e0ee0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00c3c0 tx=0x7f91ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.667+0000 7f91f4c00640 1 -- 172.21.15.31:0/1775636063 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec017070 con 0x7f91f8073b70 2024-09-11T00:56:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.667+0000 7f91f4c00640 1 -- 172.21.15.31:0/1775636063 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91ec0035f0 con 0x7f91f8073b70 2024-09-11T00:56:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.667+0000 7f91f4c00640 1 -- 172.21.15.31:0/1775636063 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec003a80 con 0x7f91f8073b70 2024-09-11T00:56:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.667+0000 7f91fe200640 1 -- 172.21.15.31:0/1775636063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f81dd070 con 0x7f91f8073b70 2024-09-11T00:56:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.667+0000 7f91fe200640 1 -- 172.21.15.31:0/1775636063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f81dd930 con 0x7f91f8073b70 2024-09-11T00:56:30.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.668+0000 7f91fe200640 1 -- 172.21.15.31:0/1775636063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91f81546a0 con 0x7f91f8073b70 2024-09-11T00:56:30.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.669+0000 7f91f4c00640 1 -- 172.21.15.31:0/1775636063 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f91ec003c20 con 0x7f91f8073b70 2024-09-11T00:56:30.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.670+0000 7f91f4c00640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f91d8077f70 0x7f91d807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:30.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.670+0000 7f91f4c00640 1 -- 172.21.15.31:0/1775636063 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f91ec09a2b0 con 0x7f91f8073b70 2024-09-11T00:56:30.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.670+0000 7f91f7e00640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f91d8077f70 0x7f91d807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:30.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.671+0000 7f91f7e00640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f91d8077f70 0x7f91d807a430 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f91f8070980 tx=0x7f91e4005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:30.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.674+0000 7f91f4c00640 1 -- 172.21.15.31:0/1775636063 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f91f81546a0 con 0x7f91f8073b70 2024-09-11T00:56:30.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.907+0000 7f91fe200640 1 -- 172.21.15.31:0/1775636063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91f81ddd10 con 0x7f91f8073b70 2024-09-11T00:56:30.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.908+0000 7f91f4c00640 1 -- 172.21.15.31:0/1775636063 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+5712 (secure 0 0 0) 0x7f91f81ddd10 con 0x7f91f8073b70 2024-09-11T00:56:30.911 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:30.911 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":184,"btime":"2024-09-11T00:56:30:045869+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:29.049168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:30.911 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 184 2024-09-11T00:56:30.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.910+0000 7f91fe200640 1 -- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f91d8077f70 msgr2=0x7f91d807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.910+0000 7f91fe200640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f91d8077f70 0x7f91d807a430 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f91f8070980 tx=0x7f91e4005f90 comp rx=0 tx=0).stop 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.910+0000 7f91fe200640 1 -- 172.21.15.31:0/1775636063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f8073b70 msgr2=0x7f91f81e0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.910+0000 7f91fe200640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f8073b70 0x7f91f81e0ee0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00c3c0 tx=0x7f91ec00c880 comp rx=0 tx=0).stop 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.911+0000 7f91fe200640 1 -- 172.21.15.31:0/1775636063 shutdown_connections 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.911+0000 7f91fe200640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f91d8077f70 0x7f91d807a430 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.911+0000 7f91fe200640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f91f80c7fe0 0x7f91f81ead80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.911+0000 7f91fe200640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91f80c3ee0 0x7f91f80adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.911+0000 7f91fe200640 1 --2- 172.21.15.31:0/1775636063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91f8073b70 0x7f91f81e0ee0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.911+0000 7f91fe200640 1 -- 172.21.15.31:0/1775636063 >> 172.21.15.31:0/1775636063 conn(0x7f91f80aa760 msgr2=0x7f91f8152d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.911+0000 7f91fe200640 1 -- 172.21.15.31:0/1775636063 shutdown_connections 2024-09-11T00:56:30.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:30.911+0000 7f91fe200640 1 -- 172.21.15.31:0/1775636063 wait complete. 2024-09-11T00:56:30.930 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:56:30:045869+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 184, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 184, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7672": { "addr": "172.21.15.204:6837/1329541959", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6836", "nonce": 1329541959, "type": "v2" }, { "addr": "172.21.15.204:6837", "nonce": 1329541959, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7672, "incarnation": 182, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:56:29.049168+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7672 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 180, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6593": { "addr": "172.21.15.204:6835/3620025884", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 3620025884, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 3620025884, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6593, "incarnation": 177, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:56:17.960286+0000", "qdb_cluster": [ 6593 ], "qdb_leader": 6593, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6593 } } } ], "standbys": [ { "addr": "172.21.15.31:6835/2337200751", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 2337200751, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 2337200751, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 166, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6503, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:56:31.085 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T00:56:31.085 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-09-11T00:56:31.148 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-09-11T00:56:31.148 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-09-11T00:56:31.158 INFO:tasks.mds_thrash:joining mds_thrasher 2024-09-11T00:56:31.159 ERROR:tasks.mds_thrash.fs.[b]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks/mds_thrash.py", line 121, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks/mds_thrash.py", line 329, in do_thrash self.sleep_unless_stopped(2) File "/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks/thrasher.py", line 50, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-11T00:56:31.159 INFO:tasks.mds_thrash:done joining 2024-09-11T00:56:31.159 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-11T00:56:31.168 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-11T00:56:31.169 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:56:31.169 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-11T00:56:31.204 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:56:31.204 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-11T00:56:31.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-11T00:56:31.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.477+0000 7f18a9a00640 1 -- 172.21.15.31:0/1094378586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18a4150020 msgr2=0x7f18a40ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:31.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.477+0000 7f18a9a00640 1 --2- 172.21.15.31:0/1094378586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18a4150020 0x7f18a40ad4e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f18a4153d20 tx=0x7f189402a510 comp rx=0 tx=0).stop 2024-09-11T00:56:31.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.477+0000 7f18a9a00640 1 -- 172.21.15.31:0/1094378586 shutdown_connections 2024-09-11T00:56:31.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.477+0000 7f18a9a00640 1 --2- 172.21.15.31:0/1094378586 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a406e850 0x7f18a4160860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.477+0000 7f18a9a00640 1 --2- 172.21.15.31:0/1094378586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18a4150020 0x7f18a40ad4e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.477+0000 7f18a9a00640 1 -- 172.21.15.31:0/1094378586 >> 172.21.15.31:0/1094378586 conn(0x7f18a4068f10 msgr2=0x7f18a40cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:31.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.477+0000 7f18a9a00640 1 -- 172.21.15.31:0/1094378586 shutdown_connections 2024-09-11T00:56:31.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.477+0000 7f18a9a00640 1 -- 172.21.15.31:0/1094378586 wait complete. 2024-09-11T00:56:31.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.479+0000 7f18a9a00640 1 Processor -- start 2024-09-11T00:56:31.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.479+0000 7f18a9a00640 1 -- start start 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a9a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a406e850 0x7f18a41556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a9a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18a40ada20 0x7f18a414ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a9a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18a4150020 0x7f18a414b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a9a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f18a4157550 con 0x7f18a40ada20 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a9a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18a41573d0 con 0x7f18a4150020 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a9a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f18a4156960 con 0x7f18a406e850 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18a40ada20 0x7f18a414ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18a40ada20 0x7f18a414ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39176/0 (socket says 172.21.15.31:39176) 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a3400640 1 -- 172.21.15.31:0/3757350415 learned_addr learned my addr 172.21.15.31:0/3757350415 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a3e00640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a406e850 0x7f18a41556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:31.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a9000640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18a4150020 0x7f18a414b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:31.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a3400640 1 -- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a406e850 msgr2=0x7f18a41556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:31.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a3400640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a406e850 0x7f18a41556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a3400640 1 -- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18a4150020 msgr2=0x7f18a414b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.480+0000 7f18a3400640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18a4150020 0x7f18a414b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.481+0000 7f18a3400640 1 -- 172.21.15.31:0/3757350415 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a40ae680 con 0x7f18a40ada20 2024-09-11T00:56:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.481+0000 7f18a3400640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18a40ada20 0x7f18a414ab40 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f189000c450 tx=0x7f189000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.481+0000 7f189be00640 1 -- 172.21.15.31:0/3757350415 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890016020 con 0x7f18a40ada20 2024-09-11T00:56:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.481+0000 7f189be00640 1 -- 172.21.15.31:0/3757350415 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1890012920 con 0x7f18a40ada20 2024-09-11T00:56:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.481+0000 7f189be00640 1 -- 172.21.15.31:0/3757350415 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890012d70 con 0x7f18a40ada20 2024-09-11T00:56:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.481+0000 7f18a9a00640 1 -- 172.21.15.31:0/3757350415 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a40ad130 con 0x7f18a40ada20 2024-09-11T00:56:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.481+0000 7f18a9a00640 1 -- 172.21.15.31:0/3757350415 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a40aec20 con 0x7f18a40ada20 2024-09-11T00:56:31.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.482+0000 7f18a9a00640 1 -- 172.21.15.31:0/3757350415 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a40cc220 con 0x7f18a40ada20 2024-09-11T00:56:31.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.484+0000 7f189be00640 1 -- 172.21.15.31:0/3757350415 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f189001b400 con 0x7f18a40ada20 2024-09-11T00:56:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.484+0000 7f189be00640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1884077fa0 0x7f188407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.485+0000 7f189be00640 1 -- 172.21.15.31:0/3757350415 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f189009af50 con 0x7f18a40ada20 2024-09-11T00:56:31.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.485+0000 7f18a3e00640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1884077fa0 0x7f188407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:31.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.486+0000 7f18a3e00640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1884077fa0 0x7f188407a460 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1894034000 tx=0x7f1894004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:31.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.489+0000 7f189be00640 1 -- 172.21.15.31:0/3757350415 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1890020020 con 0x7f18a40ada20 2024-09-11T00:56:31.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:56:31.577 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:56:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.700+0000 7f18a9a00640 1 -- 172.21.15.31:0/3757350415 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f18a40cc8e0 con 0x7f18a40ada20 2024-09-11T00:56:31.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.701+0000 7f189be00640 1 -- 172.21.15.31:0/3757350415 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 70 entries v62) ==== 81+0+4300 (secure 0 0 0) 0x7f1890062340 con 0x7f18a40ada20 2024-09-11T00:56:31.703 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/3721481399 2024-09-12T00:56:28.030054+0000 2024-09-11T00:56:31.703 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/4094068967 2024-09-12T00:55:04.553768+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/2956367495 2024-09-12T00:54:19.542253+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2689217598 2024-09-12T00:56:14.574459+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/1855324855 2024-09-12T00:52:49.521133+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1981755707 2024-09-12T00:52:39.517968+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/98007221 2024-09-12T00:41:49.383893+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/3881548149 2024-09-12T00:40:39.371058+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2163558126 2024-09-12T00:48:14.453181+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/3506150555 2024-09-12T00:40:24.367167+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/1278135080 2024-09-12T00:38:14.342086+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/2956367495 2024-09-12T00:54:19.542253+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2909466339 2024-09-12T00:43:22.735779+0000 2024-09-11T00:56:31.704 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/1278135080 2024-09-12T00:38:14.342086+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/3444890210 2024-09-12T00:37:16.369607+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/4094068967 2024-09-12T00:55:04.553768+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/645869523 2024-09-12T00:35:54.319031+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/4164016077 2024-09-12T00:50:14.476294+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2323279595 2024-09-12T00:44:44.412976+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/3506150555 2024-09-12T00:40:24.367167+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4011734390 2024-09-12T00:37:09.330669+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/3725011623 2024-09-12T00:50:23.114454+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/3441932736 2024-09-12T00:31:44.277804+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3721481399 2024-09-12T00:56:28.030054+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/2686855151 2024-09-12T00:32:24.285279+0000 2024-09-11T00:56:31.705 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/1082799504 2024-09-12T00:47:59.449917+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1981755707 2024-09-12T00:52:39.517968+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/645869523 2024-09-12T00:35:54.319031+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3925687919 2024-09-12T00:41:09.376771+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/558470224 2024-09-12T00:34:09.302673+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1801149506 2024-09-12T00:32:54.290770+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4202305151 2024-09-12T00:35:09.311954+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/3441932736 2024-09-12T00:31:44.277804+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/1082799504 2024-09-12T00:47:59.449917+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3242133790 2024-09-12T00:38:14.342154+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/3881548149 2024-09-12T00:40:39.371058+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1801149506 2024-09-12T00:32:54.290770+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4202305151 2024-09-12T00:35:09.311954+0000 2024-09-11T00:56:31.706 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2954547253 2024-09-12T00:33:34.297314+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1348312176 2024-09-12T00:45:14.418207+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/3444890210 2024-09-12T00:37:16.369607+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2689217598 2024-09-12T00:56:14.574459+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3925687919 2024-09-12T00:41:09.376771+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4011734390 2024-09-12T00:37:09.330669+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/1396366029 2024-09-12T00:45:59.426330+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2954547253 2024-09-12T00:33:34.297314+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/558470224 2024-09-12T00:34:09.302673+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/1855324855 2024-09-12T00:52:49.521133+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/98007221 2024-09-12T00:41:49.383893+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/1396366029 2024-09-12T00:45:59.426330+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/2909466339 2024-09-12T00:43:22.735779+0000 2024-09-11T00:56:31.707 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2630216714 2024-09-12T00:52:19.506830+0000 2024-09-11T00:56:31.708 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1575320768 2024-09-12T00:44:14.407953+0000 2024-09-11T00:56:31.708 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2630216714 2024-09-12T00:52:19.506830+0000 2024-09-11T00:56:31.708 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2686855151 2024-09-12T00:32:24.285279+0000 2024-09-11T00:56:31.708 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1575320768 2024-09-12T00:44:14.407953+0000 2024-09-11T00:56:31.708 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1348312176 2024-09-12T00:45:14.418207+0000 2024-09-11T00:56:31.708 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/3513641573 2024-09-12T00:45:26.774916+0000 2024-09-11T00:56:31.708 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4205320530 2024-09-12T00:43:19.398942+0000 2024-09-11T00:56:31.708 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/2113744259 2024-09-12T00:47:19.441133+0000 2024-09-11T00:56:31.708 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/2113744259 2024-09-12T00:47:19.441133+0000 2024-09-11T00:56:31.709 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2323279595 2024-09-12T00:44:44.412976+0000 2024-09-11T00:56:31.709 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/3513641573 2024-09-12T00:45:26.774916+0000 2024-09-11T00:56:31.709 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2163558126 2024-09-12T00:48:14.453181+0000 2024-09-11T00:56:31.709 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/634556689 2024-09-12T00:49:34.468674+0000 2024-09-11T00:56:31.709 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/634556689 2024-09-12T00:49:34.468674+0000 2024-09-11T00:56:31.709 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3725011623 2024-09-12T00:50:23.114454+0000 2024-09-11T00:56:31.709 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3242133790 2024-09-12T00:38:14.342154+0000 2024-09-11T00:56:31.709 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4205320530 2024-09-12T00:43:19.398942+0000 2024-09-11T00:56:31.709 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/4164016077 2024-09-12T00:50:14.476294+0000 2024-09-11T00:56:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:listed 70 entries 2024-09-11T00:56:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 -- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1884077fa0 msgr2=0x7f188407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1884077fa0 0x7f188407a460 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1894034000 tx=0x7f1894004730 comp rx=0 tx=0).stop 2024-09-11T00:56:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 -- 172.21.15.31:0/3757350415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18a40ada20 msgr2=0x7f18a414ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18a40ada20 0x7f18a414ab40 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f189000c450 tx=0x7f189000c910 comp rx=0 tx=0).stop 2024-09-11T00:56:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 -- 172.21.15.31:0/3757350415 shutdown_connections 2024-09-11T00:56:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1884077fa0 0x7f188407a460 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18a4150020 0x7f18a414b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18a40ada20 0x7f18a414ab40 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 --2- 172.21.15.31:0/3757350415 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f18a406e850 0x7f18a41556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 -- 172.21.15.31:0/3757350415 >> 172.21.15.31:0/3757350415 conn(0x7f18a4068f10 msgr2=0x7f18a414cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:31.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 -- 172.21.15.31:0/3757350415 shutdown_connections 2024-09-11T00:56:31.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.703+0000 7f18a9a00640 1 -- 172.21.15.31:0/3757350415 wait complete. 2024-09-11T00:56:31.723 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:56:31.723 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-11T00:56:31.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-11T00:56:31.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.957+0000 7fb0e0a00640 1 -- 172.21.15.31:0/4162189019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0dc072480 msgr2=0x7fb0dc1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:31.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.957+0000 7fb0e0a00640 1 --2- 172.21.15.31:0/4162189019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0dc072480 0x7fb0dc1511a0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc075260 tx=0x7fb0cc02a510 comp rx=0 tx=0).stop 2024-09-11T00:56:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.957+0000 7fb0e0a00640 1 -- 172.21.15.31:0/4162189019 shutdown_connections 2024-09-11T00:56:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.957+0000 7fb0e0a00640 1 --2- 172.21.15.31:0/4162189019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0dc072480 0x7fb0dc1511a0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.957+0000 7fb0e0a00640 1 -- 172.21.15.31:0/4162189019 >> 172.21.15.31:0/4162189019 conn(0x7fb0dc0a9750 msgr2=0x7fb0dc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.957+0000 7fb0e0a00640 1 -- 172.21.15.31:0/4162189019 shutdown_connections 2024-09-11T00:56:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.957+0000 7fb0e0a00640 1 -- 172.21.15.31:0/4162189019 wait complete. 2024-09-11T00:56:31.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.959+0000 7fb0e0a00640 1 Processor -- start 2024-09-11T00:56:31.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.959+0000 7fb0e0a00640 1 -- start start 2024-09-11T00:56:31.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0e0a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0dc072480 0x7fb0dc06f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0e0a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0dc0c23a0 0x7fb0dc0bef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0e0a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0dc153f20 0x7fb0dc0c07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0e0a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc074200 con 0x7fb0dc072480 2024-09-11T00:56:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0e0a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb0dc074080 con 0x7fb0dc153f20 2024-09-11T00:56:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0e0a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb0dc072860 con 0x7fb0dc0c23a0 2024-09-11T00:56:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0d9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0dc072480 0x7fb0dc06f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0d9e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0dc072480 0x7fb0dc06f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39190/0 (socket says 172.21.15.31:39190) 2024-09-11T00:56:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0d9e00640 1 -- 172.21.15.31:0/867678845 learned_addr learned my addr 172.21.15.31:0/867678845 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0da800640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0dc153f20 0x7fb0dc0c07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0d1400640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0dc0c23a0 0x7fb0dc0bef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0d9e00640 1 -- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0dc0c23a0 msgr2=0x7fb0dc0bef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0d9e00640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0dc0c23a0 0x7fb0dc0bef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0d9e00640 1 -- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0dc153f20 msgr2=0x7fb0dc0c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0d9e00640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0dc153f20 0x7fb0dc0c07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.960+0000 7fb0d9e00640 1 -- 172.21.15.31:0/867678845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0dc15e8f0 con 0x7fb0dc072480 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.961+0000 7fb0d9e00640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0dc072480 0x7fb0dc06f5f0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009780 tx=0x7fb0cc032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.961+0000 7fb0d2a00640 1 -- 172.21.15.31:0/867678845 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc040070 con 0x7fb0dc072480 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.961+0000 7fb0d2a00640 1 -- 172.21.15.31:0/867678845 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0cc005310 con 0x7fb0dc072480 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.961+0000 7fb0d2a00640 1 -- 172.21.15.31:0/867678845 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc003800 con 0x7fb0dc072480 2024-09-11T00:56:31.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.961+0000 7fb0e0a00640 1 -- 172.21.15.31:0/867678845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0dc0bded0 con 0x7fb0dc072480 2024-09-11T00:56:31.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.961+0000 7fb0e0a00640 1 -- 172.21.15.31:0/867678845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0dc0be490 con 0x7fb0dc072480 2024-09-11T00:56:31.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.962+0000 7fb0e0a00640 1 -- 172.21.15.31:0/867678845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0dc1511a0 con 0x7fb0dc072480 2024-09-11T00:56:31.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.963+0000 7fb0d2a00640 1 -- 172.21.15.31:0/867678845 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb0cc0039a0 con 0x7fb0dc072480 2024-09-11T00:56:31.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.964+0000 7fb0d2a00640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb0b8077f70 0x7fb0b807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:31.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.964+0000 7fb0d2a00640 1 -- 172.21.15.31:0/867678845 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fb0cc0bc260 con 0x7fb0dc072480 2024-09-11T00:56:31.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.965+0000 7fb0d1400640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb0b8077f70 0x7fb0b807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.965+0000 7fb0d1400640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb0b8077f70 0x7fb0b807a430 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0c45a0 tx=0x7fb0c4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:31.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:31.968+0000 7fb0d2a00640 1 -- 172.21.15.31:0/867678845 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb0cc083dc0 con 0x7fb0dc072480 2024-09-11T00:56:32.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.179+0000 7fb0e0a00640 1 -- 172.21.15.31:0/867678845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb0dc0c42f0 con 0x7fb0dc072480 2024-09-11T00:56:32.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.179+0000 7fb0d2a00640 1 -- 172.21.15.31:0/867678845 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 70 entries v62) ==== 81+0+4300 (secure 0 0 0) 0x7fb0cc0836c0 con 0x7fb0dc072480 2024-09-11T00:56:32.182 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/3721481399 2024-09-12T00:56:28.030054+0000 2024-09-11T00:56:32.182 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/4094068967 2024-09-12T00:55:04.553768+0000 2024-09-11T00:56:32.183 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/2956367495 2024-09-12T00:54:19.542253+0000 2024-09-11T00:56:32.183 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2689217598 2024-09-12T00:56:14.574459+0000 2024-09-11T00:56:32.183 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/1855324855 2024-09-12T00:52:49.521133+0000 2024-09-11T00:56:32.183 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1981755707 2024-09-12T00:52:39.517968+0000 2024-09-11T00:56:32.183 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/98007221 2024-09-12T00:41:49.383893+0000 2024-09-11T00:56:32.183 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/3881548149 2024-09-12T00:40:39.371058+0000 2024-09-11T00:56:32.183 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2163558126 2024-09-12T00:48:14.453181+0000 2024-09-11T00:56:32.183 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/3506150555 2024-09-12T00:40:24.367167+0000 2024-09-11T00:56:32.184 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/1278135080 2024-09-12T00:38:14.342086+0000 2024-09-11T00:56:32.184 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/2956367495 2024-09-12T00:54:19.542253+0000 2024-09-11T00:56:32.184 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2909466339 2024-09-12T00:43:22.735779+0000 2024-09-11T00:56:32.184 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/1278135080 2024-09-12T00:38:14.342086+0000 2024-09-11T00:56:32.184 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/3444890210 2024-09-12T00:37:16.369607+0000 2024-09-11T00:56:32.184 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/4094068967 2024-09-12T00:55:04.553768+0000 2024-09-11T00:56:32.184 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/645869523 2024-09-12T00:35:54.319031+0000 2024-09-11T00:56:32.184 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/4164016077 2024-09-12T00:50:14.476294+0000 2024-09-11T00:56:32.184 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2323279595 2024-09-12T00:44:44.412976+0000 2024-09-11T00:56:32.184 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/3506150555 2024-09-12T00:40:24.367167+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4011734390 2024-09-12T00:37:09.330669+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/3725011623 2024-09-12T00:50:23.114454+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/3441932736 2024-09-12T00:31:44.277804+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3721481399 2024-09-12T00:56:28.030054+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/2686855151 2024-09-12T00:32:24.285279+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/1082799504 2024-09-12T00:47:59.449917+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1981755707 2024-09-12T00:52:39.517968+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/645869523 2024-09-12T00:35:54.319031+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3925687919 2024-09-12T00:41:09.376771+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/558470224 2024-09-12T00:34:09.302673+0000 2024-09-11T00:56:32.185 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1801149506 2024-09-12T00:32:54.290770+0000 2024-09-11T00:56:32.186 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4202305151 2024-09-12T00:35:09.311954+0000 2024-09-11T00:56:32.186 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/3441932736 2024-09-12T00:31:44.277804+0000 2024-09-11T00:56:32.186 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/1082799504 2024-09-12T00:47:59.449917+0000 2024-09-11T00:56:32.186 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3242133790 2024-09-12T00:38:14.342154+0000 2024-09-11T00:56:32.186 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/3881548149 2024-09-12T00:40:39.371058+0000 2024-09-11T00:56:32.186 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1801149506 2024-09-12T00:32:54.290770+0000 2024-09-11T00:56:32.186 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4202305151 2024-09-12T00:35:09.311954+0000 2024-09-11T00:56:32.186 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2954547253 2024-09-12T00:33:34.297314+0000 2024-09-11T00:56:32.186 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1348312176 2024-09-12T00:45:14.418207+0000 2024-09-11T00:56:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/3444890210 2024-09-12T00:37:16.369607+0000 2024-09-11T00:56:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2689217598 2024-09-12T00:56:14.574459+0000 2024-09-11T00:56:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3925687919 2024-09-12T00:41:09.376771+0000 2024-09-11T00:56:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4011734390 2024-09-12T00:37:09.330669+0000 2024-09-11T00:56:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/1396366029 2024-09-12T00:45:59.426330+0000 2024-09-11T00:56:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2954547253 2024-09-12T00:33:34.297314+0000 2024-09-11T00:56:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/558470224 2024-09-12T00:34:09.302673+0000 2024-09-11T00:56:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/1855324855 2024-09-12T00:52:49.521133+0000 2024-09-11T00:56:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/98007221 2024-09-12T00:41:49.383893+0000 2024-09-11T00:56:32.187 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/1396366029 2024-09-12T00:45:59.426330+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/2909466339 2024-09-12T00:43:22.735779+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2630216714 2024-09-12T00:52:19.506830+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1575320768 2024-09-12T00:44:14.407953+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2630216714 2024-09-12T00:52:19.506830+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2686855151 2024-09-12T00:32:24.285279+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1575320768 2024-09-12T00:44:14.407953+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1348312176 2024-09-12T00:45:14.418207+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/3513641573 2024-09-12T00:45:26.774916+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4205320530 2024-09-12T00:43:19.398942+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/2113744259 2024-09-12T00:47:19.441133+0000 2024-09-11T00:56:32.188 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/2113744259 2024-09-12T00:47:19.441133+0000 2024-09-11T00:56:32.189 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2323279595 2024-09-12T00:44:44.412976+0000 2024-09-11T00:56:32.189 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/3513641573 2024-09-12T00:45:26.774916+0000 2024-09-11T00:56:32.189 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2163558126 2024-09-12T00:48:14.453181+0000 2024-09-11T00:56:32.189 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/634556689 2024-09-12T00:49:34.468674+0000 2024-09-11T00:56:32.189 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/634556689 2024-09-12T00:49:34.468674+0000 2024-09-11T00:56:32.189 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3725011623 2024-09-12T00:50:23.114454+0000 2024-09-11T00:56:32.189 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3242133790 2024-09-12T00:38:14.342154+0000 2024-09-11T00:56:32.189 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4205320530 2024-09-12T00:43:19.398942+0000 2024-09-11T00:56:32.189 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/4164016077 2024-09-12T00:50:14.476294+0000 2024-09-11T00:56:32.190 INFO:teuthology.orchestra.run.smithi031.stderr:listed 70 entries 2024-09-11T00:56:32.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 -- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb0b8077f70 msgr2=0x7fb0b807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:32.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb0b8077f70 0x7fb0b807a430 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0c45a0 tx=0x7fb0c4009040 comp rx=0 tx=0).stop 2024-09-11T00:56:32.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 -- 172.21.15.31:0/867678845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0dc072480 msgr2=0x7fb0dc06f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:32.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0dc072480 0x7fb0dc06f5f0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009780 tx=0x7fb0cc032d50 comp rx=0 tx=0).stop 2024-09-11T00:56:32.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 -- 172.21.15.31:0/867678845 shutdown_connections 2024-09-11T00:56:32.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb0b8077f70 0x7fb0b807a430 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:32.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0dc153f20 0x7fb0dc0c07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:32.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb0dc0c23a0 0x7fb0dc0bef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:32.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 --2- 172.21.15.31:0/867678845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0dc072480 0x7fb0dc06f5f0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:32.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 -- 172.21.15.31:0/867678845 >> 172.21.15.31:0/867678845 conn(0x7fb0dc0a9750 msgr2=0x7fb0dc0c30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:32.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 -- 172.21.15.31:0/867678845 shutdown_connections 2024-09-11T00:56:32.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:32.182+0000 7fb0e0a00640 1 -- 172.21.15.31:0/867678845 wait complete. 2024-09-11T00:56:32.202 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi031.front.sepia.ceph.com... 2024-09-11T00:56:32.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:56:32.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-11T00:56:32.269 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:56:32.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi031.stderr:ceph-fuse[45156]: fuse finished with error 0 and tester_r 0 2024-09-11T00:56:32.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:33.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.105+0000 7ff7c2600640 1 -- 172.21.15.31:0/2189732009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7bc1539c0 msgr2=0x7ff7bc159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:33.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.105+0000 7ff7c2600640 1 --2- 172.21.15.31:0/2189732009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7bc1539c0 0x7ff7bc159670 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4005fd0 tx=0x7ff7b402aa90 comp rx=0 tx=0).stop 2024-09-11T00:56:33.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.105+0000 7ff7c2600640 1 -- 172.21.15.31:0/2189732009 shutdown_connections 2024-09-11T00:56:33.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.105+0000 7ff7c2600640 1 --2- 172.21.15.31:0/2189732009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7bc1539c0 0x7ff7bc159670 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:33.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.105+0000 7ff7c2600640 1 --2- 172.21.15.31:0/2189732009 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff7bc156ad0 0x7ff7bc153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:33.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.105+0000 7ff7c2600640 1 -- 172.21.15.31:0/2189732009 >> 172.21.15.31:0/2189732009 conn(0x7ff7bc0a9750 msgr2=0x7ff7bc0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:33.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.105+0000 7ff7c2600640 1 -- 172.21.15.31:0/2189732009 shutdown_connections 2024-09-11T00:56:33.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.105+0000 7ff7c2600640 1 -- 172.21.15.31:0/2189732009 wait complete. 2024-09-11T00:56:33.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.107+0000 7ff7c2600640 1 Processor -- start 2024-09-11T00:56:33.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.107+0000 7ff7c2600640 1 -- start start 2024-09-11T00:56:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7c2600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7bc0c5b10 0x7ff7bc1e14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7c2600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff7bc1539c0 0x7ff7bc1efd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7c2600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7bc156ad0 0x7ff7bc14cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7c2600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc156ef0 con 0x7ff7bc156ad0 2024-09-11T00:56:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7c2600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc0c7320 con 0x7ff7bc0c5b10 2024-09-11T00:56:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7c2600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff7bc0c3950 con 0x7ff7bc1539c0 2024-09-11T00:56:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7bbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7bc156ad0 0x7ff7bc14cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7bbe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7bc156ad0 0x7ff7bc14cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39212/0 (socket says 172.21.15.31:39212) 2024-09-11T00:56:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7bbe00640 1 -- 172.21.15.31:0/1867374163 learned_addr learned my addr 172.21.15.31:0/1867374163 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:33.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7bb400640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7bc0c5b10 0x7ff7bc1e14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:33.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.108+0000 7ff7b2a00640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff7bc1539c0 0x7ff7bc1efd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:33.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7bbe00640 1 -- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff7bc1539c0 msgr2=0x7ff7bc1efd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:33.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7bbe00640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff7bc1539c0 0x7ff7bc1efd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:33.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7bbe00640 1 -- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7bc0c5b10 msgr2=0x7ff7bc1e14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:33.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7bbe00640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7bc0c5b10 0x7ff7bc1e14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:33.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7bbe00640 1 -- 172.21.15.31:0/1867374163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7bc151320 con 0x7ff7bc156ad0 2024-09-11T00:56:33.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7bbe00640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7bc156ad0 0x7ff7bc14cbf0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4006010 tx=0x7ff7b4003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:33.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7b3e00640 1 -- 172.21.15.31:0/1867374163 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4046070 con 0x7ff7bc156ad0 2024-09-11T00:56:33.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7b3e00640 1 -- 172.21.15.31:0/1867374163 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7b4004040 con 0x7ff7bc156ad0 2024-09-11T00:56:33.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7c2600640 1 -- 172.21.15.31:0/1867374163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7bc14f3e0 con 0x7ff7bc156ad0 2024-09-11T00:56:33.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7b3e00640 1 -- 172.21.15.31:0/1867374163 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4004430 con 0x7ff7bc156ad0 2024-09-11T00:56:33.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.109+0000 7ff7c2600640 1 -- 172.21.15.31:0/1867374163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7bc151030 con 0x7ff7bc156ad0 2024-09-11T00:56:33.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.110+0000 7ff7c2600640 1 -- 172.21.15.31:0/1867374163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7bc159670 con 0x7ff7bc156ad0 2024-09-11T00:56:33.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.111+0000 7ff7b3e00640 1 -- 172.21.15.31:0/1867374163 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff7b40045d0 con 0x7ff7bc156ad0 2024-09-11T00:56:33.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.112+0000 7ff7b3e00640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff798077f70 0x7ff79807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:33.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.112+0000 7ff7b3e00640 1 -- 172.21.15.31:0/1867374163 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7ff7b40bbd40 con 0x7ff7bc156ad0 2024-09-11T00:56:33.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.113+0000 7ff7bb400640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff798077f70 0x7ff79807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:33.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.113+0000 7ff7bb400640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff798077f70 0x7ff79807a430 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc152ee0 tx=0x7ff7ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:33.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.116+0000 7ff7b3e00640 1 -- 172.21.15.31:0/1867374163 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff7b4032030 con 0x7ff7bc156ad0 2024-09-11T00:56:33.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.346+0000 7ff7c2600640 1 -- 172.21.15.31:0/1867374163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7bc0c1e70 con 0x7ff7bc156ad0 2024-09-11T00:56:33.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.347+0000 7ff7b3e00640 1 -- 172.21.15.31:0/1867374163 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+5719 (secure 0 0 0) 0x7ff7b4083250 con 0x7ff7bc156ad0 2024-09-11T00:56:33.350 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:33.350 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":185,"btime":"2024-09-11T00:56:31:055058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:31.055051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7672,"qdb_cluster":[7672]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:33.350 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 185 2024-09-11T00:56:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.349+0000 7ff7c2600640 1 -- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff798077f70 msgr2=0x7ff79807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.349+0000 7ff7c2600640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff798077f70 0x7ff79807a430 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc152ee0 tx=0x7ff7ac009040 comp rx=0 tx=0).stop 2024-09-11T00:56:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.349+0000 7ff7c2600640 1 -- 172.21.15.31:0/1867374163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7bc156ad0 msgr2=0x7ff7bc14cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.349+0000 7ff7c2600640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7bc156ad0 0x7ff7bc14cbf0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4006010 tx=0x7ff7b4003ce0 comp rx=0 tx=0).stop 2024-09-11T00:56:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.349+0000 7ff7c2600640 1 -- 172.21.15.31:0/1867374163 shutdown_connections 2024-09-11T00:56:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.349+0000 7ff7c2600640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff798077f70 0x7ff79807a430 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.349+0000 7ff7c2600640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7bc156ad0 0x7ff7bc14cbf0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.349+0000 7ff7c2600640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff7bc1539c0 0x7ff7bc1efd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.349+0000 7ff7c2600640 1 --2- 172.21.15.31:0/1867374163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7bc0c5b10 0x7ff7bc1e14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:33.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.349+0000 7ff7c2600640 1 -- 172.21.15.31:0/1867374163 >> 172.21.15.31:0/1867374163 conn(0x7ff7bc0a9750 msgr2=0x7ff7bc1e21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:33.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.350+0000 7ff7c2600640 1 -- 172.21.15.31:0/1867374163 shutdown_connections 2024-09-11T00:56:33.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:33.350+0000 7ff7c2600640 1 -- 172.21.15.31:0/1867374163 wait complete. 2024-09-11T00:56:33.368 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=7522 2024-09-11T00:56:33.368 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.a 2024-09-11T00:56:34.041 INFO:tasks.ceph.mds.d.smithi204.stderr:2024-09-11T00:56:34.039+0000 7fd53ca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-11T00:56:38.371 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:56:38.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:56:38.371 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-11T00:56:38.418 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:56:38.418 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-11T00:56:38.458 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:56:38.459 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-11T00:56:38.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-11T00:56:38.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.706+0000 7f2c92a00640 1 -- 172.21.15.31:0/2278777270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c8c0ac2c0 msgr2=0x7f2c8c0ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:38.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.706+0000 7f2c92a00640 1 --2- 172.21.15.31:0/2278777270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c8c0ac2c0 0x7f2c8c0ac6a0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0c5ea0 tx=0x7f2c8002a4d0 comp rx=0 tx=0).stop 2024-09-11T00:56:38.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.707+0000 7f2c92a00640 1 -- 172.21.15.31:0/2278777270 shutdown_connections 2024-09-11T00:56:38.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.707+0000 7f2c92a00640 1 --2- 172.21.15.31:0/2278777270 >> v2:172.21.15.204:3300/0 conn(0x7f2c8c0a6e20 0x7f2c8c1510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:38.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.707+0000 7f2c92a00640 1 --2- 172.21.15.31:0/2278777270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c8c0ac2c0 0x7f2c8c0ac6a0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:38.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.707+0000 7f2c92a00640 1 -- 172.21.15.31:0/2278777270 >> 172.21.15.31:0/2278777270 conn(0x7f2c8c06caf0 msgr2=0x7f2c8c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:38.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.707+0000 7f2c92a00640 1 -- 172.21.15.31:0/2278777270 shutdown_connections 2024-09-11T00:56:38.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.707+0000 7f2c92a00640 1 -- 172.21.15.31:0/2278777270 wait complete. 2024-09-11T00:56:38.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.708+0000 7f2c92a00640 1 Processor -- start 2024-09-11T00:56:38.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.708+0000 7f2c92a00640 1 -- start start 2024-09-11T00:56:38.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.709+0000 7f2c92a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c8c0a6e20 0x7f2c8c155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:38.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.709+0000 7f2c92a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c8c0ac2c0 0x7f2c8c147d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:38.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.709+0000 7f2c92a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c8c0acbe0 0x7f2c8c157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:38.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.709+0000 7f2c92a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8c151780 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.709+0000 7f2c92a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8c151600 con 0x7f2c8c0acbe0 2024-09-11T00:56:38.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.709+0000 7f2c92a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2c8c151900 con 0x7f2c8c0ac2c0 2024-09-11T00:56:38.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.709+0000 7f2c91600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c8c0a6e20 0x7f2c8c155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:38.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.709+0000 7f2c91600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c8c0a6e20 0x7f2c8c155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50522/0 (socket says 172.21.15.31:50522) 2024-09-11T00:56:38.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.709+0000 7f2c91600640 1 -- 172.21.15.31:0/2141920982 learned_addr learned my addr 172.21.15.31:0/2141920982 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:38.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.710+0000 7f2c92000640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c8c0acbe0 0x7f2c8c157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:38.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.710+0000 7f2c90c00640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c8c0ac2c0 0x7f2c8c147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.710+0000 7f2c91600640 1 -- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c8c0ac2c0 msgr2=0x7f2c8c147d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.710+0000 7f2c91600640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c8c0ac2c0 0x7f2c8c147d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.710+0000 7f2c91600640 1 -- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c8c0acbe0 msgr2=0x7f2c8c157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.710+0000 7f2c91600640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c8c0acbe0 0x7f2c8c157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.710+0000 7f2c91600640 1 -- 172.21.15.31:0/2141920982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8c155370 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.710+0000 7f2c90c00640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c8c0ac2c0 0x7f2c8c147d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.710+0000 7f2c91600640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c8c0a6e20 0x7f2c8c155600 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f2c80009850 tx=0x7f2c80003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.711+0000 7f2c89600640 1 -- 172.21.15.31:0/2141920982 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8003f070 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.711+0000 7f2c89600640 1 -- 172.21.15.31:0/2141920982 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c800047a0 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.711+0000 7f2c92a00640 1 -- 172.21.15.31:0/2141920982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c8c0745e0 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.711+0000 7f2c89600640 1 -- 172.21.15.31:0/2141920982 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c80035030 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.711+0000 7f2c92a00640 1 -- 172.21.15.31:0/2141920982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c8c1ece20 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.712+0000 7f2c92a00640 1 -- 172.21.15.31:0/2141920982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c8c074820 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.713+0000 7f2c89600640 1 -- 172.21.15.31:0/2141920982 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2c80004bb0 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.714+0000 7f2c89600640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c6c077f70 0x7f2c6c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:38.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.714+0000 7f2c89600640 1 -- 172.21.15.31:0/2141920982 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f2c800bbd90 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.715+0000 7f2c90c00640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c6c077f70 0x7f2c6c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:38.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.715+0000 7f2c90c00640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c6c077f70 0x7f2c6c07a430 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2c780058c0 tx=0x7f2c78005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:38.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.718+0000 7f2c89600640 1 -- 172.21.15.31:0/2141920982 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2c8c074820 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.925+0000 7f2c92a00640 1 -- 172.21.15.31:0/2141920982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2c8c1ec950 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.926+0000 7f2c89600640 1 -- 172.21.15.31:0/2141920982 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 70 entries v62) ==== 81+0+4300 (secure 0 0 0) 0x7f2c800832a0 con 0x7f2c8c0a6e20 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/3721481399 2024-09-12T00:56:28.030054+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/4094068967 2024-09-12T00:55:04.553768+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/2956367495 2024-09-12T00:54:19.542253+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2689217598 2024-09-12T00:56:14.574459+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/1855324855 2024-09-12T00:52:49.521133+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1981755707 2024-09-12T00:52:39.517968+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/98007221 2024-09-12T00:41:49.383893+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/3881548149 2024-09-12T00:40:39.371058+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2163558126 2024-09-12T00:48:14.453181+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/3506150555 2024-09-12T00:40:24.367167+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/1278135080 2024-09-12T00:38:14.342086+0000 2024-09-11T00:56:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/2956367495 2024-09-12T00:54:19.542253+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2909466339 2024-09-12T00:43:22.735779+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/1278135080 2024-09-12T00:38:14.342086+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/3444890210 2024-09-12T00:37:16.369607+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/4094068967 2024-09-12T00:55:04.553768+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/645869523 2024-09-12T00:35:54.319031+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/4164016077 2024-09-12T00:50:14.476294+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2323279595 2024-09-12T00:44:44.412976+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/3506150555 2024-09-12T00:40:24.367167+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4011734390 2024-09-12T00:37:09.330669+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/3725011623 2024-09-12T00:50:23.114454+0000 2024-09-11T00:56:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/3441932736 2024-09-12T00:31:44.277804+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3721481399 2024-09-12T00:56:28.030054+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/2686855151 2024-09-12T00:32:24.285279+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/1082799504 2024-09-12T00:47:59.449917+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1981755707 2024-09-12T00:52:39.517968+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/645869523 2024-09-12T00:35:54.319031+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3925687919 2024-09-12T00:41:09.376771+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/558470224 2024-09-12T00:34:09.302673+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1801149506 2024-09-12T00:32:54.290770+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4202305151 2024-09-12T00:35:09.311954+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/3441932736 2024-09-12T00:31:44.277804+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/1082799504 2024-09-12T00:47:59.449917+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3242133790 2024-09-12T00:38:14.342154+0000 2024-09-11T00:56:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/3881548149 2024-09-12T00:40:39.371058+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1801149506 2024-09-12T00:32:54.290770+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4202305151 2024-09-12T00:35:09.311954+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2954547253 2024-09-12T00:33:34.297314+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1348312176 2024-09-12T00:45:14.418207+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/3444890210 2024-09-12T00:37:16.369607+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2689217598 2024-09-12T00:56:14.574459+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3925687919 2024-09-12T00:41:09.376771+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4011734390 2024-09-12T00:37:09.330669+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/1396366029 2024-09-12T00:45:59.426330+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2954547253 2024-09-12T00:33:34.297314+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/558470224 2024-09-12T00:34:09.302673+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/1855324855 2024-09-12T00:52:49.521133+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/98007221 2024-09-12T00:41:49.383893+0000 2024-09-11T00:56:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/1396366029 2024-09-12T00:45:59.426330+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/2909466339 2024-09-12T00:43:22.735779+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2630216714 2024-09-12T00:52:19.506830+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1575320768 2024-09-12T00:44:14.407953+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2630216714 2024-09-12T00:52:19.506830+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2686855151 2024-09-12T00:32:24.285279+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1575320768 2024-09-12T00:44:14.407953+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1348312176 2024-09-12T00:45:14.418207+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/3513641573 2024-09-12T00:45:26.774916+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4205320530 2024-09-12T00:43:19.398942+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/2113744259 2024-09-12T00:47:19.441133+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/2113744259 2024-09-12T00:47:19.441133+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2323279595 2024-09-12T00:44:44.412976+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/3513641573 2024-09-12T00:45:26.774916+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2163558126 2024-09-12T00:48:14.453181+0000 2024-09-11T00:56:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/634556689 2024-09-12T00:49:34.468674+0000 2024-09-11T00:56:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/634556689 2024-09-12T00:49:34.468674+0000 2024-09-11T00:56:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3725011623 2024-09-12T00:50:23.114454+0000 2024-09-11T00:56:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3242133790 2024-09-12T00:38:14.342154+0000 2024-09-11T00:56:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4205320530 2024-09-12T00:43:19.398942+0000 2024-09-11T00:56:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/4164016077 2024-09-12T00:50:14.476294+0000 2024-09-11T00:56:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:listed 70 entries 2024-09-11T00:56:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.928+0000 7f2c92a00640 1 -- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c6c077f70 msgr2=0x7f2c6c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.928+0000 7f2c92a00640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c6c077f70 0x7f2c6c07a430 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2c780058c0 tx=0x7f2c78005850 comp rx=0 tx=0).stop 2024-09-11T00:56:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.928+0000 7f2c92a00640 1 -- 172.21.15.31:0/2141920982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c8c0a6e20 msgr2=0x7f2c8c155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:38.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.928+0000 7f2c92a00640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c8c0a6e20 0x7f2c8c155600 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f2c80009850 tx=0x7f2c80003910 comp rx=0 tx=0).stop 2024-09-11T00:56:38.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.928+0000 7f2c92a00640 1 -- 172.21.15.31:0/2141920982 shutdown_connections 2024-09-11T00:56:38.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.928+0000 7f2c92a00640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2c6c077f70 0x7f2c6c07a430 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:38.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.928+0000 7f2c92a00640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c8c0acbe0 0x7f2c8c157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:38.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.928+0000 7f2c92a00640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2c8c0ac2c0 0x7f2c8c147d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:38.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.928+0000 7f2c92a00640 1 --2- 172.21.15.31:0/2141920982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2c8c0a6e20 0x7f2c8c155600 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:38.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.928+0000 7f2c92a00640 1 -- 172.21.15.31:0/2141920982 >> 172.21.15.31:0/2141920982 conn(0x7f2c8c06caf0 msgr2=0x7f2c8c0c0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:38.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.929+0000 7f2c92a00640 1 -- 172.21.15.31:0/2141920982 shutdown_connections 2024-09-11T00:56:38.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:38.929+0000 7f2c92a00640 1 -- 172.21.15.31:0/2141920982 wait complete. 2024-09-11T00:56:38.946 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:56:38.946 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-11T00:56:38.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-11T00:56:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.151+0000 7f46c7e00640 1 -- 172.21.15.31:0/1339569283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46ac004020 msgr2=0x7f46ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.151+0000 7f46c7e00640 1 --2- 172.21.15.31:0/1339569283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46ac004020 0x7f46ac004800 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009b20 tx=0x7f46ac005830 comp rx=0 tx=0).stop 2024-09-11T00:56:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.151+0000 7f46c7400640 1 -- 172.21.15.31:0/1339569283 reap_dead start 2024-09-11T00:56:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.151+0000 7f46c7e00640 1 -- 172.21.15.31:0/1339569283 shutdown_connections 2024-09-11T00:56:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.151+0000 7f46c7e00640 1 -- 172.21.15.31:0/1339569283 >> 172.21.15.31:0/1339569283 conn(0x7f46c80adb00 msgr2=0x7f46c80adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.151+0000 7f46c7e00640 1 -- 172.21.15.31:0/1339569283 shutdown_connections 2024-09-11T00:56:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.151+0000 7f46c7e00640 1 -- 172.21.15.31:0/1339569283 wait complete. 2024-09-11T00:56:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.152+0000 7f46c7e00640 1 Processor -- start 2024-09-11T00:56:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.152+0000 7f46c7e00640 1 -- start start 2024-09-11T00:56:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.153+0000 7f46c7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46ac004020 0x7f46c80b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.153+0000 7f46c7e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46c80c51c0 0x7f46c8146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.153+0000 7f46c7e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8147410 0x7f46c81ef870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.153+0000 7f46c7e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f46c8156810 con 0x7f46c8147410 2024-09-11T00:56:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.153+0000 7f46c7e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f46c81562d0 con 0x7f46ac004020 2024-09-11T00:56:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.153+0000 7f46c7e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f46c814cc60 con 0x7f46c80c51c0 2024-09-11T00:56:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46c7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8147410 0x7f46c81ef870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46c7400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8147410 0x7f46c81ef870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50546/0 (socket says 172.21.15.31:50546) 2024-09-11T00:56:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46c7400640 1 -- 172.21.15.31:0/2518083504 learned_addr learned my addr 172.21.15.31:0/2518083504 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46bfe00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46c80c51c0 0x7f46c8146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46c6a00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46ac004020 0x7f46c80b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46c7400640 1 -- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46c80c51c0 msgr2=0x7f46c8146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46c7400640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46c80c51c0 0x7f46c8146ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46c7400640 1 -- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46ac004020 msgr2=0x7f46c80b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46c7400640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46ac004020 0x7f46c80b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46c7400640 1 -- 172.21.15.31:0/2518083504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c8147950 con 0x7f46c8147410 2024-09-11T00:56:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.154+0000 7f46c7400640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8147410 0x7f46c81ef870 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f46c8153c80 tx=0x7f46c0006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.155+0000 7f46bea00640 1 -- 172.21.15.31:0/2518083504 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c000f880 con 0x7f46c8147410 2024-09-11T00:56:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.155+0000 7f46bea00640 1 -- 172.21.15.31:0/2518083504 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46c0016020 con 0x7f46c8147410 2024-09-11T00:56:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.155+0000 7f46bea00640 1 -- 172.21.15.31:0/2518083504 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c000fcc0 con 0x7f46c8147410 2024-09-11T00:56:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.155+0000 7f46c7e00640 1 -- 172.21.15.31:0/2518083504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c8147c10 con 0x7f46c8147410 2024-09-11T00:56:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.155+0000 7f46c7e00640 1 -- 172.21.15.31:0/2518083504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c80abde0 con 0x7f46c8147410 2024-09-11T00:56:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.156+0000 7f46c7e00640 1 -- 172.21.15.31:0/2518083504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c80adf10 con 0x7f46c8147410 2024-09-11T00:56:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.158+0000 7f46bea00640 1 -- 172.21.15.31:0/2518083504 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f46c0011430 con 0x7f46c8147410 2024-09-11T00:56:39.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.158+0000 7f46bea00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a4077fa0 0x7f46a407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:39.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.159+0000 7f46bea00640 1 -- 172.21.15.31:0/2518083504 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f46c009b740 con 0x7f46c8147410 2024-09-11T00:56:39.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.159+0000 7f46c6a00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a4077fa0 0x7f46a407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.160+0000 7f46c6a00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a4077fa0 0x7f46a407a460 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009460 tx=0x7f46b8005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.163+0000 7f46bea00640 1 -- 172.21.15.31:0/2518083504 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f46c80adf10 con 0x7f46c8147410 2024-09-11T00:56:39.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.371+0000 7f46c7e00640 1 -- 172.21.15.31:0/2518083504 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f46c80c59a0 con 0x7f46c8147410 2024-09-11T00:56:39.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.372+0000 7f46bea00640 1 -- 172.21.15.31:0/2518083504 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 70 entries v62) ==== 81+0+4300 (secure 0 0 0) 0x7f46c0020020 con 0x7f46c8147410 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/3721481399 2024-09-12T00:56:28.030054+0000 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/4094068967 2024-09-12T00:55:04.553768+0000 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/2956367495 2024-09-12T00:54:19.542253+0000 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2689217598 2024-09-12T00:56:14.574459+0000 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/1855324855 2024-09-12T00:52:49.521133+0000 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1981755707 2024-09-12T00:52:39.517968+0000 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/98007221 2024-09-12T00:41:49.383893+0000 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/3881548149 2024-09-12T00:40:39.371058+0000 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2163558126 2024-09-12T00:48:14.453181+0000 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/3506150555 2024-09-12T00:40:24.367167+0000 2024-09-11T00:56:39.374 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/1278135080 2024-09-12T00:38:14.342086+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/2956367495 2024-09-12T00:54:19.542253+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2909466339 2024-09-12T00:43:22.735779+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/1278135080 2024-09-12T00:38:14.342086+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/3444890210 2024-09-12T00:37:16.369607+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/4094068967 2024-09-12T00:55:04.553768+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/645869523 2024-09-12T00:35:54.319031+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/4164016077 2024-09-12T00:50:14.476294+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2323279595 2024-09-12T00:44:44.412976+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/3506150555 2024-09-12T00:40:24.367167+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4011734390 2024-09-12T00:37:09.330669+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/3725011623 2024-09-12T00:50:23.114454+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/3441932736 2024-09-12T00:31:44.277804+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3721481399 2024-09-12T00:56:28.030054+0000 2024-09-11T00:56:39.375 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/2686855151 2024-09-12T00:32:24.285279+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/1082799504 2024-09-12T00:47:59.449917+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1981755707 2024-09-12T00:52:39.517968+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/645869523 2024-09-12T00:35:54.319031+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3925687919 2024-09-12T00:41:09.376771+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/558470224 2024-09-12T00:34:09.302673+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1801149506 2024-09-12T00:32:54.290770+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4202305151 2024-09-12T00:35:09.311954+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/3441932736 2024-09-12T00:31:44.277804+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/1082799504 2024-09-12T00:47:59.449917+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3242133790 2024-09-12T00:38:14.342154+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/3881548149 2024-09-12T00:40:39.371058+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1801149506 2024-09-12T00:32:54.290770+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4202305151 2024-09-12T00:35:09.311954+0000 2024-09-11T00:56:39.376 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2954547253 2024-09-12T00:33:34.297314+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1348312176 2024-09-12T00:45:14.418207+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/3444890210 2024-09-12T00:37:16.369607+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2689217598 2024-09-12T00:56:14.574459+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/3925687919 2024-09-12T00:41:09.376771+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4011734390 2024-09-12T00:37:09.330669+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/1396366029 2024-09-12T00:45:59.426330+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2954547253 2024-09-12T00:33:34.297314+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/558470224 2024-09-12T00:34:09.302673+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/1855324855 2024-09-12T00:52:49.521133+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/98007221 2024-09-12T00:41:49.383893+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/1396366029 2024-09-12T00:45:59.426330+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6832/2909466339 2024-09-12T00:43:22.735779+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/2630216714 2024-09-12T00:52:19.506830+0000 2024-09-11T00:56:39.377 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/1575320768 2024-09-12T00:44:14.407953+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/2630216714 2024-09-12T00:52:19.506830+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2686855151 2024-09-12T00:32:24.285279+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1575320768 2024-09-12T00:44:14.407953+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/1348312176 2024-09-12T00:45:14.418207+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6837/3513641573 2024-09-12T00:45:26.774916+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6833/4205320530 2024-09-12T00:43:19.398942+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/2113744259 2024-09-12T00:47:19.441133+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/2113744259 2024-09-12T00:47:19.441133+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/2323279595 2024-09-12T00:44:44.412976+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/3513641573 2024-09-12T00:45:26.774916+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6834/2163558126 2024-09-12T00:48:14.453181+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6834/634556689 2024-09-12T00:49:34.468674+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6835/634556689 2024-09-12T00:49:34.468674+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3725011623 2024-09-12T00:50:23.114454+0000 2024-09-11T00:56:39.378 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/3242133790 2024-09-12T00:38:14.342154+0000 2024-09-11T00:56:39.379 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6835/4205320530 2024-09-12T00:43:19.398942+0000 2024-09-11T00:56:39.379 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.204:6836/4164016077 2024-09-12T00:50:14.476294+0000 2024-09-11T00:56:39.379 INFO:teuthology.orchestra.run.smithi031.stderr:listed 70 entries 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 -- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a4077fa0 msgr2=0x7f46a407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a4077fa0 0x7f46a407a460 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009460 tx=0x7f46b8005f90 comp rx=0 tx=0).stop 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 -- 172.21.15.31:0/2518083504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8147410 msgr2=0x7f46c81ef870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8147410 0x7f46c81ef870 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f46c8153c80 tx=0x7f46c0006720 comp rx=0 tx=0).stop 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 -- 172.21.15.31:0/2518083504 shutdown_connections 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46a4077fa0 0x7f46a407a460 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8147410 0x7f46c81ef870 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f46c80c51c0 0x7f46c8146ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 --2- 172.21.15.31:0/2518083504 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f46ac004020 0x7f46c80b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 -- 172.21.15.31:0/2518083504 >> 172.21.15.31:0/2518083504 conn(0x7f46c80adb00 msgr2=0x7f46c80acd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.374+0000 7f46c7e00640 1 -- 172.21.15.31:0/2518083504 shutdown_connections 2024-09-11T00:56:39.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:39.375+0000 7f46c7e00640 1 -- 172.21.15.31:0/2518083504 wait complete. 2024-09-11T00:56:39.392 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi204.front.sepia.ceph.com... 2024-09-11T00:56:39.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:56:39.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-11T00:56:39.466 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:56:39.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi204.stderr:ceph-fuse[41119]: fuse finished with error 0 and tester_r 0 2024-09-11T00:56:40.369 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-11T00:56:40.369 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-11T00:56:40.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-11T00:56:40.374 INFO:tasks.ceph.mds.a:Started 2024-09-11T00:56:40.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:40.513 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-11T00:56:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.544+0000 7f3de8800640 1 -- 172.21.15.31:0/1992545319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de01518b0 msgr2=0x7f3de00c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.544+0000 7f3de8800640 1 --2- 172.21.15.31:0/1992545319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de01518b0 0x7f3de00c3600 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3de00add10 tx=0x7f3ddc02a940 comp rx=0 tx=0).stop 2024-09-11T00:56:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.544+0000 7f3de8800640 1 -- 172.21.15.31:0/1992545319 shutdown_connections 2024-09-11T00:56:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.544+0000 7f3de8800640 1 --2- 172.21.15.31:0/1992545319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de01518b0 0x7f3de00c3600 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.544+0000 7f3de8800640 1 --2- 172.21.15.31:0/1992545319 >> v2:172.21.15.204:3300/0 conn(0x7f3de00c23a0 0x7f3de00c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.544+0000 7f3de8800640 1 -- 172.21.15.31:0/1992545319 >> 172.21.15.31:0/1992545319 conn(0x7f3de00a9750 msgr2=0x7f3de00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.544+0000 7f3de8800640 1 -- 172.21.15.31:0/1992545319 shutdown_connections 2024-09-11T00:56:40.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.544+0000 7f3de8800640 1 -- 172.21.15.31:0/1992545319 wait complete. 2024-09-11T00:56:40.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.546+0000 7f3de8800640 1 Processor -- start 2024-09-11T00:56:40.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.546+0000 7f3de8800640 1 -- start start 2024-09-11T00:56:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de8800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de0072480 0x7f3de0146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de8800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3de00c23a0 0x7f3de00c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de8800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3de01518b0 0x7f3de01ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de8800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3de0074200 con 0x7f3de0072480 2024-09-11T00:56:40.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de8800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3de0074080 con 0x7f3de00c23a0 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de8800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3de0072860 con 0x7f3de01518b0 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de0072480 0x7f3de0146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3de00c23a0 0x7f3de00c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de5400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3de00c23a0 0x7f3de00c0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:58036/0 (socket says 172.21.15.31:58036) 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de5400640 1 -- 172.21.15.31:0/370925740 learned_addr learned my addr 172.21.15.31:0/370925740 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de0072480 0x7f3de0146f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50600/0 (socket says 172.21.15.31:50600) 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de6800640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3de01518b0 0x7f3de01ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de5400640 1 -- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3de01518b0 msgr2=0x7f3de01ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.547+0000 7f3de5400640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3de01518b0 0x7f3de01ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.548+0000 7f3de5400640 1 -- 172.21.15.31:0/370925740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de0072480 msgr2=0x7f3de0146f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.548+0000 7f3de5400640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de0072480 0x7f3de0146f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.548+0000 7f3de5400640 1 -- 172.21.15.31:0/370925740 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de006e980 con 0x7f3de00c23a0 2024-09-11T00:56:40.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.548+0000 7f3de5e00640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de0072480 0x7f3de0146f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:56:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.548+0000 7f3de5400640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3de00c23a0 0x7f3de00c0dd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000d770 tx=0x7f3dd000dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.548+0000 7f3dda000640 1 -- 172.21.15.31:0/370925740 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0017070 con 0x7f3de00c23a0 2024-09-11T00:56:40.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.548+0000 7f3dda000640 1 -- 172.21.15.31:0/370925740 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dd0009d20 con 0x7f3de00c23a0 2024-09-11T00:56:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.548+0000 7f3de8800640 1 -- 172.21.15.31:0/370925740 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de01dffd0 con 0x7f3de00c23a0 2024-09-11T00:56:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.548+0000 7f3dda000640 1 -- 172.21.15.31:0/370925740 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd00106c0 con 0x7f3de00c23a0 2024-09-11T00:56:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.548+0000 7f3de8800640 1 -- 172.21.15.31:0/370925740 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de01474b0 con 0x7f3de00c23a0 2024-09-11T00:56:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.549+0000 7f3de8800640 1 -- 172.21.15.31:0/370925740 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de0153f20 con 0x7f3de00c23a0 2024-09-11T00:56:40.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.551+0000 7f3dda000640 1 -- 172.21.15.31:0/370925740 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3dd0010860 con 0x7f3de00c23a0 2024-09-11T00:56:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.552+0000 7f3dda000640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3dc4077fa0 0x7f3dc407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.552+0000 7f3dda000640 1 -- 172.21.15.31:0/370925740 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f3dd009b540 con 0x7f3de00c23a0 2024-09-11T00:56:40.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.552+0000 7f3de5e00640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3dc4077fa0 0x7f3dc407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:40.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.553+0000 7f3de5e00640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3dc4077fa0 0x7f3dc407a460 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4007f20 tx=0x7f3dd4007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:40.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.555+0000 7f3dda000640 1 -- 172.21.15.31:0/370925740 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3dd0062ad0 con 0x7f3de00c23a0 2024-09-11T00:56:40.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.785+0000 7f3de8800640 1 -- 172.21.15.31:0/370925740 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3de014fc50 con 0x7f3de00c23a0 2024-09-11T00:56:40.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.786+0000 7f3dda000640 1 -- 172.21.15.31:0/370925740 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+5719 (secure 0 0 0) 0x7f3dd0067500 con 0x7f3de00c23a0 2024-09-11T00:56:40.788 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:40.789 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":185,"btime":"2024-09-11T00:56:31:055058+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:31.055051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7672,"qdb_cluster":[7672]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:40.789 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 185 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 -- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3dc4077fa0 msgr2=0x7f3dc407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3dc4077fa0 0x7f3dc407a460 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4007f20 tx=0x7f3dd4007eb0 comp rx=0 tx=0).stop 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 -- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3de00c23a0 msgr2=0x7f3de00c0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3de00c23a0 0x7f3de00c0dd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000d770 tx=0x7f3dd000dc30 comp rx=0 tx=0).stop 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 -- 172.21.15.31:0/370925740 shutdown_connections 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3dc4077fa0 0x7f3dc407a460 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3de01518b0 0x7f3de01ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3de00c23a0 0x7f3de00c0dd0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 --2- 172.21.15.31:0/370925740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3de0072480 0x7f3de0146f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 -- 172.21.15.31:0/370925740 >> 172.21.15.31:0/370925740 conn(0x7f3de00a9750 msgr2=0x7f3de014a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:40.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 -- 172.21.15.31:0/370925740 shutdown_connections 2024-09-11T00:56:40.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:40.789+0000 7f3de8800640 1 -- 172.21.15.31:0/370925740 wait complete. 2024-09-11T00:56:40.807 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-11T00:56:42.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:56:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.979+0000 7f471f200640 1 -- 172.21.15.31:0/1676067309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f471806cc10 msgr2=0x7f4718159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.979+0000 7f471f200640 1 --2- 172.21.15.31:0/1676067309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f471806cc10 0x7f4718159450 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4708005fd0 tx=0x7f470802a5d0 comp rx=0 tx=0).stop 2024-09-11T00:56:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.979+0000 7f471f200640 1 -- 172.21.15.31:0/1676067309 shutdown_connections 2024-09-11T00:56:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.979+0000 7f471f200640 1 --2- 172.21.15.31:0/1676067309 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f47180caef0 0x7f4718159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.979+0000 7f471f200640 1 --2- 172.21.15.31:0/1676067309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f471806cc10 0x7f4718159450 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:42.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.979+0000 7f471f200640 1 -- 172.21.15.31:0/1676067309 >> 172.21.15.31:0/1676067309 conn(0x7f47180adb00 msgr2=0x7f47180adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:42.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.979+0000 7f471f200640 1 -- 172.21.15.31:0/1676067309 shutdown_connections 2024-09-11T00:56:42.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.979+0000 7f471f200640 1 -- 172.21.15.31:0/1676067309 wait complete. 2024-09-11T00:56:42.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.981+0000 7f471f200640 1 Processor -- start 2024-09-11T00:56:42.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.981+0000 7f471f200640 1 -- start start 2024-09-11T00:56:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f471806cc10 0x7f47181e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47180caef0 0x7f47180afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f471814c880 0x7f47181e2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4718156810 con 0x7f47180caef0 2024-09-11T00:56:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4718156530 con 0x7f471806cc10 2024-09-11T00:56:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f471814cc60 con 0x7f471814c880 2024-09-11T00:56:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47180caef0 0x7f47180afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47180caef0 0x7f47180afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50610/0 (socket says 172.21.15.31:50610) 2024-09-11T00:56:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471d400640 1 -- 172.21.15.31:0/1033914795 learned_addr learned my addr 172.21.15.31:0/1033914795 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471de00640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f471806cc10 0x7f47181e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471e800640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f471814c880 0x7f47181e2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471d400640 1 -- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f471814c880 msgr2=0x7f47181e2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471d400640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f471814c880 0x7f47181e2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471d400640 1 -- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f471806cc10 msgr2=0x7f47181e01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471d400640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f471806cc10 0x7f47181e01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.982+0000 7f471d400640 1 -- 172.21.15.31:0/1033914795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47180aef90 con 0x7f47180caef0 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.983+0000 7f471d400640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47180caef0 0x7f47180afd50 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f4708006010 tx=0x7f47080035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.983+0000 7f4712000640 1 -- 172.21.15.31:0/1033914795 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470803f070 con 0x7f47180caef0 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.983+0000 7f4712000640 1 -- 172.21.15.31:0/1033914795 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4708003ce0 con 0x7f47180caef0 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.983+0000 7f4712000640 1 -- 172.21.15.31:0/1033914795 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708035030 con 0x7f47180caef0 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.983+0000 7f471f200640 1 -- 172.21.15.31:0/1033914795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47180af270 con 0x7f47180caef0 2024-09-11T00:56:42.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.983+0000 7f471f200640 1 -- 172.21.15.31:0/1033914795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47180af830 con 0x7f47180caef0 2024-09-11T00:56:42.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.984+0000 7f471f200640 1 -- 172.21.15.31:0/1033914795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47180c8ac0 con 0x7f47180caef0 2024-09-11T00:56:42.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.985+0000 7f4712000640 1 -- 172.21.15.31:0/1033914795 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4708004a00 con 0x7f47180caef0 2024-09-11T00:56:42.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.986+0000 7f4712000640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46fc077fa0 0x7f46fc07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:42.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.986+0000 7f4712000640 1 -- 172.21.15.31:0/1033914795 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f47080bbba0 con 0x7f47180caef0 2024-09-11T00:56:42.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.987+0000 7f471de00640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46fc077fa0 0x7f46fc07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:42.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.987+0000 7f471de00640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46fc077fa0 0x7f46fc07a460 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f470c0017a0 tx=0x7f470c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:42.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:42.991+0000 7f4712000640 1 -- 172.21.15.31:0/1033914795 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4708083020 con 0x7f47180caef0 2024-09-11T00:56:43.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.223+0000 7f471f200640 1 -- 172.21.15.31:0/1033914795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47180ca760 con 0x7f47180caef0 2024-09-11T00:56:43.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.224+0000 7f4712000640 1 -- 172.21.15.31:0/1033914795 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+6545 (secure 0 0 0) 0x7f4708087a50 con 0x7f47180caef0 2024-09-11T00:56:43.226 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:43.226 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":186,"btime":"2024-09-11T00:56:41:092005+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:31.055051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7672,"qdb_cluster":[7672]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:56:43.226 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 186 2024-09-11T00:56:43.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 -- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46fc077fa0 msgr2=0x7f46fc07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:43.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46fc077fa0 0x7f46fc07a460 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f470c0017a0 tx=0x7f470c009040 comp rx=0 tx=0).stop 2024-09-11T00:56:43.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 -- 172.21.15.31:0/1033914795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47180caef0 msgr2=0x7f47180afd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:43.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47180caef0 0x7f47180afd50 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f4708006010 tx=0x7f47080035f0 comp rx=0 tx=0).stop 2024-09-11T00:56:43.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 -- 172.21.15.31:0/1033914795 shutdown_connections 2024-09-11T00:56:43.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f46fc077fa0 0x7f46fc07a460 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:43.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f471814c880 0x7f47181e2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:43.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47180caef0 0x7f47180afd50 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:43.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 --2- 172.21.15.31:0/1033914795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f471806cc10 0x7f47181e01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:43.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 -- 172.21.15.31:0/1033914795 >> 172.21.15.31:0/1033914795 conn(0x7f47180adb00 msgr2=0x7f4718154bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:43.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 -- 172.21.15.31:0/1033914795 shutdown_connections 2024-09-11T00:56:43.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:43.226+0000 7f471f200640 1 -- 172.21.15.31:0/1033914795 wait complete. 2024-09-11T00:56:43.245 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-11T00:56:43.245 INFO:tasks.mds_thrash.fs.[a]:waiting for 36 secs before thrashing 2024-09-11T00:56:45.567 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:56:45.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:56:45.568 DEBUG:teuthology.orchestra.run.smithi204:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-11T00:56:45.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:56:45.622 DEBUG:teuthology.orchestra.run.smithi204:> ip netns list 2024-09-11T00:56:45.713 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-11T00:56:45.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:56:45.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-11T00:56:45.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:56:45.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo ip link delete ceph-brx 2024-09-11T00:56:46.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:56:46.084 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-09-11T00:56:46.165 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-11T00:56:46.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:56:46.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-11T00:56:46.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:56:46.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-09-11T00:56:46.526 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T00:56:46.534 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T00:56:46.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:56:46.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.694+0000 7f66a5e00640 1 -- 172.21.15.31:0/2054418718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a0156960 msgr2=0x7f66a006e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:46.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.694+0000 7f66a5e00640 1 --2- 172.21.15.31:0/2054418718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a0156960 0x7f66a006e140 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f66a0153d20 tx=0x7f668c02a480 comp rx=0 tx=0).stop 2024-09-11T00:56:46.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.695+0000 7f66a5e00640 1 -- 172.21.15.31:0/2054418718 shutdown_connections 2024-09-11T00:56:46.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.695+0000 7f66a5e00640 1 --2- 172.21.15.31:0/2054418718 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66a00cc220 0x7f66a006e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:46.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.695+0000 7f66a5e00640 1 --2- 172.21.15.31:0/2054418718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a0156960 0x7f66a006e140 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:46.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.695+0000 7f66a5e00640 1 -- 172.21.15.31:0/2054418718 >> 172.21.15.31:0/2054418718 conn(0x7f66a0068f10 msgr2=0x7f66a00cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:46.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.695+0000 7f66a5e00640 1 -- 172.21.15.31:0/2054418718 shutdown_connections 2024-09-11T00:56:46.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.695+0000 7f66a5e00640 1 -- 172.21.15.31:0/2054418718 wait complete. 2024-09-11T00:56:46.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.696+0000 7f66a5e00640 1 Processor -- start 2024-09-11T00:56:46.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.696+0000 7f66a5e00640 1 -- start start 2024-09-11T00:56:46.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.697+0000 7f66a5e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a00cc220 0x7f66a01f0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.697+0000 7f66a5e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66a0150020 0x7f66a014e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.697+0000 7f66a5e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66a0156960 0x7f66a014af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.697+0000 7f66a5e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66a00c6ca0 con 0x7f66a00cc220 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.697+0000 7f66a5e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f66a00c6b20 con 0x7f66a0156960 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.697+0000 7f66a5e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f66a01573d0 con 0x7f66a0150020 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.697+0000 7f66a4a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a00cc220 0x7f66a01f0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.697+0000 7f66a4a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a00cc220 0x7f66a01f0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50628/0 (socket says 172.21.15.31:50628) 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.697+0000 7f66a4a00640 1 -- 172.21.15.31:0/597877954 learned_addr learned my addr 172.21.15.31:0/597877954 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.698+0000 7f66a4a00640 1 -- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66a0150020 msgr2=0x7f66a014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.698+0000 7f66a4a00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66a0150020 0x7f66a014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.698+0000 7f66a4a00640 1 -- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66a0156960 msgr2=0x7f66a014af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.698+0000 7f66a4a00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66a0156960 0x7f66a014af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.698+0000 7f66a4a00640 1 -- 172.21.15.31:0/597877954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a00b1010 con 0x7f66a00cc220 2024-09-11T00:56:46.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.698+0000 7f66a4a00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a00cc220 0x7f66a01f0050 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f669400c420 tx=0x7f669400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:46.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.698+0000 7f669cc00640 1 -- 172.21.15.31:0/597877954 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6694006890 con 0x7f66a00cc220 2024-09-11T00:56:46.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.698+0000 7f669cc00640 1 -- 172.21.15.31:0/597877954 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6694006b20 con 0x7f66a00cc220 2024-09-11T00:56:46.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.698+0000 7f66a5e00640 1 -- 172.21.15.31:0/597877954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a00b12b0 con 0x7f66a00cc220 2024-09-11T00:56:46.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.698+0000 7f669cc00640 1 -- 172.21.15.31:0/597877954 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6694007020 con 0x7f66a00cc220 2024-09-11T00:56:46.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.699+0000 7f66a5e00640 1 -- 172.21.15.31:0/597877954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a014ec70 con 0x7f66a00cc220 2024-09-11T00:56:46.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.700+0000 7f66a5e00640 1 -- 172.21.15.31:0/597877954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66a00ad4e0 con 0x7f66a00cc220 2024-09-11T00:56:46.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.701+0000 7f669cc00640 1 -- 172.21.15.31:0/597877954 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f66940071c0 con 0x7f66a00cc220 2024-09-11T00:56:46.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.702+0000 7f669cc00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6680077f70 0x7f668007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:46.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.702+0000 7f669cc00640 1 -- 172.21.15.31:0/597877954 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f669409a770 con 0x7f66a00cc220 2024-09-11T00:56:46.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.702+0000 7f669fe00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6680077f70 0x7f668007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.703+0000 7f669fe00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6680077f70 0x7f668007a430 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f668c034000 tx=0x7f668c0046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:46.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.706+0000 7f669cc00640 1 -- 172.21.15.31:0/597877954 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f66940622d0 con 0x7f66a00cc220 2024-09-11T00:56:46.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.900+0000 7f66a5e00640 1 -- 172.21.15.31:0/597877954 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f66a00cc8e0 con 0x7f6680077f70 2024-09-11T00:56:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.910+0000 7f669cc00640 1 -- 172.21.15.31:0/597877954 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493711 (secure 0 0 0) 0x7f66a00cc8e0 con 0x7f6680077f70 2024-09-11T00:56:46.917 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:46.921 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:56:46.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 -- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6680077f70 msgr2=0x7f668007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6680077f70 0x7f668007a430 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f668c034000 tx=0x7f668c0046c0 comp rx=0 tx=0).stop 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 -- 172.21.15.31:0/597877954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a00cc220 msgr2=0x7f66a01f0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a00cc220 0x7f66a01f0050 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f669400c420 tx=0x7f669400c8e0 comp rx=0 tx=0).stop 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 -- 172.21.15.31:0/597877954 shutdown_connections 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6680077f70 0x7f668007a430 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66a0156960 0x7f66a014af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66a0150020 0x7f66a014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 --2- 172.21.15.31:0/597877954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66a00cc220 0x7f66a01f0050 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 -- 172.21.15.31:0/597877954 >> 172.21.15.31:0/597877954 conn(0x7f66a0068f10 msgr2=0x7f66a0154a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 -- 172.21.15.31:0/597877954 shutdown_connections 2024-09-11T00:56:46.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:46.916+0000 7f66a5e00640 1 -- 172.21.15.31:0/597877954 wait complete. 2024-09-11T00:56:46.934 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":842,"stamp":"2024-09-11T00:56:46.843610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36448998,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":2779,"num_read_kb":341768,"num_write":2142706,"num_write_kb":34795523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":418188,"ondisk_log_size":418188,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5380088,"kb_used_data":77432,"kb_used_omap":12,"kb_used_meta":5302515,"kb_avail":749594632,"statfs":{"total":773094113280,"available":767584903168,"internally_reserved":0,"allocated":79290368,"data_stored":76164476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":5429775921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20007,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003687"},"pg_stats":[{"pgid":"5.18","version":"62'3612","reported_seq":3708,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.530101+0000","last_change":"2024-09-11T00:30:54.983412+0000","last_active":"2024-09-11T00:56:28.530101+0000","last_peered":"2024-09-11T00:56:28.530101+0000","last_clean":"2024-09-11T00:56:28.530101+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:56:28.530101+0000","last_undegraded":"2024-09-11T00:56:28.530101+0000","last_fullsized":"2024-09-11T00:56:28.530101+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:51.807846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18811,"num_write_kb":284357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"60'788","reported_seq":928,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.095736+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:56:28.095736+0000","last_peered":"2024-09-11T00:56:28.095736+0000","last_clean":"2024-09-11T00:56:28.095736+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:56:28.095736+0000","last_undegraded":"2024-09-11T00:56:28.095736+0000","last_fullsized":"2024-09-11T00:56:28.095736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":8319,"num_write":773,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"60'3033","reported_seq":3138,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059830+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:56:28.059830+0000","last_peered":"2024-09-11T00:56:28.059830+0000","last_clean":"2024-09-11T00:56:28.059830+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:56:28.059830+0000","last_undegraded":"2024-09-11T00:56:28.059830+0000","last_fullsized":"2024-09-11T00:56:28.059830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16165,"num_write_kb":269490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"61'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058244+0000","last_change":"2024-09-11T00:30:53.984316+0000","last_active":"2024-09-11T00:56:28.058244+0000","last_peered":"2024-09-11T00:56:28.058244+0000","last_clean":"2024-09-11T00:56:28.058244+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:56:28.058244+0000","last_undegraded":"2024-09-11T00:56:28.058244+0000","last_fullsized":"2024-09-11T00:56:28.058244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:14.425265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"62'3995","reported_seq":4090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.532259+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:56:28.532259+0000","last_peered":"2024-09-11T00:56:28.532259+0000","last_clean":"2024-09-11T00:56:28.532259+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:56:28.532259+0000","last_undegraded":"2024-09-11T00:56:28.532259+0000","last_fullsized":"2024-09-11T00:56:28.532259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20665,"num_write_kb":324486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"62'22","reported_seq":129,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060203+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:56:28.060203+0000","last_peered":"2024-09-11T00:56:28.060203+0000","last_clean":"2024-09-11T00:56:28.060203+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:56:28.060203+0000","last_undegraded":"2024-09-11T00:56:28.060203+0000","last_fullsized":"2024-09-11T00:56:28.060203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"60'3563","reported_seq":3674,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059819+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:56:28.059819+0000","last_peered":"2024-09-11T00:56:28.059819+0000","last_clean":"2024-09-11T00:56:28.059819+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:56:28.059819+0000","last_undegraded":"2024-09-11T00:56:28.059819+0000","last_fullsized":"2024-09-11T00:56:28.059819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":44,"num_write":18834,"num_write_kb":321224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"62'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075381+0000","last_change":"2024-09-11T00:30:53.984940+0000","last_active":"2024-09-11T00:56:28.075381+0000","last_peered":"2024-09-11T00:56:28.075381+0000","last_clean":"2024-09-11T00:56:28.075381+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:56:28.075381+0000","last_undegraded":"2024-09-11T00:56:28.075381+0000","last_fullsized":"2024-09-11T00:56:28.075381+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:19.647165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"62'3090","reported_seq":3185,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.476819+0000","last_change":"2024-09-11T00:30:54.993570+0000","last_active":"2024-09-11T00:56:28.476819+0000","last_peered":"2024-09-11T00:56:28.476819+0000","last_clean":"2024-09-11T00:56:28.476819+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:56:28.476819+0000","last_undegraded":"2024-09-11T00:56:28.476819+0000","last_fullsized":"2024-09-11T00:56:28.476819+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:45.211259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15625,"num_write_kb":254958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"62'37","reported_seq":171,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.077866+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:56:31.077866+0000","last_peered":"2024-09-11T00:56:31.077866+0000","last_clean":"2024-09-11T00:56:31.077866+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:56:31.077866+0000","last_undegraded":"2024-09-11T00:56:31.077866+0000","last_fullsized":"2024-09-11T00:56:31.077866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"60'2040","reported_seq":2144,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057996+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:56:28.057996+0000","last_peered":"2024-09-11T00:56:28.057996+0000","last_clean":"2024-09-11T00:56:28.057996+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:56:28.057996+0000","last_undegraded":"2024-09-11T00:56:28.057996+0000","last_fullsized":"2024-09-11T00:56:28.057996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":10218,"num_write_kb":171981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"61'716","reported_seq":837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075103+0000","last_change":"2024-09-11T00:30:53.974699+0000","last_active":"2024-09-11T00:56:28.075103+0000","last_peered":"2024-09-11T00:56:28.075103+0000","last_clean":"2024-09-11T00:56:28.075103+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:56:28.075103+0000","last_undegraded":"2024-09-11T00:56:28.075103+0000","last_fullsized":"2024-09-11T00:56:28.075103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:28.792487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":73385,"num_write":707,"num_write_kb":4317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"62'2671","reported_seq":2766,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.074790+0000","last_change":"2024-09-11T00:30:54.984890+0000","last_active":"2024-09-11T00:56:28.074790+0000","last_peered":"2024-09-11T00:56:28.074790+0000","last_clean":"2024-09-11T00:56:28.074790+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:56:28.074790+0000","last_undegraded":"2024-09-11T00:56:28.074790+0000","last_fullsized":"2024-09-11T00:56:28.074790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:21.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":213871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"60'3","reported_seq":136,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.069924+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:56:31.069924+0000","last_peered":"2024-09-11T00:56:31.069924+0000","last_clean":"2024-09-11T00:56:31.069924+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:56:31.069924+0000","last_undegraded":"2024-09-11T00:56:31.069924+0000","last_fullsized":"2024-09-11T00:56:31.069924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"60'3687","reported_seq":3790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068958+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:56:28.068958+0000","last_peered":"2024-09-11T00:56:28.068958+0000","last_clean":"2024-09-11T00:56:28.068958+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:56:28.068958+0000","last_undegraded":"2024-09-11T00:56:28.068958+0000","last_fullsized":"2024-09-11T00:56:28.068958+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19525,"num_write_kb":315726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"62'140","reported_seq":237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.957825+0000","last_change":"2024-09-11T00:30:53.977743+0000","last_active":"2024-09-11T00:56:34.957825+0000","last_peered":"2024-09-11T00:56:34.957825+0000","last_clean":"2024-09-11T00:56:34.957825+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:56:34.957825+0000","last_undegraded":"2024-09-11T00:56:34.957825+0000","last_fullsized":"2024-09-11T00:56:34.957825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:20.618615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1356353,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":1393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"62'3457","reported_seq":3553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.513475+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:56:28.513475+0000","last_peered":"2024-09-11T00:56:28.513475+0000","last_clean":"2024-09-11T00:56:28.513475+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:56:28.513475+0000","last_undegraded":"2024-09-11T00:56:28.513475+0000","last_fullsized":"2024-09-11T00:56:28.513475+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3457,"log_dups_size":0,"ondisk_log_size":3457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18121,"num_write_kb":297725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"60'2670","reported_seq":2774,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065793+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:56:28.065793+0000","last_peered":"2024-09-11T00:56:28.065793+0000","last_clean":"2024-09-11T00:56:28.065793+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:56:28.065793+0000","last_undegraded":"2024-09-11T00:56:28.065793+0000","last_fullsized":"2024-09-11T00:56:28.065793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2670,"log_dups_size":0,"ondisk_log_size":2670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14000,"num_write_kb":256678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"62'1368","reported_seq":1505,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.097251+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:56:28.097251+0000","last_peered":"2024-09-11T00:56:28.097251+0000","last_clean":"2024-09-11T00:56:28.097251+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:56:28.097251+0000","last_undegraded":"2024-09-11T00:56:28.097251+0000","last_fullsized":"2024-09-11T00:56:28.097251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1368,"log_dups_size":0,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":26260,"num_write":1345,"num_write_kb":4470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"61'2","reported_seq":121,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068774+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:56:28.068774+0000","last_peered":"2024-09-11T00:56:28.068774+0000","last_clean":"2024-09-11T00:56:28.068774+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:56:28.068774+0000","last_undegraded":"2024-09-11T00:56:28.068774+0000","last_fullsized":"2024-09-11T00:56:28.068774+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"62'3537","reported_seq":3633,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.531936+0000","last_change":"2024-09-11T00:30:54.997397+0000","last_active":"2024-09-11T00:56:28.531936+0000","last_peered":"2024-09-11T00:56:28.531936+0000","last_clean":"2024-09-11T00:56:28.531936+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:56:28.531936+0000","last_undegraded":"2024-09-11T00:56:28.531936+0000","last_fullsized":"2024-09-11T00:56:28.531936+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3537,"log_dups_size":0,"ondisk_log_size":3537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:53.322576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18517,"num_write_kb":279095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"60'2630","reported_seq":2735,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058909+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:56:28.058909+0000","last_peered":"2024-09-11T00:56:28.058909+0000","last_clean":"2024-09-11T00:56:28.058909+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:56:28.058909+0000","last_undegraded":"2024-09-11T00:56:28.058909+0000","last_fullsized":"2024-09-11T00:56:28.058909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":8,"num_write":13276,"num_write_kb":225109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"60'4","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051666+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:56:28.051666+0000","last_peered":"2024-09-11T00:56:28.051666+0000","last_clean":"2024-09-11T00:56:28.051666+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:56:28.051666+0000","last_undegraded":"2024-09-11T00:56:28.051666+0000","last_fullsized":"2024-09-11T00:56:28.051666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"61'2","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066019+0000","last_change":"2024-09-11T00:30:53.986858+0000","last_active":"2024-09-11T00:56:28.066019+0000","last_peered":"2024-09-11T00:56:28.066019+0000","last_clean":"2024-09-11T00:56:28.066019+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:56:28.066019+0000","last_undegraded":"2024-09-11T00:56:28.066019+0000","last_fullsized":"2024-09-11T00:56:28.066019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:33.626193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"62'3470","reported_seq":3566,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397582+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:56:28.397582+0000","last_peered":"2024-09-11T00:56:28.397582+0000","last_clean":"2024-09-11T00:56:28.397582+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:56:28.397582+0000","last_undegraded":"2024-09-11T00:56:28.397582+0000","last_fullsized":"2024-09-11T00:56:28.397582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17553,"num_write_kb":279333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"60'2695","reported_seq":2801,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.046481+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:56:28.046481+0000","last_peered":"2024-09-11T00:56:28.046481+0000","last_clean":"2024-09-11T00:56:28.046481+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:56:28.046481+0000","last_undegraded":"2024-09-11T00:56:28.046481+0000","last_fullsized":"2024-09-11T00:56:28.046481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":14,"num_write":14032,"num_write_kb":244535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"60'3","reported_seq":109,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048672+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:56:28.048672+0000","last_peered":"2024-09-11T00:56:28.048672+0000","last_clean":"2024-09-11T00:56:28.048672+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:56:28.048672+0000","last_undegraded":"2024-09-11T00:56:28.048672+0000","last_fullsized":"2024-09-11T00:56:28.048672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"62'5","reported_seq":124,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.505942+0000","last_change":"2024-09-11T00:30:53.990765+0000","last_active":"2024-09-11T00:56:28.505942+0000","last_peered":"2024-09-11T00:56:28.505942+0000","last_clean":"2024-09-11T00:56:28.505942+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:56:28.505942+0000","last_undegraded":"2024-09-11T00:56:28.505942+0000","last_fullsized":"2024-09-11T00:56:28.505942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:33.720727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"62'3622","reported_seq":3718,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.450009+0000","last_change":"2024-09-11T00:30:54.984949+0000","last_active":"2024-09-11T00:56:28.450009+0000","last_peered":"2024-09-11T00:56:28.450009+0000","last_clean":"2024-09-11T00:56:28.450009+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:56:28.450009+0000","last_undegraded":"2024-09-11T00:56:28.450009+0000","last_fullsized":"2024-09-11T00:56:28.450009+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:54.346937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18608,"num_write_kb":291609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"60'2732","reported_seq":2837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063199+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:56:28.063199+0000","last_peered":"2024-09-11T00:56:28.063199+0000","last_clean":"2024-09-11T00:56:28.063199+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:56:28.063199+0000","last_undegraded":"2024-09-11T00:56:28.063199+0000","last_fullsized":"2024-09-11T00:56:28.063199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13936,"num_write_kb":248214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"60'3","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063396+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:56:28.063396+0000","last_peered":"2024-09-11T00:56:28.063396+0000","last_clean":"2024-09-11T00:56:28.063396+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:56:28.063396+0000","last_undegraded":"2024-09-11T00:56:28.063396+0000","last_fullsized":"2024-09-11T00:56:28.063396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"62'6","reported_seq":146,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500913+0000","last_change":"2024-09-11T00:30:53.986940+0000","last_active":"2024-09-11T00:56:28.500913+0000","last_peered":"2024-09-11T00:56:28.500913+0000","last_clean":"2024-09-11T00:56:28.500913+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:56:28.500913+0000","last_undegraded":"2024-09-11T00:56:28.500913+0000","last_fullsized":"2024-09-11T00:56:28.500913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:11.942187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"62'3257","reported_seq":3355,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.520613+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:56:28.520613+0000","last_peered":"2024-09-11T00:56:28.520613+0000","last_clean":"2024-09-11T00:56:28.520613+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:56:28.520613+0000","last_undegraded":"2024-09-11T00:56:28.520613+0000","last_fullsized":"2024-09-11T00:56:28.520613+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17168,"num_write_kb":277608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'2688","reported_seq":2794,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048103+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:56:28.048103+0000","last_peered":"2024-09-11T00:56:28.048103+0000","last_clean":"2024-09-11T00:56:28.048103+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:56:28.048103+0000","last_undegraded":"2024-09-11T00:56:28.048103+0000","last_fullsized":"2024-09-11T00:56:28.048103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13441,"num_write_kb":246089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"62'5","reported_seq":141,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.085125+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:56:31.085125+0000","last_peered":"2024-09-11T00:56:31.085125+0000","last_clean":"2024-09-11T00:56:31.085125+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:56:31.085125+0000","last_undegraded":"2024-09-11T00:56:31.085125+0000","last_fullsized":"2024-09-11T00:56:31.085125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"62'65","reported_seq":245,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:37.625005+0000","last_change":"2024-09-11T00:30:53.988539+0000","last_active":"2024-09-11T00:56:37.625005+0000","last_peered":"2024-09-11T00:56:37.625005+0000","last_clean":"2024-09-11T00:56:37.625005+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:56:37.625005+0000","last_undegraded":"2024-09-11T00:56:37.625005+0000","last_fullsized":"2024-09-11T00:56:37.625005+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:19.083664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":126,"num_write":62,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"62'3386","reported_seq":3484,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.422439+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:56:28.422439+0000","last_peered":"2024-09-11T00:56:28.422439+0000","last_clean":"2024-09-11T00:56:28.422439+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:56:28.422439+0000","last_undegraded":"2024-09-11T00:56:28.422439+0000","last_fullsized":"2024-09-11T00:56:28.422439+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18005,"num_write_kb":276900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'3194","reported_seq":3298,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.050561+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:56:28.050561+0000","last_peered":"2024-09-11T00:56:28.050561+0000","last_clean":"2024-09-11T00:56:28.050561+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:56:28.050561+0000","last_undegraded":"2024-09-11T00:56:28.050561+0000","last_fullsized":"2024-09-11T00:56:28.050561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16397,"num_write_kb":262231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"62'521","reported_seq":641,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:32.047226+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:56:32.047226+0000","last_peered":"2024-09-11T00:56:32.047226+0000","last_clean":"2024-09-11T00:56:32.047226+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:56:32.047226+0000","last_undegraded":"2024-09-11T00:56:32.047226+0000","last_fullsized":"2024-09-11T00:56:32.047226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":41,"num_write":1157,"num_write_kb":372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"62'9","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.502805+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:56:28.502805+0000","last_peered":"2024-09-11T00:56:28.502805+0000","last_clean":"2024-09-11T00:56:28.502805+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:56:28.502805+0000","last_undegraded":"2024-09-11T00:56:28.502805+0000","last_fullsized":"2024-09-11T00:56:28.502805+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"61'3259","reported_seq":3357,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063309+0000","last_change":"2024-09-11T00:30:54.993118+0000","last_active":"2024-09-11T00:56:28.063309+0000","last_peered":"2024-09-11T00:56:28.063309+0000","last_clean":"2024-09-11T00:56:28.063309+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:56:28.063309+0000","last_undegraded":"2024-09-11T00:56:28.063309+0000","last_fullsized":"2024-09-11T00:56:28.063309+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:07:41.148370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16779,"num_write_kb":276080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"60'3127","reported_seq":3233,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052954+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:56:28.052954+0000","last_peered":"2024-09-11T00:56:28.052954+0000","last_clean":"2024-09-11T00:56:28.052954+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:56:28.052954+0000","last_undegraded":"2024-09-11T00:56:28.052954+0000","last_fullsized":"2024-09-11T00:56:28.052954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":12,"num_write":16580,"num_write_kb":258802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"60'4","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069515+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:56:28.069515+0000","last_peered":"2024-09-11T00:56:28.069515+0000","last_clean":"2024-09-11T00:56:28.069515+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:56:28.069515+0000","last_undegraded":"2024-09-11T00:56:28.069515+0000","last_fullsized":"2024-09-11T00:56:28.069515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'1","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052292+0000","last_change":"2024-09-11T00:30:53.986843+0000","last_active":"2024-09-11T00:56:28.052292+0000","last_peered":"2024-09-11T00:56:28.052292+0000","last_clean":"2024-09-11T00:56:28.052292+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:56:28.052292+0000","last_undegraded":"2024-09-11T00:56:28.052292+0000","last_fullsized":"2024-09-11T00:56:28.052292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:44.684154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"62'3801","reported_seq":3898,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.047541+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:56:28.047541+0000","last_peered":"2024-09-11T00:56:28.047541+0000","last_clean":"2024-09-11T00:56:28.047541+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:56:28.047541+0000","last_undegraded":"2024-09-11T00:56:28.047541+0000","last_fullsized":"2024-09-11T00:56:28.047541+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20214,"num_write_kb":289874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"60'2498","reported_seq":2605,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075698+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:56:28.075698+0000","last_peered":"2024-09-11T00:56:28.075698+0000","last_clean":"2024-09-11T00:56:28.075698+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:56:28.075698+0000","last_undegraded":"2024-09-11T00:56:28.075698+0000","last_fullsized":"2024-09-11T00:56:28.075698+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2498,"log_dups_size":0,"ondisk_log_size":2498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":13564,"num_write_kb":230524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"60'5","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059256+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:56:28.059256+0000","last_peered":"2024-09-11T00:56:28.059256+0000","last_clean":"2024-09-11T00:56:28.059256+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:56:28.059256+0000","last_undegraded":"2024-09-11T00:56:28.059256+0000","last_fullsized":"2024-09-11T00:56:28.059256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"62'5","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510149+0000","last_change":"2024-09-11T00:30:53.986896+0000","last_active":"2024-09-11T00:56:28.510149+0000","last_peered":"2024-09-11T00:56:28.510149+0000","last_clean":"2024-09-11T00:56:28.510149+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:56:28.510149+0000","last_undegraded":"2024-09-11T00:56:28.510149+0000","last_fullsized":"2024-09-11T00:56:28.510149+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:06.590278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"62'3103","reported_seq":3201,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058933+0000","last_change":"2024-09-11T00:30:55.001370+0000","last_active":"2024-09-11T00:56:28.058933+0000","last_peered":"2024-09-11T00:56:28.058933+0000","last_clean":"2024-09-11T00:56:28.058933+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:56:28.058933+0000","last_undegraded":"2024-09-11T00:56:28.058933+0000","last_fullsized":"2024-09-11T00:56:28.058933+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:56.592767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16058,"num_write_kb":244544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"60'2543","reported_seq":2648,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067860+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:56:28.067860+0000","last_peered":"2024-09-11T00:56:28.067860+0000","last_clean":"2024-09-11T00:56:28.067860+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:56:28.067860+0000","last_undegraded":"2024-09-11T00:56:28.067860+0000","last_fullsized":"2024-09-11T00:56:28.067860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":230458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"60'5","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067593+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:56:28.067593+0000","last_peered":"2024-09-11T00:56:28.067593+0000","last_clean":"2024-09-11T00:56:28.067593+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:56:28.067593+0000","last_undegraded":"2024-09-11T00:56:28.067593+0000","last_fullsized":"2024-09-11T00:56:28.067593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"62'5","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.494925+0000","last_change":"2024-09-11T00:30:53.978480+0000","last_active":"2024-09-11T00:56:28.494925+0000","last_peered":"2024-09-11T00:56:28.494925+0000","last_clean":"2024-09-11T00:56:28.494925+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:56:28.494925+0000","last_undegraded":"2024-09-11T00:56:28.494925+0000","last_fullsized":"2024-09-11T00:56:28.494925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:58.900565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"61'3199","reported_seq":3296,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058202+0000","last_change":"2024-09-11T00:30:54.997641+0000","last_active":"2024-09-11T00:56:28.058202+0000","last_peered":"2024-09-11T00:56:28.058202+0000","last_clean":"2024-09-11T00:56:28.058202+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:56:28.058202+0000","last_undegraded":"2024-09-11T00:56:28.058202+0000","last_fullsized":"2024-09-11T00:56:28.058202+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:18.437490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15924,"num_write_kb":260293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"60'3298","reported_seq":3404,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057304+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:56:28.057304+0000","last_peered":"2024-09-11T00:56:28.057304+0000","last_clean":"2024-09-11T00:56:28.057304+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:56:28.057304+0000","last_undegraded":"2024-09-11T00:56:28.057304+0000","last_fullsized":"2024-09-11T00:56:28.057304+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17655,"num_write_kb":267324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"62'73","reported_seq":262,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:33.050070+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:56:33.050070+0000","last_peered":"2024-09-11T00:56:33.050070+0000","last_clean":"2024-09-11T00:56:33.050070+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:56:33.050070+0000","last_undegraded":"2024-09-11T00:56:33.050070+0000","last_fullsized":"2024-09-11T00:56:33.050070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":129,"num_write":68,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"62'5","reported_seq":146,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500023+0000","last_change":"2024-09-11T00:30:53.979031+0000","last_active":"2024-09-11T00:56:28.500023+0000","last_peered":"2024-09-11T00:56:28.500023+0000","last_clean":"2024-09-11T00:56:28.500023+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:56:28.500023+0000","last_undegraded":"2024-09-11T00:56:28.500023+0000","last_fullsized":"2024-09-11T00:56:28.500023+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:36.604377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"62'2728","reported_seq":2824,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.475254+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:56:28.475254+0000","last_peered":"2024-09-11T00:56:28.475254+0000","last_clean":"2024-09-11T00:56:28.475254+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:56:28.475254+0000","last_undegraded":"2024-09-11T00:56:28.475254+0000","last_fullsized":"2024-09-11T00:56:28.475254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2728,"log_dups_size":0,"ondisk_log_size":2728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:39.629122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":237243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"60'2962","reported_seq":3068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051045+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:56:28.051045+0000","last_peered":"2024-09-11T00:56:28.051045+0000","last_clean":"2024-09-11T00:56:28.051045+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:56:28.051045+0000","last_undegraded":"2024-09-11T00:56:28.051045+0000","last_fullsized":"2024-09-11T00:56:28.051045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15238,"num_write_kb":265720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"62'8","reported_seq":116,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.640904+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:56:28.640904+0000","last_peered":"2024-09-11T00:56:28.640904+0000","last_clean":"2024-09-11T00:56:28.640904+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:56:28.640904+0000","last_undegraded":"2024-09-11T00:56:28.640904+0000","last_fullsized":"2024-09-11T00:56:28.640904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"62'499","reported_seq":618,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:35.590292+0000","last_change":"2024-09-11T00:30:53.989299+0000","last_active":"2024-09-11T00:56:35.590292+0000","last_peered":"2024-09-11T00:56:35.590292+0000","last_clean":"2024-09-11T00:56:35.590292+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:56:35.590292+0000","last_undegraded":"2024-09-11T00:56:35.590292+0000","last_fullsized":"2024-09-11T00:56:35.590292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:34:49.298915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":1123,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"62'3325","reported_seq":3422,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.494312+0000","last_change":"2024-09-11T00:30:54.990026+0000","last_active":"2024-09-11T00:56:28.494312+0000","last_peered":"2024-09-11T00:56:28.494312+0000","last_clean":"2024-09-11T00:56:28.494312+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:56:28.494312+0000","last_undegraded":"2024-09-11T00:56:28.494312+0000","last_fullsized":"2024-09-11T00:56:28.494312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:02:10.800530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17451,"num_write_kb":278810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"60'3332","reported_seq":3438,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051728+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:56:28.051728+0000","last_peered":"2024-09-11T00:56:28.051728+0000","last_clean":"2024-09-11T00:56:28.051728+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:56:28.051728+0000","last_undegraded":"2024-09-11T00:56:28.051728+0000","last_fullsized":"2024-09-11T00:56:28.051728+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3332,"log_dups_size":0,"ondisk_log_size":3332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17164,"num_write_kb":285768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"31'1","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.072247+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:56:28.072247+0000","last_peered":"2024-09-11T00:56:28.072247+0000","last_clean":"2024-09-11T00:56:28.072247+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:56:28.072247+0000","last_undegraded":"2024-09-11T00:56:28.072247+0000","last_fullsized":"2024-09-11T00:56:28.072247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":204,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066124+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:56:28.066124+0000","last_peered":"2024-09-11T00:56:28.066124+0000","last_clean":"2024-09-11T00:56:28.066124+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:56:28.066124+0000","last_undegraded":"2024-09-11T00:56:28.066124+0000","last_fullsized":"2024-09-11T00:56:28.066124+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"62'4","reported_seq":103,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488198+0000","last_change":"2024-09-11T00:30:53.982142+0000","last_active":"2024-09-11T00:56:28.488198+0000","last_peered":"2024-09-11T00:56:28.488198+0000","last_clean":"2024-09-11T00:56:28.488198+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:56:28.488198+0000","last_undegraded":"2024-09-11T00:56:28.488198+0000","last_fullsized":"2024-09-11T00:56:28.488198+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:41.558944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"62'2541","reported_seq":2637,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500843+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:56:28.500843+0000","last_peered":"2024-09-11T00:56:28.500843+0000","last_clean":"2024-09-11T00:56:28.500843+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:56:28.500843+0000","last_undegraded":"2024-09-11T00:56:28.500843+0000","last_fullsized":"2024-09-11T00:56:28.500843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12611,"num_write_kb":206053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"60'2809","reported_seq":2917,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069156+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:56:28.069156+0000","last_peered":"2024-09-11T00:56:28.069156+0000","last_clean":"2024-09-11T00:56:28.069156+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:56:28.069156+0000","last_undegraded":"2024-09-11T00:56:28.069156+0000","last_fullsized":"2024-09-11T00:56:28.069156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2809,"log_dups_size":0,"ondisk_log_size":2809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":34,"num_write":15096,"num_write_kb":232439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"60'2","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.053330+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:56:28.053330+0000","last_peered":"2024-09-11T00:56:28.053330+0000","last_clean":"2024-09-11T00:56:28.053330+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:56:28.053330+0000","last_undegraded":"2024-09-11T00:56:28.053330+0000","last_fullsized":"2024-09-11T00:56:28.053330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"62'4","reported_seq":102,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.498262+0000","last_change":"2024-09-11T00:30:53.989292+0000","last_active":"2024-09-11T00:56:28.498262+0000","last_peered":"2024-09-11T00:56:28.498262+0000","last_clean":"2024-09-11T00:56:28.498262+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:56:28.498262+0000","last_undegraded":"2024-09-11T00:56:28.498262+0000","last_fullsized":"2024-09-11T00:56:28.498262+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:44.071338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"62'3293","reported_seq":3389,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.437769+0000","last_change":"2024-09-11T00:30:54.981630+0000","last_active":"2024-09-11T00:56:28.437769+0000","last_peered":"2024-09-11T00:56:28.437769+0000","last_clean":"2024-09-11T00:56:28.437769+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:56:28.437769+0000","last_undegraded":"2024-09-11T00:56:28.437769+0000","last_fullsized":"2024-09-11T00:56:28.437769+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:12.832327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17661,"num_write_kb":267414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"60'3034","reported_seq":3140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066511+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:56:28.066511+0000","last_peered":"2024-09-11T00:56:28.066511+0000","last_clean":"2024-09-11T00:56:28.066511+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:56:28.066511+0000","last_undegraded":"2024-09-11T00:56:28.066511+0000","last_fullsized":"2024-09-11T00:56:28.066511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3034,"log_dups_size":0,"ondisk_log_size":3034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":15513,"num_write_kb":283129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"60'2","reported_seq":122,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060350+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:56:28.060350+0000","last_peered":"2024-09-11T00:56:28.060350+0000","last_clean":"2024-09-11T00:56:28.060350+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:56:28.060350+0000","last_undegraded":"2024-09-11T00:56:28.060350+0000","last_fullsized":"2024-09-11T00:56:28.060350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"62'1443","reported_seq":1586,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510169+0000","last_change":"2024-09-11T00:30:53.984013+0000","last_active":"2024-09-11T00:56:28.510169+0000","last_peered":"2024-09-11T00:56:28.510169+0000","last_clean":"2024-09-11T00:56:28.510169+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:56:28.510169+0000","last_undegraded":"2024-09-11T00:56:28.510169+0000","last_fullsized":"2024-09-11T00:56:28.510169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1443,"log_dups_size":0,"ondisk_log_size":1443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:49.963623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":36861,"num_write":1416,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'2904","reported_seq":3001,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.471315+0000","last_change":"2024-09-11T00:30:54.996860+0000","last_active":"2024-09-11T00:56:28.471315+0000","last_peered":"2024-09-11T00:56:28.471315+0000","last_clean":"2024-09-11T00:56:28.471315+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:56:28.471315+0000","last_undegraded":"2024-09-11T00:56:28.471315+0000","last_fullsized":"2024-09-11T00:56:28.471315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:43.516748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15203,"num_write_kb":241147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"60'3390","reported_seq":3494,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069427+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:56:28.069427+0000","last_peered":"2024-09-11T00:56:28.069427+0000","last_clean":"2024-09-11T00:56:28.069427+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:56:28.069427+0000","last_undegraded":"2024-09-11T00:56:28.069427+0000","last_fullsized":"2024-09-11T00:56:28.069427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17423,"num_write_kb":299173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"60'6","reported_seq":140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.070765+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:56:31.070765+0000","last_peered":"2024-09-11T00:56:31.070765+0000","last_clean":"2024-09-11T00:56:31.070765+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:56:31.070765+0000","last_undegraded":"2024-09-11T00:56:31.070765+0000","last_fullsized":"2024-09-11T00:56:31.070765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"62'3","reported_seq":101,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062449+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:56:28.062449+0000","last_peered":"2024-09-11T00:56:28.062449+0000","last_clean":"2024-09-11T00:56:28.062449+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:56:28.062449+0000","last_undegraded":"2024-09-11T00:56:28.062449+0000","last_fullsized":"2024-09-11T00:56:28.062449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:19.891052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"62'3177","reported_seq":3279,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.444533+0000","last_change":"2024-09-11T00:31:26.120086+0000","last_active":"2024-09-11T00:56:28.444533+0000","last_peered":"2024-09-11T00:56:28.444533+0000","last_clean":"2024-09-11T00:56:28.444533+0000","last_became_active":"2024-09-11T00:31:26.119641+0000","last_became_peered":"2024-09-11T00:31:26.119641+0000","last_unstale":"2024-09-11T00:56:28.444533+0000","last_undegraded":"2024-09-11T00:56:28.444533+0000","last_fullsized":"2024-09-11T00:56:28.444533+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3177,"log_dups_size":0,"ondisk_log_size":3177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:21:15.814831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16730,"num_write_kb":276023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"60'2906","reported_seq":3010,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.077855+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:56:28.077855+0000","last_peered":"2024-09-11T00:56:28.077855+0000","last_clean":"2024-09-11T00:56:28.077855+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:56:28.077855+0000","last_undegraded":"2024-09-11T00:56:28.077855+0000","last_fullsized":"2024-09-11T00:56:28.077855+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14712,"num_write_kb":255517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"60'5","reported_seq":125,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069857+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:56:28.069857+0000","last_peered":"2024-09-11T00:56:28.069857+0000","last_clean":"2024-09-11T00:56:28.069857+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:56:28.069857+0000","last_undegraded":"2024-09-11T00:56:28.069857+0000","last_fullsized":"2024-09-11T00:56:28.069857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"62'3","reported_seq":101,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.487423+0000","last_change":"2024-09-11T00:30:53.983352+0000","last_active":"2024-09-11T00:56:28.487423+0000","last_peered":"2024-09-11T00:56:28.487423+0000","last_clean":"2024-09-11T00:56:28.487423+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:56:28.487423+0000","last_undegraded":"2024-09-11T00:56:28.487423+0000","last_fullsized":"2024-09-11T00:56:28.487423+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:43.826114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"62'2973","reported_seq":3063,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.532651+0000","last_change":"2024-09-11T00:31:26.119816+0000","last_active":"2024-09-11T00:56:28.532651+0000","last_peered":"2024-09-11T00:56:28.532651+0000","last_clean":"2024-09-11T00:56:28.532651+0000","last_became_active":"2024-09-11T00:31:26.119440+0000","last_became_peered":"2024-09-11T00:31:26.119440+0000","last_unstale":"2024-09-11T00:56:28.532651+0000","last_undegraded":"2024-09-11T00:56:28.532651+0000","last_fullsized":"2024-09-11T00:56:28.532651+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:21:45.707321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15629,"num_write_kb":242839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"60'3355","reported_seq":3459,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062827+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:56:28.062827+0000","last_peered":"2024-09-11T00:56:28.062827+0000","last_clean":"2024-09-11T00:56:28.062827+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:56:28.062827+0000","last_undegraded":"2024-09-11T00:56:28.062827+0000","last_fullsized":"2024-09-11T00:56:28.062827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18048,"num_write_kb":289456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"62'12","reported_seq":177,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.077796+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:56:31.077796+0000","last_peered":"2024-09-11T00:56:31.077796+0000","last_clean":"2024-09-11T00:56:31.077796+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:56:31.077796+0000","last_undegraded":"2024-09-11T00:56:31.077796+0000","last_fullsized":"2024-09-11T00:56:31.077796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":177,"num_read_kb":177,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"62'2","reported_seq":99,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488958+0000","last_change":"2024-09-11T00:30:53.980703+0000","last_active":"2024-09-11T00:56:28.488958+0000","last_peered":"2024-09-11T00:56:28.488958+0000","last_clean":"2024-09-11T00:56:28.488958+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:56:28.488958+0000","last_undegraded":"2024-09-11T00:56:28.488958+0000","last_fullsized":"2024-09-11T00:56:28.488958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:51.050512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"62'2678","reported_seq":2774,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.413635+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:56:28.413635+0000","last_peered":"2024-09-11T00:56:28.413635+0000","last_clean":"2024-09-11T00:56:28.413635+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:56:28.413635+0000","last_undegraded":"2024-09-11T00:56:28.413635+0000","last_fullsized":"2024-09-11T00:56:28.413635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2678,"log_dups_size":0,"ondisk_log_size":2678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13518,"num_write_kb":242590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"60'2791","reported_seq":2895,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.056927+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:56:28.056927+0000","last_peered":"2024-09-11T00:56:28.056927+0000","last_clean":"2024-09-11T00:56:28.056927+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:56:28.056927+0000","last_undegraded":"2024-09-11T00:56:28.056927+0000","last_fullsized":"2024-09-11T00:56:28.056927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2791,"log_dups_size":0,"ondisk_log_size":2791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":254542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"60'6","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063217+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:56:28.063217+0000","last_peered":"2024-09-11T00:56:28.063217+0000","last_clean":"2024-09-11T00:56:28.063217+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:56:28.063217+0000","last_undegraded":"2024-09-11T00:56:28.063217+0000","last_fullsized":"2024-09-11T00:56:28.063217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"62'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059468+0000","last_change":"2024-09-11T00:30:53.987829+0000","last_active":"2024-09-11T00:56:28.059468+0000","last_peered":"2024-09-11T00:56:28.059468+0000","last_clean":"2024-09-11T00:56:28.059468+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:56:28.059468+0000","last_undegraded":"2024-09-11T00:56:28.059468+0000","last_fullsized":"2024-09-11T00:56:28.059468+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:57:43.792089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"62'3272","reported_seq":3367,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066748+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:56:28.066748+0000","last_peered":"2024-09-11T00:56:28.066748+0000","last_clean":"2024-09-11T00:56:28.066748+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:56:28.066748+0000","last_undegraded":"2024-09-11T00:56:28.066748+0000","last_fullsized":"2024-09-11T00:56:28.066748+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16692,"num_write_kb":275642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"60'3405","reported_seq":3511,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066672+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:56:28.066672+0000","last_peered":"2024-09-11T00:56:28.066672+0000","last_clean":"2024-09-11T00:56:28.066672+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:56:28.066672+0000","last_undegraded":"2024-09-11T00:56:28.066672+0000","last_fullsized":"2024-09-11T00:56:28.066672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3405,"log_dups_size":0,"ondisk_log_size":3405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":17487,"num_write_kb":315618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"60'5","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067380+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:56:28.067380+0000","last_peered":"2024-09-11T00:56:28.067380+0000","last_clean":"2024-09-11T00:56:28.067380+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:56:28.067380+0000","last_undegraded":"2024-09-11T00:56:28.067380+0000","last_fullsized":"2024-09-11T00:56:28.067380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"62'367","reported_seq":572,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.957442+0000","last_change":"2024-09-11T00:30:53.984057+0000","last_active":"2024-09-11T00:56:34.957442+0000","last_peered":"2024-09-11T00:56:34.957442+0000","last_clean":"2024-09-11T00:56:34.957442+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:56:34.957442+0000","last_undegraded":"2024-09-11T00:56:34.957442+0000","last_fullsized":"2024-09-11T00:56:34.957442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:10.477916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":191,"num_read_kb":78004,"num_write":360,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"62'4540","reported_seq":4637,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.544292+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:56:28.544292+0000","last_peered":"2024-09-11T00:56:28.544292+0000","last_clean":"2024-09-11T00:56:28.544292+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:56:28.544292+0000","last_undegraded":"2024-09-11T00:56:28.544292+0000","last_fullsized":"2024-09-11T00:56:28.544292+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23424,"num_write_kb":360436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"60'2652","reported_seq":2757,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051484+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:56:28.051484+0000","last_peered":"2024-09-11T00:56:28.051484+0000","last_clean":"2024-09-11T00:56:28.051484+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:56:28.051484+0000","last_undegraded":"2024-09-11T00:56:28.051484+0000","last_fullsized":"2024-09-11T00:56:28.051484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":13591,"num_write_kb":238653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"62'9","reported_seq":143,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.641347+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:56:28.641347+0000","last_peered":"2024-09-11T00:56:28.641347+0000","last_clean":"2024-09-11T00:56:28.641347+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:56:28.641347+0000","last_undegraded":"2024-09-11T00:56:28.641347+0000","last_fullsized":"2024-09-11T00:56:28.641347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":112,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"62'37","reported_seq":197,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.985643+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:56:28.985643+0000","last_peered":"2024-09-11T00:56:28.985643+0000","last_clean":"2024-09-11T00:56:28.985643+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:56:28.985643+0000","last_undegraded":"2024-09-11T00:56:28.985643+0000","last_fullsized":"2024-09-11T00:56:28.985643+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":36,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"62'4259","reported_seq":4356,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.548466+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:56:28.548466+0000","last_peered":"2024-09-11T00:56:28.548466+0000","last_clean":"2024-09-11T00:56:28.548466+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:56:28.548466+0000","last_undegraded":"2024-09-11T00:56:28.548466+0000","last_fullsized":"2024-09-11T00:56:28.548466+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22192,"num_write_kb":330506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"60'2765","reported_seq":2868,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067495+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:56:28.067495+0000","last_peered":"2024-09-11T00:56:28.067495+0000","last_clean":"2024-09-11T00:56:28.067495+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:56:28.067495+0000","last_undegraded":"2024-09-11T00:56:28.067495+0000","last_fullsized":"2024-09-11T00:56:28.067495+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14757,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"62'389","reported_seq":566,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.078750+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:56:31.078750+0000","last_peered":"2024-09-11T00:56:31.078750+0000","last_clean":"2024-09-11T00:56:31.078750+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:56:31.078750+0000","last_undegraded":"2024-09-11T00:56:31.078750+0000","last_fullsized":"2024-09-11T00:56:31.078750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":127,"num_read_kb":56252,"num_write":381,"num_write_kb":4290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"62'5","reported_seq":103,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.528773+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:56:28.528773+0000","last_peered":"2024-09-11T00:56:28.528773+0000","last_clean":"2024-09-11T00:56:28.528773+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:56:28.528773+0000","last_undegraded":"2024-09-11T00:56:28.528773+0000","last_fullsized":"2024-09-11T00:56:28.528773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"62'3220","reported_seq":3316,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.464767+0000","last_change":"2024-09-11T00:30:54.986017+0000","last_active":"2024-09-11T00:56:28.464767+0000","last_peered":"2024-09-11T00:56:28.464767+0000","last_clean":"2024-09-11T00:56:28.464767+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:56:28.464767+0000","last_undegraded":"2024-09-11T00:56:28.464767+0000","last_fullsized":"2024-09-11T00:56:28.464767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:58:12.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16450,"num_write_kb":262397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"60'3928","reported_seq":4033,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064969+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:56:28.064969+0000","last_peered":"2024-09-11T00:56:28.064969+0000","last_clean":"2024-09-11T00:56:28.064969+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:56:28.064969+0000","last_undegraded":"2024-09-11T00:56:28.064969+0000","last_fullsized":"2024-09-11T00:56:28.064969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20599,"num_write_kb":340888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"62'43","reported_seq":190,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.086970+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:56:31.086970+0000","last_peered":"2024-09-11T00:56:31.086970+0000","last_clean":"2024-09-11T00:56:31.086970+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:56:31.086970+0000","last_undegraded":"2024-09-11T00:56:31.086970+0000","last_fullsized":"2024-09-11T00:56:31.086970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":42,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"62'8","reported_seq":148,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.516812+0000","last_change":"2024-09-11T00:30:53.988462+0000","last_active":"2024-09-11T00:56:28.516812+0000","last_peered":"2024-09-11T00:56:28.516812+0000","last_clean":"2024-09-11T00:56:28.516812+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:56:28.516812+0000","last_undegraded":"2024-09-11T00:56:28.516812+0000","last_fullsized":"2024-09-11T00:56:28.516812+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:22.385557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"62'4111","reported_seq":4207,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.522560+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:56:28.522560+0000","last_peered":"2024-09-11T00:56:28.522560+0000","last_clean":"2024-09-11T00:56:28.522560+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:56:28.522560+0000","last_undegraded":"2024-09-11T00:56:28.522560+0000","last_fullsized":"2024-09-11T00:56:28.522560+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21746,"num_write_kb":328824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"60'2863","reported_seq":2972,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060043+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:56:28.060043+0000","last_peered":"2024-09-11T00:56:28.060043+0000","last_clean":"2024-09-11T00:56:28.060043+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:56:28.060043+0000","last_undegraded":"2024-09-11T00:56:28.060043+0000","last_fullsized":"2024-09-11T00:56:28.060043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2863,"log_dups_size":0,"ondisk_log_size":2863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":46,"num_write":14990,"num_write_kb":244331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"60'2","reported_seq":107,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068726+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:56:28.068726+0000","last_peered":"2024-09-11T00:56:28.068726+0000","last_clean":"2024-09-11T00:56:28.068726+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:56:28.068726+0000","last_undegraded":"2024-09-11T00:56:28.068726+0000","last_fullsized":"2024-09-11T00:56:28.068726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":203,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.264447+0000","last_change":"2024-09-11T00:30:53.986646+0000","last_active":"2024-09-11T00:56:34.264447+0000","last_peered":"2024-09-11T00:56:34.264447+0000","last_clean":"2024-09-11T00:56:34.264447+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:56:34.264447+0000","last_undegraded":"2024-09-11T00:56:34.264447+0000","last_fullsized":"2024-09-11T00:56:34.264447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:27.447940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":279,"num_read_kb":279,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"62'3641","reported_seq":3737,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514237+0000","last_change":"2024-09-11T00:30:54.982486+0000","last_active":"2024-09-11T00:56:28.514237+0000","last_peered":"2024-09-11T00:56:28.514237+0000","last_clean":"2024-09-11T00:56:28.514237+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:56:28.514237+0000","last_undegraded":"2024-09-11T00:56:28.514237+0000","last_fullsized":"2024-09-11T00:56:28.514237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:41.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19470,"num_write_kb":282742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"60'2896","reported_seq":3003,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.076032+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:56:28.076032+0000","last_peered":"2024-09-11T00:56:28.076032+0000","last_clean":"2024-09-11T00:56:28.076032+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:56:28.076032+0000","last_undegraded":"2024-09-11T00:56:28.076032+0000","last_fullsized":"2024-09-11T00:56:28.076032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":22,"num_write":14816,"num_write_kb":255894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"62'5","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064417+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:56:28.064417+0000","last_peered":"2024-09-11T00:56:28.064417+0000","last_clean":"2024-09-11T00:56:28.064417+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:56:28.064417+0000","last_undegraded":"2024-09-11T00:56:28.064417+0000","last_fullsized":"2024-09-11T00:56:28.064417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"62'6","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.504113+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:56:28.504113+0000","last_peered":"2024-09-11T00:56:28.504113+0000","last_clean":"2024-09-11T00:56:28.504113+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:56:28.504113+0000","last_undegraded":"2024-09-11T00:56:28.504113+0000","last_fullsized":"2024-09-11T00:56:28.504113+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"62'3862","reported_seq":3961,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.513928+0000","last_change":"2024-09-11T00:30:54.985091+0000","last_active":"2024-09-11T00:56:28.513928+0000","last_peered":"2024-09-11T00:56:28.513928+0000","last_clean":"2024-09-11T00:56:28.513928+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:56:28.513928+0000","last_undegraded":"2024-09-11T00:56:28.513928+0000","last_fullsized":"2024-09-11T00:56:28.513928+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:33.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20250,"num_write_kb":318736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"62'29","reported_seq":128,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.490960+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:56:28.490960+0000","last_peered":"2024-09-11T00:56:28.490960+0000","last_clean":"2024-09-11T00:56:28.490960+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:56:28.490960+0000","last_undegraded":"2024-09-11T00:56:28.490960+0000","last_fullsized":"2024-09-11T00:56:28.490960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"60'3526","reported_seq":3630,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048540+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:56:28.048540+0000","last_peered":"2024-09-11T00:56:28.048540+0000","last_clean":"2024-09-11T00:56:28.048540+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:56:28.048540+0000","last_undegraded":"2024-09-11T00:56:28.048540+0000","last_fullsized":"2024-09-11T00:56:28.048540+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":18407,"num_write_kb":311214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"60'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064114+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:56:28.064114+0000","last_peered":"2024-09-11T00:56:28.064114+0000","last_clean":"2024-09-11T00:56:28.064114+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:56:28.064114+0000","last_undegraded":"2024-09-11T00:56:28.064114+0000","last_fullsized":"2024-09-11T00:56:28.064114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"61'3026","reported_seq":3123,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.053185+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:56:28.053185+0000","last_peered":"2024-09-11T00:56:28.053185+0000","last_clean":"2024-09-11T00:56:28.053185+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:56:28.053185+0000","last_undegraded":"2024-09-11T00:56:28.053185+0000","last_fullsized":"2024-09-11T00:56:28.053185+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:11.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16026,"num_write_kb":256702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"61'972","reported_seq":1122,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065625+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:56:28.065625+0000","last_peered":"2024-09-11T00:56:28.065625+0000","last_clean":"2024-09-11T00:56:28.065625+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:56:28.065625+0000","last_undegraded":"2024-09-11T00:56:28.065625+0000","last_fullsized":"2024-09-11T00:56:28.065625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":972,"log_dups_size":0,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":7829,"num_write":949,"num_write_kb":4657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"60'3393","reported_seq":3498,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057539+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:56:28.057539+0000","last_peered":"2024-09-11T00:56:28.057539+0000","last_clean":"2024-09-11T00:56:28.057539+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:56:28.057539+0000","last_undegraded":"2024-09-11T00:56:28.057539+0000","last_fullsized":"2024-09-11T00:56:28.057539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":17621,"num_write_kb":291741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"60'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051439+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:56:28.051439+0000","last_peered":"2024-09-11T00:56:28.051439+0000","last_clean":"2024-09-11T00:56:28.051439+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:56:28.051439+0000","last_undegraded":"2024-09-11T00:56:28.051439+0000","last_fullsized":"2024-09-11T00:56:28.051439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"62'3945","reported_seq":4040,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.470154+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:56:28.470154+0000","last_peered":"2024-09-11T00:56:28.470154+0000","last_clean":"2024-09-11T00:56:28.470154+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:56:28.470154+0000","last_undegraded":"2024-09-11T00:56:28.470154+0000","last_fullsized":"2024-09-11T00:56:28.470154+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20523,"num_write_kb":340217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"62'101","reported_seq":208,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:37.634421+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:56:37.634421+0000","last_peered":"2024-09-11T00:56:37.634421+0000","last_clean":"2024-09-11T00:56:37.634421+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:56:37.634421+0000","last_undegraded":"2024-09-11T00:56:37.634421+0000","last_fullsized":"2024-09-11T00:56:37.634421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":768543,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":728,"num_write":83,"num_write_kb":803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"60'2411","reported_seq":2516,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.049073+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:56:28.049073+0000","last_peered":"2024-09-11T00:56:28.049073+0000","last_clean":"2024-09-11T00:56:28.049073+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:56:28.049073+0000","last_undegraded":"2024-09-11T00:56:28.049073+0000","last_fullsized":"2024-09-11T00:56:28.049073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":8,"num_write":12211,"num_write_kb":212619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"62'3","reported_seq":142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.049506+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:56:28.049506+0000","last_peered":"2024-09-11T00:56:28.049506+0000","last_clean":"2024-09-11T00:56:28.049506+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:56:28.049506+0000","last_undegraded":"2024-09-11T00:56:28.049506+0000","last_fullsized":"2024-09-11T00:56:28.049506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"62'2354","reported_seq":2449,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.471047+0000","last_change":"2024-09-11T00:30:54.998464+0000","last_active":"2024-09-11T00:56:28.471047+0000","last_peered":"2024-09-11T00:56:28.471047+0000","last_clean":"2024-09-11T00:56:28.471047+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:56:28.471047+0000","last_undegraded":"2024-09-11T00:56:28.471047+0000","last_fullsized":"2024-09-11T00:56:28.471047+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:56.469791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":205009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"60'676","reported_seq":797,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059574+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:56:28.059574+0000","last_peered":"2024-09-11T00:56:28.059574+0000","last_clean":"2024-09-11T00:56:28.059574+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:56:28.059574+0000","last_undegraded":"2024-09-11T00:56:28.059574+0000","last_fullsized":"2024-09-11T00:56:28.059574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":51443,"num_write":670,"num_write_kb":4291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"60'2936","reported_seq":3039,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058263+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:56:28.058263+0000","last_peered":"2024-09-11T00:56:28.058263+0000","last_clean":"2024-09-11T00:56:28.058263+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:56:28.058263+0000","last_undegraded":"2024-09-11T00:56:28.058263+0000","last_fullsized":"2024-09-11T00:56:28.058263+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2936,"log_dups_size":0,"ondisk_log_size":2936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":241742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"62'45","reported_seq":184,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488110+0000","last_change":"2024-09-11T00:30:53.987612+0000","last_active":"2024-09-11T00:56:28.488110+0000","last_peered":"2024-09-11T00:56:28.488110+0000","last_clean":"2024-09-11T00:56:28.488110+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:56:28.488110+0000","last_undegraded":"2024-09-11T00:56:28.488110+0000","last_fullsized":"2024-09-11T00:56:28.488110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:16.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"62'3441","reported_seq":3532,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.489542+0000","last_change":"2024-09-11T00:31:26.326200+0000","last_active":"2024-09-11T00:56:28.489542+0000","last_peered":"2024-09-11T00:56:28.489542+0000","last_clean":"2024-09-11T00:56:28.489542+0000","last_became_active":"2024-09-11T00:31:26.325895+0000","last_became_peered":"2024-09-11T00:31:26.325895+0000","last_unstale":"2024-09-11T00:56:28.489542+0000","last_undegraded":"2024-09-11T00:56:28.489542+0000","last_fullsized":"2024-09-11T00:56:28.489542+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:39:08.753699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18163,"num_write_kb":273760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.20","version":"60'3848","reported_seq":3953,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075938+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:56:28.075938+0000","last_peered":"2024-09-11T00:56:28.075938+0000","last_clean":"2024-09-11T00:56:28.075938+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:56:28.075938+0000","last_undegraded":"2024-09-11T00:56:28.075938+0000","last_fullsized":"2024-09-11T00:56:28.075938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20029,"num_write_kb":325938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"62'4287","reported_seq":4384,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.519505+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:56:28.519505+0000","last_peered":"2024-09-11T00:56:28.519505+0000","last_clean":"2024-09-11T00:56:28.519505+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:56:28.519505+0000","last_undegraded":"2024-09-11T00:56:28.519505+0000","last_fullsized":"2024-09-11T00:56:28.519505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22520,"num_write_kb":336110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"60'2962","reported_seq":3067,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058177+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:56:28.058177+0000","last_peered":"2024-09-11T00:56:28.058177+0000","last_clean":"2024-09-11T00:56:28.058177+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:56:28.058177+0000","last_undegraded":"2024-09-11T00:56:28.058177+0000","last_fullsized":"2024-09-11T00:56:28.058177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15172,"num_write_kb":277710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"62'2334","reported_seq":2430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075259+0000","last_change":"2024-09-11T00:30:54.979133+0000","last_active":"2024-09-11T00:56:28.075259+0000","last_peered":"2024-09-11T00:56:28.075259+0000","last_clean":"2024-09-11T00:56:28.075259+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:56:28.075259+0000","last_undegraded":"2024-09-11T00:56:28.075259+0000","last_fullsized":"2024-09-11T00:56:28.075259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2334,"log_dups_size":0,"ondisk_log_size":2334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:20.961678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11753,"num_write_kb":206776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"60'4175","reported_seq":4282,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065154+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:56:28.065154+0000","last_peered":"2024-09-11T00:56:28.065154+0000","last_clean":"2024-09-11T00:56:28.065154+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:56:28.065154+0000","last_undegraded":"2024-09-11T00:56:28.065154+0000","last_fullsized":"2024-09-11T00:56:28.065154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22002,"num_write_kb":346058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"62'3428","reported_seq":3524,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.426792+0000","last_change":"2024-09-11T00:30:54.988345+0000","last_active":"2024-09-11T00:56:28.426792+0000","last_peered":"2024-09-11T00:56:28.426792+0000","last_clean":"2024-09-11T00:56:28.426792+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:56:28.426792+0000","last_undegraded":"2024-09-11T00:56:28.426792+0000","last_fullsized":"2024-09-11T00:56:28.426792+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3428,"log_dups_size":0,"ondisk_log_size":3428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.112194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17996,"num_write_kb":289213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"60'2543","reported_seq":2648,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059170+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:56:28.059170+0000","last_peered":"2024-09-11T00:56:28.059170+0000","last_clean":"2024-09-11T00:56:28.059170+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:56:28.059170+0000","last_undegraded":"2024-09-11T00:56:28.059170+0000","last_fullsized":"2024-09-11T00:56:28.059170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13221,"num_write_kb":224922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"62'3390","reported_seq":3487,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.413384+0000","last_change":"2024-09-11T00:30:54.981185+0000","last_active":"2024-09-11T00:56:28.413384+0000","last_peered":"2024-09-11T00:56:28.413384+0000","last_clean":"2024-09-11T00:56:28.413384+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:56:28.413384+0000","last_undegraded":"2024-09-11T00:56:28.413384+0000","last_fullsized":"2024-09-11T00:56:28.413384+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:45.740467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17320,"num_write_kb":290458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"60'2853","reported_seq":2957,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052823+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:56:28.052823+0000","last_peered":"2024-09-11T00:56:28.052823+0000","last_clean":"2024-09-11T00:56:28.052823+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:56:28.052823+0000","last_undegraded":"2024-09-11T00:56:28.052823+0000","last_fullsized":"2024-09-11T00:56:28.052823+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14682,"num_write_kb":238952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"62'2740","reported_seq":2837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.499809+0000","last_change":"2024-09-11T00:30:54.992763+0000","last_active":"2024-09-11T00:56:28.499809+0000","last_peered":"2024-09-11T00:56:28.499809+0000","last_clean":"2024-09-11T00:56:28.499809+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:56:28.499809+0000","last_undegraded":"2024-09-11T00:56:28.499809+0000","last_fullsized":"2024-09-11T00:56:28.499809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2740,"log_dups_size":0,"ondisk_log_size":2740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:02.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14066,"num_write_kb":224154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"60'3660","reported_seq":3764,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052923+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:56:28.052923+0000","last_peered":"2024-09-11T00:56:28.052923+0000","last_clean":"2024-09-11T00:56:28.052923+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:56:28.052923+0000","last_undegraded":"2024-09-11T00:56:28.052923+0000","last_fullsized":"2024-09-11T00:56:28.052923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19466,"num_write_kb":307301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"62'4363","reported_seq":4460,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.478746+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:56:28.478746+0000","last_peered":"2024-09-11T00:56:28.478746+0000","last_clean":"2024-09-11T00:56:28.478746+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:56:28.478746+0000","last_undegraded":"2024-09-11T00:56:28.478746+0000","last_fullsized":"2024-09-11T00:56:28.478746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23064,"num_write_kb":350422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"60'2921","reported_seq":3025,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075658+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:56:28.075658+0000","last_peered":"2024-09-11T00:56:28.075658+0000","last_clean":"2024-09-11T00:56:28.075658+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:56:28.075658+0000","last_undegraded":"2024-09-11T00:56:28.075658+0000","last_fullsized":"2024-09-11T00:56:28.075658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15417,"num_write_kb":250464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"62'3030","reported_seq":3127,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.431858+0000","last_change":"2024-09-11T00:30:54.984447+0000","last_active":"2024-09-11T00:56:28.431858+0000","last_peered":"2024-09-11T00:56:28.431858+0000","last_clean":"2024-09-11T00:56:28.431858+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:56:28.431858+0000","last_undegraded":"2024-09-11T00:56:28.431858+0000","last_fullsized":"2024-09-11T00:56:28.431858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:49.024022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15510,"num_write_kb":262754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"60'3321","reported_seq":3427,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059533+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:56:28.059533+0000","last_peered":"2024-09-11T00:56:28.059533+0000","last_clean":"2024-09-11T00:56:28.059533+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:56:28.059533+0000","last_undegraded":"2024-09-11T00:56:28.059533+0000","last_fullsized":"2024-09-11T00:56:28.059533+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":25,"num_write":17209,"num_write_kb":314610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"62'2084","reported_seq":2181,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.543950+0000","last_change":"2024-09-11T00:30:54.996091+0000","last_active":"2024-09-11T00:56:28.543950+0000","last_peered":"2024-09-11T00:56:28.543950+0000","last_clean":"2024-09-11T00:56:28.543950+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:56:28.543950+0000","last_undegraded":"2024-09-11T00:56:28.543950+0000","last_fullsized":"2024-09-11T00:56:28.543950+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2084,"log_dups_size":0,"ondisk_log_size":2084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:42.987336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10636,"num_write_kb":173560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"60'2810","reported_seq":2914,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063723+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:56:28.063723+0000","last_peered":"2024-09-11T00:56:28.063723+0000","last_clean":"2024-09-11T00:56:28.063723+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:56:28.063723+0000","last_undegraded":"2024-09-11T00:56:28.063723+0000","last_fullsized":"2024-09-11T00:56:28.063723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2810,"log_dups_size":0,"ondisk_log_size":2810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":243007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"62'3697","reported_seq":3793,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.455458+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:56:28.455458+0000","last_peered":"2024-09-11T00:56:28.455458+0000","last_clean":"2024-09-11T00:56:28.455458+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:56:28.455458+0000","last_undegraded":"2024-09-11T00:56:28.455458+0000","last_fullsized":"2024-09-11T00:56:28.455458+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18782,"num_write_kb":312790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"60'2824","reported_seq":2928,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057103+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:56:28.057103+0000","last_peered":"2024-09-11T00:56:28.057103+0000","last_clean":"2024-09-11T00:56:28.057103+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:56:28.057103+0000","last_undegraded":"2024-09-11T00:56:28.057103+0000","last_fullsized":"2024-09-11T00:56:28.057103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2824,"log_dups_size":0,"ondisk_log_size":2824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14695,"num_write_kb":259410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"62'2917","reported_seq":3013,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510399+0000","last_change":"2024-09-11T00:30:54.981799+0000","last_active":"2024-09-11T00:56:28.510399+0000","last_peered":"2024-09-11T00:56:28.510399+0000","last_clean":"2024-09-11T00:56:28.510399+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:56:28.510399+0000","last_undegraded":"2024-09-11T00:56:28.510399+0000","last_fullsized":"2024-09-11T00:56:28.510399+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2917,"log_dups_size":0,"ondisk_log_size":2917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:42.670629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":242029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"60'3431","reported_seq":3535,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057841+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:56:28.057841+0000","last_peered":"2024-09-11T00:56:28.057841+0000","last_clean":"2024-09-11T00:56:28.057841+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:56:28.057841+0000","last_undegraded":"2024-09-11T00:56:28.057841+0000","last_fullsized":"2024-09-11T00:56:28.057841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18408,"num_write_kb":290787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"62'2945","reported_seq":3041,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514103+0000","last_change":"2024-09-11T00:30:54.992697+0000","last_active":"2024-09-11T00:56:28.514103+0000","last_peered":"2024-09-11T00:56:28.514103+0000","last_clean":"2024-09-11T00:56:28.514103+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:56:28.514103+0000","last_undegraded":"2024-09-11T00:56:28.514103+0000","last_fullsized":"2024-09-11T00:56:28.514103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:54:07.227169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15229,"num_write_kb":249300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"60'2849","reported_seq":2955,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048641+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:56:28.048641+0000","last_peered":"2024-09-11T00:56:28.048641+0000","last_clean":"2024-09-11T00:56:28.048641+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:56:28.048641+0000","last_undegraded":"2024-09-11T00:56:28.048641+0000","last_fullsized":"2024-09-11T00:56:28.048641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2849,"log_dups_size":0,"ondisk_log_size":2849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14625,"num_write_kb":263252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"62'4017","reported_seq":4113,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.477074+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:56:28.477074+0000","last_peered":"2024-09-11T00:56:28.477074+0000","last_clean":"2024-09-11T00:56:28.477074+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:56:28.477074+0000","last_undegraded":"2024-09-11T00:56:28.477074+0000","last_fullsized":"2024-09-11T00:56:28.477074+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":329319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"60'2942","reported_seq":3046,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066118+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:56:28.066118+0000","last_peered":"2024-09-11T00:56:28.066118+0000","last_clean":"2024-09-11T00:56:28.066118+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:56:28.066118+0000","last_undegraded":"2024-09-11T00:56:28.066118+0000","last_fullsized":"2024-09-11T00:56:28.066118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15436,"num_write_kb":246093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"62'3417","reported_seq":3514,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.528940+0000","last_change":"2024-09-11T00:30:54.985827+0000","last_active":"2024-09-11T00:56:28.528940+0000","last_peered":"2024-09-11T00:56:28.528940+0000","last_clean":"2024-09-11T00:56:28.528940+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:56:28.528940+0000","last_undegraded":"2024-09-11T00:56:28.528940+0000","last_fullsized":"2024-09-11T00:56:28.528940+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:32.027117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18129,"num_write_kb":269216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"60'2746","reported_seq":2850,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069377+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:56:28.069377+0000","last_peered":"2024-09-11T00:56:28.069377+0000","last_clean":"2024-09-11T00:56:28.069377+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:56:28.069377+0000","last_undegraded":"2024-09-11T00:56:28.069377+0000","last_fullsized":"2024-09-11T00:56:28.069377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13859,"num_write_kb":247924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"62'2570","reported_seq":2672,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514173+0000","last_change":"2024-09-11T00:31:26.119785+0000","last_active":"2024-09-11T00:56:28.514173+0000","last_peered":"2024-09-11T00:56:28.514173+0000","last_clean":"2024-09-11T00:56:28.514173+0000","last_became_active":"2024-09-11T00:31:26.119436+0000","last_became_peered":"2024-09-11T00:31:26.119436+0000","last_unstale":"2024-09-11T00:56:28.514173+0000","last_undegraded":"2024-09-11T00:56:28.514173+0000","last_fullsized":"2024-09-11T00:56:28.514173+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:53:49.407575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13205,"num_write_kb":216683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"60'2388","reported_seq":2494,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058894+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:56:28.058894+0000","last_peered":"2024-09-11T00:56:28.058894+0000","last_clean":"2024-09-11T00:56:28.058894+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:56:28.058894+0000","last_undegraded":"2024-09-11T00:56:28.058894+0000","last_fullsized":"2024-09-11T00:56:28.058894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2388,"log_dups_size":0,"ondisk_log_size":2388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4,"num_write":12384,"num_write_kb":205064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"62'3381","reported_seq":3477,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397849+0000","last_change":"2024-09-11T00:30:54.993156+0000","last_active":"2024-09-11T00:56:28.397849+0000","last_peered":"2024-09-11T00:56:28.397849+0000","last_clean":"2024-09-11T00:56:28.397849+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:56:28.397849+0000","last_undegraded":"2024-09-11T00:56:28.397849+0000","last_fullsized":"2024-09-11T00:56:28.397849+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:25.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17747,"num_write_kb":280034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"60'3936","reported_seq":4041,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067998+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:56:28.067998+0000","last_peered":"2024-09-11T00:56:28.067998+0000","last_clean":"2024-09-11T00:56:28.067998+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:56:28.067998+0000","last_undegraded":"2024-09-11T00:56:28.067998+0000","last_fullsized":"2024-09-11T00:56:28.067998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":20459,"num_write_kb":335990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"62'3772","reported_seq":3867,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.454399+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:56:28.454399+0000","last_peered":"2024-09-11T00:56:28.454399+0000","last_clean":"2024-09-11T00:56:28.454399+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:56:28.454399+0000","last_undegraded":"2024-09-11T00:56:28.454399+0000","last_fullsized":"2024-09-11T00:56:28.454399+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20201,"num_write_kb":302053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"60'2757","reported_seq":2861,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062570+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:56:28.062570+0000","last_peered":"2024-09-11T00:56:28.062570+0000","last_clean":"2024-09-11T00:56:28.062570+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:56:28.062570+0000","last_undegraded":"2024-09-11T00:56:28.062570+0000","last_fullsized":"2024-09-11T00:56:28.062570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14203,"num_write_kb":241057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"62'4048","reported_seq":4143,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.441208+0000","last_change":"2024-09-11T00:30:54.993461+0000","last_active":"2024-09-11T00:56:28.441208+0000","last_peered":"2024-09-11T00:56:28.441208+0000","last_clean":"2024-09-11T00:56:28.441208+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:56:28.441208+0000","last_undegraded":"2024-09-11T00:56:28.441208+0000","last_fullsized":"2024-09-11T00:56:28.441208+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:07.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21387,"num_write_kb":364007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"60'3572","reported_seq":3676,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057127+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:56:28.057127+0000","last_peered":"2024-09-11T00:56:28.057127+0000","last_clean":"2024-09-11T00:56:28.057127+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:56:28.057127+0000","last_undegraded":"2024-09-11T00:56:28.057127+0000","last_fullsized":"2024-09-11T00:56:28.057127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18317,"num_write_kb":294510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"62'2987","reported_seq":3083,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397386+0000","last_change":"2024-09-11T00:30:54.986265+0000","last_active":"2024-09-11T00:56:28.397386+0000","last_peered":"2024-09-11T00:56:28.397386+0000","last_clean":"2024-09-11T00:56:28.397386+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:56:28.397386+0000","last_undegraded":"2024-09-11T00:56:28.397386+0000","last_fullsized":"2024-09-11T00:56:28.397386+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:49.109328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15338,"num_write_kb":237611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"60'2929","reported_seq":3034,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051236+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:56:28.051236+0000","last_peered":"2024-09-11T00:56:28.051236+0000","last_clean":"2024-09-11T00:56:28.051236+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:56:28.051236+0000","last_undegraded":"2024-09-11T00:56:28.051236+0000","last_fullsized":"2024-09-11T00:56:28.051236+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2929,"log_dups_size":0,"ondisk_log_size":2929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":15867,"num_write_kb":268409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"62'3243","reported_seq":3338,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.433332+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:56:28.433332+0000","last_peered":"2024-09-11T00:56:28.433332+0000","last_clean":"2024-09-11T00:56:28.433332+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:56:28.433332+0000","last_undegraded":"2024-09-11T00:56:28.433332+0000","last_fullsized":"2024-09-11T00:56:28.433332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16324,"num_write_kb":262051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"60'2913","reported_seq":3019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052467+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:56:28.052467+0000","last_peered":"2024-09-11T00:56:28.052467+0000","last_clean":"2024-09-11T00:56:28.052467+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:56:28.052467+0000","last_undegraded":"2024-09-11T00:56:28.052467+0000","last_fullsized":"2024-09-11T00:56:28.052467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2913,"log_dups_size":0,"ondisk_log_size":2913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15444,"num_write_kb":246103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"62'3344","reported_seq":3440,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.543938+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:56:28.543938+0000","last_peered":"2024-09-11T00:56:28.543938+0000","last_clean":"2024-09-11T00:56:28.543938+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:56:28.543938+0000","last_undegraded":"2024-09-11T00:56:28.543938+0000","last_fullsized":"2024-09-11T00:56:28.543938+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3344,"log_dups_size":0,"ondisk_log_size":3344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17374,"num_write_kb":274437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"60'2266","reported_seq":2373,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065676+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:56:28.065676+0000","last_peered":"2024-09-11T00:56:28.065676+0000","last_clean":"2024-09-11T00:56:28.065676+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:56:28.065676+0000","last_undegraded":"2024-09-11T00:56:28.065676+0000","last_fullsized":"2024-09-11T00:56:28.065676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":26,"num_write":11816,"num_write_kb":219351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"62'2714","reported_seq":2811,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.547334+0000","last_change":"2024-09-11T00:30:54.994988+0000","last_active":"2024-09-11T00:56:28.547334+0000","last_peered":"2024-09-11T00:56:28.547334+0000","last_clean":"2024-09-11T00:56:28.547334+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:56:28.547334+0000","last_undegraded":"2024-09-11T00:56:28.547334+0000","last_fullsized":"2024-09-11T00:56:28.547334+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2714,"log_dups_size":0,"ondisk_log_size":2714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:31.973641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14634,"num_write_kb":230575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"60'3170","reported_seq":3259,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058097+0000","last_change":"2024-09-11T00:31:26.117501+0000","last_active":"2024-09-11T00:56:28.058097+0000","last_peered":"2024-09-11T00:56:28.058097+0000","last_clean":"2024-09-11T00:56:28.058097+0000","last_became_active":"2024-09-11T00:31:26.117146+0000","last_became_peered":"2024-09-11T00:31:26.117146+0000","last_unstale":"2024-09-11T00:56:28.058097+0000","last_undegraded":"2024-09-11T00:56:28.058097+0000","last_fullsized":"2024-09-11T00:56:28.058097+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:52.194644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15922,"num_write_kb":268523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.30","version":"62'3135","reported_seq":3231,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.469071+0000","last_change":"2024-09-11T00:30:54.984767+0000","last_active":"2024-09-11T00:56:28.469071+0000","last_peered":"2024-09-11T00:56:28.469071+0000","last_clean":"2024-09-11T00:56:28.469071+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:56:28.469071+0000","last_undegraded":"2024-09-11T00:56:28.469071+0000","last_fullsized":"2024-09-11T00:56:28.469071+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:44.833677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16038,"num_write_kb":252613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"60'3300","reported_seq":3404,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068344+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:56:28.068344+0000","last_peered":"2024-09-11T00:56:28.068344+0000","last_clean":"2024-09-11T00:56:28.068344+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:56:28.068344+0000","last_undegraded":"2024-09-11T00:56:28.068344+0000","last_fullsized":"2024-09-11T00:56:28.068344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3300,"log_dups_size":0,"ondisk_log_size":3300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":17861,"num_write_kb":288513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"62'3599","reported_seq":3695,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.492582+0000","last_change":"2024-09-11T00:30:54.986423+0000","last_active":"2024-09-11T00:56:28.492582+0000","last_peered":"2024-09-11T00:56:28.492582+0000","last_clean":"2024-09-11T00:56:28.492582+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:56:28.492582+0000","last_undegraded":"2024-09-11T00:56:28.492582+0000","last_fullsized":"2024-09-11T00:56:28.492582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:16:50.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18686,"num_write_kb":304162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"60'4168","reported_seq":4272,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063833+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:56:28.063833+0000","last_peered":"2024-09-11T00:56:28.063833+0000","last_clean":"2024-09-11T00:56:28.063833+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:56:28.063833+0000","last_undegraded":"2024-09-11T00:56:28.063833+0000","last_fullsized":"2024-09-11T00:56:28.063833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22110,"num_write_kb":362983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"62'3937","reported_seq":4034,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.544539+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:56:28.544539+0000","last_peered":"2024-09-11T00:56:28.544539+0000","last_clean":"2024-09-11T00:56:28.544539+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:56:28.544539+0000","last_undegraded":"2024-09-11T00:56:28.544539+0000","last_fullsized":"2024-09-11T00:56:28.544539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21285,"num_write_kb":343312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"60'3475","reported_seq":3579,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057082+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:56:28.057082+0000","last_peered":"2024-09-11T00:56:28.057082+0000","last_clean":"2024-09-11T00:56:28.057082+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:56:28.057082+0000","last_undegraded":"2024-09-11T00:56:28.057082+0000","last_fullsized":"2024-09-11T00:56:28.057082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3475,"log_dups_size":0,"ondisk_log_size":3475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17645,"num_write_kb":295948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"62'3302","reported_seq":3399,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.512120+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:56:28.512120+0000","last_peered":"2024-09-11T00:56:28.512120+0000","last_clean":"2024-09-11T00:56:28.512120+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:56:28.512120+0000","last_undegraded":"2024-09-11T00:56:28.512120+0000","last_fullsized":"2024-09-11T00:56:28.512120+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17558,"num_write_kb":271021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"60'3295","reported_seq":3399,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063104+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:56:28.063104+0000","last_peered":"2024-09-11T00:56:28.063104+0000","last_clean":"2024-09-11T00:56:28.063104+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:56:28.063104+0000","last_undegraded":"2024-09-11T00:56:28.063104+0000","last_fullsized":"2024-09-11T00:56:28.063104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":25,"num_write":17271,"num_write_kb":265887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"61'3033","reported_seq":3130,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048703+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:56:28.048703+0000","last_peered":"2024-09-11T00:56:28.048703+0000","last_clean":"2024-09-11T00:56:28.048703+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:56:28.048703+0000","last_undegraded":"2024-09-11T00:56:28.048703+0000","last_fullsized":"2024-09-11T00:56:28.048703+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16092,"num_write_kb":252889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"60'2629","reported_seq":2733,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.078405+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:56:28.078405+0000","last_peered":"2024-09-11T00:56:28.078405+0000","last_clean":"2024-09-11T00:56:28.078405+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:56:28.078405+0000","last_undegraded":"2024-09-11T00:56:28.078405+0000","last_fullsized":"2024-09-11T00:56:28.078405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":13758,"num_write_kb":227101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"62'3227","reported_seq":3324,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.479006+0000","last_change":"2024-09-11T00:30:54.990631+0000","last_active":"2024-09-11T00:56:28.479006+0000","last_peered":"2024-09-11T00:56:28.479006+0000","last_clean":"2024-09-11T00:56:28.479006+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:56:28.479006+0000","last_undegraded":"2024-09-11T00:56:28.479006+0000","last_fullsized":"2024-09-11T00:56:28.479006+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:02.422471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16396,"num_write_kb":258131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"60'3415","reported_seq":3520,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067480+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:56:28.067480+0000","last_peered":"2024-09-11T00:56:28.067480+0000","last_clean":"2024-09-11T00:56:28.067480+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:56:28.067480+0000","last_undegraded":"2024-09-11T00:56:28.067480+0000","last_fullsized":"2024-09-11T00:56:28.067480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":17180,"num_write_kb":310287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"62'2771","reported_seq":2866,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.447411+0000","last_change":"2024-09-11T00:30:54.991047+0000","last_active":"2024-09-11T00:56:28.447411+0000","last_peered":"2024-09-11T00:56:28.447411+0000","last_clean":"2024-09-11T00:56:28.447411+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:56:28.447411+0000","last_undegraded":"2024-09-11T00:56:28.447411+0000","last_fullsized":"2024-09-11T00:56:28.447411+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:41:48.229261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14317,"num_write_kb":221149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"60'2741","reported_seq":2846,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075991+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:56:28.075991+0000","last_peered":"2024-09-11T00:56:28.075991+0000","last_clean":"2024-09-11T00:56:28.075991+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:56:28.075991+0000","last_undegraded":"2024-09-11T00:56:28.075991+0000","last_fullsized":"2024-09-11T00:56:28.075991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2741,"log_dups_size":0,"ondisk_log_size":2741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14755,"num_write_kb":255665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"62'3704","reported_seq":3799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.547303+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:56:28.547303+0000","last_peered":"2024-09-11T00:56:28.547303+0000","last_clean":"2024-09-11T00:56:28.547303+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:56:28.547303+0000","last_undegraded":"2024-09-11T00:56:28.547303+0000","last_fullsized":"2024-09-11T00:56:28.547303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18483,"num_write_kb":307447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"60'3035","reported_seq":3140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059908+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:56:28.059908+0000","last_peered":"2024-09-11T00:56:28.059908+0000","last_clean":"2024-09-11T00:56:28.059908+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:56:28.059908+0000","last_undegraded":"2024-09-11T00:56:28.059908+0000","last_fullsized":"2024-09-11T00:56:28.059908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3035,"log_dups_size":0,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15470,"num_write_kb":262623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"62'3351","reported_seq":3447,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075584+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:56:28.075584+0000","last_peered":"2024-09-11T00:56:28.075584+0000","last_clean":"2024-09-11T00:56:28.075584+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:56:28.075584+0000","last_undegraded":"2024-09-11T00:56:28.075584+0000","last_fullsized":"2024-09-11T00:56:28.075584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17034,"num_write_kb":277088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"60'4106","reported_seq":4213,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059593+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:56:28.059593+0000","last_peered":"2024-09-11T00:56:28.059593+0000","last_clean":"2024-09-11T00:56:28.059593+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:56:28.059593+0000","last_undegraded":"2024-09-11T00:56:28.059593+0000","last_fullsized":"2024-09-11T00:56:28.059593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22024,"num_write_kb":354504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"62'3391","reported_seq":3486,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.526593+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:56:28.526593+0000","last_peered":"2024-09-11T00:56:28.526593+0000","last_clean":"2024-09-11T00:56:28.526593+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:56:28.526593+0000","last_undegraded":"2024-09-11T00:56:28.526593+0000","last_fullsized":"2024-09-11T00:56:28.526593+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18265,"num_write_kb":274031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"60'2946","reported_seq":3053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069081+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:56:28.069081+0000","last_peered":"2024-09-11T00:56:28.069081+0000","last_clean":"2024-09-11T00:56:28.069081+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:56:28.069081+0000","last_undegraded":"2024-09-11T00:56:28.069081+0000","last_fullsized":"2024-09-11T00:56:28.069081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":17,"num_write":15520,"num_write_kb":262805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1107571,"num_write_kb":17543218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212907,"ondisk_log_size":212907,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18223220,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1558,"num_read_kb":197403,"num_write":4817,"num_write_kb":19773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36511744,"data_stored":36446440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4412,"ondisk_log_size":4412,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":381,"num_write":1025649,"num_write_kb":17211858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196678,"ondisk_log_size":196678,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":17635410,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1062,"num_read_kb":143929,"num_write":4568,"num_write_kb":19046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35332096,"data_stored":35270820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4119,"ondisk_log_size":4119,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509757,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487152,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":486270,"kb_avail":93884688,"statfs":{"total":96636764160,"available":96137920512,"internally_reserved":0,"allocated":884736,"data_stored":496637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":497940938},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509759,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668164,"kb_used_data":10292,"kb_used_omap":1,"kb_used_meta":657854,"kb_avail":93703676,"statfs":{"total":96636764160,"available":95952564224,"internally_reserved":0,"allocated":10539008,"data_stored":10153943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":673642950},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509758,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685684,"kb_used_data":17828,"kb_used_omap":1,"kb_used_meta":667838,"kb_avail":93686156,"statfs":{"total":96636764160,"available":95934623744,"internally_reserved":0,"allocated":18255872,"data_stored":17864749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683866565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509757,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722280,"kb_used_data":13912,"kb_used_omap":1,"kb_used_meta":708350,"kb_avail":93649560,"statfs":{"total":96636764160,"available":95897149440,"internally_reserved":0,"allocated":14245888,"data_stored":13849304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":725350856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509758,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737056,"kb_used_data":14480,"kb_used_omap":1,"kb_used_meta":722558,"kb_avail":93634784,"statfs":{"total":96636764160,"available":95882018816,"internally_reserved":0,"allocated":14827520,"data_stored":14436462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739899845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509758,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598360,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":596990,"kb_avail":93773480,"statfs":{"total":96636764160,"available":96024043520,"internally_reserved":0,"allocated":1384448,"data_stored":998818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":611318213},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509759,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741960,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":740414,"kb_avail":93629880,"statfs":{"total":96636764160,"available":95876997120,"internally_reserved":0,"allocated":1564672,"data_stored":1176999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":758184389},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542462,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739432,"kb_used_data":17176,"kb_used_omap":1,"kb_used_meta":722238,"kb_avail":93632408,"statfs":{"total":96636764160,"available":95879585792,"internally_reserved":0,"allocated":17588224,"data_stored":17187564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739572165},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":778240,"data_stored":768599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9162752,"data_stored":9157733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4282577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5648384,"data_stored":5638872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":4283261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5562368,"data_stored":5551329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:56:46.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:56:47.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.137+0000 7f0b73e00640 1 -- 172.21.15.31:0/3185159799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b6c0c5ef0 msgr2=0x7f0b6c153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:47.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.137+0000 7f0b73e00640 1 --2- 172.21.15.31:0/3185159799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b6c0c5ef0 0x7f0b6c153480 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f0b58005fd0 tx=0x7f0b5802a5d0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.138+0000 7f0b73e00640 1 -- 172.21.15.31:0/3185159799 shutdown_connections 2024-09-11T00:56:47.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.138+0000 7f0b73e00640 1 --2- 172.21.15.31:0/3185159799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b6c0c5ef0 0x7f0b6c153480 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.138+0000 7f0b73e00640 1 --2- 172.21.15.31:0/3185159799 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0c5b10 0x7f0b6c156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.138+0000 7f0b73e00640 1 -- 172.21.15.31:0/3185159799 >> 172.21.15.31:0/3185159799 conn(0x7f0b6c0a9750 msgr2=0x7f0b6c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:47.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.138+0000 7f0b73e00640 1 -- 172.21.15.31:0/3185159799 shutdown_connections 2024-09-11T00:56:47.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.138+0000 7f0b73e00640 1 -- 172.21.15.31:0/3185159799 wait complete. 2024-09-11T00:56:47.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.139+0000 7f0b73e00640 1 Processor -- start 2024-09-11T00:56:47.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.139+0000 7f0b73e00640 1 -- start start 2024-09-11T00:56:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b73e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c0c5b10 0x7f0b6c072ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b73e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0c5ef0 0x7f0b6c074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b73e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b6c0c6310 0x7f0b6c1ef570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b73e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6c1539c0 con 0x7f0b6c0c6310 2024-09-11T00:56:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b73e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6c0c3950 con 0x7f0b6c0c5b10 2024-09-11T00:56:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b73e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6c153b40 con 0x7f0b6c0c5ef0 2024-09-11T00:56:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b71e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b6c0c6310 0x7f0b6c1ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b71e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b6c0c6310 0x7f0b6c1ef570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50644/0 (socket says 172.21.15.31:50644) 2024-09-11T00:56:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b71e00640 1 -- 172.21.15.31:0/3982301688 learned_addr learned my addr 172.21.15.31:0/3982301688 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:47.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b70a00640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0c5ef0 0x7f0b6c074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.140+0000 7f0b71400640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c0c5b10 0x7f0b6c072ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b71e00640 1 -- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0c5ef0 msgr2=0x7f0b6c074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b71e00640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0c5ef0 0x7f0b6c074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b71e00640 1 -- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c0c5b10 msgr2=0x7f0b6c072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b71e00640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c0c5b10 0x7f0b6c072ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b71e00640 1 -- 172.21.15.31:0/3982301688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6c1efe40 con 0x7f0b6c0c6310 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b71e00640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b6c0c6310 0x7f0b6c1ef570 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400c760 tx=0x7f0b6400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b69600640 1 -- 172.21.15.31:0/3982301688 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b64003b70 con 0x7f0b6c0c6310 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b69600640 1 -- 172.21.15.31:0/3982301688 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b64003d10 con 0x7f0b6c0c6310 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b73e00640 1 -- 172.21.15.31:0/3982301688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b6c1e2100 con 0x7f0b6c0c6310 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b69600640 1 -- 172.21.15.31:0/3982301688 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6400aab0 con 0x7f0b6c0c6310 2024-09-11T00:56:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.141+0000 7f0b73e00640 1 -- 172.21.15.31:0/3982301688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b6c072110 con 0x7f0b6c0c6310 2024-09-11T00:56:47.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.142+0000 7f0b73e00640 1 -- 172.21.15.31:0/3982301688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b6c156ad0 con 0x7f0b6c0c6310 2024-09-11T00:56:47.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.144+0000 7f0b69600640 1 -- 172.21.15.31:0/3982301688 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f0b6400ac50 con 0x7f0b6c0c6310 2024-09-11T00:56:47.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.144+0000 7f0b69600640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0b4c077f70 0x7f0b4c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:47.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.145+0000 7f0b69600640 1 -- 172.21.15.31:0/3982301688 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f0b6409a860 con 0x7f0b6c0c6310 2024-09-11T00:56:47.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.145+0000 7f0b71400640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0b4c077f70 0x7f0b4c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:47.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.145+0000 7f0b71400640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0b4c077f70 0x7f0b4c07a430 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c06fbc0 tx=0x7f0b60005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:47.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.148+0000 7f0b69600640 1 -- 172.21.15.31:0/3982301688 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0b64061d70 con 0x7f0b6c0c6310 2024-09-11T00:56:47.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.336+0000 7f0b73e00640 1 -- 172.21.15.31:0/3982301688 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0b6c0c1e70 con 0x7f0b4c077f70 2024-09-11T00:56:47.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.345+0000 7f0b69600640 1 -- 172.21.15.31:0/3982301688 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493711 (secure 0 0 0) 0x7f0b6c0c1e70 con 0x7f0b4c077f70 2024-09-11T00:56:47.353 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:56:47.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 -- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0b4c077f70 msgr2=0x7f0b4c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:47.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0b4c077f70 0x7f0b4c07a430 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c06fbc0 tx=0x7f0b60005f90 comp rx=0 tx=0).stop 2024-09-11T00:56:47.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 -- 172.21.15.31:0/3982301688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b6c0c6310 msgr2=0x7f0b6c1ef570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:47.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b6c0c6310 0x7f0b6c1ef570 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400c760 tx=0x7f0b6400cc20 comp rx=0 tx=0).stop 2024-09-11T00:56:47.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 -- 172.21.15.31:0/3982301688 shutdown_connections 2024-09-11T00:56:47.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0b4c077f70 0x7f0b4c07a430 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b6c0c6310 0x7f0b6c1ef570 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0b6c0c5ef0 0x7f0b6c074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 --2- 172.21.15.31:0/3982301688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0b6c0c5b10 0x7f0b6c072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 -- 172.21.15.31:0/3982301688 >> 172.21.15.31:0/3982301688 conn(0x7f0b6c0a9750 msgr2=0x7f0b6c0c4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:47.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 -- 172.21.15.31:0/3982301688 shutdown_connections 2024-09-11T00:56:47.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.350+0000 7f0b73e00640 1 -- 172.21.15.31:0/3982301688 wait complete. 2024-09-11T00:56:47.367 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":842,"stamp":"2024-09-11T00:56:46.843610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36448998,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":2779,"num_read_kb":341768,"num_write":2142706,"num_write_kb":34795523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":418188,"ondisk_log_size":418188,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5380088,"kb_used_data":77432,"kb_used_omap":12,"kb_used_meta":5302515,"kb_avail":749594632,"statfs":{"total":773094113280,"available":767584903168,"internally_reserved":0,"allocated":79290368,"data_stored":76164476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":5429775921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20007,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003687"},"pg_stats":[{"pgid":"5.18","version":"62'3612","reported_seq":3708,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.530101+0000","last_change":"2024-09-11T00:30:54.983412+0000","last_active":"2024-09-11T00:56:28.530101+0000","last_peered":"2024-09-11T00:56:28.530101+0000","last_clean":"2024-09-11T00:56:28.530101+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:56:28.530101+0000","last_undegraded":"2024-09-11T00:56:28.530101+0000","last_fullsized":"2024-09-11T00:56:28.530101+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:51.807846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18811,"num_write_kb":284357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"60'788","reported_seq":928,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.095736+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:56:28.095736+0000","last_peered":"2024-09-11T00:56:28.095736+0000","last_clean":"2024-09-11T00:56:28.095736+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:56:28.095736+0000","last_undegraded":"2024-09-11T00:56:28.095736+0000","last_fullsized":"2024-09-11T00:56:28.095736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":8319,"num_write":773,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"60'3033","reported_seq":3138,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059830+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:56:28.059830+0000","last_peered":"2024-09-11T00:56:28.059830+0000","last_clean":"2024-09-11T00:56:28.059830+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:56:28.059830+0000","last_undegraded":"2024-09-11T00:56:28.059830+0000","last_fullsized":"2024-09-11T00:56:28.059830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16165,"num_write_kb":269490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"61'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058244+0000","last_change":"2024-09-11T00:30:53.984316+0000","last_active":"2024-09-11T00:56:28.058244+0000","last_peered":"2024-09-11T00:56:28.058244+0000","last_clean":"2024-09-11T00:56:28.058244+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:56:28.058244+0000","last_undegraded":"2024-09-11T00:56:28.058244+0000","last_fullsized":"2024-09-11T00:56:28.058244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:14.425265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"62'3995","reported_seq":4090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.532259+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:56:28.532259+0000","last_peered":"2024-09-11T00:56:28.532259+0000","last_clean":"2024-09-11T00:56:28.532259+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:56:28.532259+0000","last_undegraded":"2024-09-11T00:56:28.532259+0000","last_fullsized":"2024-09-11T00:56:28.532259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20665,"num_write_kb":324486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"62'22","reported_seq":129,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060203+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:56:28.060203+0000","last_peered":"2024-09-11T00:56:28.060203+0000","last_clean":"2024-09-11T00:56:28.060203+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:56:28.060203+0000","last_undegraded":"2024-09-11T00:56:28.060203+0000","last_fullsized":"2024-09-11T00:56:28.060203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"60'3563","reported_seq":3674,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059819+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:56:28.059819+0000","last_peered":"2024-09-11T00:56:28.059819+0000","last_clean":"2024-09-11T00:56:28.059819+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:56:28.059819+0000","last_undegraded":"2024-09-11T00:56:28.059819+0000","last_fullsized":"2024-09-11T00:56:28.059819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":44,"num_write":18834,"num_write_kb":321224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"62'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075381+0000","last_change":"2024-09-11T00:30:53.984940+0000","last_active":"2024-09-11T00:56:28.075381+0000","last_peered":"2024-09-11T00:56:28.075381+0000","last_clean":"2024-09-11T00:56:28.075381+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:56:28.075381+0000","last_undegraded":"2024-09-11T00:56:28.075381+0000","last_fullsized":"2024-09-11T00:56:28.075381+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:19.647165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"62'3090","reported_seq":3185,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.476819+0000","last_change":"2024-09-11T00:30:54.993570+0000","last_active":"2024-09-11T00:56:28.476819+0000","last_peered":"2024-09-11T00:56:28.476819+0000","last_clean":"2024-09-11T00:56:28.476819+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:56:28.476819+0000","last_undegraded":"2024-09-11T00:56:28.476819+0000","last_fullsized":"2024-09-11T00:56:28.476819+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:45.211259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15625,"num_write_kb":254958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"62'37","reported_seq":171,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.077866+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:56:31.077866+0000","last_peered":"2024-09-11T00:56:31.077866+0000","last_clean":"2024-09-11T00:56:31.077866+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:56:31.077866+0000","last_undegraded":"2024-09-11T00:56:31.077866+0000","last_fullsized":"2024-09-11T00:56:31.077866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"60'2040","reported_seq":2144,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057996+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:56:28.057996+0000","last_peered":"2024-09-11T00:56:28.057996+0000","last_clean":"2024-09-11T00:56:28.057996+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:56:28.057996+0000","last_undegraded":"2024-09-11T00:56:28.057996+0000","last_fullsized":"2024-09-11T00:56:28.057996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":10218,"num_write_kb":171981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"61'716","reported_seq":837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075103+0000","last_change":"2024-09-11T00:30:53.974699+0000","last_active":"2024-09-11T00:56:28.075103+0000","last_peered":"2024-09-11T00:56:28.075103+0000","last_clean":"2024-09-11T00:56:28.075103+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:56:28.075103+0000","last_undegraded":"2024-09-11T00:56:28.075103+0000","last_fullsized":"2024-09-11T00:56:28.075103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:28.792487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":73385,"num_write":707,"num_write_kb":4317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"62'2671","reported_seq":2766,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.074790+0000","last_change":"2024-09-11T00:30:54.984890+0000","last_active":"2024-09-11T00:56:28.074790+0000","last_peered":"2024-09-11T00:56:28.074790+0000","last_clean":"2024-09-11T00:56:28.074790+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:56:28.074790+0000","last_undegraded":"2024-09-11T00:56:28.074790+0000","last_fullsized":"2024-09-11T00:56:28.074790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:21.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":213871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"60'3","reported_seq":136,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.069924+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:56:31.069924+0000","last_peered":"2024-09-11T00:56:31.069924+0000","last_clean":"2024-09-11T00:56:31.069924+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:56:31.069924+0000","last_undegraded":"2024-09-11T00:56:31.069924+0000","last_fullsized":"2024-09-11T00:56:31.069924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"60'3687","reported_seq":3790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068958+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:56:28.068958+0000","last_peered":"2024-09-11T00:56:28.068958+0000","last_clean":"2024-09-11T00:56:28.068958+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:56:28.068958+0000","last_undegraded":"2024-09-11T00:56:28.068958+0000","last_fullsized":"2024-09-11T00:56:28.068958+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19525,"num_write_kb":315726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"62'140","reported_seq":237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.957825+0000","last_change":"2024-09-11T00:30:53.977743+0000","last_active":"2024-09-11T00:56:34.957825+0000","last_peered":"2024-09-11T00:56:34.957825+0000","last_clean":"2024-09-11T00:56:34.957825+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:56:34.957825+0000","last_undegraded":"2024-09-11T00:56:34.957825+0000","last_fullsized":"2024-09-11T00:56:34.957825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:20.618615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1356353,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":1393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"62'3457","reported_seq":3553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.513475+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:56:28.513475+0000","last_peered":"2024-09-11T00:56:28.513475+0000","last_clean":"2024-09-11T00:56:28.513475+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:56:28.513475+0000","last_undegraded":"2024-09-11T00:56:28.513475+0000","last_fullsized":"2024-09-11T00:56:28.513475+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3457,"log_dups_size":0,"ondisk_log_size":3457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18121,"num_write_kb":297725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"60'2670","reported_seq":2774,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065793+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:56:28.065793+0000","last_peered":"2024-09-11T00:56:28.065793+0000","last_clean":"2024-09-11T00:56:28.065793+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:56:28.065793+0000","last_undegraded":"2024-09-11T00:56:28.065793+0000","last_fullsized":"2024-09-11T00:56:28.065793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2670,"log_dups_size":0,"ondisk_log_size":2670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14000,"num_write_kb":256678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"62'1368","reported_seq":1505,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.097251+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:56:28.097251+0000","last_peered":"2024-09-11T00:56:28.097251+0000","last_clean":"2024-09-11T00:56:28.097251+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:56:28.097251+0000","last_undegraded":"2024-09-11T00:56:28.097251+0000","last_fullsized":"2024-09-11T00:56:28.097251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1368,"log_dups_size":0,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":26260,"num_write":1345,"num_write_kb":4470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"61'2","reported_seq":121,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068774+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:56:28.068774+0000","last_peered":"2024-09-11T00:56:28.068774+0000","last_clean":"2024-09-11T00:56:28.068774+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:56:28.068774+0000","last_undegraded":"2024-09-11T00:56:28.068774+0000","last_fullsized":"2024-09-11T00:56:28.068774+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"62'3537","reported_seq":3633,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.531936+0000","last_change":"2024-09-11T00:30:54.997397+0000","last_active":"2024-09-11T00:56:28.531936+0000","last_peered":"2024-09-11T00:56:28.531936+0000","last_clean":"2024-09-11T00:56:28.531936+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:56:28.531936+0000","last_undegraded":"2024-09-11T00:56:28.531936+0000","last_fullsized":"2024-09-11T00:56:28.531936+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3537,"log_dups_size":0,"ondisk_log_size":3537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:53.322576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18517,"num_write_kb":279095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"60'2630","reported_seq":2735,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058909+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:56:28.058909+0000","last_peered":"2024-09-11T00:56:28.058909+0000","last_clean":"2024-09-11T00:56:28.058909+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:56:28.058909+0000","last_undegraded":"2024-09-11T00:56:28.058909+0000","last_fullsized":"2024-09-11T00:56:28.058909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":8,"num_write":13276,"num_write_kb":225109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"60'4","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051666+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:56:28.051666+0000","last_peered":"2024-09-11T00:56:28.051666+0000","last_clean":"2024-09-11T00:56:28.051666+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:56:28.051666+0000","last_undegraded":"2024-09-11T00:56:28.051666+0000","last_fullsized":"2024-09-11T00:56:28.051666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"61'2","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066019+0000","last_change":"2024-09-11T00:30:53.986858+0000","last_active":"2024-09-11T00:56:28.066019+0000","last_peered":"2024-09-11T00:56:28.066019+0000","last_clean":"2024-09-11T00:56:28.066019+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:56:28.066019+0000","last_undegraded":"2024-09-11T00:56:28.066019+0000","last_fullsized":"2024-09-11T00:56:28.066019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:33.626193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"62'3470","reported_seq":3566,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397582+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:56:28.397582+0000","last_peered":"2024-09-11T00:56:28.397582+0000","last_clean":"2024-09-11T00:56:28.397582+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:56:28.397582+0000","last_undegraded":"2024-09-11T00:56:28.397582+0000","last_fullsized":"2024-09-11T00:56:28.397582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17553,"num_write_kb":279333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"60'2695","reported_seq":2801,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.046481+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:56:28.046481+0000","last_peered":"2024-09-11T00:56:28.046481+0000","last_clean":"2024-09-11T00:56:28.046481+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:56:28.046481+0000","last_undegraded":"2024-09-11T00:56:28.046481+0000","last_fullsized":"2024-09-11T00:56:28.046481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":14,"num_write":14032,"num_write_kb":244535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"60'3","reported_seq":109,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048672+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:56:28.048672+0000","last_peered":"2024-09-11T00:56:28.048672+0000","last_clean":"2024-09-11T00:56:28.048672+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:56:28.048672+0000","last_undegraded":"2024-09-11T00:56:28.048672+0000","last_fullsized":"2024-09-11T00:56:28.048672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"62'5","reported_seq":124,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.505942+0000","last_change":"2024-09-11T00:30:53.990765+0000","last_active":"2024-09-11T00:56:28.505942+0000","last_peered":"2024-09-11T00:56:28.505942+0000","last_clean":"2024-09-11T00:56:28.505942+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:56:28.505942+0000","last_undegraded":"2024-09-11T00:56:28.505942+0000","last_fullsized":"2024-09-11T00:56:28.505942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:33.720727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"62'3622","reported_seq":3718,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.450009+0000","last_change":"2024-09-11T00:30:54.984949+0000","last_active":"2024-09-11T00:56:28.450009+0000","last_peered":"2024-09-11T00:56:28.450009+0000","last_clean":"2024-09-11T00:56:28.450009+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:56:28.450009+0000","last_undegraded":"2024-09-11T00:56:28.450009+0000","last_fullsized":"2024-09-11T00:56:28.450009+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:54.346937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18608,"num_write_kb":291609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"60'2732","reported_seq":2837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063199+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:56:28.063199+0000","last_peered":"2024-09-11T00:56:28.063199+0000","last_clean":"2024-09-11T00:56:28.063199+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:56:28.063199+0000","last_undegraded":"2024-09-11T00:56:28.063199+0000","last_fullsized":"2024-09-11T00:56:28.063199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13936,"num_write_kb":248214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"60'3","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063396+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:56:28.063396+0000","last_peered":"2024-09-11T00:56:28.063396+0000","last_clean":"2024-09-11T00:56:28.063396+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:56:28.063396+0000","last_undegraded":"2024-09-11T00:56:28.063396+0000","last_fullsized":"2024-09-11T00:56:28.063396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"62'6","reported_seq":146,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500913+0000","last_change":"2024-09-11T00:30:53.986940+0000","last_active":"2024-09-11T00:56:28.500913+0000","last_peered":"2024-09-11T00:56:28.500913+0000","last_clean":"2024-09-11T00:56:28.500913+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:56:28.500913+0000","last_undegraded":"2024-09-11T00:56:28.500913+0000","last_fullsized":"2024-09-11T00:56:28.500913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:11.942187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"62'3257","reported_seq":3355,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.520613+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:56:28.520613+0000","last_peered":"2024-09-11T00:56:28.520613+0000","last_clean":"2024-09-11T00:56:28.520613+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:56:28.520613+0000","last_undegraded":"2024-09-11T00:56:28.520613+0000","last_fullsized":"2024-09-11T00:56:28.520613+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17168,"num_write_kb":277608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'2688","reported_seq":2794,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048103+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:56:28.048103+0000","last_peered":"2024-09-11T00:56:28.048103+0000","last_clean":"2024-09-11T00:56:28.048103+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:56:28.048103+0000","last_undegraded":"2024-09-11T00:56:28.048103+0000","last_fullsized":"2024-09-11T00:56:28.048103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13441,"num_write_kb":246089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"62'5","reported_seq":141,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.085125+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:56:31.085125+0000","last_peered":"2024-09-11T00:56:31.085125+0000","last_clean":"2024-09-11T00:56:31.085125+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:56:31.085125+0000","last_undegraded":"2024-09-11T00:56:31.085125+0000","last_fullsized":"2024-09-11T00:56:31.085125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"62'65","reported_seq":245,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:37.625005+0000","last_change":"2024-09-11T00:30:53.988539+0000","last_active":"2024-09-11T00:56:37.625005+0000","last_peered":"2024-09-11T00:56:37.625005+0000","last_clean":"2024-09-11T00:56:37.625005+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:56:37.625005+0000","last_undegraded":"2024-09-11T00:56:37.625005+0000","last_fullsized":"2024-09-11T00:56:37.625005+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:19.083664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":126,"num_write":62,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"62'3386","reported_seq":3484,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.422439+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:56:28.422439+0000","last_peered":"2024-09-11T00:56:28.422439+0000","last_clean":"2024-09-11T00:56:28.422439+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:56:28.422439+0000","last_undegraded":"2024-09-11T00:56:28.422439+0000","last_fullsized":"2024-09-11T00:56:28.422439+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18005,"num_write_kb":276900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'3194","reported_seq":3298,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.050561+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:56:28.050561+0000","last_peered":"2024-09-11T00:56:28.050561+0000","last_clean":"2024-09-11T00:56:28.050561+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:56:28.050561+0000","last_undegraded":"2024-09-11T00:56:28.050561+0000","last_fullsized":"2024-09-11T00:56:28.050561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16397,"num_write_kb":262231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"62'521","reported_seq":641,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:32.047226+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:56:32.047226+0000","last_peered":"2024-09-11T00:56:32.047226+0000","last_clean":"2024-09-11T00:56:32.047226+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:56:32.047226+0000","last_undegraded":"2024-09-11T00:56:32.047226+0000","last_fullsized":"2024-09-11T00:56:32.047226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":41,"num_write":1157,"num_write_kb":372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"62'9","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.502805+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:56:28.502805+0000","last_peered":"2024-09-11T00:56:28.502805+0000","last_clean":"2024-09-11T00:56:28.502805+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:56:28.502805+0000","last_undegraded":"2024-09-11T00:56:28.502805+0000","last_fullsized":"2024-09-11T00:56:28.502805+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"61'3259","reported_seq":3357,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063309+0000","last_change":"2024-09-11T00:30:54.993118+0000","last_active":"2024-09-11T00:56:28.063309+0000","last_peered":"2024-09-11T00:56:28.063309+0000","last_clean":"2024-09-11T00:56:28.063309+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:56:28.063309+0000","last_undegraded":"2024-09-11T00:56:28.063309+0000","last_fullsized":"2024-09-11T00:56:28.063309+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:07:41.148370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16779,"num_write_kb":276080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"60'3127","reported_seq":3233,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052954+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:56:28.052954+0000","last_peered":"2024-09-11T00:56:28.052954+0000","last_clean":"2024-09-11T00:56:28.052954+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:56:28.052954+0000","last_undegraded":"2024-09-11T00:56:28.052954+0000","last_fullsized":"2024-09-11T00:56:28.052954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":12,"num_write":16580,"num_write_kb":258802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"60'4","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069515+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:56:28.069515+0000","last_peered":"2024-09-11T00:56:28.069515+0000","last_clean":"2024-09-11T00:56:28.069515+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:56:28.069515+0000","last_undegraded":"2024-09-11T00:56:28.069515+0000","last_fullsized":"2024-09-11T00:56:28.069515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'1","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052292+0000","last_change":"2024-09-11T00:30:53.986843+0000","last_active":"2024-09-11T00:56:28.052292+0000","last_peered":"2024-09-11T00:56:28.052292+0000","last_clean":"2024-09-11T00:56:28.052292+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:56:28.052292+0000","last_undegraded":"2024-09-11T00:56:28.052292+0000","last_fullsized":"2024-09-11T00:56:28.052292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:44.684154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"62'3801","reported_seq":3898,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.047541+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:56:28.047541+0000","last_peered":"2024-09-11T00:56:28.047541+0000","last_clean":"2024-09-11T00:56:28.047541+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:56:28.047541+0000","last_undegraded":"2024-09-11T00:56:28.047541+0000","last_fullsized":"2024-09-11T00:56:28.047541+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20214,"num_write_kb":289874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"60'2498","reported_seq":2605,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075698+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:56:28.075698+0000","last_peered":"2024-09-11T00:56:28.075698+0000","last_clean":"2024-09-11T00:56:28.075698+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:56:28.075698+0000","last_undegraded":"2024-09-11T00:56:28.075698+0000","last_fullsized":"2024-09-11T00:56:28.075698+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2498,"log_dups_size":0,"ondisk_log_size":2498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":13564,"num_write_kb":230524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"60'5","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059256+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:56:28.059256+0000","last_peered":"2024-09-11T00:56:28.059256+0000","last_clean":"2024-09-11T00:56:28.059256+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:56:28.059256+0000","last_undegraded":"2024-09-11T00:56:28.059256+0000","last_fullsized":"2024-09-11T00:56:28.059256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"62'5","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510149+0000","last_change":"2024-09-11T00:30:53.986896+0000","last_active":"2024-09-11T00:56:28.510149+0000","last_peered":"2024-09-11T00:56:28.510149+0000","last_clean":"2024-09-11T00:56:28.510149+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:56:28.510149+0000","last_undegraded":"2024-09-11T00:56:28.510149+0000","last_fullsized":"2024-09-11T00:56:28.510149+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:06.590278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"62'3103","reported_seq":3201,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058933+0000","last_change":"2024-09-11T00:30:55.001370+0000","last_active":"2024-09-11T00:56:28.058933+0000","last_peered":"2024-09-11T00:56:28.058933+0000","last_clean":"2024-09-11T00:56:28.058933+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:56:28.058933+0000","last_undegraded":"2024-09-11T00:56:28.058933+0000","last_fullsized":"2024-09-11T00:56:28.058933+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:56.592767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16058,"num_write_kb":244544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"60'2543","reported_seq":2648,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067860+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:56:28.067860+0000","last_peered":"2024-09-11T00:56:28.067860+0000","last_clean":"2024-09-11T00:56:28.067860+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:56:28.067860+0000","last_undegraded":"2024-09-11T00:56:28.067860+0000","last_fullsized":"2024-09-11T00:56:28.067860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":230458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"60'5","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067593+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:56:28.067593+0000","last_peered":"2024-09-11T00:56:28.067593+0000","last_clean":"2024-09-11T00:56:28.067593+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:56:28.067593+0000","last_undegraded":"2024-09-11T00:56:28.067593+0000","last_fullsized":"2024-09-11T00:56:28.067593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"62'5","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.494925+0000","last_change":"2024-09-11T00:30:53.978480+0000","last_active":"2024-09-11T00:56:28.494925+0000","last_peered":"2024-09-11T00:56:28.494925+0000","last_clean":"2024-09-11T00:56:28.494925+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:56:28.494925+0000","last_undegraded":"2024-09-11T00:56:28.494925+0000","last_fullsized":"2024-09-11T00:56:28.494925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:58.900565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"61'3199","reported_seq":3296,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058202+0000","last_change":"2024-09-11T00:30:54.997641+0000","last_active":"2024-09-11T00:56:28.058202+0000","last_peered":"2024-09-11T00:56:28.058202+0000","last_clean":"2024-09-11T00:56:28.058202+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:56:28.058202+0000","last_undegraded":"2024-09-11T00:56:28.058202+0000","last_fullsized":"2024-09-11T00:56:28.058202+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:18.437490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15924,"num_write_kb":260293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"60'3298","reported_seq":3404,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057304+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:56:28.057304+0000","last_peered":"2024-09-11T00:56:28.057304+0000","last_clean":"2024-09-11T00:56:28.057304+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:56:28.057304+0000","last_undegraded":"2024-09-11T00:56:28.057304+0000","last_fullsized":"2024-09-11T00:56:28.057304+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17655,"num_write_kb":267324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"62'73","reported_seq":262,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:33.050070+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:56:33.050070+0000","last_peered":"2024-09-11T00:56:33.050070+0000","last_clean":"2024-09-11T00:56:33.050070+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:56:33.050070+0000","last_undegraded":"2024-09-11T00:56:33.050070+0000","last_fullsized":"2024-09-11T00:56:33.050070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":129,"num_write":68,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"62'5","reported_seq":146,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500023+0000","last_change":"2024-09-11T00:30:53.979031+0000","last_active":"2024-09-11T00:56:28.500023+0000","last_peered":"2024-09-11T00:56:28.500023+0000","last_clean":"2024-09-11T00:56:28.500023+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:56:28.500023+0000","last_undegraded":"2024-09-11T00:56:28.500023+0000","last_fullsized":"2024-09-11T00:56:28.500023+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:36.604377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"62'2728","reported_seq":2824,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.475254+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:56:28.475254+0000","last_peered":"2024-09-11T00:56:28.475254+0000","last_clean":"2024-09-11T00:56:28.475254+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:56:28.475254+0000","last_undegraded":"2024-09-11T00:56:28.475254+0000","last_fullsized":"2024-09-11T00:56:28.475254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2728,"log_dups_size":0,"ondisk_log_size":2728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:39.629122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":237243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"60'2962","reported_seq":3068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051045+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:56:28.051045+0000","last_peered":"2024-09-11T00:56:28.051045+0000","last_clean":"2024-09-11T00:56:28.051045+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:56:28.051045+0000","last_undegraded":"2024-09-11T00:56:28.051045+0000","last_fullsized":"2024-09-11T00:56:28.051045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15238,"num_write_kb":265720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"62'8","reported_seq":116,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.640904+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:56:28.640904+0000","last_peered":"2024-09-11T00:56:28.640904+0000","last_clean":"2024-09-11T00:56:28.640904+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:56:28.640904+0000","last_undegraded":"2024-09-11T00:56:28.640904+0000","last_fullsized":"2024-09-11T00:56:28.640904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"62'499","reported_seq":618,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:35.590292+0000","last_change":"2024-09-11T00:30:53.989299+0000","last_active":"2024-09-11T00:56:35.590292+0000","last_peered":"2024-09-11T00:56:35.590292+0000","last_clean":"2024-09-11T00:56:35.590292+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:56:35.590292+0000","last_undegraded":"2024-09-11T00:56:35.590292+0000","last_fullsized":"2024-09-11T00:56:35.590292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:34:49.298915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":1123,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"62'3325","reported_seq":3422,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.494312+0000","last_change":"2024-09-11T00:30:54.990026+0000","last_active":"2024-09-11T00:56:28.494312+0000","last_peered":"2024-09-11T00:56:28.494312+0000","last_clean":"2024-09-11T00:56:28.494312+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:56:28.494312+0000","last_undegraded":"2024-09-11T00:56:28.494312+0000","last_fullsized":"2024-09-11T00:56:28.494312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:02:10.800530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17451,"num_write_kb":278810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"60'3332","reported_seq":3438,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051728+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:56:28.051728+0000","last_peered":"2024-09-11T00:56:28.051728+0000","last_clean":"2024-09-11T00:56:28.051728+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:56:28.051728+0000","last_undegraded":"2024-09-11T00:56:28.051728+0000","last_fullsized":"2024-09-11T00:56:28.051728+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3332,"log_dups_size":0,"ondisk_log_size":3332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17164,"num_write_kb":285768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"31'1","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.072247+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:56:28.072247+0000","last_peered":"2024-09-11T00:56:28.072247+0000","last_clean":"2024-09-11T00:56:28.072247+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:56:28.072247+0000","last_undegraded":"2024-09-11T00:56:28.072247+0000","last_fullsized":"2024-09-11T00:56:28.072247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":204,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066124+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:56:28.066124+0000","last_peered":"2024-09-11T00:56:28.066124+0000","last_clean":"2024-09-11T00:56:28.066124+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:56:28.066124+0000","last_undegraded":"2024-09-11T00:56:28.066124+0000","last_fullsized":"2024-09-11T00:56:28.066124+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"62'4","reported_seq":103,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488198+0000","last_change":"2024-09-11T00:30:53.982142+0000","last_active":"2024-09-11T00:56:28.488198+0000","last_peered":"2024-09-11T00:56:28.488198+0000","last_clean":"2024-09-11T00:56:28.488198+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:56:28.488198+0000","last_undegraded":"2024-09-11T00:56:28.488198+0000","last_fullsized":"2024-09-11T00:56:28.488198+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:41.558944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"62'2541","reported_seq":2637,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500843+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:56:28.500843+0000","last_peered":"2024-09-11T00:56:28.500843+0000","last_clean":"2024-09-11T00:56:28.500843+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:56:28.500843+0000","last_undegraded":"2024-09-11T00:56:28.500843+0000","last_fullsized":"2024-09-11T00:56:28.500843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12611,"num_write_kb":206053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"60'2809","reported_seq":2917,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069156+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:56:28.069156+0000","last_peered":"2024-09-11T00:56:28.069156+0000","last_clean":"2024-09-11T00:56:28.069156+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:56:28.069156+0000","last_undegraded":"2024-09-11T00:56:28.069156+0000","last_fullsized":"2024-09-11T00:56:28.069156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2809,"log_dups_size":0,"ondisk_log_size":2809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":34,"num_write":15096,"num_write_kb":232439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"60'2","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.053330+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:56:28.053330+0000","last_peered":"2024-09-11T00:56:28.053330+0000","last_clean":"2024-09-11T00:56:28.053330+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:56:28.053330+0000","last_undegraded":"2024-09-11T00:56:28.053330+0000","last_fullsized":"2024-09-11T00:56:28.053330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"62'4","reported_seq":102,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.498262+0000","last_change":"2024-09-11T00:30:53.989292+0000","last_active":"2024-09-11T00:56:28.498262+0000","last_peered":"2024-09-11T00:56:28.498262+0000","last_clean":"2024-09-11T00:56:28.498262+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:56:28.498262+0000","last_undegraded":"2024-09-11T00:56:28.498262+0000","last_fullsized":"2024-09-11T00:56:28.498262+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:44.071338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"62'3293","reported_seq":3389,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.437769+0000","last_change":"2024-09-11T00:30:54.981630+0000","last_active":"2024-09-11T00:56:28.437769+0000","last_peered":"2024-09-11T00:56:28.437769+0000","last_clean":"2024-09-11T00:56:28.437769+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:56:28.437769+0000","last_undegraded":"2024-09-11T00:56:28.437769+0000","last_fullsized":"2024-09-11T00:56:28.437769+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:12.832327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17661,"num_write_kb":267414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"60'3034","reported_seq":3140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066511+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:56:28.066511+0000","last_peered":"2024-09-11T00:56:28.066511+0000","last_clean":"2024-09-11T00:56:28.066511+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:56:28.066511+0000","last_undegraded":"2024-09-11T00:56:28.066511+0000","last_fullsized":"2024-09-11T00:56:28.066511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3034,"log_dups_size":0,"ondisk_log_size":3034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":15513,"num_write_kb":283129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"60'2","reported_seq":122,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060350+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:56:28.060350+0000","last_peered":"2024-09-11T00:56:28.060350+0000","last_clean":"2024-09-11T00:56:28.060350+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:56:28.060350+0000","last_undegraded":"2024-09-11T00:56:28.060350+0000","last_fullsized":"2024-09-11T00:56:28.060350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"62'1443","reported_seq":1586,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510169+0000","last_change":"2024-09-11T00:30:53.984013+0000","last_active":"2024-09-11T00:56:28.510169+0000","last_peered":"2024-09-11T00:56:28.510169+0000","last_clean":"2024-09-11T00:56:28.510169+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:56:28.510169+0000","last_undegraded":"2024-09-11T00:56:28.510169+0000","last_fullsized":"2024-09-11T00:56:28.510169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1443,"log_dups_size":0,"ondisk_log_size":1443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:49.963623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":36861,"num_write":1416,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'2904","reported_seq":3001,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.471315+0000","last_change":"2024-09-11T00:30:54.996860+0000","last_active":"2024-09-11T00:56:28.471315+0000","last_peered":"2024-09-11T00:56:28.471315+0000","last_clean":"2024-09-11T00:56:28.471315+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:56:28.471315+0000","last_undegraded":"2024-09-11T00:56:28.471315+0000","last_fullsized":"2024-09-11T00:56:28.471315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:43.516748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15203,"num_write_kb":241147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"60'3390","reported_seq":3494,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069427+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:56:28.069427+0000","last_peered":"2024-09-11T00:56:28.069427+0000","last_clean":"2024-09-11T00:56:28.069427+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:56:28.069427+0000","last_undegraded":"2024-09-11T00:56:28.069427+0000","last_fullsized":"2024-09-11T00:56:28.069427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17423,"num_write_kb":299173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"60'6","reported_seq":140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.070765+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:56:31.070765+0000","last_peered":"2024-09-11T00:56:31.070765+0000","last_clean":"2024-09-11T00:56:31.070765+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:56:31.070765+0000","last_undegraded":"2024-09-11T00:56:31.070765+0000","last_fullsized":"2024-09-11T00:56:31.070765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"62'3","reported_seq":101,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062449+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:56:28.062449+0000","last_peered":"2024-09-11T00:56:28.062449+0000","last_clean":"2024-09-11T00:56:28.062449+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:56:28.062449+0000","last_undegraded":"2024-09-11T00:56:28.062449+0000","last_fullsized":"2024-09-11T00:56:28.062449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:19.891052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"62'3177","reported_seq":3279,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.444533+0000","last_change":"2024-09-11T00:31:26.120086+0000","last_active":"2024-09-11T00:56:28.444533+0000","last_peered":"2024-09-11T00:56:28.444533+0000","last_clean":"2024-09-11T00:56:28.444533+0000","last_became_active":"2024-09-11T00:31:26.119641+0000","last_became_peered":"2024-09-11T00:31:26.119641+0000","last_unstale":"2024-09-11T00:56:28.444533+0000","last_undegraded":"2024-09-11T00:56:28.444533+0000","last_fullsized":"2024-09-11T00:56:28.444533+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3177,"log_dups_size":0,"ondisk_log_size":3177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:21:15.814831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16730,"num_write_kb":276023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"60'2906","reported_seq":3010,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.077855+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:56:28.077855+0000","last_peered":"2024-09-11T00:56:28.077855+0000","last_clean":"2024-09-11T00:56:28.077855+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:56:28.077855+0000","last_undegraded":"2024-09-11T00:56:28.077855+0000","last_fullsized":"2024-09-11T00:56:28.077855+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14712,"num_write_kb":255517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"60'5","reported_seq":125,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069857+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:56:28.069857+0000","last_peered":"2024-09-11T00:56:28.069857+0000","last_clean":"2024-09-11T00:56:28.069857+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:56:28.069857+0000","last_undegraded":"2024-09-11T00:56:28.069857+0000","last_fullsized":"2024-09-11T00:56:28.069857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"62'3","reported_seq":101,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.487423+0000","last_change":"2024-09-11T00:30:53.983352+0000","last_active":"2024-09-11T00:56:28.487423+0000","last_peered":"2024-09-11T00:56:28.487423+0000","last_clean":"2024-09-11T00:56:28.487423+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:56:28.487423+0000","last_undegraded":"2024-09-11T00:56:28.487423+0000","last_fullsized":"2024-09-11T00:56:28.487423+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:43.826114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"62'2973","reported_seq":3063,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.532651+0000","last_change":"2024-09-11T00:31:26.119816+0000","last_active":"2024-09-11T00:56:28.532651+0000","last_peered":"2024-09-11T00:56:28.532651+0000","last_clean":"2024-09-11T00:56:28.532651+0000","last_became_active":"2024-09-11T00:31:26.119440+0000","last_became_peered":"2024-09-11T00:31:26.119440+0000","last_unstale":"2024-09-11T00:56:28.532651+0000","last_undegraded":"2024-09-11T00:56:28.532651+0000","last_fullsized":"2024-09-11T00:56:28.532651+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:21:45.707321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15629,"num_write_kb":242839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"60'3355","reported_seq":3459,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062827+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:56:28.062827+0000","last_peered":"2024-09-11T00:56:28.062827+0000","last_clean":"2024-09-11T00:56:28.062827+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:56:28.062827+0000","last_undegraded":"2024-09-11T00:56:28.062827+0000","last_fullsized":"2024-09-11T00:56:28.062827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18048,"num_write_kb":289456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"62'12","reported_seq":177,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.077796+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:56:31.077796+0000","last_peered":"2024-09-11T00:56:31.077796+0000","last_clean":"2024-09-11T00:56:31.077796+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:56:31.077796+0000","last_undegraded":"2024-09-11T00:56:31.077796+0000","last_fullsized":"2024-09-11T00:56:31.077796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":177,"num_read_kb":177,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"62'2","reported_seq":99,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488958+0000","last_change":"2024-09-11T00:30:53.980703+0000","last_active":"2024-09-11T00:56:28.488958+0000","last_peered":"2024-09-11T00:56:28.488958+0000","last_clean":"2024-09-11T00:56:28.488958+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:56:28.488958+0000","last_undegraded":"2024-09-11T00:56:28.488958+0000","last_fullsized":"2024-09-11T00:56:28.488958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:51.050512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"62'2678","reported_seq":2774,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.413635+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:56:28.413635+0000","last_peered":"2024-09-11T00:56:28.413635+0000","last_clean":"2024-09-11T00:56:28.413635+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:56:28.413635+0000","last_undegraded":"2024-09-11T00:56:28.413635+0000","last_fullsized":"2024-09-11T00:56:28.413635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2678,"log_dups_size":0,"ondisk_log_size":2678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13518,"num_write_kb":242590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"60'2791","reported_seq":2895,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.056927+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:56:28.056927+0000","last_peered":"2024-09-11T00:56:28.056927+0000","last_clean":"2024-09-11T00:56:28.056927+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:56:28.056927+0000","last_undegraded":"2024-09-11T00:56:28.056927+0000","last_fullsized":"2024-09-11T00:56:28.056927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2791,"log_dups_size":0,"ondisk_log_size":2791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":254542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"60'6","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063217+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:56:28.063217+0000","last_peered":"2024-09-11T00:56:28.063217+0000","last_clean":"2024-09-11T00:56:28.063217+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:56:28.063217+0000","last_undegraded":"2024-09-11T00:56:28.063217+0000","last_fullsized":"2024-09-11T00:56:28.063217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"62'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059468+0000","last_change":"2024-09-11T00:30:53.987829+0000","last_active":"2024-09-11T00:56:28.059468+0000","last_peered":"2024-09-11T00:56:28.059468+0000","last_clean":"2024-09-11T00:56:28.059468+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:56:28.059468+0000","last_undegraded":"2024-09-11T00:56:28.059468+0000","last_fullsized":"2024-09-11T00:56:28.059468+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:57:43.792089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"62'3272","reported_seq":3367,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066748+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:56:28.066748+0000","last_peered":"2024-09-11T00:56:28.066748+0000","last_clean":"2024-09-11T00:56:28.066748+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:56:28.066748+0000","last_undegraded":"2024-09-11T00:56:28.066748+0000","last_fullsized":"2024-09-11T00:56:28.066748+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16692,"num_write_kb":275642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"60'3405","reported_seq":3511,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066672+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:56:28.066672+0000","last_peered":"2024-09-11T00:56:28.066672+0000","last_clean":"2024-09-11T00:56:28.066672+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:56:28.066672+0000","last_undegraded":"2024-09-11T00:56:28.066672+0000","last_fullsized":"2024-09-11T00:56:28.066672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3405,"log_dups_size":0,"ondisk_log_size":3405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":17487,"num_write_kb":315618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"60'5","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067380+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:56:28.067380+0000","last_peered":"2024-09-11T00:56:28.067380+0000","last_clean":"2024-09-11T00:56:28.067380+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:56:28.067380+0000","last_undegraded":"2024-09-11T00:56:28.067380+0000","last_fullsized":"2024-09-11T00:56:28.067380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"62'367","reported_seq":572,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.957442+0000","last_change":"2024-09-11T00:30:53.984057+0000","last_active":"2024-09-11T00:56:34.957442+0000","last_peered":"2024-09-11T00:56:34.957442+0000","last_clean":"2024-09-11T00:56:34.957442+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:56:34.957442+0000","last_undegraded":"2024-09-11T00:56:34.957442+0000","last_fullsized":"2024-09-11T00:56:34.957442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:10.477916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":191,"num_read_kb":78004,"num_write":360,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"62'4540","reported_seq":4637,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.544292+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:56:28.544292+0000","last_peered":"2024-09-11T00:56:28.544292+0000","last_clean":"2024-09-11T00:56:28.544292+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:56:28.544292+0000","last_undegraded":"2024-09-11T00:56:28.544292+0000","last_fullsized":"2024-09-11T00:56:28.544292+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23424,"num_write_kb":360436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"60'2652","reported_seq":2757,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051484+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:56:28.051484+0000","last_peered":"2024-09-11T00:56:28.051484+0000","last_clean":"2024-09-11T00:56:28.051484+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:56:28.051484+0000","last_undegraded":"2024-09-11T00:56:28.051484+0000","last_fullsized":"2024-09-11T00:56:28.051484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":13591,"num_write_kb":238653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"62'9","reported_seq":143,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.641347+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:56:28.641347+0000","last_peered":"2024-09-11T00:56:28.641347+0000","last_clean":"2024-09-11T00:56:28.641347+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:56:28.641347+0000","last_undegraded":"2024-09-11T00:56:28.641347+0000","last_fullsized":"2024-09-11T00:56:28.641347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":112,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"62'37","reported_seq":197,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.985643+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:56:28.985643+0000","last_peered":"2024-09-11T00:56:28.985643+0000","last_clean":"2024-09-11T00:56:28.985643+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:56:28.985643+0000","last_undegraded":"2024-09-11T00:56:28.985643+0000","last_fullsized":"2024-09-11T00:56:28.985643+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":36,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"62'4259","reported_seq":4356,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.548466+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:56:28.548466+0000","last_peered":"2024-09-11T00:56:28.548466+0000","last_clean":"2024-09-11T00:56:28.548466+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:56:28.548466+0000","last_undegraded":"2024-09-11T00:56:28.548466+0000","last_fullsized":"2024-09-11T00:56:28.548466+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22192,"num_write_kb":330506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"60'2765","reported_seq":2868,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067495+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:56:28.067495+0000","last_peered":"2024-09-11T00:56:28.067495+0000","last_clean":"2024-09-11T00:56:28.067495+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:56:28.067495+0000","last_undegraded":"2024-09-11T00:56:28.067495+0000","last_fullsized":"2024-09-11T00:56:28.067495+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14757,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"62'389","reported_seq":566,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.078750+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:56:31.078750+0000","last_peered":"2024-09-11T00:56:31.078750+0000","last_clean":"2024-09-11T00:56:31.078750+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:56:31.078750+0000","last_undegraded":"2024-09-11T00:56:31.078750+0000","last_fullsized":"2024-09-11T00:56:31.078750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":127,"num_read_kb":56252,"num_write":381,"num_write_kb":4290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"62'5","reported_seq":103,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.528773+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:56:28.528773+0000","last_peered":"2024-09-11T00:56:28.528773+0000","last_clean":"2024-09-11T00:56:28.528773+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:56:28.528773+0000","last_undegraded":"2024-09-11T00:56:28.528773+0000","last_fullsized":"2024-09-11T00:56:28.528773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"62'3220","reported_seq":3316,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.464767+0000","last_change":"2024-09-11T00:30:54.986017+0000","last_active":"2024-09-11T00:56:28.464767+0000","last_peered":"2024-09-11T00:56:28.464767+0000","last_clean":"2024-09-11T00:56:28.464767+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:56:28.464767+0000","last_undegraded":"2024-09-11T00:56:28.464767+0000","last_fullsized":"2024-09-11T00:56:28.464767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:58:12.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16450,"num_write_kb":262397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"60'3928","reported_seq":4033,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064969+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:56:28.064969+0000","last_peered":"2024-09-11T00:56:28.064969+0000","last_clean":"2024-09-11T00:56:28.064969+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:56:28.064969+0000","last_undegraded":"2024-09-11T00:56:28.064969+0000","last_fullsized":"2024-09-11T00:56:28.064969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20599,"num_write_kb":340888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"62'43","reported_seq":190,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.086970+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:56:31.086970+0000","last_peered":"2024-09-11T00:56:31.086970+0000","last_clean":"2024-09-11T00:56:31.086970+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:56:31.086970+0000","last_undegraded":"2024-09-11T00:56:31.086970+0000","last_fullsized":"2024-09-11T00:56:31.086970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":42,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"62'8","reported_seq":148,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.516812+0000","last_change":"2024-09-11T00:30:53.988462+0000","last_active":"2024-09-11T00:56:28.516812+0000","last_peered":"2024-09-11T00:56:28.516812+0000","last_clean":"2024-09-11T00:56:28.516812+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:56:28.516812+0000","last_undegraded":"2024-09-11T00:56:28.516812+0000","last_fullsized":"2024-09-11T00:56:28.516812+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:22.385557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"62'4111","reported_seq":4207,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.522560+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:56:28.522560+0000","last_peered":"2024-09-11T00:56:28.522560+0000","last_clean":"2024-09-11T00:56:28.522560+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:56:28.522560+0000","last_undegraded":"2024-09-11T00:56:28.522560+0000","last_fullsized":"2024-09-11T00:56:28.522560+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21746,"num_write_kb":328824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"60'2863","reported_seq":2972,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060043+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:56:28.060043+0000","last_peered":"2024-09-11T00:56:28.060043+0000","last_clean":"2024-09-11T00:56:28.060043+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:56:28.060043+0000","last_undegraded":"2024-09-11T00:56:28.060043+0000","last_fullsized":"2024-09-11T00:56:28.060043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2863,"log_dups_size":0,"ondisk_log_size":2863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":46,"num_write":14990,"num_write_kb":244331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"60'2","reported_seq":107,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068726+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:56:28.068726+0000","last_peered":"2024-09-11T00:56:28.068726+0000","last_clean":"2024-09-11T00:56:28.068726+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:56:28.068726+0000","last_undegraded":"2024-09-11T00:56:28.068726+0000","last_fullsized":"2024-09-11T00:56:28.068726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":203,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.264447+0000","last_change":"2024-09-11T00:30:53.986646+0000","last_active":"2024-09-11T00:56:34.264447+0000","last_peered":"2024-09-11T00:56:34.264447+0000","last_clean":"2024-09-11T00:56:34.264447+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:56:34.264447+0000","last_undegraded":"2024-09-11T00:56:34.264447+0000","last_fullsized":"2024-09-11T00:56:34.264447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:27.447940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":279,"num_read_kb":279,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"62'3641","reported_seq":3737,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514237+0000","last_change":"2024-09-11T00:30:54.982486+0000","last_active":"2024-09-11T00:56:28.514237+0000","last_peered":"2024-09-11T00:56:28.514237+0000","last_clean":"2024-09-11T00:56:28.514237+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:56:28.514237+0000","last_undegraded":"2024-09-11T00:56:28.514237+0000","last_fullsized":"2024-09-11T00:56:28.514237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:41.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19470,"num_write_kb":282742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"60'2896","reported_seq":3003,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.076032+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:56:28.076032+0000","last_peered":"2024-09-11T00:56:28.076032+0000","last_clean":"2024-09-11T00:56:28.076032+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:56:28.076032+0000","last_undegraded":"2024-09-11T00:56:28.076032+0000","last_fullsized":"2024-09-11T00:56:28.076032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":22,"num_write":14816,"num_write_kb":255894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"62'5","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064417+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:56:28.064417+0000","last_peered":"2024-09-11T00:56:28.064417+0000","last_clean":"2024-09-11T00:56:28.064417+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:56:28.064417+0000","last_undegraded":"2024-09-11T00:56:28.064417+0000","last_fullsized":"2024-09-11T00:56:28.064417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"62'6","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.504113+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:56:28.504113+0000","last_peered":"2024-09-11T00:56:28.504113+0000","last_clean":"2024-09-11T00:56:28.504113+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:56:28.504113+0000","last_undegraded":"2024-09-11T00:56:28.504113+0000","last_fullsized":"2024-09-11T00:56:28.504113+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"62'3862","reported_seq":3961,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.513928+0000","last_change":"2024-09-11T00:30:54.985091+0000","last_active":"2024-09-11T00:56:28.513928+0000","last_peered":"2024-09-11T00:56:28.513928+0000","last_clean":"2024-09-11T00:56:28.513928+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:56:28.513928+0000","last_undegraded":"2024-09-11T00:56:28.513928+0000","last_fullsized":"2024-09-11T00:56:28.513928+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:33.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20250,"num_write_kb":318736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"62'29","reported_seq":128,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.490960+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:56:28.490960+0000","last_peered":"2024-09-11T00:56:28.490960+0000","last_clean":"2024-09-11T00:56:28.490960+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:56:28.490960+0000","last_undegraded":"2024-09-11T00:56:28.490960+0000","last_fullsized":"2024-09-11T00:56:28.490960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"60'3526","reported_seq":3630,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048540+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:56:28.048540+0000","last_peered":"2024-09-11T00:56:28.048540+0000","last_clean":"2024-09-11T00:56:28.048540+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:56:28.048540+0000","last_undegraded":"2024-09-11T00:56:28.048540+0000","last_fullsized":"2024-09-11T00:56:28.048540+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":18407,"num_write_kb":311214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"60'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064114+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:56:28.064114+0000","last_peered":"2024-09-11T00:56:28.064114+0000","last_clean":"2024-09-11T00:56:28.064114+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:56:28.064114+0000","last_undegraded":"2024-09-11T00:56:28.064114+0000","last_fullsized":"2024-09-11T00:56:28.064114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"61'3026","reported_seq":3123,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.053185+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:56:28.053185+0000","last_peered":"2024-09-11T00:56:28.053185+0000","last_clean":"2024-09-11T00:56:28.053185+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:56:28.053185+0000","last_undegraded":"2024-09-11T00:56:28.053185+0000","last_fullsized":"2024-09-11T00:56:28.053185+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:11.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16026,"num_write_kb":256702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"61'972","reported_seq":1122,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065625+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:56:28.065625+0000","last_peered":"2024-09-11T00:56:28.065625+0000","last_clean":"2024-09-11T00:56:28.065625+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:56:28.065625+0000","last_undegraded":"2024-09-11T00:56:28.065625+0000","last_fullsized":"2024-09-11T00:56:28.065625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":972,"log_dups_size":0,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":7829,"num_write":949,"num_write_kb":4657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"60'3393","reported_seq":3498,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057539+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:56:28.057539+0000","last_peered":"2024-09-11T00:56:28.057539+0000","last_clean":"2024-09-11T00:56:28.057539+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:56:28.057539+0000","last_undegraded":"2024-09-11T00:56:28.057539+0000","last_fullsized":"2024-09-11T00:56:28.057539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":17621,"num_write_kb":291741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"60'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051439+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:56:28.051439+0000","last_peered":"2024-09-11T00:56:28.051439+0000","last_clean":"2024-09-11T00:56:28.051439+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:56:28.051439+0000","last_undegraded":"2024-09-11T00:56:28.051439+0000","last_fullsized":"2024-09-11T00:56:28.051439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"62'3945","reported_seq":4040,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.470154+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:56:28.470154+0000","last_peered":"2024-09-11T00:56:28.470154+0000","last_clean":"2024-09-11T00:56:28.470154+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:56:28.470154+0000","last_undegraded":"2024-09-11T00:56:28.470154+0000","last_fullsized":"2024-09-11T00:56:28.470154+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20523,"num_write_kb":340217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"62'101","reported_seq":208,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:37.634421+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:56:37.634421+0000","last_peered":"2024-09-11T00:56:37.634421+0000","last_clean":"2024-09-11T00:56:37.634421+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:56:37.634421+0000","last_undegraded":"2024-09-11T00:56:37.634421+0000","last_fullsized":"2024-09-11T00:56:37.634421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":768543,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":728,"num_write":83,"num_write_kb":803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"60'2411","reported_seq":2516,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.049073+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:56:28.049073+0000","last_peered":"2024-09-11T00:56:28.049073+0000","last_clean":"2024-09-11T00:56:28.049073+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:56:28.049073+0000","last_undegraded":"2024-09-11T00:56:28.049073+0000","last_fullsized":"2024-09-11T00:56:28.049073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":8,"num_write":12211,"num_write_kb":212619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"62'3","reported_seq":142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.049506+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:56:28.049506+0000","last_peered":"2024-09-11T00:56:28.049506+0000","last_clean":"2024-09-11T00:56:28.049506+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:56:28.049506+0000","last_undegraded":"2024-09-11T00:56:28.049506+0000","last_fullsized":"2024-09-11T00:56:28.049506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"62'2354","reported_seq":2449,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.471047+0000","last_change":"2024-09-11T00:30:54.998464+0000","last_active":"2024-09-11T00:56:28.471047+0000","last_peered":"2024-09-11T00:56:28.471047+0000","last_clean":"2024-09-11T00:56:28.471047+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:56:28.471047+0000","last_undegraded":"2024-09-11T00:56:28.471047+0000","last_fullsized":"2024-09-11T00:56:28.471047+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:56.469791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":205009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"60'676","reported_seq":797,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059574+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:56:28.059574+0000","last_peered":"2024-09-11T00:56:28.059574+0000","last_clean":"2024-09-11T00:56:28.059574+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:56:28.059574+0000","last_undegraded":"2024-09-11T00:56:28.059574+0000","last_fullsized":"2024-09-11T00:56:28.059574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":51443,"num_write":670,"num_write_kb":4291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"60'2936","reported_seq":3039,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058263+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:56:28.058263+0000","last_peered":"2024-09-11T00:56:28.058263+0000","last_clean":"2024-09-11T00:56:28.058263+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:56:28.058263+0000","last_undegraded":"2024-09-11T00:56:28.058263+0000","last_fullsized":"2024-09-11T00:56:28.058263+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2936,"log_dups_size":0,"ondisk_log_size":2936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":241742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"62'45","reported_seq":184,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488110+0000","last_change":"2024-09-11T00:30:53.987612+0000","last_active":"2024-09-11T00:56:28.488110+0000","last_peered":"2024-09-11T00:56:28.488110+0000","last_clean":"2024-09-11T00:56:28.488110+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:56:28.488110+0000","last_undegraded":"2024-09-11T00:56:28.488110+0000","last_fullsized":"2024-09-11T00:56:28.488110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:16.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"62'3441","reported_seq":3532,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.489542+0000","last_change":"2024-09-11T00:31:26.326200+0000","last_active":"2024-09-11T00:56:28.489542+0000","last_peered":"2024-09-11T00:56:28.489542+0000","last_clean":"2024-09-11T00:56:28.489542+0000","last_became_active":"2024-09-11T00:31:26.325895+0000","last_became_peered":"2024-09-11T00:31:26.325895+0000","last_unstale":"2024-09-11T00:56:28.489542+0000","last_undegraded":"2024-09-11T00:56:28.489542+0000","last_fullsized":"2024-09-11T00:56:28.489542+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:39:08.753699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18163,"num_write_kb":273760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.20","version":"60'3848","reported_seq":3953,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075938+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:56:28.075938+0000","last_peered":"2024-09-11T00:56:28.075938+0000","last_clean":"2024-09-11T00:56:28.075938+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:56:28.075938+0000","last_undegraded":"2024-09-11T00:56:28.075938+0000","last_fullsized":"2024-09-11T00:56:28.075938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20029,"num_write_kb":325938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"62'4287","reported_seq":4384,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.519505+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:56:28.519505+0000","last_peered":"2024-09-11T00:56:28.519505+0000","last_clean":"2024-09-11T00:56:28.519505+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:56:28.519505+0000","last_undegraded":"2024-09-11T00:56:28.519505+0000","last_fullsized":"2024-09-11T00:56:28.519505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22520,"num_write_kb":336110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"60'2962","reported_seq":3067,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058177+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:56:28.058177+0000","last_peered":"2024-09-11T00:56:28.058177+0000","last_clean":"2024-09-11T00:56:28.058177+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:56:28.058177+0000","last_undegraded":"2024-09-11T00:56:28.058177+0000","last_fullsized":"2024-09-11T00:56:28.058177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15172,"num_write_kb":277710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"62'2334","reported_seq":2430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075259+0000","last_change":"2024-09-11T00:30:54.979133+0000","last_active":"2024-09-11T00:56:28.075259+0000","last_peered":"2024-09-11T00:56:28.075259+0000","last_clean":"2024-09-11T00:56:28.075259+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:56:28.075259+0000","last_undegraded":"2024-09-11T00:56:28.075259+0000","last_fullsized":"2024-09-11T00:56:28.075259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2334,"log_dups_size":0,"ondisk_log_size":2334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:20.961678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11753,"num_write_kb":206776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"60'4175","reported_seq":4282,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065154+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:56:28.065154+0000","last_peered":"2024-09-11T00:56:28.065154+0000","last_clean":"2024-09-11T00:56:28.065154+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:56:28.065154+0000","last_undegraded":"2024-09-11T00:56:28.065154+0000","last_fullsized":"2024-09-11T00:56:28.065154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22002,"num_write_kb":346058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"62'3428","reported_seq":3524,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.426792+0000","last_change":"2024-09-11T00:30:54.988345+0000","last_active":"2024-09-11T00:56:28.426792+0000","last_peered":"2024-09-11T00:56:28.426792+0000","last_clean":"2024-09-11T00:56:28.426792+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:56:28.426792+0000","last_undegraded":"2024-09-11T00:56:28.426792+0000","last_fullsized":"2024-09-11T00:56:28.426792+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3428,"log_dups_size":0,"ondisk_log_size":3428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.112194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17996,"num_write_kb":289213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"60'2543","reported_seq":2648,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059170+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:56:28.059170+0000","last_peered":"2024-09-11T00:56:28.059170+0000","last_clean":"2024-09-11T00:56:28.059170+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:56:28.059170+0000","last_undegraded":"2024-09-11T00:56:28.059170+0000","last_fullsized":"2024-09-11T00:56:28.059170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13221,"num_write_kb":224922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"62'3390","reported_seq":3487,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.413384+0000","last_change":"2024-09-11T00:30:54.981185+0000","last_active":"2024-09-11T00:56:28.413384+0000","last_peered":"2024-09-11T00:56:28.413384+0000","last_clean":"2024-09-11T00:56:28.413384+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:56:28.413384+0000","last_undegraded":"2024-09-11T00:56:28.413384+0000","last_fullsized":"2024-09-11T00:56:28.413384+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:45.740467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17320,"num_write_kb":290458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"60'2853","reported_seq":2957,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052823+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:56:28.052823+0000","last_peered":"2024-09-11T00:56:28.052823+0000","last_clean":"2024-09-11T00:56:28.052823+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:56:28.052823+0000","last_undegraded":"2024-09-11T00:56:28.052823+0000","last_fullsized":"2024-09-11T00:56:28.052823+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14682,"num_write_kb":238952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"62'2740","reported_seq":2837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.499809+0000","last_change":"2024-09-11T00:30:54.992763+0000","last_active":"2024-09-11T00:56:28.499809+0000","last_peered":"2024-09-11T00:56:28.499809+0000","last_clean":"2024-09-11T00:56:28.499809+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:56:28.499809+0000","last_undegraded":"2024-09-11T00:56:28.499809+0000","last_fullsized":"2024-09-11T00:56:28.499809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2740,"log_dups_size":0,"ondisk_log_size":2740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:02.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14066,"num_write_kb":224154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"60'3660","reported_seq":3764,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052923+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:56:28.052923+0000","last_peered":"2024-09-11T00:56:28.052923+0000","last_clean":"2024-09-11T00:56:28.052923+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:56:28.052923+0000","last_undegraded":"2024-09-11T00:56:28.052923+0000","last_fullsized":"2024-09-11T00:56:28.052923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19466,"num_write_kb":307301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"62'4363","reported_seq":4460,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.478746+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:56:28.478746+0000","last_peered":"2024-09-11T00:56:28.478746+0000","last_clean":"2024-09-11T00:56:28.478746+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:56:28.478746+0000","last_undegraded":"2024-09-11T00:56:28.478746+0000","last_fullsized":"2024-09-11T00:56:28.478746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23064,"num_write_kb":350422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"60'2921","reported_seq":3025,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075658+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:56:28.075658+0000","last_peered":"2024-09-11T00:56:28.075658+0000","last_clean":"2024-09-11T00:56:28.075658+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:56:28.075658+0000","last_undegraded":"2024-09-11T00:56:28.075658+0000","last_fullsized":"2024-09-11T00:56:28.075658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15417,"num_write_kb":250464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"62'3030","reported_seq":3127,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.431858+0000","last_change":"2024-09-11T00:30:54.984447+0000","last_active":"2024-09-11T00:56:28.431858+0000","last_peered":"2024-09-11T00:56:28.431858+0000","last_clean":"2024-09-11T00:56:28.431858+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:56:28.431858+0000","last_undegraded":"2024-09-11T00:56:28.431858+0000","last_fullsized":"2024-09-11T00:56:28.431858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:49.024022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15510,"num_write_kb":262754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"60'3321","reported_seq":3427,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059533+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:56:28.059533+0000","last_peered":"2024-09-11T00:56:28.059533+0000","last_clean":"2024-09-11T00:56:28.059533+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:56:28.059533+0000","last_undegraded":"2024-09-11T00:56:28.059533+0000","last_fullsized":"2024-09-11T00:56:28.059533+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":25,"num_write":17209,"num_write_kb":314610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"62'2084","reported_seq":2181,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.543950+0000","last_change":"2024-09-11T00:30:54.996091+0000","last_active":"2024-09-11T00:56:28.543950+0000","last_peered":"2024-09-11T00:56:28.543950+0000","last_clean":"2024-09-11T00:56:28.543950+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:56:28.543950+0000","last_undegraded":"2024-09-11T00:56:28.543950+0000","last_fullsized":"2024-09-11T00:56:28.543950+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2084,"log_dups_size":0,"ondisk_log_size":2084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:42.987336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10636,"num_write_kb":173560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"60'2810","reported_seq":2914,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063723+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:56:28.063723+0000","last_peered":"2024-09-11T00:56:28.063723+0000","last_clean":"2024-09-11T00:56:28.063723+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:56:28.063723+0000","last_undegraded":"2024-09-11T00:56:28.063723+0000","last_fullsized":"2024-09-11T00:56:28.063723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2810,"log_dups_size":0,"ondisk_log_size":2810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":243007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"62'3697","reported_seq":3793,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.455458+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:56:28.455458+0000","last_peered":"2024-09-11T00:56:28.455458+0000","last_clean":"2024-09-11T00:56:28.455458+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:56:28.455458+0000","last_undegraded":"2024-09-11T00:56:28.455458+0000","last_fullsized":"2024-09-11T00:56:28.455458+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18782,"num_write_kb":312790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"60'2824","reported_seq":2928,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057103+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:56:28.057103+0000","last_peered":"2024-09-11T00:56:28.057103+0000","last_clean":"2024-09-11T00:56:28.057103+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:56:28.057103+0000","last_undegraded":"2024-09-11T00:56:28.057103+0000","last_fullsized":"2024-09-11T00:56:28.057103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2824,"log_dups_size":0,"ondisk_log_size":2824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14695,"num_write_kb":259410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"62'2917","reported_seq":3013,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510399+0000","last_change":"2024-09-11T00:30:54.981799+0000","last_active":"2024-09-11T00:56:28.510399+0000","last_peered":"2024-09-11T00:56:28.510399+0000","last_clean":"2024-09-11T00:56:28.510399+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:56:28.510399+0000","last_undegraded":"2024-09-11T00:56:28.510399+0000","last_fullsized":"2024-09-11T00:56:28.510399+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2917,"log_dups_size":0,"ondisk_log_size":2917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:42.670629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":242029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"60'3431","reported_seq":3535,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057841+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:56:28.057841+0000","last_peered":"2024-09-11T00:56:28.057841+0000","last_clean":"2024-09-11T00:56:28.057841+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:56:28.057841+0000","last_undegraded":"2024-09-11T00:56:28.057841+0000","last_fullsized":"2024-09-11T00:56:28.057841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18408,"num_write_kb":290787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"62'2945","reported_seq":3041,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514103+0000","last_change":"2024-09-11T00:30:54.992697+0000","last_active":"2024-09-11T00:56:28.514103+0000","last_peered":"2024-09-11T00:56:28.514103+0000","last_clean":"2024-09-11T00:56:28.514103+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:56:28.514103+0000","last_undegraded":"2024-09-11T00:56:28.514103+0000","last_fullsized":"2024-09-11T00:56:28.514103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:54:07.227169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15229,"num_write_kb":249300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"60'2849","reported_seq":2955,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048641+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:56:28.048641+0000","last_peered":"2024-09-11T00:56:28.048641+0000","last_clean":"2024-09-11T00:56:28.048641+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:56:28.048641+0000","last_undegraded":"2024-09-11T00:56:28.048641+0000","last_fullsized":"2024-09-11T00:56:28.048641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2849,"log_dups_size":0,"ondisk_log_size":2849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14625,"num_write_kb":263252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"62'4017","reported_seq":4113,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.477074+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:56:28.477074+0000","last_peered":"2024-09-11T00:56:28.477074+0000","last_clean":"2024-09-11T00:56:28.477074+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:56:28.477074+0000","last_undegraded":"2024-09-11T00:56:28.477074+0000","last_fullsized":"2024-09-11T00:56:28.477074+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":329319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"60'2942","reported_seq":3046,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066118+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:56:28.066118+0000","last_peered":"2024-09-11T00:56:28.066118+0000","last_clean":"2024-09-11T00:56:28.066118+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:56:28.066118+0000","last_undegraded":"2024-09-11T00:56:28.066118+0000","last_fullsized":"2024-09-11T00:56:28.066118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15436,"num_write_kb":246093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"62'3417","reported_seq":3514,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.528940+0000","last_change":"2024-09-11T00:30:54.985827+0000","last_active":"2024-09-11T00:56:28.528940+0000","last_peered":"2024-09-11T00:56:28.528940+0000","last_clean":"2024-09-11T00:56:28.528940+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:56:28.528940+0000","last_undegraded":"2024-09-11T00:56:28.528940+0000","last_fullsized":"2024-09-11T00:56:28.528940+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:32.027117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18129,"num_write_kb":269216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"60'2746","reported_seq":2850,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069377+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:56:28.069377+0000","last_peered":"2024-09-11T00:56:28.069377+0000","last_clean":"2024-09-11T00:56:28.069377+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:56:28.069377+0000","last_undegraded":"2024-09-11T00:56:28.069377+0000","last_fullsized":"2024-09-11T00:56:28.069377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13859,"num_write_kb":247924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"62'2570","reported_seq":2672,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514173+0000","last_change":"2024-09-11T00:31:26.119785+0000","last_active":"2024-09-11T00:56:28.514173+0000","last_peered":"2024-09-11T00:56:28.514173+0000","last_clean":"2024-09-11T00:56:28.514173+0000","last_became_active":"2024-09-11T00:31:26.119436+0000","last_became_peered":"2024-09-11T00:31:26.119436+0000","last_unstale":"2024-09-11T00:56:28.514173+0000","last_undegraded":"2024-09-11T00:56:28.514173+0000","last_fullsized":"2024-09-11T00:56:28.514173+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:53:49.407575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13205,"num_write_kb":216683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"60'2388","reported_seq":2494,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058894+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:56:28.058894+0000","last_peered":"2024-09-11T00:56:28.058894+0000","last_clean":"2024-09-11T00:56:28.058894+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:56:28.058894+0000","last_undegraded":"2024-09-11T00:56:28.058894+0000","last_fullsized":"2024-09-11T00:56:28.058894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2388,"log_dups_size":0,"ondisk_log_size":2388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4,"num_write":12384,"num_write_kb":205064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"62'3381","reported_seq":3477,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397849+0000","last_change":"2024-09-11T00:30:54.993156+0000","last_active":"2024-09-11T00:56:28.397849+0000","last_peered":"2024-09-11T00:56:28.397849+0000","last_clean":"2024-09-11T00:56:28.397849+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:56:28.397849+0000","last_undegraded":"2024-09-11T00:56:28.397849+0000","last_fullsized":"2024-09-11T00:56:28.397849+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:25.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17747,"num_write_kb":280034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"60'3936","reported_seq":4041,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067998+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:56:28.067998+0000","last_peered":"2024-09-11T00:56:28.067998+0000","last_clean":"2024-09-11T00:56:28.067998+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:56:28.067998+0000","last_undegraded":"2024-09-11T00:56:28.067998+0000","last_fullsized":"2024-09-11T00:56:28.067998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":20459,"num_write_kb":335990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"62'3772","reported_seq":3867,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.454399+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:56:28.454399+0000","last_peered":"2024-09-11T00:56:28.454399+0000","last_clean":"2024-09-11T00:56:28.454399+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:56:28.454399+0000","last_undegraded":"2024-09-11T00:56:28.454399+0000","last_fullsized":"2024-09-11T00:56:28.454399+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20201,"num_write_kb":302053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"60'2757","reported_seq":2861,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062570+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:56:28.062570+0000","last_peered":"2024-09-11T00:56:28.062570+0000","last_clean":"2024-09-11T00:56:28.062570+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:56:28.062570+0000","last_undegraded":"2024-09-11T00:56:28.062570+0000","last_fullsized":"2024-09-11T00:56:28.062570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14203,"num_write_kb":241057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"62'4048","reported_seq":4143,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.441208+0000","last_change":"2024-09-11T00:30:54.993461+0000","last_active":"2024-09-11T00:56:28.441208+0000","last_peered":"2024-09-11T00:56:28.441208+0000","last_clean":"2024-09-11T00:56:28.441208+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:56:28.441208+0000","last_undegraded":"2024-09-11T00:56:28.441208+0000","last_fullsized":"2024-09-11T00:56:28.441208+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:07.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21387,"num_write_kb":364007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"60'3572","reported_seq":3676,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057127+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:56:28.057127+0000","last_peered":"2024-09-11T00:56:28.057127+0000","last_clean":"2024-09-11T00:56:28.057127+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:56:28.057127+0000","last_undegraded":"2024-09-11T00:56:28.057127+0000","last_fullsized":"2024-09-11T00:56:28.057127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18317,"num_write_kb":294510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"62'2987","reported_seq":3083,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397386+0000","last_change":"2024-09-11T00:30:54.986265+0000","last_active":"2024-09-11T00:56:28.397386+0000","last_peered":"2024-09-11T00:56:28.397386+0000","last_clean":"2024-09-11T00:56:28.397386+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:56:28.397386+0000","last_undegraded":"2024-09-11T00:56:28.397386+0000","last_fullsized":"2024-09-11T00:56:28.397386+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:49.109328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15338,"num_write_kb":237611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"60'2929","reported_seq":3034,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051236+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:56:28.051236+0000","last_peered":"2024-09-11T00:56:28.051236+0000","last_clean":"2024-09-11T00:56:28.051236+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:56:28.051236+0000","last_undegraded":"2024-09-11T00:56:28.051236+0000","last_fullsized":"2024-09-11T00:56:28.051236+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2929,"log_dups_size":0,"ondisk_log_size":2929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":15867,"num_write_kb":268409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"62'3243","reported_seq":3338,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.433332+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:56:28.433332+0000","last_peered":"2024-09-11T00:56:28.433332+0000","last_clean":"2024-09-11T00:56:28.433332+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:56:28.433332+0000","last_undegraded":"2024-09-11T00:56:28.433332+0000","last_fullsized":"2024-09-11T00:56:28.433332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16324,"num_write_kb":262051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"60'2913","reported_seq":3019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052467+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:56:28.052467+0000","last_peered":"2024-09-11T00:56:28.052467+0000","last_clean":"2024-09-11T00:56:28.052467+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:56:28.052467+0000","last_undegraded":"2024-09-11T00:56:28.052467+0000","last_fullsized":"2024-09-11T00:56:28.052467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2913,"log_dups_size":0,"ondisk_log_size":2913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15444,"num_write_kb":246103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"62'3344","reported_seq":3440,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.543938+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:56:28.543938+0000","last_peered":"2024-09-11T00:56:28.543938+0000","last_clean":"2024-09-11T00:56:28.543938+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:56:28.543938+0000","last_undegraded":"2024-09-11T00:56:28.543938+0000","last_fullsized":"2024-09-11T00:56:28.543938+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3344,"log_dups_size":0,"ondisk_log_size":3344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17374,"num_write_kb":274437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"60'2266","reported_seq":2373,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065676+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:56:28.065676+0000","last_peered":"2024-09-11T00:56:28.065676+0000","last_clean":"2024-09-11T00:56:28.065676+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:56:28.065676+0000","last_undegraded":"2024-09-11T00:56:28.065676+0000","last_fullsized":"2024-09-11T00:56:28.065676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":26,"num_write":11816,"num_write_kb":219351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"62'2714","reported_seq":2811,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.547334+0000","last_change":"2024-09-11T00:30:54.994988+0000","last_active":"2024-09-11T00:56:28.547334+0000","last_peered":"2024-09-11T00:56:28.547334+0000","last_clean":"2024-09-11T00:56:28.547334+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:56:28.547334+0000","last_undegraded":"2024-09-11T00:56:28.547334+0000","last_fullsized":"2024-09-11T00:56:28.547334+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2714,"log_dups_size":0,"ondisk_log_size":2714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:31.973641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14634,"num_write_kb":230575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"60'3170","reported_seq":3259,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058097+0000","last_change":"2024-09-11T00:31:26.117501+0000","last_active":"2024-09-11T00:56:28.058097+0000","last_peered":"2024-09-11T00:56:28.058097+0000","last_clean":"2024-09-11T00:56:28.058097+0000","last_became_active":"2024-09-11T00:31:26.117146+0000","last_became_peered":"2024-09-11T00:31:26.117146+0000","last_unstale":"2024-09-11T00:56:28.058097+0000","last_undegraded":"2024-09-11T00:56:28.058097+0000","last_fullsized":"2024-09-11T00:56:28.058097+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:52.194644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15922,"num_write_kb":268523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.30","version":"62'3135","reported_seq":3231,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.469071+0000","last_change":"2024-09-11T00:30:54.984767+0000","last_active":"2024-09-11T00:56:28.469071+0000","last_peered":"2024-09-11T00:56:28.469071+0000","last_clean":"2024-09-11T00:56:28.469071+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:56:28.469071+0000","last_undegraded":"2024-09-11T00:56:28.469071+0000","last_fullsized":"2024-09-11T00:56:28.469071+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:44.833677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16038,"num_write_kb":252613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"60'3300","reported_seq":3404,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068344+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:56:28.068344+0000","last_peered":"2024-09-11T00:56:28.068344+0000","last_clean":"2024-09-11T00:56:28.068344+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:56:28.068344+0000","last_undegraded":"2024-09-11T00:56:28.068344+0000","last_fullsized":"2024-09-11T00:56:28.068344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3300,"log_dups_size":0,"ondisk_log_size":3300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":17861,"num_write_kb":288513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"62'3599","reported_seq":3695,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.492582+0000","last_change":"2024-09-11T00:30:54.986423+0000","last_active":"2024-09-11T00:56:28.492582+0000","last_peered":"2024-09-11T00:56:28.492582+0000","last_clean":"2024-09-11T00:56:28.492582+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:56:28.492582+0000","last_undegraded":"2024-09-11T00:56:28.492582+0000","last_fullsized":"2024-09-11T00:56:28.492582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:16:50.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18686,"num_write_kb":304162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"60'4168","reported_seq":4272,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063833+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:56:28.063833+0000","last_peered":"2024-09-11T00:56:28.063833+0000","last_clean":"2024-09-11T00:56:28.063833+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:56:28.063833+0000","last_undegraded":"2024-09-11T00:56:28.063833+0000","last_fullsized":"2024-09-11T00:56:28.063833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22110,"num_write_kb":362983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"62'3937","reported_seq":4034,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.544539+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:56:28.544539+0000","last_peered":"2024-09-11T00:56:28.544539+0000","last_clean":"2024-09-11T00:56:28.544539+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:56:28.544539+0000","last_undegraded":"2024-09-11T00:56:28.544539+0000","last_fullsized":"2024-09-11T00:56:28.544539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21285,"num_write_kb":343312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"60'3475","reported_seq":3579,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057082+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:56:28.057082+0000","last_peered":"2024-09-11T00:56:28.057082+0000","last_clean":"2024-09-11T00:56:28.057082+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:56:28.057082+0000","last_undegraded":"2024-09-11T00:56:28.057082+0000","last_fullsized":"2024-09-11T00:56:28.057082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3475,"log_dups_size":0,"ondisk_log_size":3475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17645,"num_write_kb":295948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"62'3302","reported_seq":3399,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.512120+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:56:28.512120+0000","last_peered":"2024-09-11T00:56:28.512120+0000","last_clean":"2024-09-11T00:56:28.512120+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:56:28.512120+0000","last_undegraded":"2024-09-11T00:56:28.512120+0000","last_fullsized":"2024-09-11T00:56:28.512120+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17558,"num_write_kb":271021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"60'3295","reported_seq":3399,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063104+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:56:28.063104+0000","last_peered":"2024-09-11T00:56:28.063104+0000","last_clean":"2024-09-11T00:56:28.063104+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:56:28.063104+0000","last_undegraded":"2024-09-11T00:56:28.063104+0000","last_fullsized":"2024-09-11T00:56:28.063104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":25,"num_write":17271,"num_write_kb":265887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"61'3033","reported_seq":3130,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048703+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:56:28.048703+0000","last_peered":"2024-09-11T00:56:28.048703+0000","last_clean":"2024-09-11T00:56:28.048703+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:56:28.048703+0000","last_undegraded":"2024-09-11T00:56:28.048703+0000","last_fullsized":"2024-09-11T00:56:28.048703+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16092,"num_write_kb":252889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"60'2629","reported_seq":2733,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.078405+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:56:28.078405+0000","last_peered":"2024-09-11T00:56:28.078405+0000","last_clean":"2024-09-11T00:56:28.078405+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:56:28.078405+0000","last_undegraded":"2024-09-11T00:56:28.078405+0000","last_fullsized":"2024-09-11T00:56:28.078405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":13758,"num_write_kb":227101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"62'3227","reported_seq":3324,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.479006+0000","last_change":"2024-09-11T00:30:54.990631+0000","last_active":"2024-09-11T00:56:28.479006+0000","last_peered":"2024-09-11T00:56:28.479006+0000","last_clean":"2024-09-11T00:56:28.479006+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:56:28.479006+0000","last_undegraded":"2024-09-11T00:56:28.479006+0000","last_fullsized":"2024-09-11T00:56:28.479006+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:02.422471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16396,"num_write_kb":258131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"60'3415","reported_seq":3520,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067480+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:56:28.067480+0000","last_peered":"2024-09-11T00:56:28.067480+0000","last_clean":"2024-09-11T00:56:28.067480+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:56:28.067480+0000","last_undegraded":"2024-09-11T00:56:28.067480+0000","last_fullsized":"2024-09-11T00:56:28.067480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":17180,"num_write_kb":310287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"62'2771","reported_seq":2866,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.447411+0000","last_change":"2024-09-11T00:30:54.991047+0000","last_active":"2024-09-11T00:56:28.447411+0000","last_peered":"2024-09-11T00:56:28.447411+0000","last_clean":"2024-09-11T00:56:28.447411+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:56:28.447411+0000","last_undegraded":"2024-09-11T00:56:28.447411+0000","last_fullsized":"2024-09-11T00:56:28.447411+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:41:48.229261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14317,"num_write_kb":221149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"60'2741","reported_seq":2846,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075991+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:56:28.075991+0000","last_peered":"2024-09-11T00:56:28.075991+0000","last_clean":"2024-09-11T00:56:28.075991+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:56:28.075991+0000","last_undegraded":"2024-09-11T00:56:28.075991+0000","last_fullsized":"2024-09-11T00:56:28.075991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2741,"log_dups_size":0,"ondisk_log_size":2741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14755,"num_write_kb":255665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"62'3704","reported_seq":3799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.547303+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:56:28.547303+0000","last_peered":"2024-09-11T00:56:28.547303+0000","last_clean":"2024-09-11T00:56:28.547303+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:56:28.547303+0000","last_undegraded":"2024-09-11T00:56:28.547303+0000","last_fullsized":"2024-09-11T00:56:28.547303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18483,"num_write_kb":307447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"60'3035","reported_seq":3140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059908+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:56:28.059908+0000","last_peered":"2024-09-11T00:56:28.059908+0000","last_clean":"2024-09-11T00:56:28.059908+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:56:28.059908+0000","last_undegraded":"2024-09-11T00:56:28.059908+0000","last_fullsized":"2024-09-11T00:56:28.059908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3035,"log_dups_size":0,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15470,"num_write_kb":262623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"62'3351","reported_seq":3447,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075584+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:56:28.075584+0000","last_peered":"2024-09-11T00:56:28.075584+0000","last_clean":"2024-09-11T00:56:28.075584+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:56:28.075584+0000","last_undegraded":"2024-09-11T00:56:28.075584+0000","last_fullsized":"2024-09-11T00:56:28.075584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17034,"num_write_kb":277088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"60'4106","reported_seq":4213,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059593+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:56:28.059593+0000","last_peered":"2024-09-11T00:56:28.059593+0000","last_clean":"2024-09-11T00:56:28.059593+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:56:28.059593+0000","last_undegraded":"2024-09-11T00:56:28.059593+0000","last_fullsized":"2024-09-11T00:56:28.059593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22024,"num_write_kb":354504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"62'3391","reported_seq":3486,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.526593+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:56:28.526593+0000","last_peered":"2024-09-11T00:56:28.526593+0000","last_clean":"2024-09-11T00:56:28.526593+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:56:28.526593+0000","last_undegraded":"2024-09-11T00:56:28.526593+0000","last_fullsized":"2024-09-11T00:56:28.526593+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18265,"num_write_kb":274031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"60'2946","reported_seq":3053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069081+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:56:28.069081+0000","last_peered":"2024-09-11T00:56:28.069081+0000","last_clean":"2024-09-11T00:56:28.069081+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:56:28.069081+0000","last_undegraded":"2024-09-11T00:56:28.069081+0000","last_fullsized":"2024-09-11T00:56:28.069081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":17,"num_write":15520,"num_write_kb":262805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1107571,"num_write_kb":17543218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212907,"ondisk_log_size":212907,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18223220,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1558,"num_read_kb":197403,"num_write":4817,"num_write_kb":19773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36511744,"data_stored":36446440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4412,"ondisk_log_size":4412,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":381,"num_write":1025649,"num_write_kb":17211858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196678,"ondisk_log_size":196678,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":17635410,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1062,"num_read_kb":143929,"num_write":4568,"num_write_kb":19046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35332096,"data_stored":35270820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4119,"ondisk_log_size":4119,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509757,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487152,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":486270,"kb_avail":93884688,"statfs":{"total":96636764160,"available":96137920512,"internally_reserved":0,"allocated":884736,"data_stored":496637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":497940938},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509759,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668164,"kb_used_data":10292,"kb_used_omap":1,"kb_used_meta":657854,"kb_avail":93703676,"statfs":{"total":96636764160,"available":95952564224,"internally_reserved":0,"allocated":10539008,"data_stored":10153943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":673642950},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509758,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685684,"kb_used_data":17828,"kb_used_omap":1,"kb_used_meta":667838,"kb_avail":93686156,"statfs":{"total":96636764160,"available":95934623744,"internally_reserved":0,"allocated":18255872,"data_stored":17864749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683866565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509757,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722280,"kb_used_data":13912,"kb_used_omap":1,"kb_used_meta":708350,"kb_avail":93649560,"statfs":{"total":96636764160,"available":95897149440,"internally_reserved":0,"allocated":14245888,"data_stored":13849304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":725350856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509758,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737056,"kb_used_data":14480,"kb_used_omap":1,"kb_used_meta":722558,"kb_avail":93634784,"statfs":{"total":96636764160,"available":95882018816,"internally_reserved":0,"allocated":14827520,"data_stored":14436462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739899845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509758,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598360,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":596990,"kb_avail":93773480,"statfs":{"total":96636764160,"available":96024043520,"internally_reserved":0,"allocated":1384448,"data_stored":998818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":611318213},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509759,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741960,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":740414,"kb_avail":93629880,"statfs":{"total":96636764160,"available":95876997120,"internally_reserved":0,"allocated":1564672,"data_stored":1176999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":758184389},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542462,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739432,"kb_used_data":17176,"kb_used_omap":1,"kb_used_meta":722238,"kb_avail":93632408,"statfs":{"total":96636764160,"available":95879585792,"internally_reserved":0,"allocated":17588224,"data_stored":17187564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739572165},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":778240,"data_stored":768599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9162752,"data_stored":9157733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4282577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5648384,"data_stored":5638872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":4283261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5562368,"data_stored":5551329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:56:47.375 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T00:56:47.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:56:47.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.567+0000 7f4c4de00640 1 -- 172.21.15.31:0/653829416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c480a6e20 msgr2=0x7f4c481510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:47.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.567+0000 7f4c4de00640 1 --2- 172.21.15.31:0/653829416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c480a6e20 0x7f4c481510c0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f4c34009830 tx=0x7f4c3402a3f0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.567+0000 7f4c4de00640 1 -- 172.21.15.31:0/653829416 shutdown_connections 2024-09-11T00:56:47.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.567+0000 7f4c4de00640 1 --2- 172.21.15.31:0/653829416 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c48151600 0x7f4c480c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.567+0000 7f4c4de00640 1 --2- 172.21.15.31:0/653829416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c480a6e20 0x7f4c481510c0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.567+0000 7f4c4de00640 1 -- 172.21.15.31:0/653829416 >> 172.21.15.31:0/653829416 conn(0x7f4c4806caf0 msgr2=0x7f4c4806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:47.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.567+0000 7f4c4de00640 1 -- 172.21.15.31:0/653829416 shutdown_connections 2024-09-11T00:56:47.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.567+0000 7f4c4de00640 1 -- 172.21.15.31:0/653829416 wait complete. 2024-09-11T00:56:47.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.569+0000 7f4c4de00640 1 Processor -- start 2024-09-11T00:56:47.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.569+0000 7f4c4de00640 1 -- start start 2024-09-11T00:56:47.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c4de00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c480a6e20 0x7f4c481e9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:47.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c4de00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c480ac2c0 0x7f4c480c2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:47.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c4de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c48151600 0x7f4c481496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:47.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c4de00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4c480acf30 con 0x7f4c48151600 2024-09-11T00:56:47.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c4de00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c480acdb0 con 0x7f4c480ac2c0 2024-09-11T00:56:47.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c4de00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4c480c66f0 con 0x7f4c480a6e20 2024-09-11T00:56:47.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c4d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c48151600 0x7f4c481496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c4d400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c48151600 0x7f4c481496b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50674/0 (socket says 172.21.15.31:50674) 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c4d400640 1 -- 172.21.15.31:0/55962928 learned_addr learned my addr 172.21.15.31:0/55962928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c4ca00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c480a6e20 0x7f4c481e9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.570+0000 7f4c47e00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c480ac2c0 0x7f4c480c2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c4d400640 1 -- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c480a6e20 msgr2=0x7f4c481e9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c4d400640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c480a6e20 0x7f4c481e9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c4d400640 1 -- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c480ac2c0 msgr2=0x7f4c480c2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c4d400640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c480ac2c0 0x7f4c480c2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c4d400640 1 -- 172.21.15.31:0/55962928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c481e94f0 con 0x7f4c48151600 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c4d400640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c48151600 0x7f4c481496b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000eab0 tx=0x7f4c4000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c44c00640 1 -- 172.21.15.31:0/55962928 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4000cea0 con 0x7f4c48151600 2024-09-11T00:56:47.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c44c00640 1 -- 172.21.15.31:0/55962928 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c40004b20 con 0x7f4c48151600 2024-09-11T00:56:47.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c4de00640 1 -- 172.21.15.31:0/55962928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c481573d0 con 0x7f4c48151600 2024-09-11T00:56:47.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c44c00640 1 -- 172.21.15.31:0/55962928 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c40005020 con 0x7f4c48151600 2024-09-11T00:56:47.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.571+0000 7f4c4de00640 1 -- 172.21.15.31:0/55962928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c481e9770 con 0x7f4c48151600 2024-09-11T00:56:47.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.572+0000 7f4c4de00640 1 -- 172.21.15.31:0/55962928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c480c6b50 con 0x7f4c48151600 2024-09-11T00:56:47.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.573+0000 7f4c44c00640 1 -- 172.21.15.31:0/55962928 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4c400051c0 con 0x7f4c48151600 2024-09-11T00:56:47.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.574+0000 7f4c44c00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c28077f70 0x7f4c2807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:47.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.574+0000 7f4c44c00640 1 -- 172.21.15.31:0/55962928 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f4c4009b920 con 0x7f4c48151600 2024-09-11T00:56:47.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.575+0000 7f4c4ca00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c28077f70 0x7f4c2807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:47.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.575+0000 7f4c4ca00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c28077f70 0x7f4c2807a430 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4c38007fd0 tx=0x7f4c38001430 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.578+0000 7f4c44c00640 1 -- 172.21.15.31:0/55962928 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4c481e94f0 con 0x7f4c48151600 2024-09-11T00:56:47.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.769+0000 7f4c4de00640 1 -- 172.21.15.31:0/55962928 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4c4814ef80 con 0x7f4c28077f70 2024-09-11T00:56:47.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.777+0000 7f4c44c00640 1 -- 172.21.15.31:0/55962928 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493711 (secure 0 0 0) 0x7f4c4814ef80 con 0x7f4c28077f70 2024-09-11T00:56:47.785 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:47.789 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:56:47.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 -- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c28077f70 msgr2=0x7f4c2807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:47.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c28077f70 0x7f4c2807a430 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4c38007fd0 tx=0x7f4c38001430 comp rx=0 tx=0).stop 2024-09-11T00:56:47.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 -- 172.21.15.31:0/55962928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c48151600 msgr2=0x7f4c481496b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:47.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c48151600 0x7f4c481496b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000eab0 tx=0x7f4c4000ef70 comp rx=0 tx=0).stop 2024-09-11T00:56:47.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 -- 172.21.15.31:0/55962928 shutdown_connections 2024-09-11T00:56:47.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4c28077f70 0x7f4c2807a430 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c48151600 0x7f4c481496b0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c480ac2c0 0x7f4c480c2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 --2- 172.21.15.31:0/55962928 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4c480a6e20 0x7f4c481e9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:47.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 -- 172.21.15.31:0/55962928 >> 172.21.15.31:0/55962928 conn(0x7f4c4806caf0 msgr2=0x7f4c481dd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:47.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 -- 172.21.15.31:0/55962928 shutdown_connections 2024-09-11T00:56:47.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:47.783+0000 7f4c4de00640 1 -- 172.21.15.31:0/55962928 wait complete. 2024-09-11T00:56:47.801 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":842,"stamp":"2024-09-11T00:56:46.843610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36448998,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":2779,"num_read_kb":341768,"num_write":2142706,"num_write_kb":34795523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":418188,"ondisk_log_size":418188,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5380088,"kb_used_data":77432,"kb_used_omap":12,"kb_used_meta":5302515,"kb_avail":749594632,"statfs":{"total":773094113280,"available":767584903168,"internally_reserved":0,"allocated":79290368,"data_stored":76164476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":5429775921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20007,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003687"},"pg_stats":[{"pgid":"5.18","version":"62'3612","reported_seq":3708,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.530101+0000","last_change":"2024-09-11T00:30:54.983412+0000","last_active":"2024-09-11T00:56:28.530101+0000","last_peered":"2024-09-11T00:56:28.530101+0000","last_clean":"2024-09-11T00:56:28.530101+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:56:28.530101+0000","last_undegraded":"2024-09-11T00:56:28.530101+0000","last_fullsized":"2024-09-11T00:56:28.530101+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:51.807846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18811,"num_write_kb":284357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"60'788","reported_seq":928,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.095736+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:56:28.095736+0000","last_peered":"2024-09-11T00:56:28.095736+0000","last_clean":"2024-09-11T00:56:28.095736+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:56:28.095736+0000","last_undegraded":"2024-09-11T00:56:28.095736+0000","last_fullsized":"2024-09-11T00:56:28.095736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":8319,"num_write":773,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"60'3033","reported_seq":3138,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059830+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:56:28.059830+0000","last_peered":"2024-09-11T00:56:28.059830+0000","last_clean":"2024-09-11T00:56:28.059830+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:56:28.059830+0000","last_undegraded":"2024-09-11T00:56:28.059830+0000","last_fullsized":"2024-09-11T00:56:28.059830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:52.613246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16165,"num_write_kb":269490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"61'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058244+0000","last_change":"2024-09-11T00:30:53.984316+0000","last_active":"2024-09-11T00:56:28.058244+0000","last_peered":"2024-09-11T00:56:28.058244+0000","last_clean":"2024-09-11T00:56:28.058244+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:56:28.058244+0000","last_undegraded":"2024-09-11T00:56:28.058244+0000","last_fullsized":"2024-09-11T00:56:28.058244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:14.425265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"62'3995","reported_seq":4090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.532259+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:56:28.532259+0000","last_peered":"2024-09-11T00:56:28.532259+0000","last_clean":"2024-09-11T00:56:28.532259+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:56:28.532259+0000","last_undegraded":"2024-09-11T00:56:28.532259+0000","last_fullsized":"2024-09-11T00:56:28.532259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20665,"num_write_kb":324486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"62'22","reported_seq":129,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060203+0000","last_change":"2024-09-11T00:30:44.924601+0000","last_active":"2024-09-11T00:56:28.060203+0000","last_peered":"2024-09-11T00:56:28.060203+0000","last_clean":"2024-09-11T00:56:28.060203+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:56:28.060203+0000","last_undegraded":"2024-09-11T00:56:28.060203+0000","last_fullsized":"2024-09-11T00:56:28.060203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:31.801234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"60'3563","reported_seq":3674,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059819+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:56:28.059819+0000","last_peered":"2024-09-11T00:56:28.059819+0000","last_clean":"2024-09-11T00:56:28.059819+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:56:28.059819+0000","last_undegraded":"2024-09-11T00:56:28.059819+0000","last_fullsized":"2024-09-11T00:56:28.059819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":44,"num_write":18834,"num_write_kb":321224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"62'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075381+0000","last_change":"2024-09-11T00:30:53.984940+0000","last_active":"2024-09-11T00:56:28.075381+0000","last_peered":"2024-09-11T00:56:28.075381+0000","last_clean":"2024-09-11T00:56:28.075381+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:56:28.075381+0000","last_undegraded":"2024-09-11T00:56:28.075381+0000","last_fullsized":"2024-09-11T00:56:28.075381+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:19.647165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"62'3090","reported_seq":3185,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.476819+0000","last_change":"2024-09-11T00:30:54.993570+0000","last_active":"2024-09-11T00:56:28.476819+0000","last_peered":"2024-09-11T00:56:28.476819+0000","last_clean":"2024-09-11T00:56:28.476819+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:56:28.476819+0000","last_undegraded":"2024-09-11T00:56:28.476819+0000","last_fullsized":"2024-09-11T00:56:28.476819+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:45.211259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15625,"num_write_kb":254958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"62'37","reported_seq":171,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.077866+0000","last_change":"2024-09-11T00:30:44.914641+0000","last_active":"2024-09-11T00:56:31.077866+0000","last_peered":"2024-09-11T00:56:31.077866+0000","last_clean":"2024-09-11T00:56:31.077866+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:56:31.077866+0000","last_undegraded":"2024-09-11T00:56:31.077866+0000","last_fullsized":"2024-09-11T00:56:31.077866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:12.262026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"60'2040","reported_seq":2144,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057996+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:56:28.057996+0000","last_peered":"2024-09-11T00:56:28.057996+0000","last_clean":"2024-09-11T00:56:28.057996+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:56:28.057996+0000","last_undegraded":"2024-09-11T00:56:28.057996+0000","last_fullsized":"2024-09-11T00:56:28.057996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":10218,"num_write_kb":171981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"61'716","reported_seq":837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075103+0000","last_change":"2024-09-11T00:30:53.974699+0000","last_active":"2024-09-11T00:56:28.075103+0000","last_peered":"2024-09-11T00:56:28.075103+0000","last_clean":"2024-09-11T00:56:28.075103+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:56:28.075103+0000","last_undegraded":"2024-09-11T00:56:28.075103+0000","last_fullsized":"2024-09-11T00:56:28.075103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:28.792487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":73385,"num_write":707,"num_write_kb":4317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"62'2671","reported_seq":2766,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.074790+0000","last_change":"2024-09-11T00:30:54.984890+0000","last_active":"2024-09-11T00:56:28.074790+0000","last_peered":"2024-09-11T00:56:28.074790+0000","last_clean":"2024-09-11T00:56:28.074790+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:56:28.074790+0000","last_undegraded":"2024-09-11T00:56:28.074790+0000","last_fullsized":"2024-09-11T00:56:28.074790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:21.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":213871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"60'3","reported_seq":136,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.069924+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:56:31.069924+0000","last_peered":"2024-09-11T00:56:31.069924+0000","last_clean":"2024-09-11T00:56:31.069924+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:56:31.069924+0000","last_undegraded":"2024-09-11T00:56:31.069924+0000","last_fullsized":"2024-09-11T00:56:31.069924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"60'3687","reported_seq":3790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068958+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:56:28.068958+0000","last_peered":"2024-09-11T00:56:28.068958+0000","last_clean":"2024-09-11T00:56:28.068958+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:56:28.068958+0000","last_undegraded":"2024-09-11T00:56:28.068958+0000","last_fullsized":"2024-09-11T00:56:28.068958+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19525,"num_write_kb":315726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"62'140","reported_seq":237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.957825+0000","last_change":"2024-09-11T00:30:53.977743+0000","last_active":"2024-09-11T00:56:34.957825+0000","last_peered":"2024-09-11T00:56:34.957825+0000","last_clean":"2024-09-11T00:56:34.957825+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:56:34.957825+0000","last_undegraded":"2024-09-11T00:56:34.957825+0000","last_fullsized":"2024-09-11T00:56:34.957825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:20.618615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1356353,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":1393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"62'3457","reported_seq":3553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.513475+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:56:28.513475+0000","last_peered":"2024-09-11T00:56:28.513475+0000","last_clean":"2024-09-11T00:56:28.513475+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:56:28.513475+0000","last_undegraded":"2024-09-11T00:56:28.513475+0000","last_fullsized":"2024-09-11T00:56:28.513475+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3457,"log_dups_size":0,"ondisk_log_size":3457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18121,"num_write_kb":297725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"60'2670","reported_seq":2774,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065793+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:56:28.065793+0000","last_peered":"2024-09-11T00:56:28.065793+0000","last_clean":"2024-09-11T00:56:28.065793+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:56:28.065793+0000","last_undegraded":"2024-09-11T00:56:28.065793+0000","last_fullsized":"2024-09-11T00:56:28.065793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2670,"log_dups_size":0,"ondisk_log_size":2670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14000,"num_write_kb":256678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"62'1368","reported_seq":1505,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.097251+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:56:28.097251+0000","last_peered":"2024-09-11T00:56:28.097251+0000","last_clean":"2024-09-11T00:56:28.097251+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:56:28.097251+0000","last_undegraded":"2024-09-11T00:56:28.097251+0000","last_fullsized":"2024-09-11T00:56:28.097251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1368,"log_dups_size":0,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":26260,"num_write":1345,"num_write_kb":4470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"61'2","reported_seq":121,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068774+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:56:28.068774+0000","last_peered":"2024-09-11T00:56:28.068774+0000","last_clean":"2024-09-11T00:56:28.068774+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:56:28.068774+0000","last_undegraded":"2024-09-11T00:56:28.068774+0000","last_fullsized":"2024-09-11T00:56:28.068774+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"62'3537","reported_seq":3633,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.531936+0000","last_change":"2024-09-11T00:30:54.997397+0000","last_active":"2024-09-11T00:56:28.531936+0000","last_peered":"2024-09-11T00:56:28.531936+0000","last_clean":"2024-09-11T00:56:28.531936+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:56:28.531936+0000","last_undegraded":"2024-09-11T00:56:28.531936+0000","last_fullsized":"2024-09-11T00:56:28.531936+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3537,"log_dups_size":0,"ondisk_log_size":3537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:53.322576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18517,"num_write_kb":279095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"60'2630","reported_seq":2735,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058909+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:56:28.058909+0000","last_peered":"2024-09-11T00:56:28.058909+0000","last_clean":"2024-09-11T00:56:28.058909+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:56:28.058909+0000","last_undegraded":"2024-09-11T00:56:28.058909+0000","last_fullsized":"2024-09-11T00:56:28.058909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":8,"num_write":13276,"num_write_kb":225109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"60'4","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051666+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:56:28.051666+0000","last_peered":"2024-09-11T00:56:28.051666+0000","last_clean":"2024-09-11T00:56:28.051666+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:56:28.051666+0000","last_undegraded":"2024-09-11T00:56:28.051666+0000","last_fullsized":"2024-09-11T00:56:28.051666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"61'2","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066019+0000","last_change":"2024-09-11T00:30:53.986858+0000","last_active":"2024-09-11T00:56:28.066019+0000","last_peered":"2024-09-11T00:56:28.066019+0000","last_clean":"2024-09-11T00:56:28.066019+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:56:28.066019+0000","last_undegraded":"2024-09-11T00:56:28.066019+0000","last_fullsized":"2024-09-11T00:56:28.066019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:33.626193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"62'3470","reported_seq":3566,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397582+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:56:28.397582+0000","last_peered":"2024-09-11T00:56:28.397582+0000","last_clean":"2024-09-11T00:56:28.397582+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:56:28.397582+0000","last_undegraded":"2024-09-11T00:56:28.397582+0000","last_fullsized":"2024-09-11T00:56:28.397582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17553,"num_write_kb":279333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"60'2695","reported_seq":2801,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.046481+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:56:28.046481+0000","last_peered":"2024-09-11T00:56:28.046481+0000","last_clean":"2024-09-11T00:56:28.046481+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:56:28.046481+0000","last_undegraded":"2024-09-11T00:56:28.046481+0000","last_fullsized":"2024-09-11T00:56:28.046481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":14,"num_write":14032,"num_write_kb":244535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"60'3","reported_seq":109,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048672+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:56:28.048672+0000","last_peered":"2024-09-11T00:56:28.048672+0000","last_clean":"2024-09-11T00:56:28.048672+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:56:28.048672+0000","last_undegraded":"2024-09-11T00:56:28.048672+0000","last_fullsized":"2024-09-11T00:56:28.048672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"62'5","reported_seq":124,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.505942+0000","last_change":"2024-09-11T00:30:53.990765+0000","last_active":"2024-09-11T00:56:28.505942+0000","last_peered":"2024-09-11T00:56:28.505942+0000","last_clean":"2024-09-11T00:56:28.505942+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:56:28.505942+0000","last_undegraded":"2024-09-11T00:56:28.505942+0000","last_fullsized":"2024-09-11T00:56:28.505942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:33.720727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"62'3622","reported_seq":3718,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.450009+0000","last_change":"2024-09-11T00:30:54.984949+0000","last_active":"2024-09-11T00:56:28.450009+0000","last_peered":"2024-09-11T00:56:28.450009+0000","last_clean":"2024-09-11T00:56:28.450009+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:56:28.450009+0000","last_undegraded":"2024-09-11T00:56:28.450009+0000","last_fullsized":"2024-09-11T00:56:28.450009+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:54.346937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18608,"num_write_kb":291609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"60'2732","reported_seq":2837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063199+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:56:28.063199+0000","last_peered":"2024-09-11T00:56:28.063199+0000","last_clean":"2024-09-11T00:56:28.063199+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:56:28.063199+0000","last_undegraded":"2024-09-11T00:56:28.063199+0000","last_fullsized":"2024-09-11T00:56:28.063199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13936,"num_write_kb":248214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"60'3","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063396+0000","last_change":"2024-09-11T00:30:44.921212+0000","last_active":"2024-09-11T00:56:28.063396+0000","last_peered":"2024-09-11T00:56:28.063396+0000","last_clean":"2024-09-11T00:56:28.063396+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:56:28.063396+0000","last_undegraded":"2024-09-11T00:56:28.063396+0000","last_fullsized":"2024-09-11T00:56:28.063396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:01.666636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"62'6","reported_seq":146,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500913+0000","last_change":"2024-09-11T00:30:53.986940+0000","last_active":"2024-09-11T00:56:28.500913+0000","last_peered":"2024-09-11T00:56:28.500913+0000","last_clean":"2024-09-11T00:56:28.500913+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:56:28.500913+0000","last_undegraded":"2024-09-11T00:56:28.500913+0000","last_fullsized":"2024-09-11T00:56:28.500913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:11.942187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"62'3257","reported_seq":3355,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.520613+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:56:28.520613+0000","last_peered":"2024-09-11T00:56:28.520613+0000","last_clean":"2024-09-11T00:56:28.520613+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:56:28.520613+0000","last_undegraded":"2024-09-11T00:56:28.520613+0000","last_fullsized":"2024-09-11T00:56:28.520613+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17168,"num_write_kb":277608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'2688","reported_seq":2794,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048103+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:56:28.048103+0000","last_peered":"2024-09-11T00:56:28.048103+0000","last_clean":"2024-09-11T00:56:28.048103+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:56:28.048103+0000","last_undegraded":"2024-09-11T00:56:28.048103+0000","last_fullsized":"2024-09-11T00:56:28.048103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13441,"num_write_kb":246089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"62'5","reported_seq":141,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.085125+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:56:31.085125+0000","last_peered":"2024-09-11T00:56:31.085125+0000","last_clean":"2024-09-11T00:56:31.085125+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:56:31.085125+0000","last_undegraded":"2024-09-11T00:56:31.085125+0000","last_fullsized":"2024-09-11T00:56:31.085125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"62'65","reported_seq":245,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:37.625005+0000","last_change":"2024-09-11T00:30:53.988539+0000","last_active":"2024-09-11T00:56:37.625005+0000","last_peered":"2024-09-11T00:56:37.625005+0000","last_clean":"2024-09-11T00:56:37.625005+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:56:37.625005+0000","last_undegraded":"2024-09-11T00:56:37.625005+0000","last_fullsized":"2024-09-11T00:56:37.625005+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:19.083664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":126,"num_write":62,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"62'3386","reported_seq":3484,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.422439+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:56:28.422439+0000","last_peered":"2024-09-11T00:56:28.422439+0000","last_clean":"2024-09-11T00:56:28.422439+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:56:28.422439+0000","last_undegraded":"2024-09-11T00:56:28.422439+0000","last_fullsized":"2024-09-11T00:56:28.422439+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18005,"num_write_kb":276900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'3194","reported_seq":3298,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.050561+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:56:28.050561+0000","last_peered":"2024-09-11T00:56:28.050561+0000","last_clean":"2024-09-11T00:56:28.050561+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:56:28.050561+0000","last_undegraded":"2024-09-11T00:56:28.050561+0000","last_fullsized":"2024-09-11T00:56:28.050561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16397,"num_write_kb":262231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"62'521","reported_seq":641,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:32.047226+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:56:32.047226+0000","last_peered":"2024-09-11T00:56:32.047226+0000","last_clean":"2024-09-11T00:56:32.047226+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:56:32.047226+0000","last_undegraded":"2024-09-11T00:56:32.047226+0000","last_fullsized":"2024-09-11T00:56:32.047226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":41,"num_write":1157,"num_write_kb":372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"62'9","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.502805+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:56:28.502805+0000","last_peered":"2024-09-11T00:56:28.502805+0000","last_clean":"2024-09-11T00:56:28.502805+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:56:28.502805+0000","last_undegraded":"2024-09-11T00:56:28.502805+0000","last_fullsized":"2024-09-11T00:56:28.502805+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"61'3259","reported_seq":3357,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063309+0000","last_change":"2024-09-11T00:30:54.993118+0000","last_active":"2024-09-11T00:56:28.063309+0000","last_peered":"2024-09-11T00:56:28.063309+0000","last_clean":"2024-09-11T00:56:28.063309+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:56:28.063309+0000","last_undegraded":"2024-09-11T00:56:28.063309+0000","last_fullsized":"2024-09-11T00:56:28.063309+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:07:41.148370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16779,"num_write_kb":276080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"60'3127","reported_seq":3233,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052954+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:56:28.052954+0000","last_peered":"2024-09-11T00:56:28.052954+0000","last_clean":"2024-09-11T00:56:28.052954+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:56:28.052954+0000","last_undegraded":"2024-09-11T00:56:28.052954+0000","last_fullsized":"2024-09-11T00:56:28.052954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":12,"num_write":16580,"num_write_kb":258802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"60'4","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069515+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:56:28.069515+0000","last_peered":"2024-09-11T00:56:28.069515+0000","last_clean":"2024-09-11T00:56:28.069515+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:56:28.069515+0000","last_undegraded":"2024-09-11T00:56:28.069515+0000","last_fullsized":"2024-09-11T00:56:28.069515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'1","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052292+0000","last_change":"2024-09-11T00:30:53.986843+0000","last_active":"2024-09-11T00:56:28.052292+0000","last_peered":"2024-09-11T00:56:28.052292+0000","last_clean":"2024-09-11T00:56:28.052292+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:56:28.052292+0000","last_undegraded":"2024-09-11T00:56:28.052292+0000","last_fullsized":"2024-09-11T00:56:28.052292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:44.684154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"62'3801","reported_seq":3898,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.047541+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:56:28.047541+0000","last_peered":"2024-09-11T00:56:28.047541+0000","last_clean":"2024-09-11T00:56:28.047541+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:56:28.047541+0000","last_undegraded":"2024-09-11T00:56:28.047541+0000","last_fullsized":"2024-09-11T00:56:28.047541+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20214,"num_write_kb":289874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"60'2498","reported_seq":2605,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075698+0000","last_change":"2024-09-11T00:30:45.924575+0000","last_active":"2024-09-11T00:56:28.075698+0000","last_peered":"2024-09-11T00:56:28.075698+0000","last_clean":"2024-09-11T00:56:28.075698+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:56:28.075698+0000","last_undegraded":"2024-09-11T00:56:28.075698+0000","last_fullsized":"2024-09-11T00:56:28.075698+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2498,"log_dups_size":0,"ondisk_log_size":2498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:06.520951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":13564,"num_write_kb":230524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"60'5","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059256+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:56:28.059256+0000","last_peered":"2024-09-11T00:56:28.059256+0000","last_clean":"2024-09-11T00:56:28.059256+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:56:28.059256+0000","last_undegraded":"2024-09-11T00:56:28.059256+0000","last_fullsized":"2024-09-11T00:56:28.059256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"62'5","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510149+0000","last_change":"2024-09-11T00:30:53.986896+0000","last_active":"2024-09-11T00:56:28.510149+0000","last_peered":"2024-09-11T00:56:28.510149+0000","last_clean":"2024-09-11T00:56:28.510149+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:56:28.510149+0000","last_undegraded":"2024-09-11T00:56:28.510149+0000","last_fullsized":"2024-09-11T00:56:28.510149+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:06.590278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"62'3103","reported_seq":3201,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058933+0000","last_change":"2024-09-11T00:30:55.001370+0000","last_active":"2024-09-11T00:56:28.058933+0000","last_peered":"2024-09-11T00:56:28.058933+0000","last_clean":"2024-09-11T00:56:28.058933+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:56:28.058933+0000","last_undegraded":"2024-09-11T00:56:28.058933+0000","last_fullsized":"2024-09-11T00:56:28.058933+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:56.592767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16058,"num_write_kb":244544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"60'2543","reported_seq":2648,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067860+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:56:28.067860+0000","last_peered":"2024-09-11T00:56:28.067860+0000","last_clean":"2024-09-11T00:56:28.067860+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:56:28.067860+0000","last_undegraded":"2024-09-11T00:56:28.067860+0000","last_fullsized":"2024-09-11T00:56:28.067860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":230458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"60'5","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067593+0000","last_change":"2024-09-11T00:30:44.923285+0000","last_active":"2024-09-11T00:56:28.067593+0000","last_peered":"2024-09-11T00:56:28.067593+0000","last_clean":"2024-09-11T00:56:28.067593+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:56:28.067593+0000","last_undegraded":"2024-09-11T00:56:28.067593+0000","last_fullsized":"2024-09-11T00:56:28.067593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:34:08.067084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"62'5","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.494925+0000","last_change":"2024-09-11T00:30:53.978480+0000","last_active":"2024-09-11T00:56:28.494925+0000","last_peered":"2024-09-11T00:56:28.494925+0000","last_clean":"2024-09-11T00:56:28.494925+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:56:28.494925+0000","last_undegraded":"2024-09-11T00:56:28.494925+0000","last_fullsized":"2024-09-11T00:56:28.494925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:58.900565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"61'3199","reported_seq":3296,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058202+0000","last_change":"2024-09-11T00:30:54.997641+0000","last_active":"2024-09-11T00:56:28.058202+0000","last_peered":"2024-09-11T00:56:28.058202+0000","last_clean":"2024-09-11T00:56:28.058202+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:56:28.058202+0000","last_undegraded":"2024-09-11T00:56:28.058202+0000","last_fullsized":"2024-09-11T00:56:28.058202+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:18.437490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15924,"num_write_kb":260293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"60'3298","reported_seq":3404,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057304+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:56:28.057304+0000","last_peered":"2024-09-11T00:56:28.057304+0000","last_clean":"2024-09-11T00:56:28.057304+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:56:28.057304+0000","last_undegraded":"2024-09-11T00:56:28.057304+0000","last_fullsized":"2024-09-11T00:56:28.057304+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:00.363210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17655,"num_write_kb":267324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"62'73","reported_seq":262,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:33.050070+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:56:33.050070+0000","last_peered":"2024-09-11T00:56:33.050070+0000","last_clean":"2024-09-11T00:56:33.050070+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:56:33.050070+0000","last_undegraded":"2024-09-11T00:56:33.050070+0000","last_fullsized":"2024-09-11T00:56:33.050070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":129,"num_write":68,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"62'5","reported_seq":146,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500023+0000","last_change":"2024-09-11T00:30:53.979031+0000","last_active":"2024-09-11T00:56:28.500023+0000","last_peered":"2024-09-11T00:56:28.500023+0000","last_clean":"2024-09-11T00:56:28.500023+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:56:28.500023+0000","last_undegraded":"2024-09-11T00:56:28.500023+0000","last_fullsized":"2024-09-11T00:56:28.500023+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:36.604377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"62'2728","reported_seq":2824,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.475254+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:56:28.475254+0000","last_peered":"2024-09-11T00:56:28.475254+0000","last_clean":"2024-09-11T00:56:28.475254+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:56:28.475254+0000","last_undegraded":"2024-09-11T00:56:28.475254+0000","last_fullsized":"2024-09-11T00:56:28.475254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2728,"log_dups_size":0,"ondisk_log_size":2728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:39.629122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":237243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"60'2962","reported_seq":3068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051045+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:56:28.051045+0000","last_peered":"2024-09-11T00:56:28.051045+0000","last_clean":"2024-09-11T00:56:28.051045+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:56:28.051045+0000","last_undegraded":"2024-09-11T00:56:28.051045+0000","last_fullsized":"2024-09-11T00:56:28.051045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15238,"num_write_kb":265720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"62'8","reported_seq":116,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.640904+0000","last_change":"2024-09-11T00:30:44.928517+0000","last_active":"2024-09-11T00:56:28.640904+0000","last_peered":"2024-09-11T00:56:28.640904+0000","last_clean":"2024-09-11T00:56:28.640904+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:56:28.640904+0000","last_undegraded":"2024-09-11T00:56:28.640904+0000","last_fullsized":"2024-09-11T00:56:28.640904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:20.258143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"62'499","reported_seq":618,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:35.590292+0000","last_change":"2024-09-11T00:30:53.989299+0000","last_active":"2024-09-11T00:56:35.590292+0000","last_peered":"2024-09-11T00:56:35.590292+0000","last_clean":"2024-09-11T00:56:35.590292+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:56:35.590292+0000","last_undegraded":"2024-09-11T00:56:35.590292+0000","last_fullsized":"2024-09-11T00:56:35.590292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:34:49.298915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":1123,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"62'3325","reported_seq":3422,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.494312+0000","last_change":"2024-09-11T00:30:54.990026+0000","last_active":"2024-09-11T00:56:28.494312+0000","last_peered":"2024-09-11T00:56:28.494312+0000","last_clean":"2024-09-11T00:56:28.494312+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:56:28.494312+0000","last_undegraded":"2024-09-11T00:56:28.494312+0000","last_fullsized":"2024-09-11T00:56:28.494312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:02:10.800530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17451,"num_write_kb":278810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"60'3332","reported_seq":3438,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051728+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:56:28.051728+0000","last_peered":"2024-09-11T00:56:28.051728+0000","last_clean":"2024-09-11T00:56:28.051728+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:56:28.051728+0000","last_undegraded":"2024-09-11T00:56:28.051728+0000","last_fullsized":"2024-09-11T00:56:28.051728+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3332,"log_dups_size":0,"ondisk_log_size":3332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17164,"num_write_kb":285768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"31'1","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.072247+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:56:28.072247+0000","last_peered":"2024-09-11T00:56:28.072247+0000","last_clean":"2024-09-11T00:56:28.072247+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:56:28.072247+0000","last_undegraded":"2024-09-11T00:56:28.072247+0000","last_fullsized":"2024-09-11T00:56:28.072247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":204,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066124+0000","last_change":"2024-09-11T00:30:36.028944+0000","last_active":"2024-09-11T00:56:28.066124+0000","last_peered":"2024-09-11T00:56:28.066124+0000","last_clean":"2024-09-11T00:56:28.066124+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:56:28.066124+0000","last_undegraded":"2024-09-11T00:56:28.066124+0000","last_fullsized":"2024-09-11T00:56:28.066124+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:34.776456+0000","last_clean_scrub_stamp":"2024-09-11T00:30:34.776456+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:51.684378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"62'4","reported_seq":103,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488198+0000","last_change":"2024-09-11T00:30:53.982142+0000","last_active":"2024-09-11T00:56:28.488198+0000","last_peered":"2024-09-11T00:56:28.488198+0000","last_clean":"2024-09-11T00:56:28.488198+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:56:28.488198+0000","last_undegraded":"2024-09-11T00:56:28.488198+0000","last_fullsized":"2024-09-11T00:56:28.488198+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:41.558944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"62'2541","reported_seq":2637,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500843+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:56:28.500843+0000","last_peered":"2024-09-11T00:56:28.500843+0000","last_clean":"2024-09-11T00:56:28.500843+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:56:28.500843+0000","last_undegraded":"2024-09-11T00:56:28.500843+0000","last_fullsized":"2024-09-11T00:56:28.500843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12611,"num_write_kb":206053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"60'2809","reported_seq":2917,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069156+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:56:28.069156+0000","last_peered":"2024-09-11T00:56:28.069156+0000","last_clean":"2024-09-11T00:56:28.069156+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:56:28.069156+0000","last_undegraded":"2024-09-11T00:56:28.069156+0000","last_fullsized":"2024-09-11T00:56:28.069156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2809,"log_dups_size":0,"ondisk_log_size":2809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":34,"num_write":15096,"num_write_kb":232439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"60'2","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.053330+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:56:28.053330+0000","last_peered":"2024-09-11T00:56:28.053330+0000","last_clean":"2024-09-11T00:56:28.053330+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:56:28.053330+0000","last_undegraded":"2024-09-11T00:56:28.053330+0000","last_fullsized":"2024-09-11T00:56:28.053330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"62'4","reported_seq":102,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.498262+0000","last_change":"2024-09-11T00:30:53.989292+0000","last_active":"2024-09-11T00:56:28.498262+0000","last_peered":"2024-09-11T00:56:28.498262+0000","last_clean":"2024-09-11T00:56:28.498262+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:56:28.498262+0000","last_undegraded":"2024-09-11T00:56:28.498262+0000","last_fullsized":"2024-09-11T00:56:28.498262+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:44.071338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"62'3293","reported_seq":3389,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.437769+0000","last_change":"2024-09-11T00:30:54.981630+0000","last_active":"2024-09-11T00:56:28.437769+0000","last_peered":"2024-09-11T00:56:28.437769+0000","last_clean":"2024-09-11T00:56:28.437769+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:56:28.437769+0000","last_undegraded":"2024-09-11T00:56:28.437769+0000","last_fullsized":"2024-09-11T00:56:28.437769+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:12.832327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17661,"num_write_kb":267414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"60'3034","reported_seq":3140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066511+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:56:28.066511+0000","last_peered":"2024-09-11T00:56:28.066511+0000","last_clean":"2024-09-11T00:56:28.066511+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:56:28.066511+0000","last_undegraded":"2024-09-11T00:56:28.066511+0000","last_fullsized":"2024-09-11T00:56:28.066511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3034,"log_dups_size":0,"ondisk_log_size":3034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":15513,"num_write_kb":283129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"60'2","reported_seq":122,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060350+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:56:28.060350+0000","last_peered":"2024-09-11T00:56:28.060350+0000","last_clean":"2024-09-11T00:56:28.060350+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:56:28.060350+0000","last_undegraded":"2024-09-11T00:56:28.060350+0000","last_fullsized":"2024-09-11T00:56:28.060350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"62'1443","reported_seq":1586,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510169+0000","last_change":"2024-09-11T00:30:53.984013+0000","last_active":"2024-09-11T00:56:28.510169+0000","last_peered":"2024-09-11T00:56:28.510169+0000","last_clean":"2024-09-11T00:56:28.510169+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:56:28.510169+0000","last_undegraded":"2024-09-11T00:56:28.510169+0000","last_fullsized":"2024-09-11T00:56:28.510169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1443,"log_dups_size":0,"ondisk_log_size":1443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:49.963623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":36861,"num_write":1416,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'2904","reported_seq":3001,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.471315+0000","last_change":"2024-09-11T00:30:54.996860+0000","last_active":"2024-09-11T00:56:28.471315+0000","last_peered":"2024-09-11T00:56:28.471315+0000","last_clean":"2024-09-11T00:56:28.471315+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:56:28.471315+0000","last_undegraded":"2024-09-11T00:56:28.471315+0000","last_fullsized":"2024-09-11T00:56:28.471315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:43.516748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15203,"num_write_kb":241147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"60'3390","reported_seq":3494,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069427+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:56:28.069427+0000","last_peered":"2024-09-11T00:56:28.069427+0000","last_clean":"2024-09-11T00:56:28.069427+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:56:28.069427+0000","last_undegraded":"2024-09-11T00:56:28.069427+0000","last_fullsized":"2024-09-11T00:56:28.069427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17423,"num_write_kb":299173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"60'6","reported_seq":140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.070765+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:56:31.070765+0000","last_peered":"2024-09-11T00:56:31.070765+0000","last_clean":"2024-09-11T00:56:31.070765+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:56:31.070765+0000","last_undegraded":"2024-09-11T00:56:31.070765+0000","last_fullsized":"2024-09-11T00:56:31.070765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"62'3","reported_seq":101,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062449+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:56:28.062449+0000","last_peered":"2024-09-11T00:56:28.062449+0000","last_clean":"2024-09-11T00:56:28.062449+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:56:28.062449+0000","last_undegraded":"2024-09-11T00:56:28.062449+0000","last_fullsized":"2024-09-11T00:56:28.062449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:19.891052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"62'3177","reported_seq":3279,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.444533+0000","last_change":"2024-09-11T00:31:26.120086+0000","last_active":"2024-09-11T00:56:28.444533+0000","last_peered":"2024-09-11T00:56:28.444533+0000","last_clean":"2024-09-11T00:56:28.444533+0000","last_became_active":"2024-09-11T00:31:26.119641+0000","last_became_peered":"2024-09-11T00:31:26.119641+0000","last_unstale":"2024-09-11T00:56:28.444533+0000","last_undegraded":"2024-09-11T00:56:28.444533+0000","last_fullsized":"2024-09-11T00:56:28.444533+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3177,"log_dups_size":0,"ondisk_log_size":3177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:21:15.814831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16730,"num_write_kb":276023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"60'2906","reported_seq":3010,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.077855+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:56:28.077855+0000","last_peered":"2024-09-11T00:56:28.077855+0000","last_clean":"2024-09-11T00:56:28.077855+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:56:28.077855+0000","last_undegraded":"2024-09-11T00:56:28.077855+0000","last_fullsized":"2024-09-11T00:56:28.077855+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14712,"num_write_kb":255517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"60'5","reported_seq":125,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069857+0000","last_change":"2024-09-11T00:30:44.928093+0000","last_active":"2024-09-11T00:56:28.069857+0000","last_peered":"2024-09-11T00:56:28.069857+0000","last_clean":"2024-09-11T00:56:28.069857+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:56:28.069857+0000","last_undegraded":"2024-09-11T00:56:28.069857+0000","last_fullsized":"2024-09-11T00:56:28.069857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:11:04.940285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"62'3","reported_seq":101,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.487423+0000","last_change":"2024-09-11T00:30:53.983352+0000","last_active":"2024-09-11T00:56:28.487423+0000","last_peered":"2024-09-11T00:56:28.487423+0000","last_clean":"2024-09-11T00:56:28.487423+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:56:28.487423+0000","last_undegraded":"2024-09-11T00:56:28.487423+0000","last_fullsized":"2024-09-11T00:56:28.487423+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:43.826114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"62'2973","reported_seq":3063,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.532651+0000","last_change":"2024-09-11T00:31:26.119816+0000","last_active":"2024-09-11T00:56:28.532651+0000","last_peered":"2024-09-11T00:56:28.532651+0000","last_clean":"2024-09-11T00:56:28.532651+0000","last_became_active":"2024-09-11T00:31:26.119440+0000","last_became_peered":"2024-09-11T00:31:26.119440+0000","last_unstale":"2024-09-11T00:56:28.532651+0000","last_undegraded":"2024-09-11T00:56:28.532651+0000","last_fullsized":"2024-09-11T00:56:28.532651+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:21:45.707321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15629,"num_write_kb":242839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"60'3355","reported_seq":3459,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062827+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:56:28.062827+0000","last_peered":"2024-09-11T00:56:28.062827+0000","last_clean":"2024-09-11T00:56:28.062827+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:56:28.062827+0000","last_undegraded":"2024-09-11T00:56:28.062827+0000","last_fullsized":"2024-09-11T00:56:28.062827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18048,"num_write_kb":289456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"62'12","reported_seq":177,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.077796+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:56:31.077796+0000","last_peered":"2024-09-11T00:56:31.077796+0000","last_clean":"2024-09-11T00:56:31.077796+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:56:31.077796+0000","last_undegraded":"2024-09-11T00:56:31.077796+0000","last_fullsized":"2024-09-11T00:56:31.077796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":177,"num_read_kb":177,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"62'2","reported_seq":99,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488958+0000","last_change":"2024-09-11T00:30:53.980703+0000","last_active":"2024-09-11T00:56:28.488958+0000","last_peered":"2024-09-11T00:56:28.488958+0000","last_clean":"2024-09-11T00:56:28.488958+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:56:28.488958+0000","last_undegraded":"2024-09-11T00:56:28.488958+0000","last_fullsized":"2024-09-11T00:56:28.488958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:51.050512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"62'2678","reported_seq":2774,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.413635+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:56:28.413635+0000","last_peered":"2024-09-11T00:56:28.413635+0000","last_clean":"2024-09-11T00:56:28.413635+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:56:28.413635+0000","last_undegraded":"2024-09-11T00:56:28.413635+0000","last_fullsized":"2024-09-11T00:56:28.413635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2678,"log_dups_size":0,"ondisk_log_size":2678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13518,"num_write_kb":242590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"60'2791","reported_seq":2895,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.056927+0000","last_change":"2024-09-11T00:30:45.935837+0000","last_active":"2024-09-11T00:56:28.056927+0000","last_peered":"2024-09-11T00:56:28.056927+0000","last_clean":"2024-09-11T00:56:28.056927+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:56:28.056927+0000","last_undegraded":"2024-09-11T00:56:28.056927+0000","last_fullsized":"2024-09-11T00:56:28.056927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2791,"log_dups_size":0,"ondisk_log_size":2791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:06:05.527977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":254542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"60'6","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063217+0000","last_change":"2024-09-11T00:30:44.924378+0000","last_active":"2024-09-11T00:56:28.063217+0000","last_peered":"2024-09-11T00:56:28.063217+0000","last_clean":"2024-09-11T00:56:28.063217+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:56:28.063217+0000","last_undegraded":"2024-09-11T00:56:28.063217+0000","last_fullsized":"2024-09-11T00:56:28.063217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:48.903920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"62'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059468+0000","last_change":"2024-09-11T00:30:53.987829+0000","last_active":"2024-09-11T00:56:28.059468+0000","last_peered":"2024-09-11T00:56:28.059468+0000","last_clean":"2024-09-11T00:56:28.059468+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:56:28.059468+0000","last_undegraded":"2024-09-11T00:56:28.059468+0000","last_fullsized":"2024-09-11T00:56:28.059468+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:57:43.792089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"62'3272","reported_seq":3367,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066748+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:56:28.066748+0000","last_peered":"2024-09-11T00:56:28.066748+0000","last_clean":"2024-09-11T00:56:28.066748+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:56:28.066748+0000","last_undegraded":"2024-09-11T00:56:28.066748+0000","last_fullsized":"2024-09-11T00:56:28.066748+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16692,"num_write_kb":275642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"60'3405","reported_seq":3511,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066672+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:56:28.066672+0000","last_peered":"2024-09-11T00:56:28.066672+0000","last_clean":"2024-09-11T00:56:28.066672+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:56:28.066672+0000","last_undegraded":"2024-09-11T00:56:28.066672+0000","last_fullsized":"2024-09-11T00:56:28.066672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3405,"log_dups_size":0,"ondisk_log_size":3405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":17487,"num_write_kb":315618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"60'5","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067380+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:56:28.067380+0000","last_peered":"2024-09-11T00:56:28.067380+0000","last_clean":"2024-09-11T00:56:28.067380+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:56:28.067380+0000","last_undegraded":"2024-09-11T00:56:28.067380+0000","last_fullsized":"2024-09-11T00:56:28.067380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"62'367","reported_seq":572,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.957442+0000","last_change":"2024-09-11T00:30:53.984057+0000","last_active":"2024-09-11T00:56:34.957442+0000","last_peered":"2024-09-11T00:56:34.957442+0000","last_clean":"2024-09-11T00:56:34.957442+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:56:34.957442+0000","last_undegraded":"2024-09-11T00:56:34.957442+0000","last_fullsized":"2024-09-11T00:56:34.957442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:10.477916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":191,"num_read_kb":78004,"num_write":360,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"62'4540","reported_seq":4637,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.544292+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:56:28.544292+0000","last_peered":"2024-09-11T00:56:28.544292+0000","last_clean":"2024-09-11T00:56:28.544292+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:56:28.544292+0000","last_undegraded":"2024-09-11T00:56:28.544292+0000","last_fullsized":"2024-09-11T00:56:28.544292+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23424,"num_write_kb":360436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"60'2652","reported_seq":2757,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051484+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:56:28.051484+0000","last_peered":"2024-09-11T00:56:28.051484+0000","last_clean":"2024-09-11T00:56:28.051484+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:56:28.051484+0000","last_undegraded":"2024-09-11T00:56:28.051484+0000","last_fullsized":"2024-09-11T00:56:28.051484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":13591,"num_write_kb":238653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"62'9","reported_seq":143,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.641347+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:56:28.641347+0000","last_peered":"2024-09-11T00:56:28.641347+0000","last_clean":"2024-09-11T00:56:28.641347+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:56:28.641347+0000","last_undegraded":"2024-09-11T00:56:28.641347+0000","last_fullsized":"2024-09-11T00:56:28.641347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":112,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"62'37","reported_seq":197,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.985643+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:56:28.985643+0000","last_peered":"2024-09-11T00:56:28.985643+0000","last_clean":"2024-09-11T00:56:28.985643+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:56:28.985643+0000","last_undegraded":"2024-09-11T00:56:28.985643+0000","last_fullsized":"2024-09-11T00:56:28.985643+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":36,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"62'4259","reported_seq":4356,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.548466+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:56:28.548466+0000","last_peered":"2024-09-11T00:56:28.548466+0000","last_clean":"2024-09-11T00:56:28.548466+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:56:28.548466+0000","last_undegraded":"2024-09-11T00:56:28.548466+0000","last_fullsized":"2024-09-11T00:56:28.548466+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22192,"num_write_kb":330506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"60'2765","reported_seq":2868,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067495+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:56:28.067495+0000","last_peered":"2024-09-11T00:56:28.067495+0000","last_clean":"2024-09-11T00:56:28.067495+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:56:28.067495+0000","last_undegraded":"2024-09-11T00:56:28.067495+0000","last_fullsized":"2024-09-11T00:56:28.067495+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14757,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"62'389","reported_seq":566,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.078750+0000","last_change":"2024-09-11T00:30:44.924682+0000","last_active":"2024-09-11T00:56:31.078750+0000","last_peered":"2024-09-11T00:56:31.078750+0000","last_clean":"2024-09-11T00:56:31.078750+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:56:31.078750+0000","last_undegraded":"2024-09-11T00:56:31.078750+0000","last_fullsized":"2024-09-11T00:56:31.078750+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:15:11.397251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":127,"num_read_kb":56252,"num_write":381,"num_write_kb":4290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"62'5","reported_seq":103,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.528773+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:56:28.528773+0000","last_peered":"2024-09-11T00:56:28.528773+0000","last_clean":"2024-09-11T00:56:28.528773+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:56:28.528773+0000","last_undegraded":"2024-09-11T00:56:28.528773+0000","last_fullsized":"2024-09-11T00:56:28.528773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"62'3220","reported_seq":3316,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.464767+0000","last_change":"2024-09-11T00:30:54.986017+0000","last_active":"2024-09-11T00:56:28.464767+0000","last_peered":"2024-09-11T00:56:28.464767+0000","last_clean":"2024-09-11T00:56:28.464767+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:56:28.464767+0000","last_undegraded":"2024-09-11T00:56:28.464767+0000","last_fullsized":"2024-09-11T00:56:28.464767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:58:12.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16450,"num_write_kb":262397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"60'3928","reported_seq":4033,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064969+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:56:28.064969+0000","last_peered":"2024-09-11T00:56:28.064969+0000","last_clean":"2024-09-11T00:56:28.064969+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:56:28.064969+0000","last_undegraded":"2024-09-11T00:56:28.064969+0000","last_fullsized":"2024-09-11T00:56:28.064969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20599,"num_write_kb":340888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"62'43","reported_seq":190,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.086970+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:56:31.086970+0000","last_peered":"2024-09-11T00:56:31.086970+0000","last_clean":"2024-09-11T00:56:31.086970+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:56:31.086970+0000","last_undegraded":"2024-09-11T00:56:31.086970+0000","last_fullsized":"2024-09-11T00:56:31.086970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":42,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"62'8","reported_seq":148,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.516812+0000","last_change":"2024-09-11T00:30:53.988462+0000","last_active":"2024-09-11T00:56:28.516812+0000","last_peered":"2024-09-11T00:56:28.516812+0000","last_clean":"2024-09-11T00:56:28.516812+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:56:28.516812+0000","last_undegraded":"2024-09-11T00:56:28.516812+0000","last_fullsized":"2024-09-11T00:56:28.516812+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:22.385557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"62'4111","reported_seq":4207,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.522560+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:56:28.522560+0000","last_peered":"2024-09-11T00:56:28.522560+0000","last_clean":"2024-09-11T00:56:28.522560+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:56:28.522560+0000","last_undegraded":"2024-09-11T00:56:28.522560+0000","last_fullsized":"2024-09-11T00:56:28.522560+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21746,"num_write_kb":328824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"60'2863","reported_seq":2972,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060043+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:56:28.060043+0000","last_peered":"2024-09-11T00:56:28.060043+0000","last_clean":"2024-09-11T00:56:28.060043+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:56:28.060043+0000","last_undegraded":"2024-09-11T00:56:28.060043+0000","last_fullsized":"2024-09-11T00:56:28.060043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2863,"log_dups_size":0,"ondisk_log_size":2863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":46,"num_write":14990,"num_write_kb":244331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"60'2","reported_seq":107,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068726+0000","last_change":"2024-09-11T00:30:44.913468+0000","last_active":"2024-09-11T00:56:28.068726+0000","last_peered":"2024-09-11T00:56:28.068726+0000","last_clean":"2024-09-11T00:56:28.068726+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:56:28.068726+0000","last_undegraded":"2024-09-11T00:56:28.068726+0000","last_fullsized":"2024-09-11T00:56:28.068726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:25:59.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":203,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.264447+0000","last_change":"2024-09-11T00:30:53.986646+0000","last_active":"2024-09-11T00:56:34.264447+0000","last_peered":"2024-09-11T00:56:34.264447+0000","last_clean":"2024-09-11T00:56:34.264447+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:56:34.264447+0000","last_undegraded":"2024-09-11T00:56:34.264447+0000","last_fullsized":"2024-09-11T00:56:34.264447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:27.447940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":279,"num_read_kb":279,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"62'3641","reported_seq":3737,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514237+0000","last_change":"2024-09-11T00:30:54.982486+0000","last_active":"2024-09-11T00:56:28.514237+0000","last_peered":"2024-09-11T00:56:28.514237+0000","last_clean":"2024-09-11T00:56:28.514237+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:56:28.514237+0000","last_undegraded":"2024-09-11T00:56:28.514237+0000","last_fullsized":"2024-09-11T00:56:28.514237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:41.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19470,"num_write_kb":282742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"60'2896","reported_seq":3003,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.076032+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:56:28.076032+0000","last_peered":"2024-09-11T00:56:28.076032+0000","last_clean":"2024-09-11T00:56:28.076032+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:56:28.076032+0000","last_undegraded":"2024-09-11T00:56:28.076032+0000","last_fullsized":"2024-09-11T00:56:28.076032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":22,"num_write":14816,"num_write_kb":255894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"62'5","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064417+0000","last_change":"2024-09-11T00:30:44.913672+0000","last_active":"2024-09-11T00:56:28.064417+0000","last_peered":"2024-09-11T00:56:28.064417+0000","last_clean":"2024-09-11T00:56:28.064417+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:56:28.064417+0000","last_undegraded":"2024-09-11T00:56:28.064417+0000","last_fullsized":"2024-09-11T00:56:28.064417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:38.037624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"62'6","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.504113+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:56:28.504113+0000","last_peered":"2024-09-11T00:56:28.504113+0000","last_clean":"2024-09-11T00:56:28.504113+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:56:28.504113+0000","last_undegraded":"2024-09-11T00:56:28.504113+0000","last_fullsized":"2024-09-11T00:56:28.504113+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"62'3862","reported_seq":3961,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.513928+0000","last_change":"2024-09-11T00:30:54.985091+0000","last_active":"2024-09-11T00:56:28.513928+0000","last_peered":"2024-09-11T00:56:28.513928+0000","last_clean":"2024-09-11T00:56:28.513928+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:56:28.513928+0000","last_undegraded":"2024-09-11T00:56:28.513928+0000","last_fullsized":"2024-09-11T00:56:28.513928+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:33.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20250,"num_write_kb":318736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"62'29","reported_seq":128,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.490960+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:56:28.490960+0000","last_peered":"2024-09-11T00:56:28.490960+0000","last_clean":"2024-09-11T00:56:28.490960+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:56:28.490960+0000","last_undegraded":"2024-09-11T00:56:28.490960+0000","last_fullsized":"2024-09-11T00:56:28.490960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"60'3526","reported_seq":3630,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048540+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:56:28.048540+0000","last_peered":"2024-09-11T00:56:28.048540+0000","last_clean":"2024-09-11T00:56:28.048540+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:56:28.048540+0000","last_undegraded":"2024-09-11T00:56:28.048540+0000","last_fullsized":"2024-09-11T00:56:28.048540+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":18407,"num_write_kb":311214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"60'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064114+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:56:28.064114+0000","last_peered":"2024-09-11T00:56:28.064114+0000","last_clean":"2024-09-11T00:56:28.064114+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:56:28.064114+0000","last_undegraded":"2024-09-11T00:56:28.064114+0000","last_fullsized":"2024-09-11T00:56:28.064114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"61'3026","reported_seq":3123,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.053185+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:56:28.053185+0000","last_peered":"2024-09-11T00:56:28.053185+0000","last_clean":"2024-09-11T00:56:28.053185+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:56:28.053185+0000","last_undegraded":"2024-09-11T00:56:28.053185+0000","last_fullsized":"2024-09-11T00:56:28.053185+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:11.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16026,"num_write_kb":256702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"61'972","reported_seq":1122,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065625+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:56:28.065625+0000","last_peered":"2024-09-11T00:56:28.065625+0000","last_clean":"2024-09-11T00:56:28.065625+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:56:28.065625+0000","last_undegraded":"2024-09-11T00:56:28.065625+0000","last_fullsized":"2024-09-11T00:56:28.065625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":972,"log_dups_size":0,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":7829,"num_write":949,"num_write_kb":4657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"60'3393","reported_seq":3498,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057539+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:56:28.057539+0000","last_peered":"2024-09-11T00:56:28.057539+0000","last_clean":"2024-09-11T00:56:28.057539+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:56:28.057539+0000","last_undegraded":"2024-09-11T00:56:28.057539+0000","last_fullsized":"2024-09-11T00:56:28.057539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":17621,"num_write_kb":291741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"60'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051439+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:56:28.051439+0000","last_peered":"2024-09-11T00:56:28.051439+0000","last_clean":"2024-09-11T00:56:28.051439+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:56:28.051439+0000","last_undegraded":"2024-09-11T00:56:28.051439+0000","last_fullsized":"2024-09-11T00:56:28.051439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"62'3945","reported_seq":4040,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.470154+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:56:28.470154+0000","last_peered":"2024-09-11T00:56:28.470154+0000","last_clean":"2024-09-11T00:56:28.470154+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:56:28.470154+0000","last_undegraded":"2024-09-11T00:56:28.470154+0000","last_fullsized":"2024-09-11T00:56:28.470154+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20523,"num_write_kb":340217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"62'101","reported_seq":208,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:37.634421+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:56:37.634421+0000","last_peered":"2024-09-11T00:56:37.634421+0000","last_clean":"2024-09-11T00:56:37.634421+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:56:37.634421+0000","last_undegraded":"2024-09-11T00:56:37.634421+0000","last_fullsized":"2024-09-11T00:56:37.634421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":768543,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":728,"num_write":83,"num_write_kb":803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"60'2411","reported_seq":2516,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.049073+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:56:28.049073+0000","last_peered":"2024-09-11T00:56:28.049073+0000","last_clean":"2024-09-11T00:56:28.049073+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:56:28.049073+0000","last_undegraded":"2024-09-11T00:56:28.049073+0000","last_fullsized":"2024-09-11T00:56:28.049073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":8,"num_write":12211,"num_write_kb":212619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"62'3","reported_seq":142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.049506+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:56:28.049506+0000","last_peered":"2024-09-11T00:56:28.049506+0000","last_clean":"2024-09-11T00:56:28.049506+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:56:28.049506+0000","last_undegraded":"2024-09-11T00:56:28.049506+0000","last_fullsized":"2024-09-11T00:56:28.049506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"62'2354","reported_seq":2449,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.471047+0000","last_change":"2024-09-11T00:30:54.998464+0000","last_active":"2024-09-11T00:56:28.471047+0000","last_peered":"2024-09-11T00:56:28.471047+0000","last_clean":"2024-09-11T00:56:28.471047+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:56:28.471047+0000","last_undegraded":"2024-09-11T00:56:28.471047+0000","last_fullsized":"2024-09-11T00:56:28.471047+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:56.469791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":205009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"60'676","reported_seq":797,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059574+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:56:28.059574+0000","last_peered":"2024-09-11T00:56:28.059574+0000","last_clean":"2024-09-11T00:56:28.059574+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:56:28.059574+0000","last_undegraded":"2024-09-11T00:56:28.059574+0000","last_fullsized":"2024-09-11T00:56:28.059574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":51443,"num_write":670,"num_write_kb":4291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"60'2936","reported_seq":3039,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058263+0000","last_change":"2024-09-11T00:30:45.935921+0000","last_active":"2024-09-11T00:56:28.058263+0000","last_peered":"2024-09-11T00:56:28.058263+0000","last_clean":"2024-09-11T00:56:28.058263+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:56:28.058263+0000","last_undegraded":"2024-09-11T00:56:28.058263+0000","last_fullsized":"2024-09-11T00:56:28.058263+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2936,"log_dups_size":0,"ondisk_log_size":2936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:42:52.260751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":241742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"62'45","reported_seq":184,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488110+0000","last_change":"2024-09-11T00:30:53.987612+0000","last_active":"2024-09-11T00:56:28.488110+0000","last_peered":"2024-09-11T00:56:28.488110+0000","last_clean":"2024-09-11T00:56:28.488110+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:56:28.488110+0000","last_undegraded":"2024-09-11T00:56:28.488110+0000","last_fullsized":"2024-09-11T00:56:28.488110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:16.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"62'3441","reported_seq":3532,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.489542+0000","last_change":"2024-09-11T00:31:26.326200+0000","last_active":"2024-09-11T00:56:28.489542+0000","last_peered":"2024-09-11T00:56:28.489542+0000","last_clean":"2024-09-11T00:56:28.489542+0000","last_became_active":"2024-09-11T00:31:26.325895+0000","last_became_peered":"2024-09-11T00:31:26.325895+0000","last_unstale":"2024-09-11T00:56:28.489542+0000","last_undegraded":"2024-09-11T00:56:28.489542+0000","last_fullsized":"2024-09-11T00:56:28.489542+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:39:08.753699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18163,"num_write_kb":273760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.20","version":"60'3848","reported_seq":3953,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075938+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:56:28.075938+0000","last_peered":"2024-09-11T00:56:28.075938+0000","last_clean":"2024-09-11T00:56:28.075938+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:56:28.075938+0000","last_undegraded":"2024-09-11T00:56:28.075938+0000","last_fullsized":"2024-09-11T00:56:28.075938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20029,"num_write_kb":325938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"62'4287","reported_seq":4384,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.519505+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:56:28.519505+0000","last_peered":"2024-09-11T00:56:28.519505+0000","last_clean":"2024-09-11T00:56:28.519505+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:56:28.519505+0000","last_undegraded":"2024-09-11T00:56:28.519505+0000","last_fullsized":"2024-09-11T00:56:28.519505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22520,"num_write_kb":336110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"60'2962","reported_seq":3067,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058177+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:56:28.058177+0000","last_peered":"2024-09-11T00:56:28.058177+0000","last_clean":"2024-09-11T00:56:28.058177+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:56:28.058177+0000","last_undegraded":"2024-09-11T00:56:28.058177+0000","last_fullsized":"2024-09-11T00:56:28.058177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15172,"num_write_kb":277710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"62'2334","reported_seq":2430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075259+0000","last_change":"2024-09-11T00:30:54.979133+0000","last_active":"2024-09-11T00:56:28.075259+0000","last_peered":"2024-09-11T00:56:28.075259+0000","last_clean":"2024-09-11T00:56:28.075259+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:56:28.075259+0000","last_undegraded":"2024-09-11T00:56:28.075259+0000","last_fullsized":"2024-09-11T00:56:28.075259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2334,"log_dups_size":0,"ondisk_log_size":2334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:20.961678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11753,"num_write_kb":206776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"60'4175","reported_seq":4282,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065154+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:56:28.065154+0000","last_peered":"2024-09-11T00:56:28.065154+0000","last_clean":"2024-09-11T00:56:28.065154+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:56:28.065154+0000","last_undegraded":"2024-09-11T00:56:28.065154+0000","last_fullsized":"2024-09-11T00:56:28.065154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22002,"num_write_kb":346058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"62'3428","reported_seq":3524,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.426792+0000","last_change":"2024-09-11T00:30:54.988345+0000","last_active":"2024-09-11T00:56:28.426792+0000","last_peered":"2024-09-11T00:56:28.426792+0000","last_clean":"2024-09-11T00:56:28.426792+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:56:28.426792+0000","last_undegraded":"2024-09-11T00:56:28.426792+0000","last_fullsized":"2024-09-11T00:56:28.426792+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3428,"log_dups_size":0,"ondisk_log_size":3428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.112194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17996,"num_write_kb":289213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"60'2543","reported_seq":2648,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059170+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:56:28.059170+0000","last_peered":"2024-09-11T00:56:28.059170+0000","last_clean":"2024-09-11T00:56:28.059170+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:56:28.059170+0000","last_undegraded":"2024-09-11T00:56:28.059170+0000","last_fullsized":"2024-09-11T00:56:28.059170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13221,"num_write_kb":224922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"62'3390","reported_seq":3487,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.413384+0000","last_change":"2024-09-11T00:30:54.981185+0000","last_active":"2024-09-11T00:56:28.413384+0000","last_peered":"2024-09-11T00:56:28.413384+0000","last_clean":"2024-09-11T00:56:28.413384+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:56:28.413384+0000","last_undegraded":"2024-09-11T00:56:28.413384+0000","last_fullsized":"2024-09-11T00:56:28.413384+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:45.740467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17320,"num_write_kb":290458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"60'2853","reported_seq":2957,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052823+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:56:28.052823+0000","last_peered":"2024-09-11T00:56:28.052823+0000","last_clean":"2024-09-11T00:56:28.052823+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:56:28.052823+0000","last_undegraded":"2024-09-11T00:56:28.052823+0000","last_fullsized":"2024-09-11T00:56:28.052823+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14682,"num_write_kb":238952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"62'2740","reported_seq":2837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.499809+0000","last_change":"2024-09-11T00:30:54.992763+0000","last_active":"2024-09-11T00:56:28.499809+0000","last_peered":"2024-09-11T00:56:28.499809+0000","last_clean":"2024-09-11T00:56:28.499809+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:56:28.499809+0000","last_undegraded":"2024-09-11T00:56:28.499809+0000","last_fullsized":"2024-09-11T00:56:28.499809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2740,"log_dups_size":0,"ondisk_log_size":2740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:02.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14066,"num_write_kb":224154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"60'3660","reported_seq":3764,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052923+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:56:28.052923+0000","last_peered":"2024-09-11T00:56:28.052923+0000","last_clean":"2024-09-11T00:56:28.052923+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:56:28.052923+0000","last_undegraded":"2024-09-11T00:56:28.052923+0000","last_fullsized":"2024-09-11T00:56:28.052923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19466,"num_write_kb":307301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"62'4363","reported_seq":4460,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.478746+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:56:28.478746+0000","last_peered":"2024-09-11T00:56:28.478746+0000","last_clean":"2024-09-11T00:56:28.478746+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:56:28.478746+0000","last_undegraded":"2024-09-11T00:56:28.478746+0000","last_fullsized":"2024-09-11T00:56:28.478746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23064,"num_write_kb":350422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"60'2921","reported_seq":3025,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075658+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:56:28.075658+0000","last_peered":"2024-09-11T00:56:28.075658+0000","last_clean":"2024-09-11T00:56:28.075658+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:56:28.075658+0000","last_undegraded":"2024-09-11T00:56:28.075658+0000","last_fullsized":"2024-09-11T00:56:28.075658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15417,"num_write_kb":250464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"62'3030","reported_seq":3127,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.431858+0000","last_change":"2024-09-11T00:30:54.984447+0000","last_active":"2024-09-11T00:56:28.431858+0000","last_peered":"2024-09-11T00:56:28.431858+0000","last_clean":"2024-09-11T00:56:28.431858+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:56:28.431858+0000","last_undegraded":"2024-09-11T00:56:28.431858+0000","last_fullsized":"2024-09-11T00:56:28.431858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:49.024022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15510,"num_write_kb":262754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"60'3321","reported_seq":3427,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059533+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:56:28.059533+0000","last_peered":"2024-09-11T00:56:28.059533+0000","last_clean":"2024-09-11T00:56:28.059533+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:56:28.059533+0000","last_undegraded":"2024-09-11T00:56:28.059533+0000","last_fullsized":"2024-09-11T00:56:28.059533+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":25,"num_write":17209,"num_write_kb":314610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"62'2084","reported_seq":2181,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.543950+0000","last_change":"2024-09-11T00:30:54.996091+0000","last_active":"2024-09-11T00:56:28.543950+0000","last_peered":"2024-09-11T00:56:28.543950+0000","last_clean":"2024-09-11T00:56:28.543950+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:56:28.543950+0000","last_undegraded":"2024-09-11T00:56:28.543950+0000","last_fullsized":"2024-09-11T00:56:28.543950+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2084,"log_dups_size":0,"ondisk_log_size":2084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:42.987336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10636,"num_write_kb":173560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"60'2810","reported_seq":2914,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063723+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:56:28.063723+0000","last_peered":"2024-09-11T00:56:28.063723+0000","last_clean":"2024-09-11T00:56:28.063723+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:56:28.063723+0000","last_undegraded":"2024-09-11T00:56:28.063723+0000","last_fullsized":"2024-09-11T00:56:28.063723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2810,"log_dups_size":0,"ondisk_log_size":2810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":243007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"62'3697","reported_seq":3793,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.455458+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:56:28.455458+0000","last_peered":"2024-09-11T00:56:28.455458+0000","last_clean":"2024-09-11T00:56:28.455458+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:56:28.455458+0000","last_undegraded":"2024-09-11T00:56:28.455458+0000","last_fullsized":"2024-09-11T00:56:28.455458+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18782,"num_write_kb":312790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"60'2824","reported_seq":2928,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057103+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:56:28.057103+0000","last_peered":"2024-09-11T00:56:28.057103+0000","last_clean":"2024-09-11T00:56:28.057103+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:56:28.057103+0000","last_undegraded":"2024-09-11T00:56:28.057103+0000","last_fullsized":"2024-09-11T00:56:28.057103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2824,"log_dups_size":0,"ondisk_log_size":2824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14695,"num_write_kb":259410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"62'2917","reported_seq":3013,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510399+0000","last_change":"2024-09-11T00:30:54.981799+0000","last_active":"2024-09-11T00:56:28.510399+0000","last_peered":"2024-09-11T00:56:28.510399+0000","last_clean":"2024-09-11T00:56:28.510399+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:56:28.510399+0000","last_undegraded":"2024-09-11T00:56:28.510399+0000","last_fullsized":"2024-09-11T00:56:28.510399+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2917,"log_dups_size":0,"ondisk_log_size":2917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:42.670629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":242029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"60'3431","reported_seq":3535,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057841+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:56:28.057841+0000","last_peered":"2024-09-11T00:56:28.057841+0000","last_clean":"2024-09-11T00:56:28.057841+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:56:28.057841+0000","last_undegraded":"2024-09-11T00:56:28.057841+0000","last_fullsized":"2024-09-11T00:56:28.057841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18408,"num_write_kb":290787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"62'2945","reported_seq":3041,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514103+0000","last_change":"2024-09-11T00:30:54.992697+0000","last_active":"2024-09-11T00:56:28.514103+0000","last_peered":"2024-09-11T00:56:28.514103+0000","last_clean":"2024-09-11T00:56:28.514103+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:56:28.514103+0000","last_undegraded":"2024-09-11T00:56:28.514103+0000","last_fullsized":"2024-09-11T00:56:28.514103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:54:07.227169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15229,"num_write_kb":249300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"60'2849","reported_seq":2955,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048641+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:56:28.048641+0000","last_peered":"2024-09-11T00:56:28.048641+0000","last_clean":"2024-09-11T00:56:28.048641+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:56:28.048641+0000","last_undegraded":"2024-09-11T00:56:28.048641+0000","last_fullsized":"2024-09-11T00:56:28.048641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2849,"log_dups_size":0,"ondisk_log_size":2849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14625,"num_write_kb":263252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"62'4017","reported_seq":4113,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.477074+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:56:28.477074+0000","last_peered":"2024-09-11T00:56:28.477074+0000","last_clean":"2024-09-11T00:56:28.477074+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:56:28.477074+0000","last_undegraded":"2024-09-11T00:56:28.477074+0000","last_fullsized":"2024-09-11T00:56:28.477074+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":329319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"60'2942","reported_seq":3046,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066118+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:56:28.066118+0000","last_peered":"2024-09-11T00:56:28.066118+0000","last_clean":"2024-09-11T00:56:28.066118+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:56:28.066118+0000","last_undegraded":"2024-09-11T00:56:28.066118+0000","last_fullsized":"2024-09-11T00:56:28.066118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15436,"num_write_kb":246093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"62'3417","reported_seq":3514,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.528940+0000","last_change":"2024-09-11T00:30:54.985827+0000","last_active":"2024-09-11T00:56:28.528940+0000","last_peered":"2024-09-11T00:56:28.528940+0000","last_clean":"2024-09-11T00:56:28.528940+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:56:28.528940+0000","last_undegraded":"2024-09-11T00:56:28.528940+0000","last_fullsized":"2024-09-11T00:56:28.528940+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:32.027117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18129,"num_write_kb":269216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"60'2746","reported_seq":2850,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069377+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:56:28.069377+0000","last_peered":"2024-09-11T00:56:28.069377+0000","last_clean":"2024-09-11T00:56:28.069377+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:56:28.069377+0000","last_undegraded":"2024-09-11T00:56:28.069377+0000","last_fullsized":"2024-09-11T00:56:28.069377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13859,"num_write_kb":247924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"62'2570","reported_seq":2672,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514173+0000","last_change":"2024-09-11T00:31:26.119785+0000","last_active":"2024-09-11T00:56:28.514173+0000","last_peered":"2024-09-11T00:56:28.514173+0000","last_clean":"2024-09-11T00:56:28.514173+0000","last_became_active":"2024-09-11T00:31:26.119436+0000","last_became_peered":"2024-09-11T00:31:26.119436+0000","last_unstale":"2024-09-11T00:56:28.514173+0000","last_undegraded":"2024-09-11T00:56:28.514173+0000","last_fullsized":"2024-09-11T00:56:28.514173+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:53:49.407575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13205,"num_write_kb":216683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"60'2388","reported_seq":2494,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058894+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:56:28.058894+0000","last_peered":"2024-09-11T00:56:28.058894+0000","last_clean":"2024-09-11T00:56:28.058894+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:56:28.058894+0000","last_undegraded":"2024-09-11T00:56:28.058894+0000","last_fullsized":"2024-09-11T00:56:28.058894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2388,"log_dups_size":0,"ondisk_log_size":2388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4,"num_write":12384,"num_write_kb":205064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"62'3381","reported_seq":3477,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397849+0000","last_change":"2024-09-11T00:30:54.993156+0000","last_active":"2024-09-11T00:56:28.397849+0000","last_peered":"2024-09-11T00:56:28.397849+0000","last_clean":"2024-09-11T00:56:28.397849+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:56:28.397849+0000","last_undegraded":"2024-09-11T00:56:28.397849+0000","last_fullsized":"2024-09-11T00:56:28.397849+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:25.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17747,"num_write_kb":280034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"60'3936","reported_seq":4041,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067998+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:56:28.067998+0000","last_peered":"2024-09-11T00:56:28.067998+0000","last_clean":"2024-09-11T00:56:28.067998+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:56:28.067998+0000","last_undegraded":"2024-09-11T00:56:28.067998+0000","last_fullsized":"2024-09-11T00:56:28.067998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":20459,"num_write_kb":335990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"62'3772","reported_seq":3867,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.454399+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:56:28.454399+0000","last_peered":"2024-09-11T00:56:28.454399+0000","last_clean":"2024-09-11T00:56:28.454399+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:56:28.454399+0000","last_undegraded":"2024-09-11T00:56:28.454399+0000","last_fullsized":"2024-09-11T00:56:28.454399+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20201,"num_write_kb":302053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"60'2757","reported_seq":2861,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062570+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:56:28.062570+0000","last_peered":"2024-09-11T00:56:28.062570+0000","last_clean":"2024-09-11T00:56:28.062570+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:56:28.062570+0000","last_undegraded":"2024-09-11T00:56:28.062570+0000","last_fullsized":"2024-09-11T00:56:28.062570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14203,"num_write_kb":241057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"62'4048","reported_seq":4143,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.441208+0000","last_change":"2024-09-11T00:30:54.993461+0000","last_active":"2024-09-11T00:56:28.441208+0000","last_peered":"2024-09-11T00:56:28.441208+0000","last_clean":"2024-09-11T00:56:28.441208+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:56:28.441208+0000","last_undegraded":"2024-09-11T00:56:28.441208+0000","last_fullsized":"2024-09-11T00:56:28.441208+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:07.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21387,"num_write_kb":364007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"60'3572","reported_seq":3676,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057127+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:56:28.057127+0000","last_peered":"2024-09-11T00:56:28.057127+0000","last_clean":"2024-09-11T00:56:28.057127+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:56:28.057127+0000","last_undegraded":"2024-09-11T00:56:28.057127+0000","last_fullsized":"2024-09-11T00:56:28.057127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18317,"num_write_kb":294510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"62'2987","reported_seq":3083,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397386+0000","last_change":"2024-09-11T00:30:54.986265+0000","last_active":"2024-09-11T00:56:28.397386+0000","last_peered":"2024-09-11T00:56:28.397386+0000","last_clean":"2024-09-11T00:56:28.397386+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:56:28.397386+0000","last_undegraded":"2024-09-11T00:56:28.397386+0000","last_fullsized":"2024-09-11T00:56:28.397386+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:49.109328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15338,"num_write_kb":237611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"60'2929","reported_seq":3034,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051236+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:56:28.051236+0000","last_peered":"2024-09-11T00:56:28.051236+0000","last_clean":"2024-09-11T00:56:28.051236+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:56:28.051236+0000","last_undegraded":"2024-09-11T00:56:28.051236+0000","last_fullsized":"2024-09-11T00:56:28.051236+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2929,"log_dups_size":0,"ondisk_log_size":2929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":15867,"num_write_kb":268409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"62'3243","reported_seq":3338,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.433332+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:56:28.433332+0000","last_peered":"2024-09-11T00:56:28.433332+0000","last_clean":"2024-09-11T00:56:28.433332+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:56:28.433332+0000","last_undegraded":"2024-09-11T00:56:28.433332+0000","last_fullsized":"2024-09-11T00:56:28.433332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16324,"num_write_kb":262051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"60'2913","reported_seq":3019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052467+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:56:28.052467+0000","last_peered":"2024-09-11T00:56:28.052467+0000","last_clean":"2024-09-11T00:56:28.052467+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:56:28.052467+0000","last_undegraded":"2024-09-11T00:56:28.052467+0000","last_fullsized":"2024-09-11T00:56:28.052467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2913,"log_dups_size":0,"ondisk_log_size":2913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15444,"num_write_kb":246103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"62'3344","reported_seq":3440,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.543938+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:56:28.543938+0000","last_peered":"2024-09-11T00:56:28.543938+0000","last_clean":"2024-09-11T00:56:28.543938+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:56:28.543938+0000","last_undegraded":"2024-09-11T00:56:28.543938+0000","last_fullsized":"2024-09-11T00:56:28.543938+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3344,"log_dups_size":0,"ondisk_log_size":3344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17374,"num_write_kb":274437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"60'2266","reported_seq":2373,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065676+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:56:28.065676+0000","last_peered":"2024-09-11T00:56:28.065676+0000","last_clean":"2024-09-11T00:56:28.065676+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:56:28.065676+0000","last_undegraded":"2024-09-11T00:56:28.065676+0000","last_fullsized":"2024-09-11T00:56:28.065676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":26,"num_write":11816,"num_write_kb":219351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"62'2714","reported_seq":2811,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.547334+0000","last_change":"2024-09-11T00:30:54.994988+0000","last_active":"2024-09-11T00:56:28.547334+0000","last_peered":"2024-09-11T00:56:28.547334+0000","last_clean":"2024-09-11T00:56:28.547334+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:56:28.547334+0000","last_undegraded":"2024-09-11T00:56:28.547334+0000","last_fullsized":"2024-09-11T00:56:28.547334+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2714,"log_dups_size":0,"ondisk_log_size":2714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:31.973641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14634,"num_write_kb":230575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"60'3170","reported_seq":3259,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058097+0000","last_change":"2024-09-11T00:31:26.117501+0000","last_active":"2024-09-11T00:56:28.058097+0000","last_peered":"2024-09-11T00:56:28.058097+0000","last_clean":"2024-09-11T00:56:28.058097+0000","last_became_active":"2024-09-11T00:31:26.117146+0000","last_became_peered":"2024-09-11T00:31:26.117146+0000","last_unstale":"2024-09-11T00:56:28.058097+0000","last_undegraded":"2024-09-11T00:56:28.058097+0000","last_fullsized":"2024-09-11T00:56:28.058097+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:52.194644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15922,"num_write_kb":268523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.30","version":"62'3135","reported_seq":3231,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.469071+0000","last_change":"2024-09-11T00:30:54.984767+0000","last_active":"2024-09-11T00:56:28.469071+0000","last_peered":"2024-09-11T00:56:28.469071+0000","last_clean":"2024-09-11T00:56:28.469071+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:56:28.469071+0000","last_undegraded":"2024-09-11T00:56:28.469071+0000","last_fullsized":"2024-09-11T00:56:28.469071+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:44.833677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16038,"num_write_kb":252613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"60'3300","reported_seq":3404,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068344+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:56:28.068344+0000","last_peered":"2024-09-11T00:56:28.068344+0000","last_clean":"2024-09-11T00:56:28.068344+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:56:28.068344+0000","last_undegraded":"2024-09-11T00:56:28.068344+0000","last_fullsized":"2024-09-11T00:56:28.068344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3300,"log_dups_size":0,"ondisk_log_size":3300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":17861,"num_write_kb":288513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"62'3599","reported_seq":3695,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.492582+0000","last_change":"2024-09-11T00:30:54.986423+0000","last_active":"2024-09-11T00:56:28.492582+0000","last_peered":"2024-09-11T00:56:28.492582+0000","last_clean":"2024-09-11T00:56:28.492582+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:56:28.492582+0000","last_undegraded":"2024-09-11T00:56:28.492582+0000","last_fullsized":"2024-09-11T00:56:28.492582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:16:50.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18686,"num_write_kb":304162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"60'4168","reported_seq":4272,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063833+0000","last_change":"2024-09-11T00:30:45.913458+0000","last_active":"2024-09-11T00:56:28.063833+0000","last_peered":"2024-09-11T00:56:28.063833+0000","last_clean":"2024-09-11T00:56:28.063833+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:56:28.063833+0000","last_undegraded":"2024-09-11T00:56:28.063833+0000","last_fullsized":"2024-09-11T00:56:28.063833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:12:15.900040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22110,"num_write_kb":362983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"62'3937","reported_seq":4034,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.544539+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:56:28.544539+0000","last_peered":"2024-09-11T00:56:28.544539+0000","last_clean":"2024-09-11T00:56:28.544539+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:56:28.544539+0000","last_undegraded":"2024-09-11T00:56:28.544539+0000","last_fullsized":"2024-09-11T00:56:28.544539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21285,"num_write_kb":343312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"60'3475","reported_seq":3579,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057082+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:56:28.057082+0000","last_peered":"2024-09-11T00:56:28.057082+0000","last_clean":"2024-09-11T00:56:28.057082+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:56:28.057082+0000","last_undegraded":"2024-09-11T00:56:28.057082+0000","last_fullsized":"2024-09-11T00:56:28.057082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3475,"log_dups_size":0,"ondisk_log_size":3475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17645,"num_write_kb":295948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"62'3302","reported_seq":3399,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.512120+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:56:28.512120+0000","last_peered":"2024-09-11T00:56:28.512120+0000","last_clean":"2024-09-11T00:56:28.512120+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:56:28.512120+0000","last_undegraded":"2024-09-11T00:56:28.512120+0000","last_fullsized":"2024-09-11T00:56:28.512120+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17558,"num_write_kb":271021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"60'3295","reported_seq":3399,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063104+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:56:28.063104+0000","last_peered":"2024-09-11T00:56:28.063104+0000","last_clean":"2024-09-11T00:56:28.063104+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:56:28.063104+0000","last_undegraded":"2024-09-11T00:56:28.063104+0000","last_fullsized":"2024-09-11T00:56:28.063104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":25,"num_write":17271,"num_write_kb":265887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"61'3033","reported_seq":3130,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048703+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:56:28.048703+0000","last_peered":"2024-09-11T00:56:28.048703+0000","last_clean":"2024-09-11T00:56:28.048703+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:56:28.048703+0000","last_undegraded":"2024-09-11T00:56:28.048703+0000","last_fullsized":"2024-09-11T00:56:28.048703+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16092,"num_write_kb":252889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"60'2629","reported_seq":2733,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.078405+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:56:28.078405+0000","last_peered":"2024-09-11T00:56:28.078405+0000","last_clean":"2024-09-11T00:56:28.078405+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:56:28.078405+0000","last_undegraded":"2024-09-11T00:56:28.078405+0000","last_fullsized":"2024-09-11T00:56:28.078405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":13758,"num_write_kb":227101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"62'3227","reported_seq":3324,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.479006+0000","last_change":"2024-09-11T00:30:54.990631+0000","last_active":"2024-09-11T00:56:28.479006+0000","last_peered":"2024-09-11T00:56:28.479006+0000","last_clean":"2024-09-11T00:56:28.479006+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:56:28.479006+0000","last_undegraded":"2024-09-11T00:56:28.479006+0000","last_fullsized":"2024-09-11T00:56:28.479006+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:02.422471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16396,"num_write_kb":258131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"60'3415","reported_seq":3520,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067480+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:56:28.067480+0000","last_peered":"2024-09-11T00:56:28.067480+0000","last_clean":"2024-09-11T00:56:28.067480+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:56:28.067480+0000","last_undegraded":"2024-09-11T00:56:28.067480+0000","last_fullsized":"2024-09-11T00:56:28.067480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":17180,"num_write_kb":310287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"62'2771","reported_seq":2866,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.447411+0000","last_change":"2024-09-11T00:30:54.991047+0000","last_active":"2024-09-11T00:56:28.447411+0000","last_peered":"2024-09-11T00:56:28.447411+0000","last_clean":"2024-09-11T00:56:28.447411+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:56:28.447411+0000","last_undegraded":"2024-09-11T00:56:28.447411+0000","last_fullsized":"2024-09-11T00:56:28.447411+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:41:48.229261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14317,"num_write_kb":221149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"60'2741","reported_seq":2846,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075991+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:56:28.075991+0000","last_peered":"2024-09-11T00:56:28.075991+0000","last_clean":"2024-09-11T00:56:28.075991+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:56:28.075991+0000","last_undegraded":"2024-09-11T00:56:28.075991+0000","last_fullsized":"2024-09-11T00:56:28.075991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2741,"log_dups_size":0,"ondisk_log_size":2741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14755,"num_write_kb":255665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"62'3704","reported_seq":3799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.547303+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:56:28.547303+0000","last_peered":"2024-09-11T00:56:28.547303+0000","last_clean":"2024-09-11T00:56:28.547303+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:56:28.547303+0000","last_undegraded":"2024-09-11T00:56:28.547303+0000","last_fullsized":"2024-09-11T00:56:28.547303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18483,"num_write_kb":307447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"60'3035","reported_seq":3140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059908+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:56:28.059908+0000","last_peered":"2024-09-11T00:56:28.059908+0000","last_clean":"2024-09-11T00:56:28.059908+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:56:28.059908+0000","last_undegraded":"2024-09-11T00:56:28.059908+0000","last_fullsized":"2024-09-11T00:56:28.059908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3035,"log_dups_size":0,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15470,"num_write_kb":262623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"62'3351","reported_seq":3447,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075584+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:56:28.075584+0000","last_peered":"2024-09-11T00:56:28.075584+0000","last_clean":"2024-09-11T00:56:28.075584+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:56:28.075584+0000","last_undegraded":"2024-09-11T00:56:28.075584+0000","last_fullsized":"2024-09-11T00:56:28.075584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17034,"num_write_kb":277088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"60'4106","reported_seq":4213,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059593+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:56:28.059593+0000","last_peered":"2024-09-11T00:56:28.059593+0000","last_clean":"2024-09-11T00:56:28.059593+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:56:28.059593+0000","last_undegraded":"2024-09-11T00:56:28.059593+0000","last_fullsized":"2024-09-11T00:56:28.059593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22024,"num_write_kb":354504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"62'3391","reported_seq":3486,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.526593+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:56:28.526593+0000","last_peered":"2024-09-11T00:56:28.526593+0000","last_clean":"2024-09-11T00:56:28.526593+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:56:28.526593+0000","last_undegraded":"2024-09-11T00:56:28.526593+0000","last_fullsized":"2024-09-11T00:56:28.526593+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18265,"num_write_kb":274031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"60'2946","reported_seq":3053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069081+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:56:28.069081+0000","last_peered":"2024-09-11T00:56:28.069081+0000","last_clean":"2024-09-11T00:56:28.069081+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:56:28.069081+0000","last_undegraded":"2024-09-11T00:56:28.069081+0000","last_fullsized":"2024-09-11T00:56:28.069081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":17,"num_write":15520,"num_write_kb":262805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1107571,"num_write_kb":17543218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212907,"ondisk_log_size":212907,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18223220,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1558,"num_read_kb":197403,"num_write":4817,"num_write_kb":19773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36511744,"data_stored":36446440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4412,"ondisk_log_size":4412,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":381,"num_write":1025649,"num_write_kb":17211858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196678,"ondisk_log_size":196678,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":17635410,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1062,"num_read_kb":143929,"num_write":4568,"num_write_kb":19046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35332096,"data_stored":35270820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4119,"ondisk_log_size":4119,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509757,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487152,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":486270,"kb_avail":93884688,"statfs":{"total":96636764160,"available":96137920512,"internally_reserved":0,"allocated":884736,"data_stored":496637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":497940938},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509759,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668164,"kb_used_data":10292,"kb_used_omap":1,"kb_used_meta":657854,"kb_avail":93703676,"statfs":{"total":96636764160,"available":95952564224,"internally_reserved":0,"allocated":10539008,"data_stored":10153943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":673642950},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509758,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685684,"kb_used_data":17828,"kb_used_omap":1,"kb_used_meta":667838,"kb_avail":93686156,"statfs":{"total":96636764160,"available":95934623744,"internally_reserved":0,"allocated":18255872,"data_stored":17864749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683866565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509757,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722280,"kb_used_data":13912,"kb_used_omap":1,"kb_used_meta":708350,"kb_avail":93649560,"statfs":{"total":96636764160,"available":95897149440,"internally_reserved":0,"allocated":14245888,"data_stored":13849304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":725350856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509758,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737056,"kb_used_data":14480,"kb_used_omap":1,"kb_used_meta":722558,"kb_avail":93634784,"statfs":{"total":96636764160,"available":95882018816,"internally_reserved":0,"allocated":14827520,"data_stored":14436462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739899845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509758,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598360,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":596990,"kb_avail":93773480,"statfs":{"total":96636764160,"available":96024043520,"internally_reserved":0,"allocated":1384448,"data_stored":998818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":611318213},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509759,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741960,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":740414,"kb_avail":93629880,"statfs":{"total":96636764160,"available":95876997120,"internally_reserved":0,"allocated":1564672,"data_stored":1176999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":758184389},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542462,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739432,"kb_used_data":17176,"kb_used_omap":1,"kb_used_meta":722238,"kb_avail":93632408,"statfs":{"total":96636764160,"available":95879585792,"internally_reserved":0,"allocated":17588224,"data_stored":17187564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739572165},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":778240,"data_stored":768599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9162752,"data_stored":9157733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4282577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5648384,"data_stored":5638872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":4283261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5562368,"data_stored":5551329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:56:47.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:56:48.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.001+0000 7f75c0000640 1 -- 172.21.15.31:0/4669972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a00051c0 msgr2=0x7f75a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:48.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.001+0000 7f75c0000640 1 --2- 172.21.15.31:0/4669972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a00051c0 0x7f75a000b9c0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f75b0005fd0 tx=0x7f75a0005830 comp rx=0 tx=0).stop 2024-09-11T00:56:48.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.001+0000 7f75be000640 1 -- 172.21.15.31:0/4669972 reap_dead start 2024-09-11T00:56:48.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.001+0000 7f75c0000640 1 -- 172.21.15.31:0/4669972 shutdown_connections 2024-09-11T00:56:48.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.001+0000 7f75c0000640 1 -- 172.21.15.31:0/4669972 >> 172.21.15.31:0/4669972 conn(0x7f75b80a9750 msgr2=0x7f75b80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:48.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.001+0000 7f75c0000640 1 -- 172.21.15.31:0/4669972 shutdown_connections 2024-09-11T00:56:48.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.001+0000 7f75c0000640 1 -- 172.21.15.31:0/4669972 wait complete. 2024-09-11T00:56:48.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.002+0000 7f75c0000640 1 Processor -- start 2024-09-11T00:56:48.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.003+0000 7f75c0000640 1 -- start start 2024-09-11T00:56:48.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.003+0000 7f75c0000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75a00051c0 0x7f75b814c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:48.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.003+0000 7f75c0000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b80aaa50 0x7f75b81de170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:48.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.003+0000 7f75c0000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b814cc90 0x7f75b81de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:48.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.003+0000 7f75c0000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f75b8074200 con 0x7f75b80aaa50 2024-09-11T00:56:48.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.003+0000 7f75c0000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f75b8074080 con 0x7f75b814cc90 2024-09-11T00:56:48.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75c0000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f75b8072860 con 0x7f75a00051c0 2024-09-11T00:56:48.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75bcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b80aaa50 0x7f75b81de170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75bcc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b80aaa50 0x7f75b81de170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38502/0 (socket says 172.21.15.31:38502) 2024-09-11T00:56:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75bcc00640 1 -- 172.21.15.31:0/1926838046 learned_addr learned my addr 172.21.15.31:0/1926838046 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75bd600640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75a00051c0 0x7f75b814c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75be000640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b814cc90 0x7f75b81de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75bcc00640 1 -- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75a00051c0 msgr2=0x7f75b814c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75bcc00640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75a00051c0 0x7f75b814c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75bcc00640 1 -- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b814cc90 msgr2=0x7f75b81de6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75bcc00640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b814cc90 0x7f75b81de6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.004+0000 7f75bcc00640 1 -- 172.21.15.31:0/1926838046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b80ad530 con 0x7f75b80aaa50 2024-09-11T00:56:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.005+0000 7f75bcc00640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b80aaa50 0x7f75b81de170 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f75a4011a10 tx=0x7f75a4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:48.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.005+0000 7f75b5600640 1 -- 172.21.15.31:0/1926838046 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a400fcf0 con 0x7f75b80aaa50 2024-09-11T00:56:48.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.005+0000 7f75b5600640 1 -- 172.21.15.31:0/1926838046 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75a4016070 con 0x7f75b80aaa50 2024-09-11T00:56:48.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.005+0000 7f75b5600640 1 -- 172.21.15.31:0/1926838046 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a40126b0 con 0x7f75b80aaa50 2024-09-11T00:56:48.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.005+0000 7f75c0000640 1 -- 172.21.15.31:0/1926838046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b80ad810 con 0x7f75b80aaa50 2024-09-11T00:56:48.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.005+0000 7f75c0000640 1 -- 172.21.15.31:0/1926838046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b8154400 con 0x7f75b80aaa50 2024-09-11T00:56:48.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.006+0000 7f75c0000640 1 -- 172.21.15.31:0/1926838046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b80ada50 con 0x7f75b80aaa50 2024-09-11T00:56:48.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.007+0000 7f75b5600640 1 -- 172.21.15.31:0/1926838046 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f75a4012850 con 0x7f75b80aaa50 2024-09-11T00:56:48.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.008+0000 7f75b5600640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7598077f70 0x7f759807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:48.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.008+0000 7f75b5600640 1 -- 172.21.15.31:0/1926838046 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f75a409b7d0 con 0x7f75b80aaa50 2024-09-11T00:56:48.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.008+0000 7f75bd600640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7598077f70 0x7f759807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:48.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.009+0000 7f75bd600640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7598077f70 0x7f759807a430 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f75ac006c30 tx=0x7f75ac0067a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:48.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.012+0000 7f75b5600640 1 -- 172.21.15.31:0/1926838046 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f75b80ada50 con 0x7f75b80aaa50 2024-09-11T00:56:48.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.209+0000 7f75c0000640 1 -- 172.21.15.31:0/1926838046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75b8073630 con 0x7f75b80aaa50 2024-09-11T00:56:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.211+0000 7f75b5600640 1 -- 172.21.15.31:0/1926838046 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+25258 (secure 0 0 0) 0x7f75b8073630 con 0x7f75b80aaa50 2024-09-11T00:56:48.213 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:48.213 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":62,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:56:28.030290+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.35","mappings":[{"from":0,"to":7}]},{"pgid":"5.8","mappings":[{"from":4,"to":5}]},{"pgid":"5.17","mappings":[{"from":4,"to":3}]},{"pgid":"5.26","mappings":[{"from":4,"to":5}]},{"pgid":"5.28","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6832/3721481399":"2024-09-12T00:56:28.030054+0000","172.21.15.204:6835/4094068967":"2024-09-12T00:55:04.553768+0000","172.21.15.31:6833/2956367495":"2024-09-12T00:54:19.542253+0000","172.21.15.204:6837/2689217598":"2024-09-12T00:56:14.574459+0000","172.21.15.31:6834/1855324855":"2024-09-12T00:52:49.521133+0000","172.21.15.204:6835/1981755707":"2024-09-12T00:52:39.517968+0000","172.21.15.204:6834/98007221":"2024-09-12T00:41:49.383893+0000","172.21.15.31:6832/3881548149":"2024-09-12T00:40:39.371058+0000","172.21.15.31:6835/2163558126":"2024-09-12T00:48:14.453181+0000","172.21.15.204:6835/3506150555":"2024-09-12T00:40:24.367167+0000","172.21.15.204:6837/1278135080":"2024-09-12T00:38:14.342086+0000","172.21.15.31:6832/2956367495":"2024-09-12T00:54:19.542253+0000","172.21.15.31:6834/2909466339":"2024-09-12T00:43:22.735779+0000","172.21.15.204:6836/1278135080":"2024-09-12T00:38:14.342086+0000","172.21.15.204:6835/3444890210":"2024-09-12T00:37:16.369607+0000","172.21.15.204:6834/4094068967":"2024-09-12T00:55:04.553768+0000","172.21.15.204:6837/645869523":"2024-09-12T00:35:54.319031+0000","172.21.15.204:6837/4164016077":"2024-09-12T00:50:14.476294+0000","172.21.15.31:6834/2323279595":"2024-09-12T00:44:44.412976+0000","172.21.15.204:6834/3506150555":"2024-09-12T00:40:24.367167+0000","172.21.15.31:6833/4011734390":"2024-09-12T00:37:09.330669+0000","172.21.15.31:6834/3725011623":"2024-09-12T00:50:23.114454+0000","172.21.15.204:6837/3441932736":"2024-09-12T00:31:44.277804+0000","172.21.15.31:6833/3721481399":"2024-09-12T00:56:28.030054+0000","172.21.15.31:6833/2686855151":"2024-09-12T00:32:24.285279+0000","172.21.15.204:6837/1082799504":"2024-09-12T00:47:59.449917+0000","172.21.15.204:6834/1981755707":"2024-09-12T00:52:39.517968+0000","172.21.15.204:6836/645869523":"2024-09-12T00:35:54.319031+0000","172.21.15.31:6835/3925687919":"2024-09-12T00:41:09.376771+0000","172.21.15.204:6835/558470224":"2024-09-12T00:34:09.302673+0000","172.21.15.204:6835/1801149506":"2024-09-12T00:32:54.290770+0000","172.21.15.31:6833/4202305151":"2024-09-12T00:35:09.311954+0000","172.21.15.204:6836/3441932736":"2024-09-12T00:31:44.277804+0000","172.21.15.204:6836/1082799504":"2024-09-12T00:47:59.449917+0000","172.21.15.31:6833/3242133790":"2024-09-12T00:38:14.342154+0000","172.21.15.31:6834/3881548149":"2024-09-12T00:40:39.371058+0000","172.21.15.204:6834/1801149506":"2024-09-12T00:32:54.290770+0000","172.21.15.31:6835/4202305151":"2024-09-12T00:35:09.311954+0000","172.21.15.204:6836/2954547253":"2024-09-12T00:33:34.297314+0000","172.21.15.204:6834/1348312176":"2024-09-12T00:45:14.418207+0000","172.21.15.204:6834/3444890210":"2024-09-12T00:37:16.369607+0000","172.21.15.204:6836/2689217598":"2024-09-12T00:56:14.574459+0000","172.21.15.31:6833/3925687919":"2024-09-12T00:41:09.376771+0000","172.21.15.31:6835/4011734390":"2024-09-12T00:37:09.330669+0000","172.21.15.31:6835/1396366029":"2024-09-12T00:45:59.426330+0000","172.21.15.204:6837/2954547253":"2024-09-12T00:33:34.297314+0000","172.21.15.204:6834/558470224":"2024-09-12T00:34:09.302673+0000","172.21.15.31:6835/1855324855":"2024-09-12T00:52:49.521133+0000","172.21.15.204:6835/98007221":"2024-09-12T00:41:49.383893+0000","172.21.15.31:6834/1396366029":"2024-09-12T00:45:59.426330+0000","172.21.15.31:6832/2909466339":"2024-09-12T00:43:22.735779+0000","172.21.15.204:6837/2630216714":"2024-09-12T00:52:19.506830+0000","172.21.15.204:6834/1575320768":"2024-09-12T00:44:14.407953+0000","172.21.15.204:6836/2630216714":"2024-09-12T00:52:19.506830+0000","172.21.15.31:6835/2686855151":"2024-09-12T00:32:24.285279+0000","172.21.15.204:6835/1575320768":"2024-09-12T00:44:14.407953+0000","172.21.15.204:6835/1348312176":"2024-09-12T00:45:14.418207+0000","172.21.15.204:6837/3513641573":"2024-09-12T00:45:26.774916+0000","172.21.15.31:6833/4205320530":"2024-09-12T00:43:19.398942+0000","172.21.15.204:6834/2113744259":"2024-09-12T00:47:19.441133+0000","172.21.15.204:6835/2113744259":"2024-09-12T00:47:19.441133+0000","172.21.15.31:6835/2323279595":"2024-09-12T00:44:44.412976+0000","172.21.15.204:6836/3513641573":"2024-09-12T00:45:26.774916+0000","172.21.15.31:6834/2163558126":"2024-09-12T00:48:14.453181+0000","172.21.15.204:6834/634556689":"2024-09-12T00:49:34.468674+0000","172.21.15.204:6835/634556689":"2024-09-12T00:49:34.468674+0000","172.21.15.31:6835/3725011623":"2024-09-12T00:50:23.114454+0000","172.21.15.31:6835/3242133790":"2024-09-12T00:38:14.342154+0000","172.21.15.31:6835/4205320530":"2024-09-12T00:43:19.398942+0000","172.21.15.204:6836/4164016077":"2024-09-12T00:50:14.476294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 -- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7598077f70 msgr2=0x7f759807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7598077f70 0x7f759807a430 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f75ac006c30 tx=0x7f75ac0067a0 comp rx=0 tx=0).stop 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 -- 172.21.15.31:0/1926838046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b80aaa50 msgr2=0x7f75b81de170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b80aaa50 0x7f75b81de170 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f75a4011a10 tx=0x7f75a4011ed0 comp rx=0 tx=0).stop 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 -- 172.21.15.31:0/1926838046 shutdown_connections 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7598077f70 0x7f759807a430 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f75b814cc90 0x7f75b81de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75b80aaa50 0x7f75b81de170 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 --2- 172.21.15.31:0/1926838046 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f75a00051c0 0x7f75b814c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 -- 172.21.15.31:0/1926838046 >> 172.21.15.31:0/1926838046 conn(0x7f75b80a9750 msgr2=0x7f75b8150330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:48.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 -- 172.21.15.31:0/1926838046 shutdown_connections 2024-09-11T00:56:48.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:48.214+0000 7f75c0000640 1 -- 172.21.15.31:0/1926838046 wait complete. 2024-09-11T00:56:49.234 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T00:56:49.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T00:56:49.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.409+0000 7f2964800640 1 -- 172.21.15.31:0/3262923962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295c0c23a0 msgr2=0x7f295c0c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.409+0000 7f2964800640 1 --2- 172.21.15.31:0/3262923962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295c0c23a0 0x7f295c0c30c0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f294c005fd0 tx=0x7f294c02a670 comp rx=0 tx=0).stop 2024-09-11T00:56:49.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.409+0000 7f2964800640 1 -- 172.21.15.31:0/3262923962 shutdown_connections 2024-09-11T00:56:49.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.409+0000 7f2964800640 1 --2- 172.21.15.31:0/3262923962 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f295c1518b0 0x7f295c0c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.409+0000 7f2964800640 1 --2- 172.21.15.31:0/3262923962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295c0c23a0 0x7f295c0c30c0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.409+0000 7f2964800640 1 -- 172.21.15.31:0/3262923962 >> 172.21.15.31:0/3262923962 conn(0x7f295c0a9750 msgr2=0x7f295c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:49.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.409+0000 7f2964800640 1 -- 172.21.15.31:0/3262923962 shutdown_connections 2024-09-11T00:56:49.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.409+0000 7f2964800640 1 -- 172.21.15.31:0/3262923962 wait complete. 2024-09-11T00:56:49.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.411+0000 7f2964800640 1 Processor -- start 2024-09-11T00:56:49.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.411+0000 7f2964800640 1 -- start start 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.412+0000 7f2964800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295c072480 0x7f295c06f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.412+0000 7f2964800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f295c0c23a0 0x7f295c1efad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.412+0000 7f2964800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f295c1518b0 0x7f295c1e1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.412+0000 7f2964800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f295c074200 con 0x7f295c072480 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.412+0000 7f2964800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f295c074080 con 0x7f295c0c23a0 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.412+0000 7f2964800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f295c072860 con 0x7f295c1518b0 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.412+0000 7f2961e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295c072480 0x7f295c06f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.412+0000 7f2961e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295c072480 0x7f295c06f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38516/0 (socket says 172.21.15.31:38516) 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.412+0000 7f2961e00640 1 -- 172.21.15.31:0/3302057806 learned_addr learned my addr 172.21.15.31:0/3302057806 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2962800640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f295c1518b0 0x7f295c1e1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2961400640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f295c0c23a0 0x7f295c1efad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:49.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2961e00640 1 -- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f295c1518b0 msgr2=0x7f295c1e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2961e00640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f295c1518b0 0x7f295c1e1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2961e00640 1 -- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f295c0c23a0 msgr2=0x7f295c1efad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2961e00640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f295c0c23a0 0x7f295c1efad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2961e00640 1 -- 172.21.15.31:0/3302057806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295c14b1f0 con 0x7f295c072480 2024-09-11T00:56:49.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2962800640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f295c1518b0 0x7f295c1e1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:49.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2961400640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f295c0c23a0 0x7f295c1efad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:49.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2961e00640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295c072480 0x7f295c06f5a0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f295000c530 tx=0x7f295000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:49.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2956000640 1 -- 172.21.15.31:0/3302057806 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29500052d0 con 0x7f295c072480 2024-09-11T00:56:49.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2956000640 1 -- 172.21.15.31:0/3302057806 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2950005470 con 0x7f295c072480 2024-09-11T00:56:49.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.413+0000 7f2956000640 1 -- 172.21.15.31:0/3302057806 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950012870 con 0x7f295c072480 2024-09-11T00:56:49.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.414+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f295c0c6970 con 0x7f295c072480 2024-09-11T00:56:49.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.414+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f295c14aa70 con 0x7f295c072480 2024-09-11T00:56:49.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.415+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f295c0c3b90 con 0x7f295c072480 2024-09-11T00:56:49.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.416+0000 7f2956000640 1 -- 172.21.15.31:0/3302057806 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f29500035f0 con 0x7f295c072480 2024-09-11T00:56:49.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.417+0000 7f2956000640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2940077fa0 0x7f294007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:49.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.417+0000 7f2956000640 1 -- 172.21.15.31:0/3302057806 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f295009a8d0 con 0x7f295c072480 2024-09-11T00:56:49.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.417+0000 7f2956000640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] conn(0x7f29400846c0 0x7f2940086b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:49.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.417+0000 7f2956000640 1 -- 172.21.15.31:0/3302057806 --> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2950013e70 con 0x7f29400846c0 2024-09-11T00:56:49.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.417+0000 7f2956000640 1 -- 172.21.15.31:0/3302057806 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f295009f2a0 con 0x7f295c072480 2024-09-11T00:56:49.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.417+0000 7f2961400640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2940077fa0 0x7f294007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:49.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.417+0000 7f2962800640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] conn(0x7f29400846c0 0x7f2940086b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:49.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.418+0000 7f2962800640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] conn(0x7f29400846c0 0x7f2940086b60 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:49.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.418+0000 7f2961400640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2940077fa0 0x7f294007a460 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f294c033000 tx=0x7f294c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:49.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.419+0000 7f2956000640 1 -- 172.21.15.31:0/3302057806 <== osd.0 v2:172.21.15.31:6808/2913875274 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2950013e70 con 0x7f29400846c0 2024-09-11T00:56:49.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.432+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 --> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f295c150a90 con 0x7f29400846c0 2024-09-11T00:56:49.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.432+0000 7f2956000640 1 -- 172.21.15.31:0/3302057806 <== osd.0 v2:172.21.15.31:6808/2913875274 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f295c150a90 con 0x7f29400846c0 2024-09-11T00:56:49.434 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:56:49.434 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T00:56:49.434 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:56:49.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 >> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] conn(0x7f29400846c0 msgr2=0x7f2940086b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2964800640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.31:6808/2913875274,v1:172.21.15.31:6809/2913875274] conn(0x7f29400846c0 0x7f2940086b60 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2940077fa0 msgr2=0x7f294007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2964800640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2940077fa0 0x7f294007a460 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f294c033000 tx=0x7f294c004730 comp rx=0 tx=0).stop 2024-09-11T00:56:49.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295c072480 msgr2=0x7f295c06f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2964800640 1 --2- 172.21.15.31:0/3302057806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f295c072480 0x7f295c06f5a0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f295000c530 tx=0x7f295000c9f0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2962800640 1 -- 172.21.15.31:0/3302057806 reap_dead start 2024-09-11T00:56:49.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 shutdown_connections 2024-09-11T00:56:49.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 >> 172.21.15.31:0/3302057806 conn(0x7f295c0a9750 msgr2=0x7f295c0c0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:49.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 shutdown_connections 2024-09-11T00:56:49.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.433+0000 7f2964800640 1 -- 172.21.15.31:0/3302057806 wait complete. 2024-09-11T00:56:49.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T00:56:49.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.609+0000 7f1345400640 1 -- 172.21.15.31:0/2594249407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13401518b0 msgr2=0x7f13400c37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.609+0000 7f1345400640 1 --2- 172.21.15.31:0/2594249407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13401518b0 0x7f13400c37d0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f1338009830 tx=0x7f133802a640 comp rx=0 tx=0).stop 2024-09-11T00:56:49.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.609+0000 7f1345400640 1 -- 172.21.15.31:0/2594249407 shutdown_connections 2024-09-11T00:56:49.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.609+0000 7f1345400640 1 --2- 172.21.15.31:0/2594249407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13401518b0 0x7f13400c37d0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.609+0000 7f1345400640 1 -- 172.21.15.31:0/2594249407 >> 172.21.15.31:0/2594249407 conn(0x7f13400a9750 msgr2=0x7f13400a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:49.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.609+0000 7f1345400640 1 -- 172.21.15.31:0/2594249407 shutdown_connections 2024-09-11T00:56:49.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.609+0000 7f1345400640 1 -- 172.21.15.31:0/2594249407 wait complete. 2024-09-11T00:56:49.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.611+0000 7f1345400640 1 Processor -- start 2024-09-11T00:56:49.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.611+0000 7f1345400640 1 -- start start 2024-09-11T00:56:49.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.612+0000 7f1345400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1340072480 0x7f134006e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:49.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.612+0000 7f1345400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13400c23a0 0x7f13400c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:49.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.612+0000 7f1345400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13401518b0 0x7f1340145a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:49.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.612+0000 7f1345400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1340074080 con 0x7f13401518b0 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.612+0000 7f1345400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13400c3d50 con 0x7f1340072480 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.612+0000 7f1345400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f1340074200 con 0x7f13400c23a0 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.612+0000 7f133f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13401518b0 0x7f1340145a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.612+0000 7f133f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13401518b0 0x7f1340145a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38538/0 (socket says 172.21.15.31:38538) 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.612+0000 7f133f200640 1 -- 172.21.15.31:0/1115184496 learned_addr learned my addr 172.21.15.31:0/1115184496 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f133e800640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1340072480 0x7f134006e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f1335e00640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13400c23a0 0x7f13400c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f133f200640 1 -- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13400c23a0 msgr2=0x7f13400c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f133f200640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13400c23a0 0x7f13400c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f133f200640 1 -- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1340072480 msgr2=0x7f134006e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f133f200640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1340072480 0x7f134006e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f133f200640 1 -- 172.21.15.31:0/1115184496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13401ec5c0 con 0x7f13401518b0 2024-09-11T00:56:49.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f1335e00640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13400c23a0 0x7f13400c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:56:49.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f133f200640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13401518b0 0x7f1340145a10 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f133802ac30 tx=0x7f1338032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:49.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f1337400640 1 -- 172.21.15.31:0/1115184496 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338040070 con 0x7f13401518b0 2024-09-11T00:56:49.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f1337400640 1 -- 172.21.15.31:0/1115184496 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f133802ae60 con 0x7f13401518b0 2024-09-11T00:56:49.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f1337400640 1 -- 172.21.15.31:0/1115184496 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1338005280 con 0x7f13401518b0 2024-09-11T00:56:49.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f1345400640 1 -- 172.21.15.31:0/1115184496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1340147810 con 0x7f13401518b0 2024-09-11T00:56:49.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.613+0000 7f1345400640 1 -- 172.21.15.31:0/1115184496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13400701c0 con 0x7f13401518b0 2024-09-11T00:56:49.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.615+0000 7f1345400640 1 -- 172.21.15.31:0/1115184496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13400a91a0 con 0x7f13401518b0 2024-09-11T00:56:49.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.616+0000 7f1337400640 1 -- 172.21.15.31:0/1115184496 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f1338005420 con 0x7f13401518b0 2024-09-11T00:56:49.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.617+0000 7f1337400640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f131c077f70 0x7f131c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:49.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.617+0000 7f1337400640 1 -- 172.21.15.31:0/1115184496 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f13380bbec0 con 0x7f13401518b0 2024-09-11T00:56:49.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.617+0000 7f133e800640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f131c077f70 0x7f131c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:49.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.618+0000 7f133e800640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f131c077f70 0x7f131c07a430 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f13300046b0 tx=0x7f1330009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:49.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.621+0000 7f1337400640 1 -- 172.21.15.31:0/1115184496 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1338083a80 con 0x7f13401518b0 2024-09-11T00:56:49.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.888+0000 7f1345400640 1 -- 172.21.15.31:0/1115184496 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f13401475c0 con 0x7f131c077f70 2024-09-11T00:56:49.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.889+0000 7f1337400640 1 -- 172.21.15.31:0/1115184496 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f13401475c0 con 0x7f131c077f70 2024-09-11T00:56:49.891 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T00:56:49.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.891+0000 7f1345400640 1 -- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f131c077f70 msgr2=0x7f131c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.891+0000 7f1345400640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f131c077f70 0x7f131c07a430 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f13300046b0 tx=0x7f1330009040 comp rx=0 tx=0).stop 2024-09-11T00:56:49.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.891+0000 7f1345400640 1 -- 172.21.15.31:0/1115184496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13401518b0 msgr2=0x7f1340145a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:49.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.891+0000 7f1345400640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13401518b0 0x7f1340145a10 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f133802ac30 tx=0x7f1338032d60 comp rx=0 tx=0).stop 2024-09-11T00:56:49.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.891+0000 7f1345400640 1 -- 172.21.15.31:0/1115184496 shutdown_connections 2024-09-11T00:56:49.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.891+0000 7f1345400640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f131c077f70 0x7f131c07a430 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.891+0000 7f1345400640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13401518b0 0x7f1340145a10 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.891+0000 7f1345400640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f13400c23a0 0x7f13400c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.891+0000 7f1345400640 1 --2- 172.21.15.31:0/1115184496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1340072480 0x7f134006e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:49.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.891+0000 7f1345400640 1 -- 172.21.15.31:0/1115184496 >> 172.21.15.31:0/1115184496 conn(0x7f13400a9750 msgr2=0x7f134014ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:49.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.892+0000 7f1345400640 1 -- 172.21.15.31:0/1115184496 shutdown_connections 2024-09-11T00:56:49.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:49.892+0000 7f1345400640 1 -- 172.21.15.31:0/1115184496 wait complete. 2024-09-11T00:56:49.909 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T00:56:49.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T00:56:50.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.073+0000 7fdcbb000640 1 -- 172.21.15.31:0/3586140878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcb40c62e0 msgr2=0x7fdcb40ac6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.073+0000 7fdcbb000640 1 --2- 172.21.15.31:0/3586140878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcb40c62e0 0x7fdcb40ac6d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fdcac005fd0 tx=0x7fdcac02a8b0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.073+0000 7fdcbb000640 1 -- 172.21.15.31:0/3586140878 shutdown_connections 2024-09-11T00:56:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.073+0000 7fdcbb000640 1 --2- 172.21.15.31:0/3586140878 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcb40c62e0 0x7fdcb40ac6d0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.073+0000 7fdcbb000640 1 -- 172.21.15.31:0/3586140878 >> 172.21.15.31:0/3586140878 conn(0x7fdcb406caf0 msgr2=0x7fdcb406ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.073+0000 7fdcbb000640 1 -- 172.21.15.31:0/3586140878 shutdown_connections 2024-09-11T00:56:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.073+0000 7fdcbb000640 1 -- 172.21.15.31:0/3586140878 wait complete. 2024-09-11T00:56:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.075+0000 7fdcbb000640 1 Processor -- start 2024-09-11T00:56:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.075+0000 7fdcbb000640 1 -- start start 2024-09-11T00:56:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcbb000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcb40c1db0 0x7fdcb414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcbb000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcb40c62e0 0x7fdcb41e0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcbb000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcb41511a0 0x7fdcb414b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcbb000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb40acc10 con 0x7fdcb40c62e0 2024-09-11T00:56:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcbb000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb4154b70 con 0x7fdcb41511a0 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcbb000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb4151580 con 0x7fdcb40c1db0 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcb9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcb40c62e0 0x7fdcb41e0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcb9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcb40c62e0 0x7fdcb41e0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38552/0 (socket says 172.21.15.31:38552) 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcb9200640 1 -- 172.21.15.31:0/1150080707 learned_addr learned my addr 172.21.15.31:0/1150080707 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcb9c00640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcb40c1db0 0x7fdcb414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.076+0000 7fdcba600640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcb41511a0 0x7fdcb414b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcb9200640 1 -- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcb40c1db0 msgr2=0x7fdcb414cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcb9200640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fdcb40c1db0 0x7fdcb414cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcb9200640 1 -- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcb41511a0 msgr2=0x7fdcb414b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcb9200640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcb41511a0 0x7fdcb414b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcb9200640 1 -- 172.21.15.31:0/1150080707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb4151c80 con 0x7fdcb40c62e0 2024-09-11T00:56:50.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcb9200640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcb40c62e0 0x7fdcb41e0b10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fdca000c490 tx=0x7fdca000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:50.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcb1600640 1 -- 172.21.15.31:0/1150080707 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca0016020 con 0x7fdcb40c62e0 2024-09-11T00:56:50.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcb1600640 1 -- 172.21.15.31:0/1150080707 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdca0012940 con 0x7fdcb40c62e0 2024-09-11T00:56:50.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcb1600640 1 -- 172.21.15.31:0/1150080707 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca0012d90 con 0x7fdcb40c62e0 2024-09-11T00:56:50.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb4151f60 con 0x7fdcb40c62e0 2024-09-11T00:56:50.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.077+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb4155060 con 0x7fdcb40c62e0 2024-09-11T00:56:50.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.078+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdcb4152160 con 0x7fdcb40c62e0 2024-09-11T00:56:50.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.079+0000 7fdcb1600640 1 -- 172.21.15.31:0/1150080707 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fdca001a3b0 con 0x7fdcb40c62e0 2024-09-11T00:56:50.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.080+0000 7fdcb1600640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc94077fa0 0x7fdc9407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.080+0000 7fdcb1600640 1 -- 172.21.15.31:0/1150080707 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fdca009ac80 con 0x7fdcb40c62e0 2024-09-11T00:56:50.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.081+0000 7fdcb1600640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] conn(0x7fdc940846c0 0x7fdc94086b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.081+0000 7fdcb1600640 1 -- 172.21.15.31:0/1150080707 --> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc94087230 con 0x7fdc940846c0 2024-09-11T00:56:50.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.081+0000 7fdcb1600640 1 -- 172.21.15.31:0/1150080707 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fdca00a1020 con 0x7fdcb40c62e0 2024-09-11T00:56:50.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.081+0000 7fdcb9c00640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc94077fa0 0x7fdc9407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.081+0000 7fdcba600640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] conn(0x7fdc940846c0 0x7fdc94086b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.082+0000 7fdcb9c00640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc94077fa0 0x7fdc9407a460 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fdca80015e0 tx=0x7fdca8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:50.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.082+0000 7fdcba600640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] conn(0x7fdc940846c0 0x7fdc94086b60 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:50.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.082+0000 7fdcb1600640 1 -- 172.21.15.31:0/1150080707 <== osd.1 v2:172.21.15.31:6800/3912540156 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc94087230 con 0x7fdc940846c0 2024-09-11T00:56:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.096+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 --> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdcb41580e0 con 0x7fdc940846c0 2024-09-11T00:56:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.096+0000 7fdcb1600640 1 -- 172.21.15.31:0/1150080707 <== osd.1 v2:172.21.15.31:6800/3912540156 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdcac031440 con 0x7fdc940846c0 2024-09-11T00:56:50.098 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:56:50.098 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T00:56:50.098 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:56:50.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.096+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 >> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] conn(0x7fdc940846c0 msgr2=0x7fdc94086b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.096+0000 7fdcbb000640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.31:6800/3912540156,v1:172.21.15.31:6801/3912540156] conn(0x7fdc940846c0 0x7fdc94086b60 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.096+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc94077fa0 msgr2=0x7fdc9407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.096+0000 7fdcbb000640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fdc94077fa0 0x7fdc9407a460 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fdca80015e0 tx=0x7fdca8009040 comp rx=0 tx=0).stop 2024-09-11T00:56:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.096+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcb40c62e0 msgr2=0x7fdcb41e0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.096+0000 7fdcbb000640 1 --2- 172.21.15.31:0/1150080707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcb40c62e0 0x7fdcb41e0b10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fdca000c490 tx=0x7fdca000c950 comp rx=0 tx=0).stop 2024-09-11T00:56:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.096+0000 7fdcba600640 1 -- 172.21.15.31:0/1150080707 reap_dead start 2024-09-11T00:56:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.097+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 shutdown_connections 2024-09-11T00:56:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.097+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 >> 172.21.15.31:0/1150080707 conn(0x7fdcb406caf0 msgr2=0x7fdcb414dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.097+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 shutdown_connections 2024-09-11T00:56:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.097+0000 7fdcbb000640 1 -- 172.21.15.31:0/1150080707 wait complete. 2024-09-11T00:56:50.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T00:56:50.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.272+0000 7f59e6600640 1 -- 172.21.15.31:0/2002121453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e0153f20 msgr2=0x7f59e01516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.272+0000 7f59e6600640 1 --2- 172.21.15.31:0/2002121453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e0153f20 0x7f59e01516e0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f59d8005fd0 tx=0x7f59d802a940 comp rx=0 tx=0).stop 2024-09-11T00:56:50.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.272+0000 7f59e6600640 1 -- 172.21.15.31:0/2002121453 shutdown_connections 2024-09-11T00:56:50.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.272+0000 7f59e6600640 1 --2- 172.21.15.31:0/2002121453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e0153f20 0x7f59e01516e0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.272+0000 7f59e6600640 1 --2- 172.21.15.31:0/2002121453 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59e0072480 0x7f59e01511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.272+0000 7f59e6600640 1 -- 172.21.15.31:0/2002121453 >> 172.21.15.31:0/2002121453 conn(0x7f59e00a9750 msgr2=0x7f59e00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:50.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.272+0000 7f59e6600640 1 -- 172.21.15.31:0/2002121453 shutdown_connections 2024-09-11T00:56:50.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.272+0000 7f59e6600640 1 -- 172.21.15.31:0/2002121453 wait complete. 2024-09-11T00:56:50.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.274+0000 7f59e6600640 1 Processor -- start 2024-09-11T00:56:50.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.274+0000 7f59e6600640 1 -- start start 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59e6600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59e0072480 0x7f59e00c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59e6600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59e00c23a0 0x7f59e00c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59e6600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e0153f20 0x7f59e014a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59e6600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f59e0074200 con 0x7f59e0153f20 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59e6600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f59e0074080 con 0x7f59e0072480 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59e6600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f59e0072860 con 0x7f59e00c23a0 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59dfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e0153f20 0x7f59e014a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59dfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e0153f20 0x7f59e014a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38560/0 (socket says 172.21.15.31:38560) 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59dfe00640 1 -- 172.21.15.31:0/2649078366 learned_addr learned my addr 172.21.15.31:0/2649078366 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59df400640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59e0072480 0x7f59e00c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59d6a00640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59e00c23a0 0x7f59e00c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59dfe00640 1 -- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59e00c23a0 msgr2=0x7f59e00c6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59dfe00640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59e00c23a0 0x7f59e00c6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59dfe00640 1 -- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59e0072480 msgr2=0x7f59e00c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.275+0000 7f59dfe00640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59e0072480 0x7f59e00c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.276+0000 7f59dfe00640 1 -- 172.21.15.31:0/2649078366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e00c69c0 con 0x7f59e0153f20 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.276+0000 7f59dfe00640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e0153f20 0x7f59e014a690 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f59d8009cd0 tx=0x7f59d8031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.276+0000 7f59d7e00640 1 -- 172.21.15.31:0/2649078366 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d803e070 con 0x7f59e0153f20 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.276+0000 7f59d7e00640 1 -- 172.21.15.31:0/2649078366 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59d8004540 con 0x7f59e0153f20 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.276+0000 7f59d7e00640 1 -- 172.21.15.31:0/2649078366 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d8030430 con 0x7f59e0153f20 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.276+0000 7f59e6600640 1 -- 172.21.15.31:0/2649078366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e0144dc0 con 0x7f59e0153f20 2024-09-11T00:56:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.276+0000 7f59e6600640 1 -- 172.21.15.31:0/2649078366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e014abd0 con 0x7f59e0153f20 2024-09-11T00:56:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.277+0000 7f59e6600640 1 -- 172.21.15.31:0/2649078366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e01511a0 con 0x7f59e0153f20 2024-09-11T00:56:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.278+0000 7f59d7e00640 1 -- 172.21.15.31:0/2649078366 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f59d802f070 con 0x7f59e0153f20 2024-09-11T00:56:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.279+0000 7f59d7e00640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f59bc077fa0 0x7f59bc07a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.279+0000 7f59d7e00640 1 -- 172.21.15.31:0/2649078366 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f59d80bc420 con 0x7f59e0153f20 2024-09-11T00:56:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.280+0000 7f59df400640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f59bc077fa0 0x7f59bc07a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.281+0000 7f59df400640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f59bc077fa0 0x7f59bc07a460 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f59d0005fd0 tx=0x7f59d0005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:50.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.283+0000 7f59d7e00640 1 -- 172.21.15.31:0/2649078366 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f59e00c69c0 con 0x7f59e0153f20 2024-09-11T00:56:50.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.478+0000 7f59e6600640 1 -- 172.21.15.31:0/2649078366 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f59e00c2160 con 0x7f59bc077fa0 2024-09-11T00:56:50.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.479+0000 7f59d7e00640 1 -- 172.21.15.31:0/2649078366 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f59e00c2160 con 0x7f59bc077fa0 2024-09-11T00:56:50.480 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T00:56:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.481+0000 7f59e6600640 1 -- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f59bc077fa0 msgr2=0x7f59bc07a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.481+0000 7f59e6600640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f59bc077fa0 0x7f59bc07a460 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f59d0005fd0 tx=0x7f59d0005850 comp rx=0 tx=0).stop 2024-09-11T00:56:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.481+0000 7f59e6600640 1 -- 172.21.15.31:0/2649078366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e0153f20 msgr2=0x7f59e014a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.481+0000 7f59e6600640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e0153f20 0x7f59e014a690 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f59d8009cd0 tx=0x7f59d8031ea0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.481+0000 7f59e6600640 1 -- 172.21.15.31:0/2649078366 shutdown_connections 2024-09-11T00:56:50.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.481+0000 7f59e6600640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f59bc077fa0 0x7f59bc07a460 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.481+0000 7f59e6600640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e0153f20 0x7f59e014a690 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.481+0000 7f59e6600640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f59e00c23a0 0x7f59e00c6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.481+0000 7f59e6600640 1 --2- 172.21.15.31:0/2649078366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59e0072480 0x7f59e00c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.481+0000 7f59e6600640 1 -- 172.21.15.31:0/2649078366 >> 172.21.15.31:0/2649078366 conn(0x7f59e00a9750 msgr2=0x7f59e0152050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:50.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.482+0000 7f59e6600640 1 -- 172.21.15.31:0/2649078366 shutdown_connections 2024-09-11T00:56:50.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.482+0000 7f59e6600640 1 -- 172.21.15.31:0/2649078366 wait complete. 2024-09-11T00:56:50.500 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T00:56:50.500 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T00:56:50.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.664+0000 7f3407c00640 1 -- 172.21.15.31:0/3414149613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3400072480 msgr2=0x7f34001511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.664+0000 7f3407c00640 1 --2- 172.21.15.31:0/3414149613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3400072480 0x7f34001511a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f34000c5500 tx=0x7f33f002a6d0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.664+0000 7f3407c00640 1 -- 172.21.15.31:0/3414149613 shutdown_connections 2024-09-11T00:56:50.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.664+0000 7f3407c00640 1 --2- 172.21.15.31:0/3414149613 >> v2:172.21.15.204:3300/0 conn(0x7f3400153f20 0x7f34001516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.664+0000 7f3407c00640 1 --2- 172.21.15.31:0/3414149613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3400072480 0x7f34001511a0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.664+0000 7f3407c00640 1 -- 172.21.15.31:0/3414149613 >> 172.21.15.31:0/3414149613 conn(0x7f34000a9750 msgr2=0x7f34000a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:50.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.664+0000 7f3407c00640 1 -- 172.21.15.31:0/3414149613 shutdown_connections 2024-09-11T00:56:50.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.664+0000 7f3407c00640 1 -- 172.21.15.31:0/3414149613 wait complete. 2024-09-11T00:56:50.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.666+0000 7f3407c00640 1 Processor -- start 2024-09-11T00:56:50.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.666+0000 7f3407c00640 1 -- start start 2024-09-11T00:56:50.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3407c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3400072480 0x7f34000c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3407c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34000c23a0 0x7f340014ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3407c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3400153f20 0x7f340014b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3407c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3400074200 con 0x7f3400153f20 2024-09-11T00:56:50.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3407c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3400074080 con 0x7f3400072480 2024-09-11T00:56:50.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3407c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3400072860 con 0x7f34000c23a0 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3405c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3400153f20 0x7f340014b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3405200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3400072480 0x7f34000c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3405c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3400153f20 0x7f340014b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38578/0 (socket says 172.21.15.31:38578) 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3405c00640 1 -- 172.21.15.31:0/661576206 learned_addr learned my addr 172.21.15.31:0/661576206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f33f7e00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34000c23a0 0x7f340014ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3405c00640 1 -- 172.21.15.31:0/661576206 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34000c23a0 msgr2=0x7f340014ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3405c00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f34000c23a0 0x7f340014ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3405c00640 1 -- 172.21.15.31:0/661576206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3400072480 msgr2=0x7f34000c4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3405c00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3400072480 0x7f34000c4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.667+0000 7f3405c00640 1 -- 172.21.15.31:0/661576206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34000c4b20 con 0x7f3400153f20 2024-09-11T00:56:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.668+0000 7f3405c00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3400153f20 0x7f340014b1c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f33f800e990 tx=0x7f33f800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:50.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.668+0000 7f33fd600640 1 -- 172.21.15.31:0/661576206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8004d20 con 0x7f3400153f20 2024-09-11T00:56:50.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.668+0000 7f33fd600640 1 -- 172.21.15.31:0/661576206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33f800cb00 con 0x7f3400153f20 2024-09-11T00:56:50.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.668+0000 7f33fd600640 1 -- 172.21.15.31:0/661576206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8005020 con 0x7f3400153f20 2024-09-11T00:56:50.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.668+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34001ecf30 con 0x7f3400153f20 2024-09-11T00:56:50.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.668+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34001ecc40 con 0x7f3400153f20 2024-09-11T00:56:50.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.669+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3400073630 con 0x7f3400153f20 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.670+0000 7f33fd600640 1 -- 172.21.15.31:0/661576206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f33f800f040 con 0x7f3400153f20 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.671+0000 7f33fd600640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33dc077f70 0x7f33dc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.672+0000 7f33fd600640 1 -- 172.21.15.31:0/661576206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f33f809b2d0 con 0x7f3400153f20 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.672+0000 7f33fd600640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f33dc084690 0x7f33dc086b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.672+0000 7f33fd600640 1 -- 172.21.15.31:0/661576206 --> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33dc087200 con 0x7f33dc084690 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.672+0000 7f33fd600640 1 -- 172.21.15.31:0/661576206 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f33f80a2050 con 0x7f3400153f20 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.672+0000 7f3405200640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33dc077f70 0x7f33dc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.672+0000 7f33f7e00640 1 -- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f33dc084690 msgr2=0x7f33dc086b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.672+0000 7f33f7e00640 1 -- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f33dc084690 msgr2=0x7f33dc086b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.672+0000 7f33f7e00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f33dc084690 0x7f33dc086b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-11T00:56:50.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.672+0000 7f33f7e00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f33dc084690 0x7f33dc086b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T00:56:50.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.673+0000 7f3405200640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33dc077f70 0x7f33dc07a430 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f33f0033000 tx=0x7f33f0004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:50.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.874+0000 7f33f7e00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f33dc084690 0x7f33dc086b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:50.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:50.874+0000 7f33f7e00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f33dc084690 0x7f33dc086b30 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:51.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.765+0000 7f33fd600640 1 -- 172.21.15.31:0/661576206 <== osd.2 v2:172.21.15.31:6816/2777449146 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f33dc087200 con 0x7f33dc084690 2024-09-11T00:56:51.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.780+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 --> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f34001511a0 con 0x7f33dc084690 2024-09-11T00:56:51.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.781+0000 7f33fd600640 1 -- 172.21.15.31:0/661576206 <== osd.2 v2:172.21.15.31:6816/2777449146 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f34001511a0 con 0x7f33dc084690 2024-09-11T00:56:51.783 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:56:51.783 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T00:56:51.783 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.781+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f33dc084690 msgr2=0x7f33dc086b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.781+0000 7f3407c00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:6816/2777449146,v1:172.21.15.31:6817/2777449146] conn(0x7f33dc084690 0x7f33dc086b30 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.781+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33dc077f70 msgr2=0x7f33dc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.781+0000 7f3407c00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f33dc077f70 0x7f33dc07a430 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f33f0033000 tx=0x7f33f0004730 comp rx=0 tx=0).stop 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.781+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3400153f20 msgr2=0x7f340014b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.781+0000 7f3407c00640 1 --2- 172.21.15.31:0/661576206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3400153f20 0x7f340014b1c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f33f800e990 tx=0x7f33f800ee50 comp rx=0 tx=0).stop 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.782+0000 7f3405c00640 1 -- 172.21.15.31:0/661576206 reap_dead start 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.782+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 shutdown_connections 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.782+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 >> 172.21.15.31:0/661576206 conn(0x7f34000a9750 msgr2=0x7f34001529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.782+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 shutdown_connections 2024-09-11T00:56:51.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.782+0000 7f3407c00640 1 -- 172.21.15.31:0/661576206 wait complete. 2024-09-11T00:56:51.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T00:56:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.968+0000 7f27a3a00640 1 -- 172.21.15.31:0/2883003342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f279c0c23a0 msgr2=0x7f279c0c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.968+0000 7f27a3a00640 1 --2- 172.21.15.31:0/2883003342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f279c0c23a0 0x7f279c0c30c0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f2788009830 tx=0x7f278802a5a0 comp rx=0 tx=0).stop 2024-09-11T00:56:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.968+0000 7f27a3a00640 1 -- 172.21.15.31:0/2883003342 shutdown_connections 2024-09-11T00:56:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.968+0000 7f27a3a00640 1 --2- 172.21.15.31:0/2883003342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f279c0c23a0 0x7f279c0c30c0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.968+0000 7f27a3a00640 1 -- 172.21.15.31:0/2883003342 >> 172.21.15.31:0/2883003342 conn(0x7f279c0a9750 msgr2=0x7f279c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.968+0000 7f27a3a00640 1 -- 172.21.15.31:0/2883003342 shutdown_connections 2024-09-11T00:56:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.968+0000 7f27a3a00640 1 -- 172.21.15.31:0/2883003342 wait complete. 2024-09-11T00:56:51.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.969+0000 7f27a3a00640 1 Processor -- start 2024-09-11T00:56:51.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.970+0000 7f27a3a00640 1 -- start start 2024-09-11T00:56:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.970+0000 7f27a3a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f279c072480 0x7f279c1dea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.970+0000 7f27a3a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f279c0c23a0 0x7f279c143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.970+0000 7f27a3a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f279c1518b0 0x7f279c1438c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.970+0000 7f27a3a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f279c074080 con 0x7f279c1518b0 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.970+0000 7f27a3a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f279c0c3d50 con 0x7f279c0c23a0 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.970+0000 7f27a3a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f279c074200 con 0x7f279c072480 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.971+0000 7f27a1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f279c1518b0 0x7f279c1438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.971+0000 7f27a1000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f279c072480 0x7f279c1dea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.971+0000 7f279be00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f279c0c23a0 0x7f279c143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.971+0000 7f27a1a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f279c1518b0 0x7f279c1438c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38602/0 (socket says 172.21.15.31:38602) 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.971+0000 7f27a1a00640 1 -- 172.21.15.31:0/1786771508 learned_addr learned my addr 172.21.15.31:0/1786771508 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.971+0000 7f27a1a00640 1 -- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f279c072480 msgr2=0x7f279c1dea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.971+0000 7f27a1a00640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f279c072480 0x7f279c1dea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.971+0000 7f27a1a00640 1 -- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f279c0c23a0 msgr2=0x7f279c143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.971+0000 7f27a1a00640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f279c0c23a0 0x7f279c143380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.971+0000 7f27a1a00640 1 -- 172.21.15.31:0/1786771508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279c1ed500 con 0x7f279c1518b0 2024-09-11T00:56:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.972+0000 7f27a1a00640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f279c1518b0 0x7f279c1438c0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f279400c780 tx=0x7f279400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.972+0000 7f2798c00640 1 -- 172.21.15.31:0/1786771508 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794004b60 con 0x7f279c1518b0 2024-09-11T00:56:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.972+0000 7f2798c00640 1 -- 172.21.15.31:0/1786771508 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2794004df0 con 0x7f279c1518b0 2024-09-11T00:56:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.972+0000 7f2798c00640 1 -- 172.21.15.31:0/1786771508 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279400aa70 con 0x7f279c1518b0 2024-09-11T00:56:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.972+0000 7f27a3a00640 1 -- 172.21.15.31:0/1786771508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f279c1ed7e0 con 0x7f279c1518b0 2024-09-11T00:56:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.972+0000 7f27a3a00640 1 -- 172.21.15.31:0/1786771508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f279c1edda0 con 0x7f279c1518b0 2024-09-11T00:56:51.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.973+0000 7f27a3a00640 1 -- 172.21.15.31:0/1786771508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f279c153f20 con 0x7f279c1518b0 2024-09-11T00:56:51.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.974+0000 7f2798c00640 1 -- 172.21.15.31:0/1786771508 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2794023060 con 0x7f279c1518b0 2024-09-11T00:56:51.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.975+0000 7f2798c00640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f277c077f70 0x7f277c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:51.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.975+0000 7f2798c00640 1 -- 172.21.15.31:0/1786771508 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f279409ae10 con 0x7f279c1518b0 2024-09-11T00:56:51.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.976+0000 7f27a1000640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f277c077f70 0x7f277c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:51.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.976+0000 7f27a1000640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f277c077f70 0x7f277c07a430 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2790000f60 tx=0x7f2790009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:51.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:51.979+0000 7f2798c00640 1 -- 172.21.15.31:0/1786771508 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f279409e030 con 0x7f279c1518b0 2024-09-11T00:56:52.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.172+0000 7f27a3a00640 1 -- 172.21.15.31:0/1786771508 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f279c14e6a0 con 0x7f277c077f70 2024-09-11T00:56:52.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.173+0000 7f2798c00640 1 -- 172.21.15.31:0/1786771508 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f279c14e6a0 con 0x7f277c077f70 2024-09-11T00:56:52.175 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T00:56:52.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 -- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f277c077f70 msgr2=0x7f277c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f277c077f70 0x7f277c07a430 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2790000f60 tx=0x7f2790009040 comp rx=0 tx=0).stop 2024-09-11T00:56:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 -- 172.21.15.31:0/1786771508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f279c1518b0 msgr2=0x7f279c1438c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f279c1518b0 0x7f279c1438c0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f279400c780 tx=0x7f279400cc40 comp rx=0 tx=0).stop 2024-09-11T00:56:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 -- 172.21.15.31:0/1786771508 shutdown_connections 2024-09-11T00:56:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f277c077f70 0x7f277c07a430 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f279c1518b0 0x7f279c1438c0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f279c0c23a0 0x7f279c143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 --2- 172.21.15.31:0/1786771508 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f279c072480 0x7f279c1dea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 -- 172.21.15.31:0/1786771508 >> 172.21.15.31:0/1786771508 conn(0x7f279c0a9750 msgr2=0x7f279c14df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.175+0000 7f27a3a00640 1 -- 172.21.15.31:0/1786771508 shutdown_connections 2024-09-11T00:56:52.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.176+0000 7f27a3a00640 1 -- 172.21.15.31:0/1786771508 wait complete. 2024-09-11T00:56:52.194 INFO:tasks.ceph:Scrubbing osd.3 2024-09-11T00:56:52.195 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-11T00:56:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.357+0000 7f6335c00640 1 -- 172.21.15.31:0/1191577698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63301532c0 msgr2=0x7f63300c6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.357+0000 7f6335c00640 1 --2- 172.21.15.31:0/1191577698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63301532c0 0x7f63300c6f40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6320009830 tx=0x7f632002a380 comp rx=0 tx=0).stop 2024-09-11T00:56:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.357+0000 7f6335c00640 1 -- 172.21.15.31:0/1191577698 shutdown_connections 2024-09-11T00:56:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.357+0000 7f6335c00640 1 --2- 172.21.15.31:0/1191577698 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f633006ff40 0x7f63300c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.357+0000 7f6335c00640 1 --2- 172.21.15.31:0/1191577698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63301532c0 0x7f63300c6f40 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.357+0000 7f6335c00640 1 -- 172.21.15.31:0/1191577698 >> 172.21.15.31:0/1191577698 conn(0x7f63300a9750 msgr2=0x7f63300a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.357+0000 7f6335c00640 1 -- 172.21.15.31:0/1191577698 shutdown_connections 2024-09-11T00:56:52.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.357+0000 7f6335c00640 1 -- 172.21.15.31:0/1191577698 wait complete. 2024-09-11T00:56:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.359+0000 7f6335c00640 1 Processor -- start 2024-09-11T00:56:52.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.359+0000 7f6335c00640 1 -- start start 2024-09-11T00:56:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f6335c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f633006ff40 0x7f63301552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f6335c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63300c3f20 0x7f63300ac6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f6335c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63301532c0 0x7f63300acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f6335c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f63300a7300 con 0x7f63301532c0 2024-09-11T00:56:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f6335c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f633006f340 con 0x7f633006ff40 2024-09-11T00:56:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f6335c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f63300a7480 con 0x7f63300c3f20 2024-09-11T00:56:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f632f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63301532c0 0x7f63300acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f632f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63301532c0 0x7f63300acc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38606/0 (socket says 172.21.15.31:38606) 2024-09-11T00:56:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f632f400640 1 -- 172.21.15.31:0/179585839 learned_addr learned my addr 172.21.15.31:0/179585839 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f632e000640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63300c3f20 0x7f63300ac6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f632ea00640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f633006ff40 0x7f63301552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f632f400640 1 -- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63300c3f20 msgr2=0x7f63300ac6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f632f400640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f63300c3f20 0x7f63300ac6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f632f400640 1 -- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f633006ff40 msgr2=0x7f63301552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f632f400640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f633006ff40 0x7f63301552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.360+0000 7f632f400640 1 -- 172.21.15.31:0/179585839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63301567d0 con 0x7f63301532c0 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.361+0000 7f632f400640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63301532c0 0x7f63300acc20 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f632800caa0 tx=0x7f632800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.361+0000 7f6326a00640 1 -- 172.21.15.31:0/179585839 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6328017070 con 0x7f63301532c0 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.361+0000 7f6326a00640 1 -- 172.21.15.31:0/179585839 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f632800a6d0 con 0x7f63301532c0 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.361+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63301ed0f0 con 0x7f63301532c0 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.361+0000 7f6326a00640 1 -- 172.21.15.31:0/179585839 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632800ab20 con 0x7f63301532c0 2024-09-11T00:56:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.361+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63300c6020 con 0x7f63301532c0 2024-09-11T00:56:52.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.362+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6330156580 con 0x7f63301532c0 2024-09-11T00:56:52.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.363+0000 7f6326a00640 1 -- 172.21.15.31:0/179585839 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f632800acc0 con 0x7f63301532c0 2024-09-11T00:56:52.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.364+0000 7f6326a00640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6310077fa0 0x7f631007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.364+0000 7f6326a00640 1 -- 172.21.15.31:0/179585839 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f632809a730 con 0x7f63301532c0 2024-09-11T00:56:52.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.365+0000 7f6326a00640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] conn(0x7f63100846b0 0x7f6310086b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.365+0000 7f6326a00640 1 -- 172.21.15.31:0/179585839 --> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6328005800 con 0x7f63100846b0 2024-09-11T00:56:52.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.365+0000 7f6326a00640 1 -- 172.21.15.31:0/179585839 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f63280a1050 con 0x7f63301532c0 2024-09-11T00:56:52.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.365+0000 7f632ea00640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6310077fa0 0x7f631007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.365+0000 7f632e000640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] conn(0x7f63100846b0 0x7f6310086b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.365+0000 7f632e000640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] conn(0x7f63100846b0 0x7f6310086b50 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:52.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.365+0000 7f632ea00640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6310077fa0 0x7f631007a460 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6320034000 tx=0x7f6320004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:52.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.366+0000 7f6326a00640 1 -- 172.21.15.31:0/179585839 <== osd.3 v2:172.21.15.31:6824/4217184896 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6328005800 con 0x7f63100846b0 2024-09-11T00:56:52.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.379+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 --> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f63300c75f0 con 0x7f63100846b0 2024-09-11T00:56:52.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f6326a00640 1 -- 172.21.15.31:0/179585839 <== osd.3 v2:172.21.15.31:6824/4217184896 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f63300c75f0 con 0x7f63100846b0 2024-09-11T00:56:52.382 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:56:52.382 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T00:56:52.382 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:56:52.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 >> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] conn(0x7f63100846b0 msgr2=0x7f6310086b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f6335c00640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.31:6824/4217184896,v1:172.21.15.31:6825/4217184896] conn(0x7f63100846b0 0x7f6310086b50 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6310077fa0 msgr2=0x7f631007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f6335c00640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6310077fa0 0x7f631007a460 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6320034000 tx=0x7f6320004730 comp rx=0 tx=0).stop 2024-09-11T00:56:52.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63301532c0 msgr2=0x7f63300acc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f6335c00640 1 --2- 172.21.15.31:0/179585839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f63301532c0 0x7f63300acc20 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f632800caa0 tx=0x7f632800cf60 comp rx=0 tx=0).stop 2024-09-11T00:56:52.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f632f400640 1 -- 172.21.15.31:0/179585839 reap_dead start 2024-09-11T00:56:52.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 shutdown_connections 2024-09-11T00:56:52.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 >> 172.21.15.31:0/179585839 conn(0x7f63300a9750 msgr2=0x7f63300c57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:52.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.380+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 shutdown_connections 2024-09-11T00:56:52.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.381+0000 7f6335c00640 1 -- 172.21.15.31:0/179585839 wait complete. 2024-09-11T00:56:52.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-11T00:56:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.557+0000 7fb6b3200640 1 -- 172.21.15.31:0/3873480887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6ac0abde0 msgr2=0x7fb6ac156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.557+0000 7fb6b3200640 1 --2- 172.21.15.31:0/3873480887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6ac0abde0 0x7fb6ac156350 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac06e990 tx=0x7fb6a402aa30 comp rx=0 tx=0).stop 2024-09-11T00:56:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.557+0000 7fb6b3200640 1 -- 172.21.15.31:0/3873480887 shutdown_connections 2024-09-11T00:56:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.557+0000 7fb6b3200640 1 --2- 172.21.15.31:0/3873480887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6ac0abde0 0x7fb6ac156350 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.557+0000 7fb6b3200640 1 --2- 172.21.15.31:0/3873480887 >> v2:172.21.15.204:3300/0 conn(0x7fb6ac074f50 0x7fb6ac06db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.557+0000 7fb6b3200640 1 --2- 172.21.15.31:0/3873480887 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6ac074630 0x7fb6ac074a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.557+0000 7fb6b3200640 1 -- 172.21.15.31:0/3873480887 >> 172.21.15.31:0/3873480887 conn(0x7fb6ac0c8320 msgr2=0x7fb6ac0c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.557+0000 7fb6b3200640 1 -- 172.21.15.31:0/3873480887 shutdown_connections 2024-09-11T00:56:52.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.557+0000 7fb6b3200640 1 -- 172.21.15.31:0/3873480887 wait complete. 2024-09-11T00:56:52.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.558+0000 7fb6b3200640 1 Processor -- start 2024-09-11T00:56:52.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.559+0000 7fb6b3200640 1 -- start start 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.559+0000 7fb6b3200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6ac074630 0x7fb6ac1459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.559+0000 7fb6b3200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6ac074f50 0x7fb6ac13fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.559+0000 7fb6b3200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6ac0abde0 0x7fb6ac140240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.559+0000 7fb6b3200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac153b70 con 0x7fb6ac074f50 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.559+0000 7fb6b3200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac0c2d20 con 0x7fb6ac0abde0 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.559+0000 7fb6b3200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ac1441d0 con 0x7fb6ac074630 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6ab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6ac074f50 0x7fb6ac13fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6ab400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6ac074f50 0x7fb6ac13fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38622/0 (socket says 172.21.15.31:38622) 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6ab400640 1 -- 172.21.15.31:0/80300994 learned_addr learned my addr 172.21.15.31:0/80300994 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6abe00640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6ac074630 0x7fb6ac1459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6b1200640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6ac0abde0 0x7fb6ac140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6ab400640 1 -- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6ac074630 msgr2=0x7fb6ac1459a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6ab400640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6ac074630 0x7fb6ac1459a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6ab400640 1 -- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6ac0abde0 msgr2=0x7fb6ac140240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6ab400640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6ac0abde0 0x7fb6ac140240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6ab400640 1 -- 172.21.15.31:0/80300994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ac147070 con 0x7fb6ac074f50 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6b1200640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6ac0abde0 0x7fb6ac140240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.560+0000 7fb6ab400640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6ac074f50 0x7fb6ac13fd00 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fb69800d770 tx=0x7fb69800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.561+0000 7fb6a3e00640 1 -- 172.21.15.31:0/80300994 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698017070 con 0x7fb6ac074f50 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.561+0000 7fb6a3e00640 1 -- 172.21.15.31:0/80300994 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb698009d20 con 0x7fb6ac074f50 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.561+0000 7fb6a3e00640 1 -- 172.21.15.31:0/80300994 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6980106c0 con 0x7fb6ac074f50 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.561+0000 7fb6b3200640 1 -- 172.21.15.31:0/80300994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ac1f07e0 con 0x7fb6ac074f50 2024-09-11T00:56:52.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.561+0000 7fb6b3200640 1 -- 172.21.15.31:0/80300994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ac06cdd0 con 0x7fb6ac074f50 2024-09-11T00:56:52.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.562+0000 7fb6b3200640 1 -- 172.21.15.31:0/80300994 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6ac074a10 con 0x7fb6ac074f50 2024-09-11T00:56:52.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.563+0000 7fb6a3e00640 1 -- 172.21.15.31:0/80300994 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb698010860 con 0x7fb6ac074f50 2024-09-11T00:56:52.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.564+0000 7fb6a3e00640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb68c077f70 0x7fb68c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.564+0000 7fb6a3e00640 1 -- 172.21.15.31:0/80300994 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fb69809b110 con 0x7fb6ac074f50 2024-09-11T00:56:52.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.565+0000 7fb6abe00640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb68c077f70 0x7fb68c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.565+0000 7fb6abe00640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb68c077f70 0x7fb68c07a430 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac1521f0 tx=0x7fb69c00b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:52.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.568+0000 7fb6a3e00640 1 -- 172.21.15.31:0/80300994 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb698062620 con 0x7fb6ac074f50 2024-09-11T00:56:52.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.763+0000 7fb6b3200640 1 -- 172.21.15.31:0/80300994 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fb6ac1ec520 con 0x7fb68c077f70 2024-09-11T00:56:52.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.764+0000 7fb6a3e00640 1 -- 172.21.15.31:0/80300994 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb6ac1ec520 con 0x7fb68c077f70 2024-09-11T00:56:52.765 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 3 to deep-scrub 2024-09-11T00:56:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 -- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb68c077f70 msgr2=0x7fb68c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb68c077f70 0x7fb68c07a430 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac1521f0 tx=0x7fb69c00b2e0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 -- 172.21.15.31:0/80300994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6ac074f50 msgr2=0x7fb6ac13fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6ac074f50 0x7fb6ac13fd00 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fb69800d770 tx=0x7fb69800dc30 comp rx=0 tx=0).stop 2024-09-11T00:56:52.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 -- 172.21.15.31:0/80300994 shutdown_connections 2024-09-11T00:56:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb68c077f70 0x7fb68c07a430 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6ac0abde0 0x7fb6ac140240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6ac074f50 0x7fb6ac13fd00 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 --2- 172.21.15.31:0/80300994 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6ac074630 0x7fb6ac1459a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 -- 172.21.15.31:0/80300994 >> 172.21.15.31:0/80300994 conn(0x7fb6ac0c8320 msgr2=0x7fb6ac152590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 -- 172.21.15.31:0/80300994 shutdown_connections 2024-09-11T00:56:52.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.766+0000 7fb6b3200640 1 -- 172.21.15.31:0/80300994 wait complete. 2024-09-11T00:56:52.784 INFO:tasks.ceph:Scrubbing osd.4 2024-09-11T00:56:52.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-11T00:56:52.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.948+0000 7fab9d600640 1 -- 172.21.15.31:0/1879668100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab98154a80 msgr2=0x7fab9806d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.948+0000 7fab9d600640 1 --2- 172.21.15.31:0/1879668100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab98154a80 0x7fab9806d5e0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fab98151a30 tx=0x7fab8802a4d0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.948+0000 7fab9d600640 1 -- 172.21.15.31:0/1879668100 shutdown_connections 2024-09-11T00:56:52.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.948+0000 7fab9d600640 1 --2- 172.21.15.31:0/1879668100 >> v2:172.21.15.204:3300/0 conn(0x7fab9806db20 0x7fab98074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.948+0000 7fab9d600640 1 --2- 172.21.15.31:0/1879668100 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab98153e60 0x7fab98157ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.948+0000 7fab9d600640 1 --2- 172.21.15.31:0/1879668100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab98154a80 0x7fab9806d5e0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.948+0000 7fab9d600640 1 -- 172.21.15.31:0/1879668100 >> 172.21.15.31:0/1879668100 conn(0x7fab9806caf0 msgr2=0x7fab9806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:52.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.948+0000 7fab9d600640 1 -- 172.21.15.31:0/1879668100 shutdown_connections 2024-09-11T00:56:52.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.948+0000 7fab9d600640 1 -- 172.21.15.31:0/1879668100 wait complete. 2024-09-11T00:56:52.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.950+0000 7fab9d600640 1 Processor -- start 2024-09-11T00:56:52.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.950+0000 7fab9d600640 1 -- start start 2024-09-11T00:56:52.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab9d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9806db20 0x7fab981efde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab9d600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab98153e60 0x7fab981ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab9d600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab98154a80 0x7fab9814d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab9d600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab980728b0 con 0x7fab9806db20 2024-09-11T00:56:52.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab9d600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fab980bef10 con 0x7fab98154a80 2024-09-11T00:56:52.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab9d600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fab98072a30 con 0x7fab98153e60 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab97e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9806db20 0x7fab981efde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab97e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9806db20 0x7fab981efde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38634/0 (socket says 172.21.15.31:38634) 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab97e00640 1 -- 172.21.15.31:0/1689197767 learned_addr learned my addr 172.21.15.31:0/1689197767 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab97400640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab98153e60 0x7fab981ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab9cc00640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab98154a80 0x7fab9814d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab97e00640 1 -- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab98153e60 msgr2=0x7fab981ebed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab97e00640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fab98153e60 0x7fab981ebed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab97e00640 1 -- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab98154a80 msgr2=0x7fab9814d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab97e00640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fab98154a80 0x7fab9814d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.951+0000 7fab97e00640 1 -- 172.21.15.31:0/1689197767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab981f0320 con 0x7fab9806db20 2024-09-11T00:56:52.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.952+0000 7fab97e00640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9806db20 0x7fab981efde0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fab88034040 tx=0x7fab88003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.952+0000 7fab8fe00640 1 -- 172.21.15.31:0/1689197767 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8803e030 con 0x7fab9806db20 2024-09-11T00:56:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.952+0000 7fab8fe00640 1 -- 172.21.15.31:0/1689197767 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab88003e00 con 0x7fab9806db20 2024-09-11T00:56:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.952+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9806f920 con 0x7fab9806db20 2024-09-11T00:56:52.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.952+0000 7fab8fe00640 1 -- 172.21.15.31:0/1689197767 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88035040 con 0x7fab9806db20 2024-09-11T00:56:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.952+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab981ef5e0 con 0x7fab9806db20 2024-09-11T00:56:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.953+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fab9806f6d0 con 0x7fab9806db20 2024-09-11T00:56:52.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.954+0000 7fab8fe00640 1 -- 172.21.15.31:0/1689197767 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fab880049c0 con 0x7fab9806db20 2024-09-11T00:56:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.955+0000 7fab8fe00640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fab78077f70 0x7fab7807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.955+0000 7fab8fe00640 1 -- 172.21.15.31:0/1689197767 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fab880bbc10 con 0x7fab9806db20 2024-09-11T00:56:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.956+0000 7fab8fe00640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] conn(0x7fab78084680 0x7fab78086b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.956+0000 7fab97400640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fab78077f70 0x7fab7807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.956+0000 7fab8fe00640 1 -- 172.21.15.31:0/1689197767 --> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab8800b0b0 con 0x7fab78084680 2024-09-11T00:56:52.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.956+0000 7fab8fe00640 1 -- 172.21.15.31:0/1689197767 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fab880bc060 con 0x7fab9806db20 2024-09-11T00:56:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.956+0000 7fab9cc00640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] conn(0x7fab78084680 0x7fab78086b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.956+0000 7fab97400640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fab78077f70 0x7fab7807a430 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fab981e0490 tx=0x7fab8400b390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:52.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.957+0000 7fab9cc00640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] conn(0x7fab78084680 0x7fab78086b20 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:52.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.958+0000 7fab8fe00640 1 -- 172.21.15.31:0/1689197767 <== osd.4 v2:172.21.15.204:6802/635888001 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fab8800b0b0 con 0x7fab78084680 2024-09-11T00:56:52.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.971+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 --> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fab981e01e0 con 0x7fab78084680 2024-09-11T00:56:52.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab8fe00640 1 -- 172.21.15.31:0/1689197767 <== osd.4 v2:172.21.15.204:6802/635888001 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fab981e01e0 con 0x7fab78084680 2024-09-11T00:56:52.973 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:56:52.974 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T00:56:52.974 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] conn(0x7fab78084680 msgr2=0x7fab78086b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9d600640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:6802/635888001,v1:172.21.15.204:6803/635888001] conn(0x7fab78084680 0x7fab78086b20 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fab78077f70 msgr2=0x7fab7807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9d600640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fab78077f70 0x7fab7807a430 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fab981e0490 tx=0x7fab8400b390 comp rx=0 tx=0).stop 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9806db20 msgr2=0x7fab981efde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9d600640 1 --2- 172.21.15.31:0/1689197767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9806db20 0x7fab981efde0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fab88034040 tx=0x7fab88003910 comp rx=0 tx=0).stop 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9cc00640 1 -- 172.21.15.31:0/1689197767 reap_dead start 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 shutdown_connections 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 >> 172.21.15.31:0/1689197767 conn(0x7fab9806caf0 msgr2=0x7fab981df5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 shutdown_connections 2024-09-11T00:56:52.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:52.972+0000 7fab9d600640 1 -- 172.21.15.31:0/1689197767 wait complete. 2024-09-11T00:56:52.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-11T00:56:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.153+0000 7f07f7800640 1 -- 172.21.15.31:0/1862130147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01518d0 msgr2=0x7f07f00c3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.153+0000 7f07f7800640 1 --2- 172.21.15.31:0/1862130147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01518d0 0x7f07f00c3830 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f07e8005fd0 tx=0x7f07e802a9d0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.153+0000 7f07f7800640 1 -- 172.21.15.31:0/1862130147 shutdown_connections 2024-09-11T00:56:53.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.153+0000 7f07f7800640 1 --2- 172.21.15.31:0/1862130147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01518d0 0x7f07f00c3830 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.153+0000 7f07f7800640 1 -- 172.21.15.31:0/1862130147 >> 172.21.15.31:0/1862130147 conn(0x7f07f00a9750 msgr2=0x7f07f00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:53.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.153+0000 7f07f7800640 1 -- 172.21.15.31:0/1862130147 shutdown_connections 2024-09-11T00:56:53.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.153+0000 7f07f7800640 1 -- 172.21.15.31:0/1862130147 wait complete. 2024-09-11T00:56:53.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.155+0000 7f07f7800640 1 Processor -- start 2024-09-11T00:56:53.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.155+0000 7f07f7800640 1 -- start start 2024-09-11T00:56:53.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07f7800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07f0072480 0x7f07f01ef980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07f7800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07f00c2410 0x7f07f00c1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07f7800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01518d0 0x7f07f0152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07f7800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f07f0072860 con 0x7f07f01518d0 2024-09-11T00:56:53.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07f7800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f07f00729e0 con 0x7f07f0072480 2024-09-11T00:56:53.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07f7800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f07f0074080 con 0x7f07f00c2410 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07f5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01518d0 0x7f07f0152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07f5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01518d0 0x7f07f0152140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38652/0 (socket says 172.21.15.31:38652) 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07f5800640 1 -- 172.21.15.31:0/1118419918 learned_addr learned my addr 172.21.15.31:0/1118419918 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07f4e00640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07f0072480 0x7f07f01ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.156+0000 7f07e7e00640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07f00c2410 0x7f07f00c1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07f5800640 1 -- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07f00c2410 msgr2=0x7f07f00c1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07f5800640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07f00c2410 0x7f07f00c1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07f5800640 1 -- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07f0072480 msgr2=0x7f07f01ef980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07f5800640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07f0072480 0x7f07f01ef980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07f5800640 1 -- 172.21.15.31:0/1118419918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f006f370 con 0x7f07f01518d0 2024-09-11T00:56:53.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07e7e00640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07f00c2410 0x7f07f00c1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:53.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07f5800640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01518d0 0x7f07f0152140 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009cd0 tx=0x7f07e8031df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:53.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07ed600640 1 -- 172.21.15.31:0/1118419918 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e8045070 con 0x7f07f01518d0 2024-09-11T00:56:53.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07ed600640 1 -- 172.21.15.31:0/1118419918 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07e802fc70 con 0x7f07f01518d0 2024-09-11T00:56:53.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07ed600640 1 -- 172.21.15.31:0/1118419918 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e8003bd0 con 0x7f07f01518d0 2024-09-11T00:56:53.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07f7800640 1 -- 172.21.15.31:0/1118419918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f006e5c0 con 0x7f07f01518d0 2024-09-11T00:56:53.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.157+0000 7f07f7800640 1 -- 172.21.15.31:0/1118419918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f0153950 con 0x7f07f01518d0 2024-09-11T00:56:53.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.159+0000 7f07f7800640 1 -- 172.21.15.31:0/1118419918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f0153f20 con 0x7f07f01518d0 2024-09-11T00:56:53.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.160+0000 7f07ed600640 1 -- 172.21.15.31:0/1118419918 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f07e80310d0 con 0x7f07f01518d0 2024-09-11T00:56:53.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.160+0000 7f07ed600640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f07cc077f70 0x7f07cc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.161+0000 7f07ed600640 1 -- 172.21.15.31:0/1118419918 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f07e80bb5f0 con 0x7f07f01518d0 2024-09-11T00:56:53.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.161+0000 7f07f4e00640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f07cc077f70 0x7f07cc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.162+0000 7f07f4e00640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f07cc077f70 0x7f07cc07a430 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f07e0001810 tx=0x7f07e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:53.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.165+0000 7f07ed600640 1 -- 172.21.15.31:0/1118419918 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f07e8033020 con 0x7f07f01518d0 2024-09-11T00:56:53.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.358+0000 7f07f7800640 1 -- 172.21.15.31:0/1118419918 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f07f0074ea0 con 0x7f07cc077f70 2024-09-11T00:56:53.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.359+0000 7f07ed600640 1 -- 172.21.15.31:0/1118419918 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f07e00014b0 con 0x7f07cc077f70 2024-09-11T00:56:53.361 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 4 to deep-scrub 2024-09-11T00:56:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 -- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f07cc077f70 msgr2=0x7f07cc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f07cc077f70 0x7f07cc07a430 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f07e0001810 tx=0x7f07e0009040 comp rx=0 tx=0).stop 2024-09-11T00:56:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 -- 172.21.15.31:0/1118419918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01518d0 msgr2=0x7f07f0152140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01518d0 0x7f07f0152140 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009cd0 tx=0x7f07e8031df0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 -- 172.21.15.31:0/1118419918 shutdown_connections 2024-09-11T00:56:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f07cc077f70 0x7f07cc07a430 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07f01518d0 0x7f07f0152140 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07f00c2410 0x7f07f00c1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 --2- 172.21.15.31:0/1118419918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07f0072480 0x7f07f01ef980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 -- 172.21.15.31:0/1118419918 >> 172.21.15.31:0/1118419918 conn(0x7f07f00a9750 msgr2=0x7f07f0143b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 -- 172.21.15.31:0/1118419918 shutdown_connections 2024-09-11T00:56:53.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.362+0000 7f07f7800640 1 -- 172.21.15.31:0/1118419918 wait complete. 2024-09-11T00:56:53.380 INFO:tasks.ceph:Scrubbing osd.5 2024-09-11T00:56:53.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-11T00:56:53.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.544+0000 7faa79200640 1 -- 172.21.15.31:0/2102872099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa74159690 msgr2=0x7faa74159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.544+0000 7faa79200640 1 --2- 172.21.15.31:0/2102872099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa74159690 0x7faa74159a70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faa64005fd0 tx=0x7faa6402a820 comp rx=0 tx=0).stop 2024-09-11T00:56:53.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.544+0000 7faa79200640 1 -- 172.21.15.31:0/2102872099 shutdown_connections 2024-09-11T00:56:53.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.544+0000 7faa79200640 1 --2- 172.21.15.31:0/2102872099 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa740c9980 0x7faa740b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.544+0000 7faa79200640 1 --2- 172.21.15.31:0/2102872099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa74159690 0x7faa74159a70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.544+0000 7faa79200640 1 -- 172.21.15.31:0/2102872099 >> 172.21.15.31:0/2102872099 conn(0x7faa740ad810 msgr2=0x7faa740adc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:53.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.544+0000 7faa79200640 1 -- 172.21.15.31:0/2102872099 shutdown_connections 2024-09-11T00:56:53.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.544+0000 7faa79200640 1 -- 172.21.15.31:0/2102872099 wait complete. 2024-09-11T00:56:53.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.546+0000 7faa79200640 1 Processor -- start 2024-09-11T00:56:53.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.546+0000 7faa79200640 1 -- start start 2024-09-11T00:56:53.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa79200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa740c9980 0x7faa740cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa79200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa740c9e90 0x7faa74152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa79200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa74159690 0x7faa74152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa79200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faa7406e070 con 0x7faa740c9e90 2024-09-11T00:56:53.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa79200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa740c6a30 con 0x7faa74159690 2024-09-11T00:56:53.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa79200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7faa7406e1f0 con 0x7faa740c9980 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa6aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa740c9e90 0x7faa74152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa6aa00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa740c9e90 0x7faa74152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38666/0 (socket says 172.21.15.31:38666) 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa6aa00640 1 -- 172.21.15.31:0/3506894161 learned_addr learned my addr 172.21.15.31:0/3506894161 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa73400640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa740c9980 0x7faa740cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.547+0000 7faa73e00640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa74159690 0x7faa74152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa6aa00640 1 -- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa740c9980 msgr2=0x7faa740cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa6aa00640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7faa740c9980 0x7faa740cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa6aa00640 1 -- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa74159690 msgr2=0x7faa74152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa6aa00640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa74159690 0x7faa74152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa6aa00640 1 -- 172.21.15.31:0/3506894161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa741ef2b0 con 0x7faa740c9e90 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa6aa00640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa740c9e90 0x7faa74152110 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00e9a0 tx=0x7faa5c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa6be00640 1 -- 172.21.15.31:0/3506894161 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c00cd80 con 0x7faa740c9e90 2024-09-11T00:56:53.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa6be00640 1 -- 172.21.15.31:0/3506894161 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa5c010430 con 0x7faa740c9e90 2024-09-11T00:56:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa6be00640 1 -- 172.21.15.31:0/3506894161 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c010880 con 0x7faa740c9e90 2024-09-11T00:56:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa741ecb80 con 0x7faa740c9e90 2024-09-11T00:56:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.548+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa741ef830 con 0x7faa740c9e90 2024-09-11T00:56:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.549+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faa74159a70 con 0x7faa740c9e90 2024-09-11T00:56:53.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.551+0000 7faa6be00640 1 -- 172.21.15.31:0/3506894161 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7faa5c010a20 con 0x7faa740c9e90 2024-09-11T00:56:53.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.552+0000 7faa6be00640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faa50077fa0 0x7faa5007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.552+0000 7faa6be00640 1 -- 172.21.15.31:0/3506894161 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7faa5c09be80 con 0x7faa740c9e90 2024-09-11T00:56:53.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.552+0000 7faa6be00640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] conn(0x7faa500846b0 0x7faa50086b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.552+0000 7faa6be00640 1 -- 172.21.15.31:0/3506894161 --> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa50087220 con 0x7faa500846b0 2024-09-11T00:56:53.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.552+0000 7faa6be00640 1 -- 172.21.15.31:0/3506894161 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7faa5c09c230 con 0x7faa740c9e90 2024-09-11T00:56:53.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.552+0000 7faa73400640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faa50077fa0 0x7faa5007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.552+0000 7faa73e00640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] conn(0x7faa500846b0 0x7faa50086b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.553+0000 7faa73e00640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] conn(0x7faa500846b0 0x7faa50086b50 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:53.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.553+0000 7faa73400640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faa50077fa0 0x7faa5007a460 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7faa64034000 tx=0x7faa64004ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:53.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.554+0000 7faa6be00640 1 -- 172.21.15.31:0/3506894161 <== osd.5 v2:172.21.15.204:6810/482649985 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faa50087220 con 0x7faa500846b0 2024-09-11T00:56:53.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.567+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 --> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faa740ad370 con 0x7faa500846b0 2024-09-11T00:56:53.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.568+0000 7faa6be00640 1 -- 172.21.15.31:0/3506894161 <== osd.5 v2:172.21.15.204:6810/482649985 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7faa740ad370 con 0x7faa500846b0 2024-09-11T00:56:53.570 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:56:53.570 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T00:56:53.570 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:56:53.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] conn(0x7faa500846b0 msgr2=0x7faa50086b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa79200640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:6810/482649985,v1:172.21.15.204:6811/482649985] conn(0x7faa500846b0 0x7faa50086b50 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faa50077fa0 msgr2=0x7faa5007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa79200640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7faa50077fa0 0x7faa5007a460 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7faa64034000 tx=0x7faa64004ab0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa740c9e90 msgr2=0x7faa74152110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa79200640 1 --2- 172.21.15.31:0/3506894161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa740c9e90 0x7faa74152110 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00e9a0 tx=0x7faa5c00ee60 comp rx=0 tx=0).stop 2024-09-11T00:56:53.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa73e00640 1 -- 172.21.15.31:0/3506894161 reap_dead start 2024-09-11T00:56:53.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 shutdown_connections 2024-09-11T00:56:53.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 >> 172.21.15.31:0/3506894161 conn(0x7faa740ad810 msgr2=0x7faa740ae6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:53.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 shutdown_connections 2024-09-11T00:56:53.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.569+0000 7faa79200640 1 -- 172.21.15.31:0/3506894161 wait complete. 2024-09-11T00:56:53.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-11T00:56:53.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.746+0000 7f5aca000640 1 -- 172.21.15.31:0/2967789283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40c5a20 msgr2=0x7f5ac40bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.746+0000 7f5aca000640 1 --2- 172.21.15.31:0/2967789283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40c5a20 0x7f5ac40bf600 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0005fd0 tx=0x7f5ab002a670 comp rx=0 tx=0).stop 2024-09-11T00:56:53.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.746+0000 7f5aca000640 1 -- 172.21.15.31:0/2967789283 shutdown_connections 2024-09-11T00:56:53.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.746+0000 7f5aca000640 1 --2- 172.21.15.31:0/2967789283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40c5a20 0x7f5ac40bf600 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.746+0000 7f5aca000640 1 --2- 172.21.15.31:0/2967789283 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ac40c6760 0x7f5ac40c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.746+0000 7f5aca000640 1 -- 172.21.15.31:0/2967789283 >> 172.21.15.31:0/2967789283 conn(0x7f5ac40694f0 msgr2=0x7f5ac40c9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:53.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.746+0000 7f5aca000640 1 -- 172.21.15.31:0/2967789283 shutdown_connections 2024-09-11T00:56:53.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.746+0000 7f5aca000640 1 -- 172.21.15.31:0/2967789283 wait complete. 2024-09-11T00:56:53.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.748+0000 7f5aca000640 1 Processor -- start 2024-09-11T00:56:53.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.748+0000 7f5aca000640 1 -- start start 2024-09-11T00:56:53.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.748+0000 7f5aca000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac40c5a20 0x7f5ac406f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.748+0000 7f5aca000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40c6760 0x7f5ac406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5aca000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ac40c6b40 0x7f5ac406e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5aca000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac40c9460 con 0x7f5ac40c6760 2024-09-11T00:56:53.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5aca000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac40c92e0 con 0x7f5ac40c5a20 2024-09-11T00:56:53.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5aca000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac4072ab0 con 0x7f5ac40c6b40 2024-09-11T00:56:53.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5ac3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40c6760 0x7f5ac406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5ac3e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40c6760 0x7f5ac406dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38680/0 (socket says 172.21.15.31:38680) 2024-09-11T00:56:53.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5ac3e00640 1 -- 172.21.15.31:0/116182768 learned_addr learned my addr 172.21.15.31:0/116182768 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:53.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5ac8c00640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac40c5a20 0x7f5ac406f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5ac9600640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ac40c6b40 0x7f5ac406e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5ac3e00640 1 -- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ac40c6b40 msgr2=0x7f5ac406e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5ac3e00640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ac40c6b40 0x7f5ac406e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5ac3e00640 1 -- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac40c5a20 msgr2=0x7f5ac406f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5ac3e00640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac40c5a20 0x7f5ac406f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.749+0000 7f5ac3e00640 1 -- 172.21.15.31:0/116182768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac41ebfa0 con 0x7f5ac40c6760 2024-09-11T00:56:53.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.750+0000 7f5ac3e00640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40c6760 0x7f5ac406dc00 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0034040 tx=0x7f5ab00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:53.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.750+0000 7f5ac0c00640 1 -- 172.21.15.31:0/116182768 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab003e030 con 0x7f5ac40c6760 2024-09-11T00:56:53.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.750+0000 7f5ac0c00640 1 -- 172.21.15.31:0/116182768 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ab0003d90 con 0x7f5ac40c6760 2024-09-11T00:56:53.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.750+0000 7f5ac0c00640 1 -- 172.21.15.31:0/116182768 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0035040 con 0x7f5ac40c6760 2024-09-11T00:56:53.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.750+0000 7f5aca000640 1 -- 172.21.15.31:0/116182768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac41ec280 con 0x7f5ac40c6760 2024-09-11T00:56:53.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.750+0000 7f5aca000640 1 -- 172.21.15.31:0/116182768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac406f740 con 0x7f5ac40c6760 2024-09-11T00:56:53.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.751+0000 7f5aca000640 1 -- 172.21.15.31:0/116182768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac40bfcb0 con 0x7f5ac40c6760 2024-09-11T00:56:53.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.752+0000 7f5ac0c00640 1 -- 172.21.15.31:0/116182768 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5ab00049e0 con 0x7f5ac40c6760 2024-09-11T00:56:53.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.753+0000 7f5ac0c00640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5aa4077f70 0x7f5aa407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:53.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.753+0000 7f5ac0c00640 1 -- 172.21.15.31:0/116182768 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f5ab00bb990 con 0x7f5ac40c6760 2024-09-11T00:56:53.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.754+0000 7f5ac8c00640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5aa4077f70 0x7f5aa407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:53.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.754+0000 7f5ac8c00640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5aa4077f70 0x7f5aa407a430 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5ac409c2c0 tx=0x7f5ab800b410 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:53.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.757+0000 7f5ac0c00640 1 -- 172.21.15.31:0/116182768 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5ab0082ea0 con 0x7f5ac40c6760 2024-09-11T00:56:53.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.956+0000 7f5aca000640 1 -- 172.21.15.31:0/116182768 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f5ac40c1310 con 0x7f5aa4077f70 2024-09-11T00:56:53.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.957+0000 7f5ac0c00640 1 -- 172.21.15.31:0/116182768 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5ac40c1310 con 0x7f5aa4077f70 2024-09-11T00:56:53.959 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 5 to deep-scrub 2024-09-11T00:56:53.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 -- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5aa4077f70 msgr2=0x7f5aa407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5aa4077f70 0x7f5aa407a430 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5ac409c2c0 tx=0x7f5ab800b410 comp rx=0 tx=0).stop 2024-09-11T00:56:53.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 -- 172.21.15.31:0/116182768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40c6760 msgr2=0x7f5ac406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:53.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40c6760 0x7f5ac406dc00 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0034040 tx=0x7f5ab00038a0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 -- 172.21.15.31:0/116182768 shutdown_connections 2024-09-11T00:56:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5aa4077f70 0x7f5aa407a430 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5ac40c6b40 0x7f5ac406e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ac40c6760 0x7f5ac406dc00 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 --2- 172.21.15.31:0/116182768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ac40c5a20 0x7f5ac406f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 -- 172.21.15.31:0/116182768 >> 172.21.15.31:0/116182768 conn(0x7f5ac40694f0 msgr2=0x7f5ac4156ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 -- 172.21.15.31:0/116182768 shutdown_connections 2024-09-11T00:56:53.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:53.960+0000 7f5aca000640 1 -- 172.21.15.31:0/116182768 wait complete. 2024-09-11T00:56:53.978 INFO:tasks.ceph:Scrubbing osd.6 2024-09-11T00:56:53.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-11T00:56:54.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.145+0000 7f5433e00640 1 -- 172.21.15.31:0/1075931503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5434159690 msgr2=0x7f5434159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.145+0000 7f5433e00640 1 --2- 172.21.15.31:0/1075931503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5434159690 0x7f5434159a70 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f543406cb80 tx=0x7f542402a4a0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.145+0000 7f5433e00640 1 -- 172.21.15.31:0/1075931503 shutdown_connections 2024-09-11T00:56:54.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.145+0000 7f5433e00640 1 --2- 172.21.15.31:0/1075931503 >> v2:172.21.15.204:3300/0 conn(0x7f54340c9e90 0x7f543406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.145+0000 7f5433e00640 1 --2- 172.21.15.31:0/1075931503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5434159690 0x7f5434159a70 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.145+0000 7f5433e00640 1 -- 172.21.15.31:0/1075931503 >> 172.21.15.31:0/1075931503 conn(0x7f54340ad810 msgr2=0x7f54340adc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:54.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.145+0000 7f5433e00640 1 -- 172.21.15.31:0/1075931503 shutdown_connections 2024-09-11T00:56:54.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.145+0000 7f5433e00640 1 -- 172.21.15.31:0/1075931503 wait complete. 2024-09-11T00:56:54.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.146+0000 7f5433e00640 1 Processor -- start 2024-09-11T00:56:54.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.147+0000 7f5433e00640 1 -- start start 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.147+0000 7f5433e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54340c9980 0x7f54340cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.147+0000 7f5433e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f54340c9e90 0x7f5434157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.147+0000 7f5433e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5434159690 0x7f543414a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.147+0000 7f5433e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f54340b1420 con 0x7f54340c9980 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.147+0000 7f5433e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f54340c6a30 con 0x7f5434159690 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.147+0000 7f5433e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f54340b15a0 con 0x7f54340c9e90 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54340c9980 0x7f54340cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54340c9980 0x7f54340cbbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38694/0 (socket says 172.21.15.31:38694) 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432a00640 1 -- 172.21.15.31:0/1395910607 learned_addr learned my addr 172.21.15.31:0/1395910607 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5433400640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5434159690 0x7f543414a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432000640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f54340c9e90 0x7f5434157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432a00640 1 -- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f54340c9e90 msgr2=0x7f5434157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432a00640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f54340c9e90 0x7f5434157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432a00640 1 -- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5434159690 msgr2=0x7f543414a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432a00640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5434159690 0x7f543414a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432a00640 1 -- 172.21.15.31:0/1395910607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543414c0e0 con 0x7f54340c9980 2024-09-11T00:56:54.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5433400640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5434159690 0x7f543414a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:54.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432000640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f54340c9e90 0x7f5434157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:56:54.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.148+0000 7f5432a00640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54340c9980 0x7f54340cbbb0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f5424009920 tx=0x7f5424033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:54.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.149+0000 7f542aa00640 1 -- 172.21.15.31:0/1395910607 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424031d60 con 0x7f54340c9980 2024-09-11T00:56:54.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.149+0000 7f542aa00640 1 -- 172.21.15.31:0/1395910607 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5424033e10 con 0x7f54340c9980 2024-09-11T00:56:54.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.149+0000 7f542aa00640 1 -- 172.21.15.31:0/1395910607 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424005260 con 0x7f54340c9980 2024-09-11T00:56:54.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.149+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5434149f90 con 0x7f54340c9980 2024-09-11T00:56:54.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.149+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54341ecaa0 con 0x7f54340c9980 2024-09-11T00:56:54.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.150+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f543406d960 con 0x7f54340c9980 2024-09-11T00:56:54.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.151+0000 7f542aa00640 1 -- 172.21.15.31:0/1395910607 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5424005400 con 0x7f54340c9980 2024-09-11T00:56:54.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.152+0000 7f542aa00640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5414077f70 0x7f541407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.152+0000 7f542aa00640 1 -- 172.21.15.31:0/1395910607 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f54240bbd60 con 0x7f54340c9980 2024-09-11T00:56:54.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.152+0000 7f542aa00640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] conn(0x7f5414084690 0x7f5414086b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.152+0000 7f542aa00640 1 -- 172.21.15.31:0/1395910607 --> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54240317a0 con 0x7f5414084690 2024-09-11T00:56:54.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.152+0000 7f542aa00640 1 -- 172.21.15.31:0/1395910607 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f54240be380 con 0x7f54340c9980 2024-09-11T00:56:54.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.152+0000 7f5432000640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5414077f70 0x7f541407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.153+0000 7f5433400640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] conn(0x7f5414084690 0x7f5414086b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.153+0000 7f5432000640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5414077f70 0x7f541407a430 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5420007a50 tx=0x7f5420009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:54.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.153+0000 7f5433400640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] conn(0x7f5414084690 0x7f5414086b30 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:54.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.154+0000 7f542aa00640 1 -- 172.21.15.31:0/1395910607 <== osd.6 v2:172.21.15.204:6818/1935657097 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f54240317a0 con 0x7f5414084690 2024-09-11T00:56:54.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.168+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 --> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f54340ae070 con 0x7f5414084690 2024-09-11T00:56:54.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.168+0000 7f542aa00640 1 -- 172.21.15.31:0/1395910607 <== osd.6 v2:172.21.15.204:6818/1935657097 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f54340ae070 con 0x7f5414084690 2024-09-11T00:56:54.170 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:56:54.170 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T00:56:54.170 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:56:54.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.168+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] conn(0x7f5414084690 msgr2=0x7f5414086b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.168+0000 7f5433e00640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:6818/1935657097,v1:172.21.15.204:6819/1935657097] conn(0x7f5414084690 0x7f5414086b30 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.169+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5414077f70 msgr2=0x7f541407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.169+0000 7f5433e00640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5414077f70 0x7f541407a430 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5420007a50 tx=0x7f5420009040 comp rx=0 tx=0).stop 2024-09-11T00:56:54.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.169+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54340c9980 msgr2=0x7f54340cbbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.169+0000 7f5433e00640 1 --2- 172.21.15.31:0/1395910607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54340c9980 0x7f54340cbbb0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f5424009920 tx=0x7f5424033c20 comp rx=0 tx=0).stop 2024-09-11T00:56:54.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.169+0000 7f5433400640 1 -- 172.21.15.31:0/1395910607 reap_dead start 2024-09-11T00:56:54.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.169+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 shutdown_connections 2024-09-11T00:56:54.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.169+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 >> 172.21.15.31:0/1395910607 conn(0x7f54340ad810 msgr2=0x7f54340af400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:54.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.169+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 shutdown_connections 2024-09-11T00:56:54.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.169+0000 7f5433e00640 1 -- 172.21.15.31:0/1395910607 wait complete. 2024-09-11T00:56:54.185 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-11T00:56:54.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.346+0000 7fea8fe00640 1 -- 172.21.15.31:0/233465492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea8806e480 msgr2=0x7fea8806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.346+0000 7fea8fe00640 1 --2- 172.21.15.31:0/233465492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea8806e480 0x7fea8806e860 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fea880c2060 tx=0x7fea7c02a6d0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.346+0000 7fea8fe00640 1 -- 172.21.15.31:0/233465492 shutdown_connections 2024-09-11T00:56:54.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.346+0000 7fea8fe00640 1 --2- 172.21.15.31:0/233465492 >> v2:172.21.15.204:3300/0 conn(0x7fea880c2ff0 0x7fea880c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.346+0000 7fea8fe00640 1 --2- 172.21.15.31:0/233465492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea8806e480 0x7fea8806e860 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.346+0000 7fea8fe00640 1 -- 172.21.15.31:0/233465492 >> 172.21.15.31:0/233465492 conn(0x7fea880a9750 msgr2=0x7fea880a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:54.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.346+0000 7fea8fe00640 1 -- 172.21.15.31:0/233465492 shutdown_connections 2024-09-11T00:56:54.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.346+0000 7fea8fe00640 1 -- 172.21.15.31:0/233465492 wait complete. 2024-09-11T00:56:54.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.348+0000 7fea8fe00640 1 Processor -- start 2024-09-11T00:56:54.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.348+0000 7fea8fe00640 1 -- start start 2024-09-11T00:56:54.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8fe00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8806e480 0x7fea88154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8fe00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea880c2ff0 0x7fea880abdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8fe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea880c5de0 0x7fea880ac310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8fe00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fea8814fb20 con 0x7fea880c5de0 2024-09-11T00:56:54.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8fe00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fea8806ef90 con 0x7fea8806e480 2024-09-11T00:56:54.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8fe00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fea880c7690 con 0x7fea880c2ff0 2024-09-11T00:56:54.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea880c5de0 0x7fea880ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea880c5de0 0x7fea880ac310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38712/0 (socket says 172.21.15.31:38712) 2024-09-11T00:56:54.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8de00640 1 -- 172.21.15.31:0/15696976 learned_addr learned my addr 172.21.15.31:0/15696976 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:54.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8d400640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8806e480 0x7fea88154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8ca00640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea880c2ff0 0x7fea880abdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8de00640 1 -- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea880c2ff0 msgr2=0x7fea880abdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8de00640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea880c2ff0 0x7fea880abdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8de00640 1 -- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8806e480 msgr2=0x7fea88154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8de00640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8806e480 0x7fea88154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.349+0000 7fea8de00640 1 -- 172.21.15.31:0/15696976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea881eb0d0 con 0x7fea880c5de0 2024-09-11T00:56:54.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.350+0000 7fea8de00640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea880c5de0 0x7fea880ac310 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fea8000c700 tx=0x7fea8000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.350+0000 7fea85600640 1 -- 172.21.15.31:0/15696976 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80003b70 con 0x7fea880c5de0 2024-09-11T00:56:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.350+0000 7fea85600640 1 -- 172.21.15.31:0/15696976 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea80003d10 con 0x7fea880c5de0 2024-09-11T00:56:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.350+0000 7fea8fe00640 1 -- 172.21.15.31:0/15696976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea88155a40 con 0x7fea880c5de0 2024-09-11T00:56:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.350+0000 7fea85600640 1 -- 172.21.15.31:0/15696976 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8000aab0 con 0x7fea880c5de0 2024-09-11T00:56:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.350+0000 7fea8fe00640 1 -- 172.21.15.31:0/15696976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea880aab50 con 0x7fea880c5de0 2024-09-11T00:56:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.351+0000 7fea8fe00640 1 -- 172.21.15.31:0/15696976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea880aad50 con 0x7fea880c5de0 2024-09-11T00:56:54.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.352+0000 7fea85600640 1 -- 172.21.15.31:0/15696976 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fea8000ac50 con 0x7fea880c5de0 2024-09-11T00:56:54.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.353+0000 7fea85600640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea68077f70 0x7fea6807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.353+0000 7fea85600640 1 -- 172.21.15.31:0/15696976 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fea8009a920 con 0x7fea880c5de0 2024-09-11T00:56:54.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.354+0000 7fea8d400640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea68077f70 0x7fea6807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.354+0000 7fea8d400640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea68077f70 0x7fea6807a430 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fea7c033000 tx=0x7fea7c004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:54.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.357+0000 7fea85600640 1 -- 172.21.15.31:0/15696976 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fea880aad50 con 0x7fea880c5de0 2024-09-11T00:56:54.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.552+0000 7fea8fe00640 1 -- 172.21.15.31:0/15696976 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fea8814f740 con 0x7fea68077f70 2024-09-11T00:56:54.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.552+0000 7fea85600640 1 -- 172.21.15.31:0/15696976 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fea8814f740 con 0x7fea68077f70 2024-09-11T00:56:54.554 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 6 to deep-scrub 2024-09-11T00:56:54.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 -- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea68077f70 msgr2=0x7fea6807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea68077f70 0x7fea6807a430 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fea7c033000 tx=0x7fea7c004730 comp rx=0 tx=0).stop 2024-09-11T00:56:54.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 -- 172.21.15.31:0/15696976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea880c5de0 msgr2=0x7fea880ac310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea880c5de0 0x7fea880ac310 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fea8000c700 tx=0x7fea8000cbc0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 -- 172.21.15.31:0/15696976 shutdown_connections 2024-09-11T00:56:54.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fea68077f70 0x7fea6807a430 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea880c5de0 0x7fea880ac310 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fea880c2ff0 0x7fea880abdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 --2- 172.21.15.31:0/15696976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fea8806e480 0x7fea88154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 -- 172.21.15.31:0/15696976 >> 172.21.15.31:0/15696976 conn(0x7fea880a9750 msgr2=0x7fea880a8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:54.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 -- 172.21.15.31:0/15696976 shutdown_connections 2024-09-11T00:56:54.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.555+0000 7fea8fe00640 1 -- 172.21.15.31:0/15696976 wait complete. 2024-09-11T00:56:54.573 INFO:tasks.ceph:Scrubbing osd.7 2024-09-11T00:56:54.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-11T00:56:54.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.737+0000 7fe246c00640 1 -- 172.21.15.31:0/1549494624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2400c2b10 msgr2=0x7fe240156bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.737+0000 7fe246c00640 1 --2- 172.21.15.31:0/1549494624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2400c2b10 0x7fe240156bd0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe238005fd0 tx=0x7fe23802a940 comp rx=0 tx=0).stop 2024-09-11T00:56:54.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.737+0000 7fe246c00640 1 -- 172.21.15.31:0/1549494624 shutdown_connections 2024-09-11T00:56:54.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.737+0000 7fe246c00640 1 --2- 172.21.15.31:0/1549494624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2400c2b10 0x7fe240156bd0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.737+0000 7fe246c00640 1 --2- 172.21.15.31:0/1549494624 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2401536a0 0x7fe2400c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.737+0000 7fe246c00640 1 -- 172.21.15.31:0/1549494624 >> 172.21.15.31:0/1549494624 conn(0x7fe24006caf0 msgr2=0x7fe24006ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:54.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.737+0000 7fe246c00640 1 -- 172.21.15.31:0/1549494624 shutdown_connections 2024-09-11T00:56:54.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.737+0000 7fe246c00640 1 -- 172.21.15.31:0/1549494624 wait complete. 2024-09-11T00:56:54.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.739+0000 7fe246c00640 1 Processor -- start 2024-09-11T00:56:54.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.739+0000 7fe246c00640 1 -- start start 2024-09-11T00:56:54.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe246c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2400c2b10 0x7fe2401ec500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe246c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2400c4c00 0x7fe2400ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe246c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2401536a0 0x7fe2400ab750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe246c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe2400c0d60 con 0x7fe2401536a0 2024-09-11T00:56:54.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe246c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe2400c6080 con 0x7fe2400c4c00 2024-09-11T00:56:54.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe246c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe240153a80 con 0x7fe2400c2b10 2024-09-11T00:56:54.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe246200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2401536a0 0x7fe2400ab750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe246200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2401536a0 0x7fe2400ab750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38728/0 (socket says 172.21.15.31:38728) 2024-09-11T00:56:54.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe246200640 1 -- 172.21.15.31:0/586074131 learned_addr learned my addr 172.21.15.31:0/586074131 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:54.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe245800640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2400c2b10 0x7fe2401ec500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.740+0000 7fe244e00640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2400c4c00 0x7fe2400ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe246200640 1 -- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2400c2b10 msgr2=0x7fe2401ec500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe246200640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe2400c2b10 0x7fe2401ec500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe246200640 1 -- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2400c4c00 msgr2=0x7fe2400ab210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe246200640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe2400c4c00 0x7fe2400ab210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe246200640 1 -- 172.21.15.31:0/586074131 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe24014ce80 con 0x7fe2401536a0 2024-09-11T00:56:54.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe246200640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2401536a0 0x7fe2400ab750 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fe238009cd0 tx=0x7fe238031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:54.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe23d600640 1 -- 172.21.15.31:0/586074131 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe238030430 con 0x7fe2401536a0 2024-09-11T00:56:54.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe23d600640 1 -- 172.21.15.31:0/586074131 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe238004240 con 0x7fe2401536a0 2024-09-11T00:56:54.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe24014d330 con 0x7fe2401536a0 2024-09-11T00:56:54.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe23d600640 1 -- 172.21.15.31:0/586074131 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe238004650 con 0x7fe2401536a0 2024-09-11T00:56:54.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.741+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2400aa950 con 0x7fe2401536a0 2024-09-11T00:56:54.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.742+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe2400a96e0 con 0x7fe2401536a0 2024-09-11T00:56:54.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.743+0000 7fe23d600640 1 -- 172.21.15.31:0/586074131 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe23802f070 con 0x7fe2401536a0 2024-09-11T00:56:54.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.744+0000 7fe23d600640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe220077f70 0x7fe22007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.745+0000 7fe23d600640 1 -- 172.21.15.31:0/586074131 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fe2380bc460 con 0x7fe2401536a0 2024-09-11T00:56:54.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.745+0000 7fe23d600640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] conn(0x7fe220084690 0x7fe220086b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.745+0000 7fe23d600640 1 -- 172.21.15.31:0/586074131 --> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe23802eec0 con 0x7fe220084690 2024-09-11T00:56:54.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.745+0000 7fe23d600640 1 -- 172.21.15.31:0/586074131 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fe2380c3050 con 0x7fe2401536a0 2024-09-11T00:56:54.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.745+0000 7fe245800640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe220077f70 0x7fe22007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.745+0000 7fe244e00640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] conn(0x7fe220084690 0x7fe220086b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.746+0000 7fe245800640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe220077f70 0x7fe22007a430 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe234009ac0 tx=0x7fe234005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:54.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.746+0000 7fe244e00640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] conn(0x7fe220084690 0x7fe220086b30 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:54.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.747+0000 7fe23d600640 1 -- 172.21.15.31:0/586074131 <== osd.7 v2:172.21.15.204:6826/276504597 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe23802eec0 con 0x7fe220084690 2024-09-11T00:56:54.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.760+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 --> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe2400c6440 con 0x7fe220084690 2024-09-11T00:56:54.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.761+0000 7fe23d600640 1 -- 172.21.15.31:0/586074131 <== osd.7 v2:172.21.15.204:6826/276504597 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe2400c6440 con 0x7fe220084690 2024-09-11T00:56:54.763 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-09-11T00:56:54.763 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T00:56:54.763 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-09-11T00:56:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.761+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] conn(0x7fe220084690 msgr2=0x7fe220086b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.761+0000 7fe246c00640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:6826/276504597,v1:172.21.15.204:6827/276504597] conn(0x7fe220084690 0x7fe220086b30 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.761+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe220077f70 msgr2=0x7fe22007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.761+0000 7fe246c00640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe220077f70 0x7fe22007a430 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe234009ac0 tx=0x7fe234005f90 comp rx=0 tx=0).stop 2024-09-11T00:56:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.761+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2401536a0 msgr2=0x7fe2400ab750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.761+0000 7fe246c00640 1 --2- 172.21.15.31:0/586074131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2401536a0 0x7fe2400ab750 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fe238009cd0 tx=0x7fe238031e20 comp rx=0 tx=0).stop 2024-09-11T00:56:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.761+0000 7fe246200640 1 -- 172.21.15.31:0/586074131 reap_dead start 2024-09-11T00:56:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.762+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 shutdown_connections 2024-09-11T00:56:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.762+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 >> 172.21.15.31:0/586074131 conn(0x7fe24006caf0 msgr2=0x7fe2400c41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:54.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.762+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 shutdown_connections 2024-09-11T00:56:54.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.762+0000 7fe246c00640 1 -- 172.21.15.31:0/586074131 wait complete. 2024-09-11T00:56:54.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-11T00:56:54.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.937+0000 7fe78c600640 1 -- 172.21.15.31:0/1207524967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe784156ad0 msgr2=0x7fe784153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.937+0000 7fe78c600640 1 --2- 172.21.15.31:0/1207524967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe784156ad0 0x7fe784153480 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe770009830 tx=0x7fe77002a500 comp rx=0 tx=0).stop 2024-09-11T00:56:54.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.937+0000 7fe78c600640 1 -- 172.21.15.31:0/1207524967 shutdown_connections 2024-09-11T00:56:54.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.937+0000 7fe78c600640 1 --2- 172.21.15.31:0/1207524967 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe7841539c0 0x7fe784159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.937+0000 7fe78c600640 1 --2- 172.21.15.31:0/1207524967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe784156ad0 0x7fe784153480 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.937+0000 7fe78c600640 1 -- 172.21.15.31:0/1207524967 >> 172.21.15.31:0/1207524967 conn(0x7fe7840a9750 msgr2=0x7fe7840a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:54.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.937+0000 7fe78c600640 1 -- 172.21.15.31:0/1207524967 shutdown_connections 2024-09-11T00:56:54.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.937+0000 7fe78c600640 1 -- 172.21.15.31:0/1207524967 wait complete. 2024-09-11T00:56:54.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.939+0000 7fe78c600640 1 Processor -- start 2024-09-11T00:56:54.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.939+0000 7fe78c600640 1 -- start start 2024-09-11T00:56:54.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe78c600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7840c5b10 0x7fe7841df6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe78c600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7841539c0 0x7fe7841521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe78c600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe784156ad0 0x7fe784154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe78c600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe784156f20 con 0x7fe7840c5b10 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe78c600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe7840c7320 con 0x7fe7841539c0 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe78c600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe7840c3950 con 0x7fe784156ad0 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe789c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7840c5b10 0x7fe7841df6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe789c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7840c5b10 0x7fe7841df6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38734/0 (socket says 172.21.15.31:38734) 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe789c00640 1 -- 172.21.15.31:0/678754050 learned_addr learned my addr 172.21.15.31:0/678754050 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe78a600640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe784156ad0 0x7fe784154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.940+0000 7fe789200640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7841539c0 0x7fe7841521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe789c00640 1 -- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe784156ad0 msgr2=0x7fe784154a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe789c00640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe784156ad0 0x7fe784154a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe789c00640 1 -- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7841539c0 msgr2=0x7fe7841521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:54.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe789c00640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7841539c0 0x7fe7841521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:54.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe789c00640 1 -- 172.21.15.31:0/678754050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7841e2070 con 0x7fe7840c5b10 2024-09-11T00:56:54.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe789c00640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7840c5b10 0x7fe7841df6e0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fe77800c490 tx=0x7fe77800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:54.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe781600640 1 -- 172.21.15.31:0/678754050 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe778016020 con 0x7fe7840c5b10 2024-09-11T00:56:54.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe781600640 1 -- 172.21.15.31:0/678754050 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe778012940 con 0x7fe7840c5b10 2024-09-11T00:56:54.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe781600640 1 -- 172.21.15.31:0/678754050 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe778012d90 con 0x7fe7840c5b10 2024-09-11T00:56:54.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe78c600640 1 -- 172.21.15.31:0/678754050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe784153fb0 con 0x7fe7840c5b10 2024-09-11T00:56:54.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.941+0000 7fe78c600640 1 -- 172.21.15.31:0/678754050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe784154570 con 0x7fe7840c5b10 2024-09-11T00:56:54.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.942+0000 7fe78c600640 1 -- 172.21.15.31:0/678754050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe784077240 con 0x7fe7840c5b10 2024-09-11T00:56:54.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.944+0000 7fe781600640 1 -- 172.21.15.31:0/678754050 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe77801a3b0 con 0x7fe7840c5b10 2024-09-11T00:56:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.945+0000 7fe781600640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe764077f70 0x7fe76407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:54.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.945+0000 7fe781600640 1 -- 172.21.15.31:0/678754050 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fe77809aa80 con 0x7fe7840c5b10 2024-09-11T00:56:54.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.945+0000 7fe789200640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe764077f70 0x7fe76407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:54.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.946+0000 7fe789200640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe764077f70 0x7fe76407a430 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe770034000 tx=0x7fe770004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:54.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:54.949+0000 7fe781600640 1 -- 172.21.15.31:0/678754050 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe784077240 con 0x7fe7840c5b10 2024-09-11T00:56:55.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.148+0000 7fe78c600640 1 -- 172.21.15.31:0/678754050 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fe784150940 con 0x7fe764077f70 2024-09-11T00:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.148+0000 7fe781600640 1 -- 172.21.15.31:0/678754050 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe784150940 con 0x7fe764077f70 2024-09-11T00:56:55.150 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 7 to deep-scrub 2024-09-11T00:56:55.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 -- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe764077f70 msgr2=0x7fe76407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe764077f70 0x7fe76407a430 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe770034000 tx=0x7fe770004730 comp rx=0 tx=0).stop 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 -- 172.21.15.31:0/678754050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7840c5b10 msgr2=0x7fe7841df6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7840c5b10 0x7fe7841df6e0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fe77800c490 tx=0x7fe77800c950 comp rx=0 tx=0).stop 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 -- 172.21.15.31:0/678754050 shutdown_connections 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe764077f70 0x7fe76407a430 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe784156ad0 0x7fe784154a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7841539c0 0x7fe7841521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 --2- 172.21.15.31:0/678754050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe7840c5b10 0x7fe7841df6e0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 -- 172.21.15.31:0/678754050 >> 172.21.15.31:0/678754050 conn(0x7fe7840a9750 msgr2=0x7fe7840a7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 -- 172.21.15.31:0/678754050 shutdown_connections 2024-09-11T00:56:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.151+0000 7fe78c600640 1 -- 172.21.15.31:0/678754050 wait complete. 2024-09-11T00:56:55.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:56:55.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.329+0000 7fa495a00640 1 -- 172.21.15.31:0/1014516509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa474004800 msgr2=0x7fa474004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:55.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.329+0000 7fa495a00640 1 --2- 172.21.15.31:0/1014516509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa474004800 0x7fa474004c80 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009b20 tx=0x7fa474005830 comp rx=0 tx=0).stop 2024-09-11T00:56:55.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.329+0000 7fa48f400640 1 -- 172.21.15.31:0/1014516509 reap_dead start 2024-09-11T00:56:55.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.329+0000 7fa495a00640 1 -- 172.21.15.31:0/1014516509 shutdown_connections 2024-09-11T00:56:55.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.329+0000 7fa495a00640 1 -- 172.21.15.31:0/1014516509 >> 172.21.15.31:0/1014516509 conn(0x7fa4900a9750 msgr2=0x7fa4900a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:55.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.329+0000 7fa495a00640 1 -- 172.21.15.31:0/1014516509 shutdown_connections 2024-09-11T00:56:55.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.329+0000 7fa495a00640 1 -- 172.21.15.31:0/1014516509 wait complete. 2024-09-11T00:56:55.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.331+0000 7fa495a00640 1 Processor -- start 2024-09-11T00:56:55.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.331+0000 7fa495a00640 1 -- start start 2024-09-11T00:56:55.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.331+0000 7fa495a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa474004800 0x7fa4901ed0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:55.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.331+0000 7fa495a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa490073c40 0x7fa4901551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:55.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa495a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa490072b60 0x7fa49014cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:55.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa495a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900c7320 con 0x7fa474004800 2024-09-11T00:56:55.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa495a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900c3950 con 0x7fa490072b60 2024-09-11T00:56:55.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa495a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa490153480 con 0x7fa490073c40 2024-09-11T00:56:55.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa48ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa474004800 0x7fa4901ed0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:55.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa48ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa474004800 0x7fa4901ed0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38748/0 (socket says 172.21.15.31:38748) 2024-09-11T00:56:55.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa48ea00640 1 -- 172.21.15.31:0/4079188163 learned_addr learned my addr 172.21.15.31:0/4079188163 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:56:55.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa48e000640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa490073c40 0x7fa4901551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:55.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa48f400640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa490072b60 0x7fa49014cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:55.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa48ea00640 1 -- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa490073c40 msgr2=0x7fa4901551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:55.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa48ea00640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa490073c40 0x7fa4901551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:55.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa48ea00640 1 -- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa490072b60 msgr2=0x7fa49014cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:55.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa48ea00640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa490072b60 0x7fa49014cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:55.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.332+0000 7fa48ea00640 1 -- 172.21.15.31:0/4079188163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4901e2070 con 0x7fa474004800 2024-09-11T00:56:55.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.333+0000 7fa48ea00640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa474004800 0x7fa4901ed0b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fa4800066e0 tx=0x7fa480003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:55.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.333+0000 7fa486a00640 1 -- 172.21.15.31:0/4079188163 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48000f720 con 0x7fa474004800 2024-09-11T00:56:55.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.333+0000 7fa486a00640 1 -- 172.21.15.31:0/4079188163 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa480016020 con 0x7fa474004800 2024-09-11T00:56:55.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.333+0000 7fa495a00640 1 -- 172.21.15.31:0/4079188163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa490159eb0 con 0x7fa474004800 2024-09-11T00:56:55.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.333+0000 7fa486a00640 1 -- 172.21.15.31:0/4079188163 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48000fb60 con 0x7fa474004800 2024-09-11T00:56:55.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.333+0000 7fa495a00640 1 -- 172.21.15.31:0/4079188163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa490072640 con 0x7fa474004800 2024-09-11T00:56:55.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.334+0000 7fa495a00640 1 -- 172.21.15.31:0/4079188163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4900c5ef0 con 0x7fa474004800 2024-09-11T00:56:55.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.335+0000 7fa486a00640 1 -- 172.21.15.31:0/4079188163 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa48000fd00 con 0x7fa474004800 2024-09-11T00:56:55.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.336+0000 7fa486a00640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa470077f70 0x7fa47007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:56:55.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.336+0000 7fa486a00640 1 -- 172.21.15.31:0/4079188163 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fa48009b760 con 0x7fa474004800 2024-09-11T00:56:55.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.337+0000 7fa48e000640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa470077f70 0x7fa47007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:56:55.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.337+0000 7fa48e000640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa470077f70 0x7fa47007a430 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa47c035000 tx=0x7fa47c02d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:56:55.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.340+0000 7fa486a00640 1 -- 172.21.15.31:0/4079188163 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa480020020 con 0x7fa474004800 2024-09-11T00:56:55.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.531+0000 7fa495a00640 1 -- 172.21.15.31:0/4079188163 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa4900c1e70 con 0x7fa470077f70 2024-09-11T00:56:55.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.540+0000 7fa486a00640 1 -- 172.21.15.31:0/4079188163 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493511 (secure 0 0 0) 0x7fa4900c1e70 con 0x7fa470077f70 2024-09-11T00:56:55.547 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:56:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:56:55.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.545+0000 7fa495a00640 1 -- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa470077f70 msgr2=0x7fa47007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:55.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.545+0000 7fa495a00640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa470077f70 0x7fa47007a430 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa47c035000 tx=0x7fa47c02d040 comp rx=0 tx=0).stop 2024-09-11T00:56:55.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.545+0000 7fa495a00640 1 -- 172.21.15.31:0/4079188163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa474004800 msgr2=0x7fa4901ed0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:56:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.545+0000 7fa495a00640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa474004800 0x7fa4901ed0b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fa4800066e0 tx=0x7fa480003a80 comp rx=0 tx=0).stop 2024-09-11T00:56:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.545+0000 7fa495a00640 1 -- 172.21.15.31:0/4079188163 shutdown_connections 2024-09-11T00:56:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.545+0000 7fa495a00640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa470077f70 0x7fa47007a430 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.545+0000 7fa495a00640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa490072b60 0x7fa49014cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.545+0000 7fa495a00640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa490073c40 0x7fa4901551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.545+0000 7fa495a00640 1 --2- 172.21.15.31:0/4079188163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa474004800 0x7fa4901ed0b0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:56:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.545+0000 7fa495a00640 1 -- 172.21.15.31:0/4079188163 >> 172.21.15.31:0/4079188163 conn(0x7fa4900a9750 msgr2=0x7fa49006f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:56:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.546+0000 7fa495a00640 1 -- 172.21.15.31:0/4079188163 shutdown_connections 2024-09-11T00:56:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:56:55.546+0000 7fa495a00640 1 -- 172.21.15.31:0/4079188163 wait complete. 2024-09-11T00:56:55.563 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":846,"stamp":"2024-09-11T00:56:54.846152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36448998,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":2779,"num_read_kb":341768,"num_write":2142706,"num_write_kb":34795523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":418195,"ondisk_log_size":418195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5380088,"kb_used_data":77432,"kb_used_omap":12,"kb_used_meta":5302515,"kb_avail":749594632,"statfs":{"total":773094113280,"available":767584903168,"internally_reserved":0,"allocated":79290368,"data_stored":76164476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":5429775921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003763"},"pg_stats":[{"pgid":"5.18","version":"62'3612","reported_seq":3708,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.530101+0000","last_change":"2024-09-11T00:30:54.983412+0000","last_active":"2024-09-11T00:56:28.530101+0000","last_peered":"2024-09-11T00:56:28.530101+0000","last_clean":"2024-09-11T00:56:28.530101+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:56:28.530101+0000","last_undegraded":"2024-09-11T00:56:28.530101+0000","last_fullsized":"2024-09-11T00:56:28.530101+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:13:51.807846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18811,"num_write_kb":284357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"60'788","reported_seq":928,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.095736+0000","last_change":"2024-09-11T00:30:44.922567+0000","last_active":"2024-09-11T00:56:28.095736+0000","last_peered":"2024-09-11T00:56:28.095736+0000","last_clean":"2024-09-11T00:56:28.095736+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:56:28.095736+0000","last_undegraded":"2024-09-11T00:56:28.095736+0000","last_fullsized":"2024-09-11T00:56:28.095736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:03.999472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":8319,"num_write":773,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"60'3033","reported_seq":3139,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059856+0000","last_change":"2024-09-11T00:30:45.938642+0000","last_active":"2024-09-11T00:56:28.059856+0000","last_peered":"2024-09-11T00:56:28.059856+0000","last_clean":"2024-09-11T00:56:28.059856+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:56:28.059856+0000","last_undegraded":"2024-09-11T00:56:28.059856+0000","last_fullsized":"2024-09-11T00:56:28.059856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16165,"num_write_kb":269490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"61'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058244+0000","last_change":"2024-09-11T00:30:53.984316+0000","last_active":"2024-09-11T00:56:28.058244+0000","last_peered":"2024-09-11T00:56:28.058244+0000","last_clean":"2024-09-11T00:56:28.058244+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:56:28.058244+0000","last_undegraded":"2024-09-11T00:56:28.058244+0000","last_fullsized":"2024-09-11T00:56:28.058244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:14.425265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"62'3995","reported_seq":4090,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.532259+0000","last_change":"2024-09-11T00:30:54.987537+0000","last_active":"2024-09-11T00:56:28.532259+0000","last_peered":"2024-09-11T00:56:28.532259+0000","last_clean":"2024-09-11T00:56:28.532259+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:56:28.532259+0000","last_undegraded":"2024-09-11T00:56:28.532259+0000","last_fullsized":"2024-09-11T00:56:28.532259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:02.445797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20665,"num_write_kb":324486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"62'22","reported_seq":138,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.046603+0000","last_change":"2024-09-11T00:56:54.046603+0000","last_active":"2024-09-11T00:56:54.046603+0000","last_peered":"2024-09-11T00:56:54.046603+0000","last_clean":"2024-09-11T00:56:54.046603+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:56:54.046603+0000","last_undegraded":"2024-09-11T00:56:54.046603+0000","last_fullsized":"2024-09-11T00:56:54.046603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'22","last_scrub_stamp":"2024-09-11T00:56:54.046518+0000","last_deep_scrub":"62'22","last_deep_scrub_stamp":"2024-09-11T00:56:54.046518+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.046518+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:50:58.895853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"60'3563","reported_seq":3674,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059819+0000","last_change":"2024-09-11T00:30:45.934073+0000","last_active":"2024-09-11T00:56:28.059819+0000","last_peered":"2024-09-11T00:56:28.059819+0000","last_clean":"2024-09-11T00:56:28.059819+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:56:28.059819+0000","last_undegraded":"2024-09-11T00:56:28.059819+0000","last_fullsized":"2024-09-11T00:56:28.059819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:42.611467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":44,"num_write":18834,"num_write_kb":321224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"62'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075381+0000","last_change":"2024-09-11T00:30:53.984940+0000","last_active":"2024-09-11T00:56:28.075381+0000","last_peered":"2024-09-11T00:56:28.075381+0000","last_clean":"2024-09-11T00:56:28.075381+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:56:28.075381+0000","last_undegraded":"2024-09-11T00:56:28.075381+0000","last_fullsized":"2024-09-11T00:56:28.075381+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:19.647165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"62'3090","reported_seq":3185,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.476819+0000","last_change":"2024-09-11T00:30:54.993570+0000","last_active":"2024-09-11T00:56:28.476819+0000","last_peered":"2024-09-11T00:56:28.476819+0000","last_clean":"2024-09-11T00:56:28.476819+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:56:28.476819+0000","last_undegraded":"2024-09-11T00:56:28.476819+0000","last_fullsized":"2024-09-11T00:56:28.476819+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:45.211259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15625,"num_write_kb":254958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"62'38","reported_seq":180,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.714721+0000","last_change":"2024-09-11T00:56:54.714721+0000","last_active":"2024-09-11T00:56:54.714721+0000","last_peered":"2024-09-11T00:56:54.714721+0000","last_clean":"2024-09-11T00:56:54.714721+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:56:54.714721+0000","last_undegraded":"2024-09-11T00:56:54.714721+0000","last_fullsized":"2024-09-11T00:56:54.714721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'38","last_scrub_stamp":"2024-09-11T00:56:54.714635+0000","last_deep_scrub":"62'38","last_deep_scrub_stamp":"2024-09-11T00:56:54.714635+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.714635+0000","objects_scrubbed":1,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:59:22.322810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"60'2040","reported_seq":2144,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057996+0000","last_change":"2024-09-11T00:30:45.921353+0000","last_active":"2024-09-11T00:56:28.057996+0000","last_peered":"2024-09-11T00:56:28.057996+0000","last_clean":"2024-09-11T00:56:28.057996+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:56:28.057996+0000","last_undegraded":"2024-09-11T00:56:28.057996+0000","last_fullsized":"2024-09-11T00:56:28.057996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:23:35.741494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":10218,"num_write_kb":171981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"61'716","reported_seq":837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075103+0000","last_change":"2024-09-11T00:30:53.974699+0000","last_active":"2024-09-11T00:56:28.075103+0000","last_peered":"2024-09-11T00:56:28.075103+0000","last_clean":"2024-09-11T00:56:28.075103+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:56:28.075103+0000","last_undegraded":"2024-09-11T00:56:28.075103+0000","last_fullsized":"2024-09-11T00:56:28.075103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:32:28.792487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":73385,"num_write":707,"num_write_kb":4317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"62'2671","reported_seq":2766,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.074790+0000","last_change":"2024-09-11T00:30:54.984890+0000","last_active":"2024-09-11T00:56:28.074790+0000","last_peered":"2024-09-11T00:56:28.074790+0000","last_clean":"2024-09-11T00:56:28.074790+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:56:28.074790+0000","last_undegraded":"2024-09-11T00:56:28.074790+0000","last_fullsized":"2024-09-11T00:56:28.074790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:04:21.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":213871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"60'3","reported_seq":136,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.069924+0000","last_change":"2024-09-11T00:30:44.915213+0000","last_active":"2024-09-11T00:56:31.069924+0000","last_peered":"2024-09-11T00:56:31.069924+0000","last_clean":"2024-09-11T00:56:31.069924+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:56:31.069924+0000","last_undegraded":"2024-09-11T00:56:31.069924+0000","last_fullsized":"2024-09-11T00:56:31.069924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:16:25.061269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"60'3687","reported_seq":3790,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068958+0000","last_change":"2024-09-11T00:30:45.923755+0000","last_active":"2024-09-11T00:56:28.068958+0000","last_peered":"2024-09-11T00:56:28.068958+0000","last_clean":"2024-09-11T00:56:28.068958+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:56:28.068958+0000","last_undegraded":"2024-09-11T00:56:28.068958+0000","last_fullsized":"2024-09-11T00:56:28.068958+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:22.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19525,"num_write_kb":315726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"62'140","reported_seq":237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.957825+0000","last_change":"2024-09-11T00:30:53.977743+0000","last_active":"2024-09-11T00:56:34.957825+0000","last_peered":"2024-09-11T00:56:34.957825+0000","last_clean":"2024-09-11T00:56:34.957825+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:56:34.957825+0000","last_undegraded":"2024-09-11T00:56:34.957825+0000","last_fullsized":"2024-09-11T00:56:34.957825+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:19:20.618615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1356353,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":1393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"62'3457","reported_seq":3553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.513475+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:56:28.513475+0000","last_peered":"2024-09-11T00:56:28.513475+0000","last_clean":"2024-09-11T00:56:28.513475+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:56:28.513475+0000","last_undegraded":"2024-09-11T00:56:28.513475+0000","last_fullsized":"2024-09-11T00:56:28.513475+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3457,"log_dups_size":0,"ondisk_log_size":3457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18121,"num_write_kb":297725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"60'2670","reported_seq":2775,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065840+0000","last_change":"2024-09-11T00:30:45.930552+0000","last_active":"2024-09-11T00:56:28.065840+0000","last_peered":"2024-09-11T00:56:28.065840+0000","last_clean":"2024-09-11T00:56:28.065840+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:56:28.065840+0000","last_undegraded":"2024-09-11T00:56:28.065840+0000","last_fullsized":"2024-09-11T00:56:28.065840+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2670,"log_dups_size":0,"ondisk_log_size":2670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:16.728445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14000,"num_write_kb":256678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"62'1368","reported_seq":1505,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.097251+0000","last_change":"2024-09-11T00:30:44.923332+0000","last_active":"2024-09-11T00:56:28.097251+0000","last_peered":"2024-09-11T00:56:28.097251+0000","last_clean":"2024-09-11T00:56:28.097251+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:56:28.097251+0000","last_undegraded":"2024-09-11T00:56:28.097251+0000","last_fullsized":"2024-09-11T00:56:28.097251+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1368,"log_dups_size":0,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:31:12.016719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":26260,"num_write":1345,"num_write_kb":4470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"61'2","reported_seq":121,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068774+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:56:28.068774+0000","last_peered":"2024-09-11T00:56:28.068774+0000","last_clean":"2024-09-11T00:56:28.068774+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:56:28.068774+0000","last_undegraded":"2024-09-11T00:56:28.068774+0000","last_fullsized":"2024-09-11T00:56:28.068774+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"62'3537","reported_seq":3633,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.531936+0000","last_change":"2024-09-11T00:30:54.997397+0000","last_active":"2024-09-11T00:56:28.531936+0000","last_peered":"2024-09-11T00:56:28.531936+0000","last_clean":"2024-09-11T00:56:28.531936+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:56:28.531936+0000","last_undegraded":"2024-09-11T00:56:28.531936+0000","last_fullsized":"2024-09-11T00:56:28.531936+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3537,"log_dups_size":0,"ondisk_log_size":3537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:49:53.322576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18517,"num_write_kb":279095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"60'2630","reported_seq":2735,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058909+0000","last_change":"2024-09-11T00:30:45.925186+0000","last_active":"2024-09-11T00:56:28.058909+0000","last_peered":"2024-09-11T00:56:28.058909+0000","last_clean":"2024-09-11T00:56:28.058909+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:56:28.058909+0000","last_undegraded":"2024-09-11T00:56:28.058909+0000","last_fullsized":"2024-09-11T00:56:28.058909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:57:38.236346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":8,"num_write":13276,"num_write_kb":225109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"60'4","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051688+0000","last_change":"2024-09-11T00:30:44.927691+0000","last_active":"2024-09-11T00:56:28.051688+0000","last_peered":"2024-09-11T00:56:28.051688+0000","last_clean":"2024-09-11T00:56:28.051688+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:56:28.051688+0000","last_undegraded":"2024-09-11T00:56:28.051688+0000","last_fullsized":"2024-09-11T00:56:28.051688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:37.527075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"61'2","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066019+0000","last_change":"2024-09-11T00:30:53.986858+0000","last_active":"2024-09-11T00:56:28.066019+0000","last_peered":"2024-09-11T00:56:28.066019+0000","last_clean":"2024-09-11T00:56:28.066019+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:56:28.066019+0000","last_undegraded":"2024-09-11T00:56:28.066019+0000","last_fullsized":"2024-09-11T00:56:28.066019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:33.626193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"62'3470","reported_seq":3566,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397582+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:56:28.397582+0000","last_peered":"2024-09-11T00:56:28.397582+0000","last_clean":"2024-09-11T00:56:28.397582+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:56:28.397582+0000","last_undegraded":"2024-09-11T00:56:28.397582+0000","last_fullsized":"2024-09-11T00:56:28.397582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17553,"num_write_kb":279333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"60'2695","reported_seq":2802,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.046511+0000","last_change":"2024-09-11T00:30:45.930934+0000","last_active":"2024-09-11T00:56:28.046511+0000","last_peered":"2024-09-11T00:56:28.046511+0000","last_clean":"2024-09-11T00:56:28.046511+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:56:28.046511+0000","last_undegraded":"2024-09-11T00:56:28.046511+0000","last_fullsized":"2024-09-11T00:56:28.046511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:38.157097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":14,"num_write":14032,"num_write_kb":244535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"60'3","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048721+0000","last_change":"2024-09-11T00:30:44.917420+0000","last_active":"2024-09-11T00:56:28.048721+0000","last_peered":"2024-09-11T00:56:28.048721+0000","last_clean":"2024-09-11T00:56:28.048721+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:56:28.048721+0000","last_undegraded":"2024-09-11T00:56:28.048721+0000","last_fullsized":"2024-09-11T00:56:28.048721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:41:58.048598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"62'5","reported_seq":124,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.505942+0000","last_change":"2024-09-11T00:30:53.990765+0000","last_active":"2024-09-11T00:56:28.505942+0000","last_peered":"2024-09-11T00:56:28.505942+0000","last_clean":"2024-09-11T00:56:28.505942+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:56:28.505942+0000","last_undegraded":"2024-09-11T00:56:28.505942+0000","last_fullsized":"2024-09-11T00:56:28.505942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:33.720727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"62'3622","reported_seq":3718,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.450009+0000","last_change":"2024-09-11T00:30:54.984949+0000","last_active":"2024-09-11T00:56:28.450009+0000","last_peered":"2024-09-11T00:56:28.450009+0000","last_clean":"2024-09-11T00:56:28.450009+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:56:28.450009+0000","last_undegraded":"2024-09-11T00:56:28.450009+0000","last_fullsized":"2024-09-11T00:56:28.450009+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:54.346937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18608,"num_write_kb":291609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"60'2732","reported_seq":2838,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063247+0000","last_change":"2024-09-11T00:30:45.931986+0000","last_active":"2024-09-11T00:56:28.063247+0000","last_peered":"2024-09-11T00:56:28.063247+0000","last_clean":"2024-09-11T00:56:28.063247+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:56:28.063247+0000","last_undegraded":"2024-09-11T00:56:28.063247+0000","last_fullsized":"2024-09-11T00:56:28.063247+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13936,"num_write_kb":248214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"60'3","reported_seq":118,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:50.663864+0000","last_change":"2024-09-11T00:56:50.663864+0000","last_active":"2024-09-11T00:56:50.663864+0000","last_peered":"2024-09-11T00:56:50.663864+0000","last_clean":"2024-09-11T00:56:50.663864+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:56:50.663864+0000","last_undegraded":"2024-09-11T00:56:50.663864+0000","last_fullsized":"2024-09-11T00:56:50.663864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3","last_scrub_stamp":"2024-09-11T00:56:50.663784+0000","last_deep_scrub":"60'3","last_deep_scrub_stamp":"2024-09-11T00:56:50.663784+0000","last_clean_scrub_stamp":"2024-09-11T00:56:50.663784+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:41:39.646786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"62'6","reported_seq":146,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500913+0000","last_change":"2024-09-11T00:30:53.986940+0000","last_active":"2024-09-11T00:56:28.500913+0000","last_peered":"2024-09-11T00:56:28.500913+0000","last_clean":"2024-09-11T00:56:28.500913+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:56:28.500913+0000","last_undegraded":"2024-09-11T00:56:28.500913+0000","last_fullsized":"2024-09-11T00:56:28.500913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:11.942187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"62'3257","reported_seq":3355,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.520613+0000","last_change":"2024-09-11T00:30:54.983486+0000","last_active":"2024-09-11T00:56:28.520613+0000","last_peered":"2024-09-11T00:56:28.520613+0000","last_clean":"2024-09-11T00:56:28.520613+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:56:28.520613+0000","last_undegraded":"2024-09-11T00:56:28.520613+0000","last_fullsized":"2024-09-11T00:56:28.520613+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:55:08.654100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17168,"num_write_kb":277608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'2688","reported_seq":2795,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048142+0000","last_change":"2024-09-11T00:30:45.924301+0000","last_active":"2024-09-11T00:56:28.048142+0000","last_peered":"2024-09-11T00:56:28.048142+0000","last_clean":"2024-09-11T00:56:28.048142+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:56:28.048142+0000","last_undegraded":"2024-09-11T00:56:28.048142+0000","last_fullsized":"2024-09-11T00:56:28.048142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:11.561302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13441,"num_write_kb":246089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"62'5","reported_seq":141,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.085125+0000","last_change":"2024-09-11T00:30:44.929472+0000","last_active":"2024-09-11T00:56:31.085125+0000","last_peered":"2024-09-11T00:56:31.085125+0000","last_clean":"2024-09-11T00:56:31.085125+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:56:31.085125+0000","last_undegraded":"2024-09-11T00:56:31.085125+0000","last_fullsized":"2024-09-11T00:56:31.085125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:18.136598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"62'65","reported_seq":245,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:37.625005+0000","last_change":"2024-09-11T00:30:53.988539+0000","last_active":"2024-09-11T00:56:37.625005+0000","last_peered":"2024-09-11T00:56:37.625005+0000","last_clean":"2024-09-11T00:56:37.625005+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:56:37.625005+0000","last_undegraded":"2024-09-11T00:56:37.625005+0000","last_fullsized":"2024-09-11T00:56:37.625005+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:19.083664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":126,"num_write":62,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"62'3386","reported_seq":3484,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.422439+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:56:28.422439+0000","last_peered":"2024-09-11T00:56:28.422439+0000","last_clean":"2024-09-11T00:56:28.422439+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:56:28.422439+0000","last_undegraded":"2024-09-11T00:56:28.422439+0000","last_fullsized":"2024-09-11T00:56:28.422439+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18005,"num_write_kb":276900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'3194","reported_seq":3299,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.050607+0000","last_change":"2024-09-11T00:30:45.925504+0000","last_active":"2024-09-11T00:56:28.050607+0000","last_peered":"2024-09-11T00:56:28.050607+0000","last_clean":"2024-09-11T00:56:28.050607+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:56:28.050607+0000","last_undegraded":"2024-09-11T00:56:28.050607+0000","last_fullsized":"2024-09-11T00:56:28.050607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:46:08.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16397,"num_write_kb":262231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"62'521","reported_seq":642,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:33.047686+0000","last_change":"2024-09-11T00:30:44.918612+0000","last_active":"2024-09-11T00:56:33.047686+0000","last_peered":"2024-09-11T00:56:33.047686+0000","last_clean":"2024-09-11T00:56:33.047686+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:56:33.047686+0000","last_undegraded":"2024-09-11T00:56:33.047686+0000","last_fullsized":"2024-09-11T00:56:33.047686+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:51.207360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":41,"num_write":1157,"num_write_kb":372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"62'9","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.502805+0000","last_change":"2024-09-11T00:30:53.982497+0000","last_active":"2024-09-11T00:56:28.502805+0000","last_peered":"2024-09-11T00:56:28.502805+0000","last_clean":"2024-09-11T00:56:28.502805+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:56:28.502805+0000","last_undegraded":"2024-09-11T00:56:28.502805+0000","last_fullsized":"2024-09-11T00:56:28.502805+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:22:21.437255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"61'3259","reported_seq":3357,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063309+0000","last_change":"2024-09-11T00:30:54.993118+0000","last_active":"2024-09-11T00:56:28.063309+0000","last_peered":"2024-09-11T00:56:28.063309+0000","last_clean":"2024-09-11T00:56:28.063309+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:56:28.063309+0000","last_undegraded":"2024-09-11T00:56:28.063309+0000","last_fullsized":"2024-09-11T00:56:28.063309+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:07:41.148370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16779,"num_write_kb":276080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"60'3127","reported_seq":3233,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052954+0000","last_change":"2024-09-11T00:30:45.936207+0000","last_active":"2024-09-11T00:56:28.052954+0000","last_peered":"2024-09-11T00:56:28.052954+0000","last_clean":"2024-09-11T00:56:28.052954+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:56:28.052954+0000","last_undegraded":"2024-09-11T00:56:28.052954+0000","last_fullsized":"2024-09-11T00:56:28.052954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:57.613008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":12,"num_write":16580,"num_write_kb":258802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"60'4","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069515+0000","last_change":"2024-09-11T00:30:44.924928+0000","last_active":"2024-09-11T00:56:28.069515+0000","last_peered":"2024-09-11T00:56:28.069515+0000","last_clean":"2024-09-11T00:56:28.069515+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:56:28.069515+0000","last_undegraded":"2024-09-11T00:56:28.069515+0000","last_fullsized":"2024-09-11T00:56:28.069515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:39:21.890917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'1","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052292+0000","last_change":"2024-09-11T00:30:53.986843+0000","last_active":"2024-09-11T00:56:28.052292+0000","last_peered":"2024-09-11T00:56:28.052292+0000","last_clean":"2024-09-11T00:56:28.052292+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:56:28.052292+0000","last_undegraded":"2024-09-11T00:56:28.052292+0000","last_fullsized":"2024-09-11T00:56:28.052292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:48:44.684154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"62'3801","reported_seq":3898,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.047541+0000","last_change":"2024-09-11T00:30:54.987116+0000","last_active":"2024-09-11T00:56:28.047541+0000","last_peered":"2024-09-11T00:56:28.047541+0000","last_clean":"2024-09-11T00:56:28.047541+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:56:28.047541+0000","last_undegraded":"2024-09-11T00:56:28.047541+0000","last_fullsized":"2024-09-11T00:56:28.047541+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:42.410621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20214,"num_write_kb":289874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"60'2498","reported_seq":2613,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.133953+0000","last_change":"2024-09-11T00:56:54.133953+0000","last_active":"2024-09-11T00:56:54.133953+0000","last_peered":"2024-09-11T00:56:54.133953+0000","last_clean":"2024-09-11T00:56:54.133953+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:56:54.133953+0000","last_undegraded":"2024-09-11T00:56:54.133953+0000","last_fullsized":"2024-09-11T00:56:54.133953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2498","last_scrub_stamp":"2024-09-11T00:56:54.133840+0000","last_deep_scrub":"60'2498","last_deep_scrub_stamp":"2024-09-11T00:56:54.133840+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.133840+0000","objects_scrubbed":0,"log_size":2498,"log_dups_size":0,"ondisk_log_size":2498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:46:40.030654+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":13564,"num_write_kb":230524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"60'5","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059256+0000","last_change":"2024-09-11T00:30:44.926701+0000","last_active":"2024-09-11T00:56:28.059256+0000","last_peered":"2024-09-11T00:56:28.059256+0000","last_clean":"2024-09-11T00:56:28.059256+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:56:28.059256+0000","last_undegraded":"2024-09-11T00:56:28.059256+0000","last_fullsized":"2024-09-11T00:56:28.059256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:43.549040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"62'5","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510149+0000","last_change":"2024-09-11T00:30:53.986896+0000","last_active":"2024-09-11T00:56:28.510149+0000","last_peered":"2024-09-11T00:56:28.510149+0000","last_clean":"2024-09-11T00:56:28.510149+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:56:28.510149+0000","last_undegraded":"2024-09-11T00:56:28.510149+0000","last_fullsized":"2024-09-11T00:56:28.510149+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:53:06.590278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"62'3103","reported_seq":3201,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058933+0000","last_change":"2024-09-11T00:30:55.001370+0000","last_active":"2024-09-11T00:56:28.058933+0000","last_peered":"2024-09-11T00:56:28.058933+0000","last_clean":"2024-09-11T00:56:28.058933+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:56:28.058933+0000","last_undegraded":"2024-09-11T00:56:28.058933+0000","last_fullsized":"2024-09-11T00:56:28.058933+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:56.592767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16058,"num_write_kb":244544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"60'2543","reported_seq":2648,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067860+0000","last_change":"2024-09-11T00:30:45.938048+0000","last_active":"2024-09-11T00:56:28.067860+0000","last_peered":"2024-09-11T00:56:28.067860+0000","last_clean":"2024-09-11T00:56:28.067860+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:56:28.067860+0000","last_undegraded":"2024-09-11T00:56:28.067860+0000","last_fullsized":"2024-09-11T00:56:28.067860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:05:42.162059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":230458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"60'5","reported_seq":120,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.289576+0000","last_change":"2024-09-11T00:56:54.289576+0000","last_active":"2024-09-11T00:56:54.289576+0000","last_peered":"2024-09-11T00:56:54.289576+0000","last_clean":"2024-09-11T00:56:54.289576+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:56:54.289576+0000","last_undegraded":"2024-09-11T00:56:54.289576+0000","last_fullsized":"2024-09-11T00:56:54.289576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'5","last_scrub_stamp":"2024-09-11T00:56:54.289506+0000","last_deep_scrub":"60'5","last_deep_scrub_stamp":"2024-09-11T00:56:54.289506+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.289506+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:31:35.983150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"62'5","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.494925+0000","last_change":"2024-09-11T00:30:53.978480+0000","last_active":"2024-09-11T00:56:28.494925+0000","last_peered":"2024-09-11T00:56:28.494925+0000","last_clean":"2024-09-11T00:56:28.494925+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:56:28.494925+0000","last_undegraded":"2024-09-11T00:56:28.494925+0000","last_fullsized":"2024-09-11T00:56:28.494925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:58.900565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"61'3199","reported_seq":3296,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058202+0000","last_change":"2024-09-11T00:30:54.997641+0000","last_active":"2024-09-11T00:56:28.058202+0000","last_peered":"2024-09-11T00:56:28.058202+0000","last_clean":"2024-09-11T00:56:28.058202+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:56:28.058202+0000","last_undegraded":"2024-09-11T00:56:28.058202+0000","last_fullsized":"2024-09-11T00:56:28.058202+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:55:18.437490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15924,"num_write_kb":260293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"60'3298","reported_seq":3405,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057329+0000","last_change":"2024-09-11T00:30:45.928456+0000","last_active":"2024-09-11T00:56:28.057329+0000","last_peered":"2024-09-11T00:56:28.057329+0000","last_clean":"2024-09-11T00:56:28.057329+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:56:28.057329+0000","last_undegraded":"2024-09-11T00:56:28.057329+0000","last_fullsized":"2024-09-11T00:56:28.057329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17655,"num_write_kb":267324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"62'73","reported_seq":262,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:33.050070+0000","last_change":"2024-09-11T00:30:44.928021+0000","last_active":"2024-09-11T00:56:33.050070+0000","last_peered":"2024-09-11T00:56:33.050070+0000","last_clean":"2024-09-11T00:56:33.050070+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:56:33.050070+0000","last_undegraded":"2024-09-11T00:56:33.050070+0000","last_fullsized":"2024-09-11T00:56:33.050070+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:38.391668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":129,"num_write":68,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"62'5","reported_seq":147,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.544700+0000","last_change":"2024-09-11T00:30:53.979031+0000","last_active":"2024-09-11T00:56:28.544700+0000","last_peered":"2024-09-11T00:56:28.544700+0000","last_clean":"2024-09-11T00:56:28.544700+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:56:28.544700+0000","last_undegraded":"2024-09-11T00:56:28.544700+0000","last_fullsized":"2024-09-11T00:56:28.544700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"62'2728","reported_seq":2824,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.475254+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:56:28.475254+0000","last_peered":"2024-09-11T00:56:28.475254+0000","last_clean":"2024-09-11T00:56:28.475254+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:56:28.475254+0000","last_undegraded":"2024-09-11T00:56:28.475254+0000","last_fullsized":"2024-09-11T00:56:28.475254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2728,"log_dups_size":0,"ondisk_log_size":2728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:02:39.629122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":237243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"60'2962","reported_seq":3068,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051045+0000","last_change":"2024-09-11T00:30:45.916630+0000","last_active":"2024-09-11T00:56:28.051045+0000","last_peered":"2024-09-11T00:56:28.051045+0000","last_clean":"2024-09-11T00:56:28.051045+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:56:28.051045+0000","last_undegraded":"2024-09-11T00:56:28.051045+0000","last_fullsized":"2024-09-11T00:56:28.051045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:53.435836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15238,"num_write_kb":265720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"62'8","reported_seq":124,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:53.315719+0000","last_change":"2024-09-11T00:56:53.315719+0000","last_active":"2024-09-11T00:56:53.315719+0000","last_peered":"2024-09-11T00:56:53.315719+0000","last_clean":"2024-09-11T00:56:53.315719+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:56:53.315719+0000","last_undegraded":"2024-09-11T00:56:53.315719+0000","last_fullsized":"2024-09-11T00:56:53.315719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-09-11T00:56:53.315634+0000","last_deep_scrub":"62'8","last_deep_scrub_stamp":"2024-09-11T00:56:53.315634+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.315634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:27:19.909262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"62'499","reported_seq":619,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:37.590769+0000","last_change":"2024-09-11T00:30:53.989299+0000","last_active":"2024-09-11T00:56:37.590769+0000","last_peered":"2024-09-11T00:56:37.590769+0000","last_clean":"2024-09-11T00:56:37.590769+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:56:37.590769+0000","last_undegraded":"2024-09-11T00:56:37.590769+0000","last_fullsized":"2024-09-11T00:56:37.590769+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":1123,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"62'3325","reported_seq":3422,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.494312+0000","last_change":"2024-09-11T00:30:54.990026+0000","last_active":"2024-09-11T00:56:28.494312+0000","last_peered":"2024-09-11T00:56:28.494312+0000","last_clean":"2024-09-11T00:56:28.494312+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:56:28.494312+0000","last_undegraded":"2024-09-11T00:56:28.494312+0000","last_fullsized":"2024-09-11T00:56:28.494312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:02:10.800530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17451,"num_write_kb":278810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"60'3332","reported_seq":3438,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051728+0000","last_change":"2024-09-11T00:30:45.919816+0000","last_active":"2024-09-11T00:56:28.051728+0000","last_peered":"2024-09-11T00:56:28.051728+0000","last_clean":"2024-09-11T00:56:28.051728+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:56:28.051728+0000","last_undegraded":"2024-09-11T00:56:28.051728+0000","last_fullsized":"2024-09-11T00:56:28.051728+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3332,"log_dups_size":0,"ondisk_log_size":3332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:24:26.950871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17164,"num_write_kb":285768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"31'1","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.072247+0000","last_change":"2024-09-11T00:30:44.929689+0000","last_active":"2024-09-11T00:56:28.072247+0000","last_peered":"2024-09-11T00:56:28.072247+0000","last_clean":"2024-09-11T00:56:28.072247+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:56:28.072247+0000","last_undegraded":"2024-09-11T00:56:28.072247+0000","last_fullsized":"2024-09-11T00:56:28.072247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:47:22.577086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"62'74","reported_seq":214,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:50.518182+0000","last_change":"2024-09-11T00:56:50.518182+0000","last_active":"2024-09-11T00:56:50.518182+0000","last_peered":"2024-09-11T00:56:50.518182+0000","last_clean":"2024-09-11T00:56:50.518182+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:56:50.518182+0000","last_undegraded":"2024-09-11T00:56:50.518182+0000","last_fullsized":"2024-09-11T00:56:50.518182+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'74","last_scrub_stamp":"2024-09-11T00:56:50.518060+0000","last_deep_scrub":"62'74","last_deep_scrub_stamp":"2024-09-11T00:56:50.518060+0000","last_clean_scrub_stamp":"2024-09-11T00:56:50.518060+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:18.176495+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"62'4","reported_seq":103,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488198+0000","last_change":"2024-09-11T00:30:53.982142+0000","last_active":"2024-09-11T00:56:28.488198+0000","last_peered":"2024-09-11T00:56:28.488198+0000","last_clean":"2024-09-11T00:56:28.488198+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:56:28.488198+0000","last_undegraded":"2024-09-11T00:56:28.488198+0000","last_fullsized":"2024-09-11T00:56:28.488198+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:41.558944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"62'2541","reported_seq":2637,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500843+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:56:28.500843+0000","last_peered":"2024-09-11T00:56:28.500843+0000","last_clean":"2024-09-11T00:56:28.500843+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:56:28.500843+0000","last_undegraded":"2024-09-11T00:56:28.500843+0000","last_fullsized":"2024-09-11T00:56:28.500843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12611,"num_write_kb":206053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"60'2809","reported_seq":2917,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069156+0000","last_change":"2024-09-11T00:30:45.924464+0000","last_active":"2024-09-11T00:56:28.069156+0000","last_peered":"2024-09-11T00:56:28.069156+0000","last_clean":"2024-09-11T00:56:28.069156+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:56:28.069156+0000","last_undegraded":"2024-09-11T00:56:28.069156+0000","last_fullsized":"2024-09-11T00:56:28.069156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2809,"log_dups_size":0,"ondisk_log_size":2809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:24.677879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":34,"num_write":15096,"num_write_kb":232439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"60'2","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.053330+0000","last_change":"2024-09-11T00:30:44.915904+0000","last_active":"2024-09-11T00:56:28.053330+0000","last_peered":"2024-09-11T00:56:28.053330+0000","last_clean":"2024-09-11T00:56:28.053330+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:56:28.053330+0000","last_undegraded":"2024-09-11T00:56:28.053330+0000","last_fullsized":"2024-09-11T00:56:28.053330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:22.415388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"62'4","reported_seq":103,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.889909+0000","last_change":"2024-09-11T00:30:53.989292+0000","last_active":"2024-09-11T00:56:28.889909+0000","last_peered":"2024-09-11T00:56:28.889909+0000","last_clean":"2024-09-11T00:56:28.889909+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:56:28.889909+0000","last_undegraded":"2024-09-11T00:56:28.889909+0000","last_fullsized":"2024-09-11T00:56:28.889909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"62'3293","reported_seq":3389,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.437769+0000","last_change":"2024-09-11T00:30:54.981630+0000","last_active":"2024-09-11T00:56:28.437769+0000","last_peered":"2024-09-11T00:56:28.437769+0000","last_clean":"2024-09-11T00:56:28.437769+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:56:28.437769+0000","last_undegraded":"2024-09-11T00:56:28.437769+0000","last_fullsized":"2024-09-11T00:56:28.437769+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:12.832327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17661,"num_write_kb":267414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"60'3034","reported_seq":3140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066511+0000","last_change":"2024-09-11T00:30:45.927117+0000","last_active":"2024-09-11T00:56:28.066511+0000","last_peered":"2024-09-11T00:56:28.066511+0000","last_clean":"2024-09-11T00:56:28.066511+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:56:28.066511+0000","last_undegraded":"2024-09-11T00:56:28.066511+0000","last_fullsized":"2024-09-11T00:56:28.066511+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3034,"log_dups_size":0,"ondisk_log_size":3034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:13.340402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":15513,"num_write_kb":283129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"60'2","reported_seq":122,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060350+0000","last_change":"2024-09-11T00:30:44.925000+0000","last_active":"2024-09-11T00:56:28.060350+0000","last_peered":"2024-09-11T00:56:28.060350+0000","last_clean":"2024-09-11T00:56:28.060350+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:56:28.060350+0000","last_undegraded":"2024-09-11T00:56:28.060350+0000","last_fullsized":"2024-09-11T00:56:28.060350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:49.805057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"62'1443","reported_seq":1586,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510169+0000","last_change":"2024-09-11T00:30:53.984013+0000","last_active":"2024-09-11T00:56:28.510169+0000","last_peered":"2024-09-11T00:56:28.510169+0000","last_clean":"2024-09-11T00:56:28.510169+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:56:28.510169+0000","last_undegraded":"2024-09-11T00:56:28.510169+0000","last_fullsized":"2024-09-11T00:56:28.510169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":1443,"log_dups_size":0,"ondisk_log_size":1443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:18:49.963623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":36861,"num_write":1416,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'2904","reported_seq":3001,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.471315+0000","last_change":"2024-09-11T00:30:54.996860+0000","last_active":"2024-09-11T00:56:28.471315+0000","last_peered":"2024-09-11T00:56:28.471315+0000","last_clean":"2024-09-11T00:56:28.471315+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:56:28.471315+0000","last_undegraded":"2024-09-11T00:56:28.471315+0000","last_fullsized":"2024-09-11T00:56:28.471315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:46:43.516748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15203,"num_write_kb":241147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"60'3390","reported_seq":3494,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069427+0000","last_change":"2024-09-11T00:30:45.930078+0000","last_active":"2024-09-11T00:56:28.069427+0000","last_peered":"2024-09-11T00:56:28.069427+0000","last_clean":"2024-09-11T00:56:28.069427+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:56:28.069427+0000","last_undegraded":"2024-09-11T00:56:28.069427+0000","last_fullsized":"2024-09-11T00:56:28.069427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:44.556323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17423,"num_write_kb":299173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"60'6","reported_seq":140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.070765+0000","last_change":"2024-09-11T00:30:44.929920+0000","last_active":"2024-09-11T00:56:31.070765+0000","last_peered":"2024-09-11T00:56:31.070765+0000","last_clean":"2024-09-11T00:56:31.070765+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:56:31.070765+0000","last_undegraded":"2024-09-11T00:56:31.070765+0000","last_fullsized":"2024-09-11T00:56:31.070765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:18:31.813626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"62'3","reported_seq":102,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.493628+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:56:28.493628+0000","last_peered":"2024-09-11T00:56:28.493628+0000","last_clean":"2024-09-11T00:56:28.493628+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:56:28.493628+0000","last_undegraded":"2024-09-11T00:56:28.493628+0000","last_fullsized":"2024-09-11T00:56:28.493628+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"62'3177","reported_seq":3279,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.444533+0000","last_change":"2024-09-11T00:31:26.120086+0000","last_active":"2024-09-11T00:56:28.444533+0000","last_peered":"2024-09-11T00:56:28.444533+0000","last_clean":"2024-09-11T00:56:28.444533+0000","last_became_active":"2024-09-11T00:31:26.119641+0000","last_became_peered":"2024-09-11T00:31:26.119641+0000","last_unstale":"2024-09-11T00:56:28.444533+0000","last_undegraded":"2024-09-11T00:56:28.444533+0000","last_fullsized":"2024-09-11T00:56:28.444533+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3177,"log_dups_size":0,"ondisk_log_size":3177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:21:15.814831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16730,"num_write_kb":276023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"60'2906","reported_seq":3010,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.077855+0000","last_change":"2024-09-11T00:30:45.923521+0000","last_active":"2024-09-11T00:56:28.077855+0000","last_peered":"2024-09-11T00:56:28.077855+0000","last_clean":"2024-09-11T00:56:28.077855+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:56:28.077855+0000","last_undegraded":"2024-09-11T00:56:28.077855+0000","last_fullsized":"2024-09-11T00:56:28.077855+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:06.514410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14712,"num_write_kb":255517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"62'6","reported_seq":134,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:51.684558+0000","last_change":"2024-09-11T00:56:51.684558+0000","last_active":"2024-09-11T00:56:51.684558+0000","last_peered":"2024-09-11T00:56:51.684558+0000","last_clean":"2024-09-11T00:56:51.684558+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:56:51.684558+0000","last_undegraded":"2024-09-11T00:56:51.684558+0000","last_fullsized":"2024-09-11T00:56:51.684558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:56:51.684468+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:56:51.684468+0000","last_clean_scrub_stamp":"2024-09-11T00:56:51.684468+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:43.127053+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"62'3","reported_seq":102,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.506920+0000","last_change":"2024-09-11T00:30:53.983352+0000","last_active":"2024-09-11T00:56:28.506920+0000","last_peered":"2024-09-11T00:56:28.506920+0000","last_clean":"2024-09-11T00:56:28.506920+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:56:28.506920+0000","last_undegraded":"2024-09-11T00:56:28.506920+0000","last_fullsized":"2024-09-11T00:56:28.506920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"62'2973","reported_seq":3063,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.532651+0000","last_change":"2024-09-11T00:31:26.119816+0000","last_active":"2024-09-11T00:56:28.532651+0000","last_peered":"2024-09-11T00:56:28.532651+0000","last_clean":"2024-09-11T00:56:28.532651+0000","last_became_active":"2024-09-11T00:31:26.119440+0000","last_became_peered":"2024-09-11T00:31:26.119440+0000","last_unstale":"2024-09-11T00:56:28.532651+0000","last_undegraded":"2024-09-11T00:56:28.532651+0000","last_fullsized":"2024-09-11T00:56:28.532651+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:21:45.707321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15629,"num_write_kb":242839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"60'3355","reported_seq":3459,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062827+0000","last_change":"2024-09-11T00:30:45.931685+0000","last_active":"2024-09-11T00:56:28.062827+0000","last_peered":"2024-09-11T00:56:28.062827+0000","last_clean":"2024-09-11T00:56:28.062827+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:56:28.062827+0000","last_undegraded":"2024-09-11T00:56:28.062827+0000","last_fullsized":"2024-09-11T00:56:28.062827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:14.754120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18048,"num_write_kb":289456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"62'12","reported_seq":177,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.077796+0000","last_change":"2024-09-11T00:30:44.913102+0000","last_active":"2024-09-11T00:56:31.077796+0000","last_peered":"2024-09-11T00:56:31.077796+0000","last_clean":"2024-09-11T00:56:31.077796+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:56:31.077796+0000","last_undegraded":"2024-09-11T00:56:31.077796+0000","last_fullsized":"2024-09-11T00:56:31.077796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:37:25.592357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":177,"num_read_kb":177,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"62'2","reported_seq":99,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488958+0000","last_change":"2024-09-11T00:30:53.980703+0000","last_active":"2024-09-11T00:56:28.488958+0000","last_peered":"2024-09-11T00:56:28.488958+0000","last_clean":"2024-09-11T00:56:28.488958+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:56:28.488958+0000","last_undegraded":"2024-09-11T00:56:28.488958+0000","last_fullsized":"2024-09-11T00:56:28.488958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:51.050512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"62'2678","reported_seq":2774,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.413635+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:56:28.413635+0000","last_peered":"2024-09-11T00:56:28.413635+0000","last_clean":"2024-09-11T00:56:28.413635+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:56:28.413635+0000","last_undegraded":"2024-09-11T00:56:28.413635+0000","last_fullsized":"2024-09-11T00:56:28.413635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2678,"log_dups_size":0,"ondisk_log_size":2678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13518,"num_write_kb":242590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"60'2791","reported_seq":2903,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:53.489186+0000","last_change":"2024-09-11T00:56:53.489186+0000","last_active":"2024-09-11T00:56:53.489186+0000","last_peered":"2024-09-11T00:56:53.489186+0000","last_clean":"2024-09-11T00:56:53.489186+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:56:53.489186+0000","last_undegraded":"2024-09-11T00:56:53.489186+0000","last_fullsized":"2024-09-11T00:56:53.489186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2791","last_scrub_stamp":"2024-09-11T00:56:53.489113+0000","last_deep_scrub":"60'2791","last_deep_scrub_stamp":"2024-09-11T00:56:53.489113+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.489113+0000","objects_scrubbed":0,"log_size":2791,"log_dups_size":0,"ondisk_log_size":2791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:13.547085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":254542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"60'6","reported_seq":120,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:52.719391+0000","last_change":"2024-09-11T00:56:52.719391+0000","last_active":"2024-09-11T00:56:52.719391+0000","last_peered":"2024-09-11T00:56:52.719391+0000","last_clean":"2024-09-11T00:56:52.719391+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:56:52.719391+0000","last_undegraded":"2024-09-11T00:56:52.719391+0000","last_fullsized":"2024-09-11T00:56:52.719391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'6","last_scrub_stamp":"2024-09-11T00:56:52.719331+0000","last_deep_scrub":"60'6","last_deep_scrub_stamp":"2024-09-11T00:56:52.719331+0000","last_clean_scrub_stamp":"2024-09-11T00:56:52.719331+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:18.018393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"62'3","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059468+0000","last_change":"2024-09-11T00:30:53.987829+0000","last_active":"2024-09-11T00:56:28.059468+0000","last_peered":"2024-09-11T00:56:28.059468+0000","last_clean":"2024-09-11T00:56:28.059468+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:56:28.059468+0000","last_undegraded":"2024-09-11T00:56:28.059468+0000","last_fullsized":"2024-09-11T00:56:28.059468+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:57:43.792089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"62'3272","reported_seq":3367,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066748+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:56:28.066748+0000","last_peered":"2024-09-11T00:56:28.066748+0000","last_clean":"2024-09-11T00:56:28.066748+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:56:28.066748+0000","last_undegraded":"2024-09-11T00:56:28.066748+0000","last_fullsized":"2024-09-11T00:56:28.066748+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16692,"num_write_kb":275642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"60'3405","reported_seq":3511,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066672+0000","last_change":"2024-09-11T00:30:45.923043+0000","last_active":"2024-09-11T00:56:28.066672+0000","last_peered":"2024-09-11T00:56:28.066672+0000","last_clean":"2024-09-11T00:56:28.066672+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:56:28.066672+0000","last_undegraded":"2024-09-11T00:56:28.066672+0000","last_fullsized":"2024-09-11T00:56:28.066672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3405,"log_dups_size":0,"ondisk_log_size":3405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:08.699400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":17487,"num_write_kb":315618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"60'5","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067380+0000","last_change":"2024-09-11T00:30:44.921061+0000","last_active":"2024-09-11T00:56:28.067380+0000","last_peered":"2024-09-11T00:56:28.067380+0000","last_clean":"2024-09-11T00:56:28.067380+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:56:28.067380+0000","last_undegraded":"2024-09-11T00:56:28.067380+0000","last_fullsized":"2024-09-11T00:56:28.067380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:35.956831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"62'367","reported_seq":572,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.957442+0000","last_change":"2024-09-11T00:30:53.984057+0000","last_active":"2024-09-11T00:56:34.957442+0000","last_peered":"2024-09-11T00:56:34.957442+0000","last_clean":"2024-09-11T00:56:34.957442+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:56:34.957442+0000","last_undegraded":"2024-09-11T00:56:34.957442+0000","last_fullsized":"2024-09-11T00:56:34.957442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:10.477916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":191,"num_read_kb":78004,"num_write":360,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"62'4540","reported_seq":4637,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.544292+0000","last_change":"2024-09-11T00:30:54.989646+0000","last_active":"2024-09-11T00:56:28.544292+0000","last_peered":"2024-09-11T00:56:28.544292+0000","last_clean":"2024-09-11T00:56:28.544292+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:56:28.544292+0000","last_undegraded":"2024-09-11T00:56:28.544292+0000","last_fullsized":"2024-09-11T00:56:28.544292+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:23.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23424,"num_write_kb":360436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"60'2652","reported_seq":2757,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051484+0000","last_change":"2024-09-11T00:30:45.915657+0000","last_active":"2024-09-11T00:56:28.051484+0000","last_peered":"2024-09-11T00:56:28.051484+0000","last_clean":"2024-09-11T00:56:28.051484+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:56:28.051484+0000","last_undegraded":"2024-09-11T00:56:28.051484+0000","last_fullsized":"2024-09-11T00:56:28.051484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:16.277815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":13591,"num_write_kb":238653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"62'9","reported_seq":143,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.641347+0000","last_change":"2024-09-11T00:30:44.920079+0000","last_active":"2024-09-11T00:56:28.641347+0000","last_peered":"2024-09-11T00:56:28.641347+0000","last_clean":"2024-09-11T00:56:28.641347+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:56:28.641347+0000","last_undegraded":"2024-09-11T00:56:28.641347+0000","last_fullsized":"2024-09-11T00:56:28.641347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:17:30.671129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":112,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"62'37","reported_seq":197,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.985643+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:56:28.985643+0000","last_peered":"2024-09-11T00:56:28.985643+0000","last_clean":"2024-09-11T00:56:28.985643+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:56:28.985643+0000","last_undegraded":"2024-09-11T00:56:28.985643+0000","last_fullsized":"2024-09-11T00:56:28.985643+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":36,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"62'4259","reported_seq":4356,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.548466+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:56:28.548466+0000","last_peered":"2024-09-11T00:56:28.548466+0000","last_clean":"2024-09-11T00:56:28.548466+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:56:28.548466+0000","last_undegraded":"2024-09-11T00:56:28.548466+0000","last_fullsized":"2024-09-11T00:56:28.548466+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22192,"num_write_kb":330506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"60'2765","reported_seq":2868,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067495+0000","last_change":"2024-09-11T00:30:45.927482+0000","last_active":"2024-09-11T00:56:28.067495+0000","last_peered":"2024-09-11T00:56:28.067495+0000","last_clean":"2024-09-11T00:56:28.067495+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:56:28.067495+0000","last_undegraded":"2024-09-11T00:56:28.067495+0000","last_fullsized":"2024-09-11T00:56:28.067495+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:52.941636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14757,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"62'392","reported_seq":577,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:53.146103+0000","last_change":"2024-09-11T00:56:53.146103+0000","last_active":"2024-09-11T00:56:53.146103+0000","last_peered":"2024-09-11T00:56:53.146103+0000","last_clean":"2024-09-11T00:56:53.146103+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:56:53.146103+0000","last_undegraded":"2024-09-11T00:56:53.146103+0000","last_fullsized":"2024-09-11T00:56:53.146103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'392","last_scrub_stamp":"2024-09-11T00:56:53.146012+0000","last_deep_scrub":"62'392","last_deep_scrub_stamp":"2024-09-11T00:56:53.146012+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.146012+0000","objects_scrubbed":4,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:09:09.764335+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":127,"num_read_kb":56252,"num_write":381,"num_write_kb":4290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"62'5","reported_seq":103,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.528773+0000","last_change":"2024-09-11T00:30:53.987025+0000","last_active":"2024-09-11T00:56:28.528773+0000","last_peered":"2024-09-11T00:56:28.528773+0000","last_clean":"2024-09-11T00:56:28.528773+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:56:28.528773+0000","last_undegraded":"2024-09-11T00:56:28.528773+0000","last_fullsized":"2024-09-11T00:56:28.528773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:48.846824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"62'3220","reported_seq":3316,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.464767+0000","last_change":"2024-09-11T00:30:54.986017+0000","last_active":"2024-09-11T00:56:28.464767+0000","last_peered":"2024-09-11T00:56:28.464767+0000","last_clean":"2024-09-11T00:56:28.464767+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:56:28.464767+0000","last_undegraded":"2024-09-11T00:56:28.464767+0000","last_fullsized":"2024-09-11T00:56:28.464767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:58:12.595770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16450,"num_write_kb":262397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"60'3928","reported_seq":4033,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064969+0000","last_change":"2024-09-11T00:30:45.943035+0000","last_active":"2024-09-11T00:56:28.064969+0000","last_peered":"2024-09-11T00:56:28.064969+0000","last_clean":"2024-09-11T00:56:28.064969+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:56:28.064969+0000","last_undegraded":"2024-09-11T00:56:28.064969+0000","last_fullsized":"2024-09-11T00:56:28.064969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:16.832815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20599,"num_write_kb":340888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"62'43","reported_seq":190,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:31.086970+0000","last_change":"2024-09-11T00:30:44.917869+0000","last_active":"2024-09-11T00:56:31.086970+0000","last_peered":"2024-09-11T00:56:31.086970+0000","last_clean":"2024-09-11T00:56:31.086970+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:56:31.086970+0000","last_undegraded":"2024-09-11T00:56:31.086970+0000","last_fullsized":"2024-09-11T00:56:31.086970+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:34.758510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":42,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"62'8","reported_seq":148,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.516812+0000","last_change":"2024-09-11T00:30:53.988462+0000","last_active":"2024-09-11T00:56:28.516812+0000","last_peered":"2024-09-11T00:56:28.516812+0000","last_clean":"2024-09-11T00:56:28.516812+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:56:28.516812+0000","last_undegraded":"2024-09-11T00:56:28.516812+0000","last_fullsized":"2024-09-11T00:56:28.516812+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:22.385557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"62'4111","reported_seq":4207,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.522560+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:56:28.522560+0000","last_peered":"2024-09-11T00:56:28.522560+0000","last_clean":"2024-09-11T00:56:28.522560+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:56:28.522560+0000","last_undegraded":"2024-09-11T00:56:28.522560+0000","last_fullsized":"2024-09-11T00:56:28.522560+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21746,"num_write_kb":328824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"60'2863","reported_seq":2972,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.060043+0000","last_change":"2024-09-11T00:30:45.944095+0000","last_active":"2024-09-11T00:56:28.060043+0000","last_peered":"2024-09-11T00:56:28.060043+0000","last_clean":"2024-09-11T00:56:28.060043+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:56:28.060043+0000","last_undegraded":"2024-09-11T00:56:28.060043+0000","last_fullsized":"2024-09-11T00:56:28.060043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2863,"log_dups_size":0,"ondisk_log_size":2863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:25.660074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":46,"num_write":14990,"num_write_kb":244331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"60'2","reported_seq":115,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:53.729230+0000","last_change":"2024-09-11T00:56:53.729230+0000","last_active":"2024-09-11T00:56:53.729230+0000","last_peered":"2024-09-11T00:56:53.729230+0000","last_clean":"2024-09-11T00:56:53.729230+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:56:53.729230+0000","last_undegraded":"2024-09-11T00:56:53.729230+0000","last_fullsized":"2024-09-11T00:56:53.729230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-11T00:56:53.729158+0000","last_deep_scrub":"60'2","last_deep_scrub_stamp":"2024-09-11T00:56:53.729158+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.729158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:06.778420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"62'12","reported_seq":203,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:34.264447+0000","last_change":"2024-09-11T00:30:53.986646+0000","last_active":"2024-09-11T00:56:34.264447+0000","last_peered":"2024-09-11T00:56:34.264447+0000","last_clean":"2024-09-11T00:56:34.264447+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:56:34.264447+0000","last_undegraded":"2024-09-11T00:56:34.264447+0000","last_fullsized":"2024-09-11T00:56:34.264447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:27.447940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":279,"num_read_kb":279,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"62'3641","reported_seq":3737,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514237+0000","last_change":"2024-09-11T00:30:54.982486+0000","last_active":"2024-09-11T00:56:28.514237+0000","last_peered":"2024-09-11T00:56:28.514237+0000","last_clean":"2024-09-11T00:56:28.514237+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:56:28.514237+0000","last_undegraded":"2024-09-11T00:56:28.514237+0000","last_fullsized":"2024-09-11T00:56:28.514237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:11:41.475164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19470,"num_write_kb":282742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"60'2896","reported_seq":3003,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.076032+0000","last_change":"2024-09-11T00:30:45.944827+0000","last_active":"2024-09-11T00:56:28.076032+0000","last_peered":"2024-09-11T00:56:28.076032+0000","last_clean":"2024-09-11T00:56:28.076032+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:56:28.076032+0000","last_undegraded":"2024-09-11T00:56:28.076032+0000","last_fullsized":"2024-09-11T00:56:28.076032+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:38:12.263806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":22,"num_write":14816,"num_write_kb":255894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"62'5","reported_seq":118,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:51.508473+0000","last_change":"2024-09-11T00:56:51.508473+0000","last_active":"2024-09-11T00:56:51.508473+0000","last_peered":"2024-09-11T00:56:51.508473+0000","last_clean":"2024-09-11T00:56:51.508473+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:56:51.508473+0000","last_undegraded":"2024-09-11T00:56:51.508473+0000","last_fullsized":"2024-09-11T00:56:51.508473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2024-09-11T00:56:51.508373+0000","last_deep_scrub":"62'5","last_deep_scrub_stamp":"2024-09-11T00:56:51.508373+0000","last_clean_scrub_stamp":"2024-09-11T00:56:51.508373+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:38:34.105569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"62'6","reported_seq":104,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.504113+0000","last_change":"2024-09-11T00:30:53.981388+0000","last_active":"2024-09-11T00:56:28.504113+0000","last_peered":"2024-09-11T00:56:28.504113+0000","last_clean":"2024-09-11T00:56:28.504113+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:56:28.504113+0000","last_undegraded":"2024-09-11T00:56:28.504113+0000","last_fullsized":"2024-09-11T00:56:28.504113+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:10:28.078067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"62'3862","reported_seq":3961,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.513928+0000","last_change":"2024-09-11T00:30:54.985091+0000","last_active":"2024-09-11T00:56:28.513928+0000","last_peered":"2024-09-11T00:56:28.513928+0000","last_clean":"2024-09-11T00:56:28.513928+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:56:28.513928+0000","last_undegraded":"2024-09-11T00:56:28.513928+0000","last_fullsized":"2024-09-11T00:56:28.513928+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:06:33.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20250,"num_write_kb":318736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"62'29","reported_seq":128,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.490960+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:56:28.490960+0000","last_peered":"2024-09-11T00:56:28.490960+0000","last_clean":"2024-09-11T00:56:28.490960+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:56:28.490960+0000","last_undegraded":"2024-09-11T00:56:28.490960+0000","last_fullsized":"2024-09-11T00:56:28.490960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:40:22.799068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"60'3526","reported_seq":3630,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048540+0000","last_change":"2024-09-11T00:30:45.913905+0000","last_active":"2024-09-11T00:56:28.048540+0000","last_peered":"2024-09-11T00:56:28.048540+0000","last_clean":"2024-09-11T00:56:28.048540+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:56:28.048540+0000","last_undegraded":"2024-09-11T00:56:28.048540+0000","last_fullsized":"2024-09-11T00:56:28.048540+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:10:35.427655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":18407,"num_write_kb":311214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"60'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.064114+0000","last_change":"2024-09-11T00:30:44.921909+0000","last_active":"2024-09-11T00:56:28.064114+0000","last_peered":"2024-09-11T00:56:28.064114+0000","last_clean":"2024-09-11T00:56:28.064114+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:56:28.064114+0000","last_undegraded":"2024-09-11T00:56:28.064114+0000","last_fullsized":"2024-09-11T00:56:28.064114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:51:09.268479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"61'3026","reported_seq":3123,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.053185+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:56:28.053185+0000","last_peered":"2024-09-11T00:56:28.053185+0000","last_clean":"2024-09-11T00:56:28.053185+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:56:28.053185+0000","last_undegraded":"2024-09-11T00:56:28.053185+0000","last_fullsized":"2024-09-11T00:56:28.053185+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:11.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16026,"num_write_kb":256702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"61'972","reported_seq":1122,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065625+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:56:28.065625+0000","last_peered":"2024-09-11T00:56:28.065625+0000","last_clean":"2024-09-11T00:56:28.065625+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:56:28.065625+0000","last_undegraded":"2024-09-11T00:56:28.065625+0000","last_fullsized":"2024-09-11T00:56:28.065625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":972,"log_dups_size":0,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":7829,"num_write":949,"num_write_kb":4657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"60'3393","reported_seq":3498,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057539+0000","last_change":"2024-09-11T00:30:45.931992+0000","last_active":"2024-09-11T00:56:28.057539+0000","last_peered":"2024-09-11T00:56:28.057539+0000","last_clean":"2024-09-11T00:56:28.057539+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:56:28.057539+0000","last_undegraded":"2024-09-11T00:56:28.057539+0000","last_fullsized":"2024-09-11T00:56:28.057539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:16.219353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":17621,"num_write_kb":291741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"60'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051439+0000","last_change":"2024-09-11T00:30:44.914915+0000","last_active":"2024-09-11T00:56:28.051439+0000","last_peered":"2024-09-11T00:56:28.051439+0000","last_clean":"2024-09-11T00:56:28.051439+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:56:28.051439+0000","last_undegraded":"2024-09-11T00:56:28.051439+0000","last_fullsized":"2024-09-11T00:56:28.051439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:26.730187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"62'3945","reported_seq":4040,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.470154+0000","last_change":"2024-09-11T00:30:54.997688+0000","last_active":"2024-09-11T00:56:28.470154+0000","last_peered":"2024-09-11T00:56:28.470154+0000","last_clean":"2024-09-11T00:56:28.470154+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:56:28.470154+0000","last_undegraded":"2024-09-11T00:56:28.470154+0000","last_fullsized":"2024-09-11T00:56:28.470154+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:58:06.843111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20523,"num_write_kb":340217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"62'101","reported_seq":208,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:37.634421+0000","last_change":"2024-09-11T00:30:44.927329+0000","last_active":"2024-09-11T00:56:37.634421+0000","last_peered":"2024-09-11T00:56:37.634421+0000","last_clean":"2024-09-11T00:56:37.634421+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:56:37.634421+0000","last_undegraded":"2024-09-11T00:56:37.634421+0000","last_fullsized":"2024-09-11T00:56:37.634421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:42:03.519814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":768543,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":728,"num_write":83,"num_write_kb":803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"60'2411","reported_seq":2516,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.049073+0000","last_change":"2024-09-11T00:30:45.919723+0000","last_active":"2024-09-11T00:56:28.049073+0000","last_peered":"2024-09-11T00:56:28.049073+0000","last_clean":"2024-09-11T00:56:28.049073+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:56:28.049073+0000","last_undegraded":"2024-09-11T00:56:28.049073+0000","last_fullsized":"2024-09-11T00:56:28.049073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:28.216186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":8,"num_write":12211,"num_write_kb":212619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"62'3","reported_seq":142,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.049506+0000","last_change":"2024-09-11T00:30:53.983335+0000","last_active":"2024-09-11T00:56:28.049506+0000","last_peered":"2024-09-11T00:56:28.049506+0000","last_clean":"2024-09-11T00:56:28.049506+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:56:28.049506+0000","last_undegraded":"2024-09-11T00:56:28.049506+0000","last_fullsized":"2024-09-11T00:56:28.049506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:14.315167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"62'2354","reported_seq":2449,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.471047+0000","last_change":"2024-09-11T00:30:54.998464+0000","last_active":"2024-09-11T00:56:28.471047+0000","last_peered":"2024-09-11T00:56:28.471047+0000","last_clean":"2024-09-11T00:56:28.471047+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:56:28.471047+0000","last_undegraded":"2024-09-11T00:56:28.471047+0000","last_fullsized":"2024-09-11T00:56:28.471047+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:45:56.469791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":205009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"60'676","reported_seq":797,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059574+0000","last_change":"2024-09-11T00:30:44.924499+0000","last_active":"2024-09-11T00:56:28.059574+0000","last_peered":"2024-09-11T00:56:28.059574+0000","last_clean":"2024-09-11T00:56:28.059574+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:56:28.059574+0000","last_undegraded":"2024-09-11T00:56:28.059574+0000","last_fullsized":"2024-09-11T00:56:28.059574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:43.878159+0000","last_clean_scrub_stamp":"2024-09-11T00:30:43.878159+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:18.833989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":51443,"num_write":670,"num_write_kb":4291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"60'2936","reported_seq":3047,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:52.479641+0000","last_change":"2024-09-11T00:56:52.479641+0000","last_active":"2024-09-11T00:56:52.479641+0000","last_peered":"2024-09-11T00:56:52.479641+0000","last_clean":"2024-09-11T00:56:52.479641+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:56:52.479641+0000","last_undegraded":"2024-09-11T00:56:52.479641+0000","last_fullsized":"2024-09-11T00:56:52.479641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2936","last_scrub_stamp":"2024-09-11T00:56:52.479567+0000","last_deep_scrub":"60'2936","last_deep_scrub_stamp":"2024-09-11T00:56:52.479567+0000","last_clean_scrub_stamp":"2024-09-11T00:56:52.479567+0000","objects_scrubbed":0,"log_size":2936,"log_dups_size":0,"ondisk_log_size":2936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:12.543004+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":241742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"62'45","reported_seq":184,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.488110+0000","last_change":"2024-09-11T00:30:53.987612+0000","last_active":"2024-09-11T00:56:28.488110+0000","last_peered":"2024-09-11T00:56:28.488110+0000","last_clean":"2024-09-11T00:56:28.488110+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:56:28.488110+0000","last_undegraded":"2024-09-11T00:56:28.488110+0000","last_fullsized":"2024-09-11T00:56:28.488110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:16.340969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"62'3441","reported_seq":3532,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.489542+0000","last_change":"2024-09-11T00:31:26.326200+0000","last_active":"2024-09-11T00:56:28.489542+0000","last_peered":"2024-09-11T00:56:28.489542+0000","last_clean":"2024-09-11T00:56:28.489542+0000","last_became_active":"2024-09-11T00:31:26.325895+0000","last_became_peered":"2024-09-11T00:31:26.325895+0000","last_unstale":"2024-09-11T00:56:28.489542+0000","last_undegraded":"2024-09-11T00:56:28.489542+0000","last_fullsized":"2024-09-11T00:56:28.489542+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:39:08.753699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18163,"num_write_kb":273760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.20","version":"60'3848","reported_seq":3953,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075938+0000","last_change":"2024-09-11T00:30:45.923910+0000","last_active":"2024-09-11T00:56:28.075938+0000","last_peered":"2024-09-11T00:56:28.075938+0000","last_clean":"2024-09-11T00:56:28.075938+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:56:28.075938+0000","last_undegraded":"2024-09-11T00:56:28.075938+0000","last_fullsized":"2024-09-11T00:56:28.075938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:14:48.367926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20029,"num_write_kb":325938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"62'4287","reported_seq":4384,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.519505+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:56:28.519505+0000","last_peered":"2024-09-11T00:56:28.519505+0000","last_clean":"2024-09-11T00:56:28.519505+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:56:28.519505+0000","last_undegraded":"2024-09-11T00:56:28.519505+0000","last_fullsized":"2024-09-11T00:56:28.519505+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:08:48.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22520,"num_write_kb":336110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"60'2962","reported_seq":3067,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058177+0000","last_change":"2024-09-11T00:30:45.922210+0000","last_active":"2024-09-11T00:56:28.058177+0000","last_peered":"2024-09-11T00:56:28.058177+0000","last_clean":"2024-09-11T00:56:28.058177+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:56:28.058177+0000","last_undegraded":"2024-09-11T00:56:28.058177+0000","last_fullsized":"2024-09-11T00:56:28.058177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:13:30.883035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15172,"num_write_kb":277710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"62'2334","reported_seq":2430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075259+0000","last_change":"2024-09-11T00:30:54.979133+0000","last_active":"2024-09-11T00:56:28.075259+0000","last_peered":"2024-09-11T00:56:28.075259+0000","last_clean":"2024-09-11T00:56:28.075259+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:56:28.075259+0000","last_undegraded":"2024-09-11T00:56:28.075259+0000","last_fullsized":"2024-09-11T00:56:28.075259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2334,"log_dups_size":0,"ondisk_log_size":2334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:20.961678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11753,"num_write_kb":206776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"60'4175","reported_seq":4282,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065154+0000","last_change":"2024-09-11T00:30:45.935229+0000","last_active":"2024-09-11T00:56:28.065154+0000","last_peered":"2024-09-11T00:56:28.065154+0000","last_clean":"2024-09-11T00:56:28.065154+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:56:28.065154+0000","last_undegraded":"2024-09-11T00:56:28.065154+0000","last_fullsized":"2024-09-11T00:56:28.065154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:58.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22002,"num_write_kb":346058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"62'3428","reported_seq":3524,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.426792+0000","last_change":"2024-09-11T00:30:54.988345+0000","last_active":"2024-09-11T00:56:28.426792+0000","last_peered":"2024-09-11T00:56:28.426792+0000","last_clean":"2024-09-11T00:56:28.426792+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:56:28.426792+0000","last_undegraded":"2024-09-11T00:56:28.426792+0000","last_fullsized":"2024-09-11T00:56:28.426792+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3428,"log_dups_size":0,"ondisk_log_size":3428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:19.112194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17996,"num_write_kb":289213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"60'2543","reported_seq":2648,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059170+0000","last_change":"2024-09-11T00:30:45.925822+0000","last_active":"2024-09-11T00:56:28.059170+0000","last_peered":"2024-09-11T00:56:28.059170+0000","last_clean":"2024-09-11T00:56:28.059170+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:56:28.059170+0000","last_undegraded":"2024-09-11T00:56:28.059170+0000","last_fullsized":"2024-09-11T00:56:28.059170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:02:23.800721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13221,"num_write_kb":224922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"62'3390","reported_seq":3487,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.413384+0000","last_change":"2024-09-11T00:30:54.981185+0000","last_active":"2024-09-11T00:56:28.413384+0000","last_peered":"2024-09-11T00:56:28.413384+0000","last_clean":"2024-09-11T00:56:28.413384+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:56:28.413384+0000","last_undegraded":"2024-09-11T00:56:28.413384+0000","last_fullsized":"2024-09-11T00:56:28.413384+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:43:45.740467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17320,"num_write_kb":290458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"60'2853","reported_seq":2958,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052870+0000","last_change":"2024-09-11T00:30:45.930291+0000","last_active":"2024-09-11T00:56:28.052870+0000","last_peered":"2024-09-11T00:56:28.052870+0000","last_clean":"2024-09-11T00:56:28.052870+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:56:28.052870+0000","last_undegraded":"2024-09-11T00:56:28.052870+0000","last_fullsized":"2024-09-11T00:56:28.052870+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:17.432587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14682,"num_write_kb":238952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"62'2740","reported_seq":2837,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.499809+0000","last_change":"2024-09-11T00:30:54.992763+0000","last_active":"2024-09-11T00:56:28.499809+0000","last_peered":"2024-09-11T00:56:28.499809+0000","last_clean":"2024-09-11T00:56:28.499809+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:56:28.499809+0000","last_undegraded":"2024-09-11T00:56:28.499809+0000","last_fullsized":"2024-09-11T00:56:28.499809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2740,"log_dups_size":0,"ondisk_log_size":2740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:02.879645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14066,"num_write_kb":224154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"60'3660","reported_seq":3765,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052970+0000","last_change":"2024-09-11T00:30:45.929433+0000","last_active":"2024-09-11T00:56:28.052970+0000","last_peered":"2024-09-11T00:56:28.052970+0000","last_clean":"2024-09-11T00:56:28.052970+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:56:28.052970+0000","last_undegraded":"2024-09-11T00:56:28.052970+0000","last_fullsized":"2024-09-11T00:56:28.052970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:13:54.323247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19466,"num_write_kb":307301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"62'4363","reported_seq":4460,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.478746+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:56:28.478746+0000","last_peered":"2024-09-11T00:56:28.478746+0000","last_clean":"2024-09-11T00:56:28.478746+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:56:28.478746+0000","last_undegraded":"2024-09-11T00:56:28.478746+0000","last_fullsized":"2024-09-11T00:56:28.478746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23064,"num_write_kb":350422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"60'2921","reported_seq":3026,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075718+0000","last_change":"2024-09-11T00:30:45.944927+0000","last_active":"2024-09-11T00:56:28.075718+0000","last_peered":"2024-09-11T00:56:28.075718+0000","last_clean":"2024-09-11T00:56:28.075718+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:56:28.075718+0000","last_undegraded":"2024-09-11T00:56:28.075718+0000","last_fullsized":"2024-09-11T00:56:28.075718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:20:51.849508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15417,"num_write_kb":250464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"62'3030","reported_seq":3127,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.431858+0000","last_change":"2024-09-11T00:30:54.984447+0000","last_active":"2024-09-11T00:56:28.431858+0000","last_peered":"2024-09-11T00:56:28.431858+0000","last_clean":"2024-09-11T00:56:28.431858+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:56:28.431858+0000","last_undegraded":"2024-09-11T00:56:28.431858+0000","last_fullsized":"2024-09-11T00:56:28.431858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:18:49.024022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15510,"num_write_kb":262754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"60'3321","reported_seq":3428,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059575+0000","last_change":"2024-09-11T00:30:45.934334+0000","last_active":"2024-09-11T00:56:28.059575+0000","last_peered":"2024-09-11T00:56:28.059575+0000","last_clean":"2024-09-11T00:56:28.059575+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:56:28.059575+0000","last_undegraded":"2024-09-11T00:56:28.059575+0000","last_fullsized":"2024-09-11T00:56:28.059575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:08.280316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":25,"num_write":17209,"num_write_kb":314610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"62'2084","reported_seq":2181,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.543950+0000","last_change":"2024-09-11T00:30:54.996091+0000","last_active":"2024-09-11T00:56:28.543950+0000","last_peered":"2024-09-11T00:56:28.543950+0000","last_clean":"2024-09-11T00:56:28.543950+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:56:28.543950+0000","last_undegraded":"2024-09-11T00:56:28.543950+0000","last_fullsized":"2024-09-11T00:56:28.543950+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2084,"log_dups_size":0,"ondisk_log_size":2084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:55:42.987336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10636,"num_write_kb":173560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"60'2810","reported_seq":2915,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063775+0000","last_change":"2024-09-11T00:30:45.938458+0000","last_active":"2024-09-11T00:56:28.063775+0000","last_peered":"2024-09-11T00:56:28.063775+0000","last_clean":"2024-09-11T00:56:28.063775+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:56:28.063775+0000","last_undegraded":"2024-09-11T00:56:28.063775+0000","last_fullsized":"2024-09-11T00:56:28.063775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2810,"log_dups_size":0,"ondisk_log_size":2810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:53:14.740551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":243007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"62'3697","reported_seq":3793,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.455458+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:56:28.455458+0000","last_peered":"2024-09-11T00:56:28.455458+0000","last_clean":"2024-09-11T00:56:28.455458+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:56:28.455458+0000","last_undegraded":"2024-09-11T00:56:28.455458+0000","last_fullsized":"2024-09-11T00:56:28.455458+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18782,"num_write_kb":312790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"60'2824","reported_seq":2928,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057103+0000","last_change":"2024-09-11T00:30:45.941064+0000","last_active":"2024-09-11T00:56:28.057103+0000","last_peered":"2024-09-11T00:56:28.057103+0000","last_clean":"2024-09-11T00:56:28.057103+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:56:28.057103+0000","last_undegraded":"2024-09-11T00:56:28.057103+0000","last_fullsized":"2024-09-11T00:56:28.057103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2824,"log_dups_size":0,"ondisk_log_size":2824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:46:20.160997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14695,"num_write_kb":259410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"62'2917","reported_seq":3013,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.510399+0000","last_change":"2024-09-11T00:30:54.981799+0000","last_active":"2024-09-11T00:56:28.510399+0000","last_peered":"2024-09-11T00:56:28.510399+0000","last_clean":"2024-09-11T00:56:28.510399+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:56:28.510399+0000","last_undegraded":"2024-09-11T00:56:28.510399+0000","last_fullsized":"2024-09-11T00:56:28.510399+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2917,"log_dups_size":0,"ondisk_log_size":2917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:42.670629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":242029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"60'3431","reported_seq":3535,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057841+0000","last_change":"2024-09-11T00:30:45.919406+0000","last_active":"2024-09-11T00:56:28.057841+0000","last_peered":"2024-09-11T00:56:28.057841+0000","last_clean":"2024-09-11T00:56:28.057841+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:56:28.057841+0000","last_undegraded":"2024-09-11T00:56:28.057841+0000","last_fullsized":"2024-09-11T00:56:28.057841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:59.960699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18408,"num_write_kb":290787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"62'2945","reported_seq":3041,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514103+0000","last_change":"2024-09-11T00:30:54.992697+0000","last_active":"2024-09-11T00:56:28.514103+0000","last_peered":"2024-09-11T00:56:28.514103+0000","last_clean":"2024-09-11T00:56:28.514103+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:56:28.514103+0000","last_undegraded":"2024-09-11T00:56:28.514103+0000","last_fullsized":"2024-09-11T00:56:28.514103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:54:07.227169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15229,"num_write_kb":249300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"60'2849","reported_seq":2955,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048641+0000","last_change":"2024-09-11T00:30:45.930479+0000","last_active":"2024-09-11T00:56:28.048641+0000","last_peered":"2024-09-11T00:56:28.048641+0000","last_clean":"2024-09-11T00:56:28.048641+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:56:28.048641+0000","last_undegraded":"2024-09-11T00:56:28.048641+0000","last_fullsized":"2024-09-11T00:56:28.048641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2849,"log_dups_size":0,"ondisk_log_size":2849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:27.141120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14625,"num_write_kb":263252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"62'4017","reported_seq":4113,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.477074+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:56:28.477074+0000","last_peered":"2024-09-11T00:56:28.477074+0000","last_clean":"2024-09-11T00:56:28.477074+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:56:28.477074+0000","last_undegraded":"2024-09-11T00:56:28.477074+0000","last_fullsized":"2024-09-11T00:56:28.477074+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":329319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"60'2942","reported_seq":3046,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066118+0000","last_change":"2024-09-11T00:30:45.929677+0000","last_active":"2024-09-11T00:56:28.066118+0000","last_peered":"2024-09-11T00:56:28.066118+0000","last_clean":"2024-09-11T00:56:28.066118+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:56:28.066118+0000","last_undegraded":"2024-09-11T00:56:28.066118+0000","last_fullsized":"2024-09-11T00:56:28.066118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:04:40.572202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15436,"num_write_kb":246093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"62'3417","reported_seq":3514,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.528940+0000","last_change":"2024-09-11T00:30:54.985827+0000","last_active":"2024-09-11T00:56:28.528940+0000","last_peered":"2024-09-11T00:56:28.528940+0000","last_clean":"2024-09-11T00:56:28.528940+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:56:28.528940+0000","last_undegraded":"2024-09-11T00:56:28.528940+0000","last_fullsized":"2024-09-11T00:56:28.528940+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:32.027117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18129,"num_write_kb":269216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"60'2746","reported_seq":2850,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069377+0000","last_change":"2024-09-11T00:30:45.940434+0000","last_active":"2024-09-11T00:56:28.069377+0000","last_peered":"2024-09-11T00:56:28.069377+0000","last_clean":"2024-09-11T00:56:28.069377+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:56:28.069377+0000","last_undegraded":"2024-09-11T00:56:28.069377+0000","last_fullsized":"2024-09-11T00:56:28.069377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:44.724545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13859,"num_write_kb":247924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"62'2570","reported_seq":2672,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.514173+0000","last_change":"2024-09-11T00:31:26.119785+0000","last_active":"2024-09-11T00:56:28.514173+0000","last_peered":"2024-09-11T00:56:28.514173+0000","last_clean":"2024-09-11T00:56:28.514173+0000","last_became_active":"2024-09-11T00:31:26.119436+0000","last_became_peered":"2024-09-11T00:31:26.119436+0000","last_unstale":"2024-09-11T00:56:28.514173+0000","last_undegraded":"2024-09-11T00:56:28.514173+0000","last_fullsized":"2024-09-11T00:56:28.514173+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:53:49.407575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13205,"num_write_kb":216683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"60'2388","reported_seq":2494,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058894+0000","last_change":"2024-09-11T00:30:45.926412+0000","last_active":"2024-09-11T00:56:28.058894+0000","last_peered":"2024-09-11T00:56:28.058894+0000","last_clean":"2024-09-11T00:56:28.058894+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:56:28.058894+0000","last_undegraded":"2024-09-11T00:56:28.058894+0000","last_fullsized":"2024-09-11T00:56:28.058894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2388,"log_dups_size":0,"ondisk_log_size":2388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:19.077799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4,"num_write":12384,"num_write_kb":205064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"62'3381","reported_seq":3477,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397849+0000","last_change":"2024-09-11T00:30:54.993156+0000","last_active":"2024-09-11T00:56:28.397849+0000","last_peered":"2024-09-11T00:56:28.397849+0000","last_clean":"2024-09-11T00:56:28.397849+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:56:28.397849+0000","last_undegraded":"2024-09-11T00:56:28.397849+0000","last_fullsized":"2024-09-11T00:56:28.397849+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:21:25.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17747,"num_write_kb":280034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"60'3936","reported_seq":4041,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067998+0000","last_change":"2024-09-11T00:30:45.935583+0000","last_active":"2024-09-11T00:56:28.067998+0000","last_peered":"2024-09-11T00:56:28.067998+0000","last_clean":"2024-09-11T00:56:28.067998+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:56:28.067998+0000","last_undegraded":"2024-09-11T00:56:28.067998+0000","last_fullsized":"2024-09-11T00:56:28.067998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:55:50.196100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":20459,"num_write_kb":335990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"62'3772","reported_seq":3867,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.454399+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:56:28.454399+0000","last_peered":"2024-09-11T00:56:28.454399+0000","last_clean":"2024-09-11T00:56:28.454399+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:56:28.454399+0000","last_undegraded":"2024-09-11T00:56:28.454399+0000","last_fullsized":"2024-09-11T00:56:28.454399+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20201,"num_write_kb":302053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"60'2757","reported_seq":2861,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.062570+0000","last_change":"2024-09-11T00:30:45.938750+0000","last_active":"2024-09-11T00:56:28.062570+0000","last_peered":"2024-09-11T00:56:28.062570+0000","last_clean":"2024-09-11T00:56:28.062570+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:56:28.062570+0000","last_undegraded":"2024-09-11T00:56:28.062570+0000","last_fullsized":"2024-09-11T00:56:28.062570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:00:34.741313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14203,"num_write_kb":241057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"62'4048","reported_seq":4143,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.441208+0000","last_change":"2024-09-11T00:30:54.993461+0000","last_active":"2024-09-11T00:56:28.441208+0000","last_peered":"2024-09-11T00:56:28.441208+0000","last_clean":"2024-09-11T00:56:28.441208+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:56:28.441208+0000","last_undegraded":"2024-09-11T00:56:28.441208+0000","last_fullsized":"2024-09-11T00:56:28.441208+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:19:07.096547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21387,"num_write_kb":364007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"60'3572","reported_seq":3676,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057127+0000","last_change":"2024-09-11T00:30:45.933752+0000","last_active":"2024-09-11T00:56:28.057127+0000","last_peered":"2024-09-11T00:56:28.057127+0000","last_clean":"2024-09-11T00:56:28.057127+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:56:28.057127+0000","last_undegraded":"2024-09-11T00:56:28.057127+0000","last_fullsized":"2024-09-11T00:56:28.057127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:46.533232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18317,"num_write_kb":294510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"62'2987","reported_seq":3083,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397386+0000","last_change":"2024-09-11T00:30:54.986265+0000","last_active":"2024-09-11T00:56:28.397386+0000","last_peered":"2024-09-11T00:56:28.397386+0000","last_clean":"2024-09-11T00:56:28.397386+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:56:28.397386+0000","last_undegraded":"2024-09-11T00:56:28.397386+0000","last_fullsized":"2024-09-11T00:56:28.397386+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:03:49.109328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15338,"num_write_kb":237611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"60'2929","reported_seq":3034,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.051236+0000","last_change":"2024-09-11T00:30:45.920559+0000","last_active":"2024-09-11T00:56:28.051236+0000","last_peered":"2024-09-11T00:56:28.051236+0000","last_clean":"2024-09-11T00:56:28.051236+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:56:28.051236+0000","last_undegraded":"2024-09-11T00:56:28.051236+0000","last_fullsized":"2024-09-11T00:56:28.051236+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2929,"log_dups_size":0,"ondisk_log_size":2929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:03.499274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":15867,"num_write_kb":268409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"62'3243","reported_seq":3338,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.433332+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:56:28.433332+0000","last_peered":"2024-09-11T00:56:28.433332+0000","last_clean":"2024-09-11T00:56:28.433332+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:56:28.433332+0000","last_undegraded":"2024-09-11T00:56:28.433332+0000","last_fullsized":"2024-09-11T00:56:28.433332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16324,"num_write_kb":262051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"60'2913","reported_seq":3019,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.052467+0000","last_change":"2024-09-11T00:30:45.930417+0000","last_active":"2024-09-11T00:56:28.052467+0000","last_peered":"2024-09-11T00:56:28.052467+0000","last_clean":"2024-09-11T00:56:28.052467+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:56:28.052467+0000","last_undegraded":"2024-09-11T00:56:28.052467+0000","last_fullsized":"2024-09-11T00:56:28.052467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2913,"log_dups_size":0,"ondisk_log_size":2913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:17.067954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15444,"num_write_kb":246103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"62'3344","reported_seq":3440,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.543938+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:56:28.543938+0000","last_peered":"2024-09-11T00:56:28.543938+0000","last_clean":"2024-09-11T00:56:28.543938+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:56:28.543938+0000","last_undegraded":"2024-09-11T00:56:28.543938+0000","last_fullsized":"2024-09-11T00:56:28.543938+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3344,"log_dups_size":0,"ondisk_log_size":3344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17374,"num_write_kb":274437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"60'2266","reported_seq":2373,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065676+0000","last_change":"2024-09-11T00:30:45.926718+0000","last_active":"2024-09-11T00:56:28.065676+0000","last_peered":"2024-09-11T00:56:28.065676+0000","last_clean":"2024-09-11T00:56:28.065676+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:56:28.065676+0000","last_undegraded":"2024-09-11T00:56:28.065676+0000","last_fullsized":"2024-09-11T00:56:28.065676+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:29:30.299299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":26,"num_write":11816,"num_write_kb":219351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"62'2714","reported_seq":2811,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.547334+0000","last_change":"2024-09-11T00:30:54.994988+0000","last_active":"2024-09-11T00:56:28.547334+0000","last_peered":"2024-09-11T00:56:28.547334+0000","last_clean":"2024-09-11T00:56:28.547334+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:56:28.547334+0000","last_undegraded":"2024-09-11T00:56:28.547334+0000","last_fullsized":"2024-09-11T00:56:28.547334+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2714,"log_dups_size":0,"ondisk_log_size":2714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:31.973641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14634,"num_write_kb":230575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"60'3170","reported_seq":3259,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.058097+0000","last_change":"2024-09-11T00:31:26.117501+0000","last_active":"2024-09-11T00:56:28.058097+0000","last_peered":"2024-09-11T00:56:28.058097+0000","last_clean":"2024-09-11T00:56:28.058097+0000","last_became_active":"2024-09-11T00:31:26.117146+0000","last_became_peered":"2024-09-11T00:31:26.117146+0000","last_unstale":"2024-09-11T00:56:28.058097+0000","last_undegraded":"2024-09-11T00:56:28.058097+0000","last_fullsized":"2024-09-11T00:56:28.058097+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:52.194644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15922,"num_write_kb":268523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.30","version":"62'3135","reported_seq":3231,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.469071+0000","last_change":"2024-09-11T00:30:54.984767+0000","last_active":"2024-09-11T00:56:28.469071+0000","last_peered":"2024-09-11T00:56:28.469071+0000","last_clean":"2024-09-11T00:56:28.469071+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:56:28.469071+0000","last_undegraded":"2024-09-11T00:56:28.469071+0000","last_fullsized":"2024-09-11T00:56:28.469071+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:44.833677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16038,"num_write_kb":252613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"60'3300","reported_seq":3404,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068344+0000","last_change":"2024-09-11T00:30:45.930662+0000","last_active":"2024-09-11T00:56:28.068344+0000","last_peered":"2024-09-11T00:56:28.068344+0000","last_clean":"2024-09-11T00:56:28.068344+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:56:28.068344+0000","last_undegraded":"2024-09-11T00:56:28.068344+0000","last_fullsized":"2024-09-11T00:56:28.068344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3300,"log_dups_size":0,"ondisk_log_size":3300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:10.360418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":17861,"num_write_kb":288513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"62'3599","reported_seq":3695,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.492582+0000","last_change":"2024-09-11T00:30:54.986423+0000","last_active":"2024-09-11T00:56:28.492582+0000","last_peered":"2024-09-11T00:56:28.492582+0000","last_clean":"2024-09-11T00:56:28.492582+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:56:28.492582+0000","last_undegraded":"2024-09-11T00:56:28.492582+0000","last_fullsized":"2024-09-11T00:56:28.492582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:16:50.684479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18686,"num_write_kb":304162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"60'4168","reported_seq":4280,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.445154+0000","last_change":"2024-09-11T00:56:54.445154+0000","last_active":"2024-09-11T00:56:54.445154+0000","last_peered":"2024-09-11T00:56:54.445154+0000","last_clean":"2024-09-11T00:56:54.445154+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:56:54.445154+0000","last_undegraded":"2024-09-11T00:56:54.445154+0000","last_fullsized":"2024-09-11T00:56:54.445154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4168","last_scrub_stamp":"2024-09-11T00:56:54.445072+0000","last_deep_scrub":"60'4168","last_deep_scrub_stamp":"2024-09-11T00:56:54.445072+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.445072+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:52.768326+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22110,"num_write_kb":362983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"62'3937","reported_seq":4034,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.544539+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:56:28.544539+0000","last_peered":"2024-09-11T00:56:28.544539+0000","last_clean":"2024-09-11T00:56:28.544539+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:56:28.544539+0000","last_undegraded":"2024-09-11T00:56:28.544539+0000","last_fullsized":"2024-09-11T00:56:28.544539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21285,"num_write_kb":343312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"60'3475","reported_seq":3579,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.057082+0000","last_change":"2024-09-11T00:30:45.934925+0000","last_active":"2024-09-11T00:56:28.057082+0000","last_peered":"2024-09-11T00:56:28.057082+0000","last_clean":"2024-09-11T00:56:28.057082+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:56:28.057082+0000","last_undegraded":"2024-09-11T00:56:28.057082+0000","last_fullsized":"2024-09-11T00:56:28.057082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3475,"log_dups_size":0,"ondisk_log_size":3475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:05.451820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17645,"num_write_kb":295948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"62'3302","reported_seq":3399,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.512120+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:56:28.512120+0000","last_peered":"2024-09-11T00:56:28.512120+0000","last_clean":"2024-09-11T00:56:28.512120+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:56:28.512120+0000","last_undegraded":"2024-09-11T00:56:28.512120+0000","last_fullsized":"2024-09-11T00:56:28.512120+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17558,"num_write_kb":271021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"60'3295","reported_seq":3399,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.063104+0000","last_change":"2024-09-11T00:30:45.927916+0000","last_active":"2024-09-11T00:56:28.063104+0000","last_peered":"2024-09-11T00:56:28.063104+0000","last_clean":"2024-09-11T00:56:28.063104+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:56:28.063104+0000","last_undegraded":"2024-09-11T00:56:28.063104+0000","last_fullsized":"2024-09-11T00:56:28.063104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:04.308789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":25,"num_write":17271,"num_write_kb":265887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"61'3033","reported_seq":3130,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048703+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:56:28.048703+0000","last_peered":"2024-09-11T00:56:28.048703+0000","last_clean":"2024-09-11T00:56:28.048703+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:56:28.048703+0000","last_undegraded":"2024-09-11T00:56:28.048703+0000","last_fullsized":"2024-09-11T00:56:28.048703+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:09.748798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16092,"num_write_kb":252889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"60'2629","reported_seq":2733,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.078405+0000","last_change":"2024-09-11T00:30:45.930080+0000","last_active":"2024-09-11T00:56:28.078405+0000","last_peered":"2024-09-11T00:56:28.078405+0000","last_clean":"2024-09-11T00:56:28.078405+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:56:28.078405+0000","last_undegraded":"2024-09-11T00:56:28.078405+0000","last_fullsized":"2024-09-11T00:56:28.078405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:01:38.637490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":13758,"num_write_kb":227101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"62'3227","reported_seq":3324,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.479006+0000","last_change":"2024-09-11T00:30:54.990631+0000","last_active":"2024-09-11T00:56:28.479006+0000","last_peered":"2024-09-11T00:56:28.479006+0000","last_clean":"2024-09-11T00:56:28.479006+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:56:28.479006+0000","last_undegraded":"2024-09-11T00:56:28.479006+0000","last_fullsized":"2024-09-11T00:56:28.479006+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:02.422471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16396,"num_write_kb":258131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"60'3415","reported_seq":3520,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.067480+0000","last_change":"2024-09-11T00:30:45.933671+0000","last_active":"2024-09-11T00:56:28.067480+0000","last_peered":"2024-09-11T00:56:28.067480+0000","last_clean":"2024-09-11T00:56:28.067480+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:56:28.067480+0000","last_undegraded":"2024-09-11T00:56:28.067480+0000","last_fullsized":"2024-09-11T00:56:28.067480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:20:06.146586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":17180,"num_write_kb":310287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"62'2771","reported_seq":2866,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.447411+0000","last_change":"2024-09-11T00:30:54.991047+0000","last_active":"2024-09-11T00:56:28.447411+0000","last_peered":"2024-09-11T00:56:28.447411+0000","last_clean":"2024-09-11T00:56:28.447411+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:56:28.447411+0000","last_undegraded":"2024-09-11T00:56:28.447411+0000","last_fullsized":"2024-09-11T00:56:28.447411+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:41:48.229261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14317,"num_write_kb":221149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"60'2741","reported_seq":2847,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.076040+0000","last_change":"2024-09-11T00:30:45.936984+0000","last_active":"2024-09-11T00:56:28.076040+0000","last_peered":"2024-09-11T00:56:28.076040+0000","last_clean":"2024-09-11T00:56:28.076040+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:56:28.076040+0000","last_undegraded":"2024-09-11T00:56:28.076040+0000","last_fullsized":"2024-09-11T00:56:28.076040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2741,"log_dups_size":0,"ondisk_log_size":2741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:55.951417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14755,"num_write_kb":255665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"62'3704","reported_seq":3799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.547303+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:56:28.547303+0000","last_peered":"2024-09-11T00:56:28.547303+0000","last_clean":"2024-09-11T00:56:28.547303+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:56:28.547303+0000","last_undegraded":"2024-09-11T00:56:28.547303+0000","last_fullsized":"2024-09-11T00:56:28.547303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18483,"num_write_kb":307447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"60'3035","reported_seq":3140,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059908+0000","last_change":"2024-09-11T00:30:45.940570+0000","last_active":"2024-09-11T00:56:28.059908+0000","last_peered":"2024-09-11T00:56:28.059908+0000","last_clean":"2024-09-11T00:56:28.059908+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:56:28.059908+0000","last_undegraded":"2024-09-11T00:56:28.059908+0000","last_fullsized":"2024-09-11T00:56:28.059908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":3035,"log_dups_size":0,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:32:19.387286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15470,"num_write_kb":262623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"62'3351","reported_seq":3447,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075584+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:56:28.075584+0000","last_peered":"2024-09-11T00:56:28.075584+0000","last_clean":"2024-09-11T00:56:28.075584+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:56:28.075584+0000","last_undegraded":"2024-09-11T00:56:28.075584+0000","last_fullsized":"2024-09-11T00:56:28.075584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17034,"num_write_kb":277088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"60'4106","reported_seq":4213,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.059593+0000","last_change":"2024-09-11T00:30:45.937618+0000","last_active":"2024-09-11T00:56:28.059593+0000","last_peered":"2024-09-11T00:56:28.059593+0000","last_clean":"2024-09-11T00:56:28.059593+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:56:28.059593+0000","last_undegraded":"2024-09-11T00:56:28.059593+0000","last_fullsized":"2024-09-11T00:56:28.059593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:00:54.444561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22024,"num_write_kb":354504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"62'3391","reported_seq":3486,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.526593+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:56:28.526593+0000","last_peered":"2024-09-11T00:56:28.526593+0000","last_clean":"2024-09-11T00:56:28.526593+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:56:28.526593+0000","last_undegraded":"2024-09-11T00:56:28.526593+0000","last_fullsized":"2024-09-11T00:56:28.526593+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18265,"num_write_kb":274031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"60'2946","reported_seq":3053,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.069081+0000","last_change":"2024-09-11T00:30:45.941698+0000","last_active":"2024-09-11T00:56:28.069081+0000","last_peered":"2024-09-11T00:56:28.069081+0000","last_clean":"2024-09-11T00:56:28.069081+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:56:28.069081+0000","last_undegraded":"2024-09-11T00:56:28.069081+0000","last_fullsized":"2024-09-11T00:56:28.069081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:44.885442+0000","last_clean_scrub_stamp":"2024-09-11T00:30:44.885442+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:19:25.229403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":17,"num_write":15520,"num_write_kb":262805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1107571,"num_write_kb":17543218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212907,"ondisk_log_size":212907,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18223220,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1558,"num_read_kb":197403,"num_write":4817,"num_write_kb":19773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36511744,"data_stored":36446440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4412,"ondisk_log_size":4412,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":381,"num_write":1025649,"num_write_kb":17211858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196678,"ondisk_log_size":196678,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":17635410,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1062,"num_read_kb":143929,"num_write":4568,"num_write_kb":19046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35332096,"data_stored":35270820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4124,"ondisk_log_size":4124,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509759,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487152,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":486270,"kb_avail":93884688,"statfs":{"total":96636764160,"available":96137920512,"internally_reserved":0,"allocated":884736,"data_stored":496637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":497940938},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509761,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668164,"kb_used_data":10292,"kb_used_omap":1,"kb_used_meta":657854,"kb_avail":93703676,"statfs":{"total":96636764160,"available":95952564224,"internally_reserved":0,"allocated":10539008,"data_stored":10153943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":673642950},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509760,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685684,"kb_used_data":17828,"kb_used_omap":1,"kb_used_meta":667838,"kb_avail":93686156,"statfs":{"total":96636764160,"available":95934623744,"internally_reserved":0,"allocated":18255872,"data_stored":17864749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683866565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509759,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722280,"kb_used_data":13912,"kb_used_omap":1,"kb_used_meta":708350,"kb_avail":93649560,"statfs":{"total":96636764160,"available":95897149440,"internally_reserved":0,"allocated":14245888,"data_stored":13849304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":725350856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509760,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737056,"kb_used_data":14480,"kb_used_omap":1,"kb_used_meta":722558,"kb_avail":93634784,"statfs":{"total":96636764160,"available":95882018816,"internally_reserved":0,"allocated":14827520,"data_stored":14436462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739899845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509760,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598360,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":596990,"kb_avail":93773480,"statfs":{"total":96636764160,"available":96024043520,"internally_reserved":0,"allocated":1384448,"data_stored":998818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":611318213},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509761,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741960,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":740414,"kb_avail":93629880,"statfs":{"total":96636764160,"available":95876997120,"internally_reserved":0,"allocated":1564672,"data_stored":1176999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":758184389},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542464,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739432,"kb_used_data":17176,"kb_used_omap":1,"kb_used_meta":722238,"kb_avail":93632408,"statfs":{"total":96636764160,"available":95879585792,"internally_reserved":0,"allocated":17588224,"data_stored":17187564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739572165},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":778240,"data_stored":768599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9162752,"data_stored":9157733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4282577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5648384,"data_stored":5638872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":4283261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5562368,"data_stored":5551329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:56:55.573 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.574 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.574 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.574 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.574 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.574 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.574 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.574 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.575 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.575 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.575 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.575 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.575 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.575 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.575 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.575 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.576 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.576 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.576 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.576 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.576 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.576 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.576 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.576 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.576 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.577 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.577 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.577 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.577 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.577 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.577 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.577 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.577 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.577 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.578 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.578 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.578 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.578 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.578 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.578 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.578 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.578 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.579 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.579 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.579 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.579 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.579 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.579 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.579 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.579 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.579 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.579 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.580 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.580 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.580 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.580 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.580 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.580 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.580 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.580 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.580 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.581 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.581 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.581 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.581 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.581 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.581 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.581 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.581 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.581 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.582 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.582 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.582 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.582 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.582 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.582 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.582 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.582 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.582 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.583 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.583 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.583 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.583 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.583 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.583 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.583 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.583 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.583 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.584 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.584 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.584 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.584 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.584 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.584 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.584 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.584 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.585 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.585 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.585 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.585 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.585 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.585 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.585 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.585 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.585 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.586 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.586 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.586 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.586 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.586 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.586 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.586 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.586 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.586 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-09-11T00:30:43.878159+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=43, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.587 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.587 INFO:tasks.ceph:pgid 5.26 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.587 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.587 INFO:tasks.ceph:pgid 5.27 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.587 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.587 INFO:tasks.ceph:pgid 5.24 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.587 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.587 INFO:tasks.ceph:pgid 5.25 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.587 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.588 INFO:tasks.ceph:pgid 5.22 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.588 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.588 INFO:tasks.ceph:pgid 5.23 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.588 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.588 INFO:tasks.ceph:pgid 5.20 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.588 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.588 INFO:tasks.ceph:pgid 5.21 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.588 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.589 INFO:tasks.ceph:pgid 5.2e last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.589 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.589 INFO:tasks.ceph:pgid 5.2f last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.589 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.589 INFO:tasks.ceph:pgid 5.2c last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.589 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.589 INFO:tasks.ceph:pgid 5.2d last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.589 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.589 INFO:tasks.ceph:pgid 5.2a last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.590 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.590 INFO:tasks.ceph:pgid 5.2b last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.590 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.590 INFO:tasks.ceph:pgid 5.28 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.590 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.590 INFO:tasks.ceph:pgid 5.29 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.590 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.590 INFO:tasks.ceph:pgid 5.36 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.590 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.591 INFO:tasks.ceph:pgid 5.37 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.591 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.591 INFO:tasks.ceph:pgid 5.34 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.591 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.591 INFO:tasks.ceph:pgid 5.35 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.591 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.591 INFO:tasks.ceph:pgid 5.32 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.591 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.591 INFO:tasks.ceph:pgid 5.33 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.591 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.592 INFO:tasks.ceph:pgid 5.30 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.592 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.592 INFO:tasks.ceph:pgid 5.31 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.592 INFO:tasks.ceph:pgid 5.3e last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.592 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.592 INFO:tasks.ceph:pgid 5.3f last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.592 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.592 INFO:tasks.ceph:pgid 5.3c last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.592 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.593 INFO:tasks.ceph:pgid 5.3d last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.593 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.593 INFO:tasks.ceph:pgid 5.3a last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.593 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.593 INFO:tasks.ceph:pgid 5.3b last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.593 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.593 INFO:tasks.ceph:pgid 5.38 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.593 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.594 INFO:tasks.ceph:pgid 5.39 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.594 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-09-11T00:30:44.885442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:56:55.594 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T00:57:01.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:57:01.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:57:15.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:57:15.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.768+0000 7f53e6a00640 1 -- 172.21.15.31:0/3653931047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53c4004d40 msgr2=0x7f53c40051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.768+0000 7f53e6a00640 1 --2- 172.21.15.31:0/3653931047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53c4004d40 0x7f53c40051a0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009ed0 tx=0x7f53c4005830 comp rx=0 tx=0).stop 2024-09-11T00:57:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.768+0000 7f53e4a00640 1 -- 172.21.15.31:0/3653931047 reap_dead start 2024-09-11T00:57:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.768+0000 7f53e6a00640 1 -- 172.21.15.31:0/3653931047 shutdown_connections 2024-09-11T00:57:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.768+0000 7f53e6a00640 1 -- 172.21.15.31:0/3653931047 >> 172.21.15.31:0/3653931047 conn(0x7f53e00a9750 msgr2=0x7f53e00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.768+0000 7f53e6a00640 1 -- 172.21.15.31:0/3653931047 shutdown_connections 2024-09-11T00:57:15.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.769+0000 7f53e6a00640 1 -- 172.21.15.31:0/3653931047 wait complete. 2024-09-11T00:57:15.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.770+0000 7f53e6a00640 1 Processor -- start 2024-09-11T00:57:15.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.770+0000 7f53e6a00640 1 -- start start 2024-09-11T00:57:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.771+0000 7f53e6a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53c4004d40 0x7f53e00c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:15.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.771+0000 7f53e6a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53e0155e20 0x7f53e01562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.771+0000 7f53e6a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53e014bc70 0x7f53e014c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.771+0000 7f53e6a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f53e0074240 con 0x7f53c4004d40 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.771+0000 7f53e6a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f53e00740c0 con 0x7f53e0155e20 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.771+0000 7f53e6a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f53e0072860 con 0x7f53e014bc70 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.771+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53c4004d40 0x7f53e00c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.771+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53c4004d40 0x7f53e00c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37230/0 (socket says 172.21.15.31:37230) 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.771+0000 7f53dfe00640 1 -- 172.21.15.31:0/3023961130 learned_addr learned my addr 172.21.15.31:0/3023961130 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.772+0000 7f53d7400640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53e0155e20 0x7f53e01562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.772+0000 7f53e4a00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53e014bc70 0x7f53e014c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.772+0000 7f53dfe00640 1 -- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53e014bc70 msgr2=0x7f53e014c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.772+0000 7f53dfe00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53e014bc70 0x7f53e014c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:15.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.772+0000 7f53dfe00640 1 -- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53e0155e20 msgr2=0x7f53e01562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.772+0000 7f53dfe00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53e0155e20 0x7f53e01562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.772+0000 7f53dfe00640 1 -- 172.21.15.31:0/3023961130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e01e2570 con 0x7f53c4004d40 2024-09-11T00:57:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.772+0000 7f53e4a00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53e014bc70 0x7f53e014c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.772+0000 7f53dfe00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53c4004d40 0x7f53e00c6a60 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f53d0001760 tx=0x7f53d00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.772+0000 7f53dcc00640 1 -- 172.21.15.31:0/3023961130 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0015070 con 0x7f53c4004d40 2024-09-11T00:57:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.773+0000 7f53dcc00640 1 -- 172.21.15.31:0/3023961130 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f53d0011920 con 0x7f53c4004d40 2024-09-11T00:57:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.773+0000 7f53e6a00640 1 -- 172.21.15.31:0/3023961130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e01e2160 con 0x7f53c4004d40 2024-09-11T00:57:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.773+0000 7f53dcc00640 1 -- 172.21.15.31:0/3023961130 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0011d70 con 0x7f53c4004d40 2024-09-11T00:57:15.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.773+0000 7f53e6a00640 1 -- 172.21.15.31:0/3023961130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e00c6fa0 con 0x7f53c4004d40 2024-09-11T00:57:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.774+0000 7f53e6a00640 1 -- 172.21.15.31:0/3023961130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53e0154090 con 0x7f53c4004d40 2024-09-11T00:57:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.775+0000 7f53dcc00640 1 -- 172.21.15.31:0/3023961130 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f53d0010420 con 0x7f53c4004d40 2024-09-11T00:57:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.776+0000 7f53dcc00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53bc077f70 0x7f53bc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.776+0000 7f53dcc00640 1 -- 172.21.15.31:0/3023961130 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f53d009bd90 con 0x7f53c4004d40 2024-09-11T00:57:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.776+0000 7f53d7400640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53bc077f70 0x7f53bc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.777+0000 7f53d7400640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53bc077f70 0x7f53bc07a430 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f53e00707d0 tx=0x7f53c8006b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.780+0000 7f53dcc00640 1 -- 172.21.15.31:0/3023961130 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f53d0063870 con 0x7f53c4004d40 2024-09-11T00:57:15.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.973+0000 7f53e6a00640 1 -- 172.21.15.31:0/3023961130 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f53e01e2830 con 0x7f53bc077f70 2024-09-11T00:57:15.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.982+0000 7f53dcc00640 1 -- 172.21.15.31:0/3023961130 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+494537 (secure 0 0 0) 0x7f53e01e2830 con 0x7f53bc077f70 2024-09-11T00:57:15.990 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:15.993 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:57:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.987+0000 7f53e6a00640 1 -- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53bc077f70 msgr2=0x7f53bc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.987+0000 7f53e6a00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53bc077f70 0x7f53bc07a430 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f53e00707d0 tx=0x7f53c8006b70 comp rx=0 tx=0).stop 2024-09-11T00:57:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.987+0000 7f53e6a00640 1 -- 172.21.15.31:0/3023961130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53c4004d40 msgr2=0x7f53e00c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.987+0000 7f53e6a00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53c4004d40 0x7f53e00c6a60 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f53d0001760 tx=0x7f53d00039c0 comp rx=0 tx=0).stop 2024-09-11T00:57:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.987+0000 7f53e6a00640 1 -- 172.21.15.31:0/3023961130 shutdown_connections 2024-09-11T00:57:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.987+0000 7f53e6a00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f53bc077f70 0x7f53bc07a430 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.988+0000 7f53e6a00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f53e014bc70 0x7f53e014c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.988+0000 7f53e6a00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53e0155e20 0x7f53e01562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:15.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.988+0000 7f53e6a00640 1 --2- 172.21.15.31:0/3023961130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53c4004d40 0x7f53e00c6a60 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:15.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.988+0000 7f53e6a00640 1 -- 172.21.15.31:0/3023961130 >> 172.21.15.31:0/3023961130 conn(0x7f53e00a9750 msgr2=0x7f53e00c0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:15.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.988+0000 7f53e6a00640 1 -- 172.21.15.31:0/3023961130 shutdown_connections 2024-09-11T00:57:15.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:15.988+0000 7f53e6a00640 1 -- 172.21.15.31:0/3023961130 wait complete. 2024-09-11T00:57:16.006 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":856,"stamp":"2024-09-11T00:57:14.852435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36448998,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":2779,"num_read_kb":341768,"num_write":2142706,"num_write_kb":34795523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":418231,"ondisk_log_size":418231,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5380088,"kb_used_data":77432,"kb_used_omap":12,"kb_used_meta":5302515,"kb_avail":749594632,"statfs":{"total":773094113280,"available":767584903168,"internally_reserved":0,"allocated":79290368,"data_stored":76164476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":5429775921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003493"},"pg_stats":[{"pgid":"5.18","version":"62'3612","reported_seq":3716,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:00.452987+0000","last_change":"2024-09-11T00:57:00.452987+0000","last_active":"2024-09-11T00:57:00.452987+0000","last_peered":"2024-09-11T00:57:00.452987+0000","last_clean":"2024-09-11T00:57:00.452987+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:57:00.452987+0000","last_undegraded":"2024-09-11T00:57:00.452987+0000","last_fullsized":"2024-09-11T00:57:00.452987+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3612","last_scrub_stamp":"2024-09-11T00:57:00.452887+0000","last_deep_scrub":"62'3612","last_deep_scrub_stamp":"2024-09-11T00:57:00.452887+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.452887+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:10:50.430948+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18811,"num_write_kb":284357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"62'791","reported_seq":939,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:07.828297+0000","last_change":"2024-09-11T00:57:07.828297+0000","last_active":"2024-09-11T00:57:07.828297+0000","last_peered":"2024-09-11T00:57:07.828297+0000","last_clean":"2024-09-11T00:57:07.828297+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:57:07.828297+0000","last_undegraded":"2024-09-11T00:57:07.828297+0000","last_fullsized":"2024-09-11T00:57:07.828297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'791","last_scrub_stamp":"2024-09-11T00:57:07.828194+0000","last_deep_scrub":"62'791","last_deep_scrub_stamp":"2024-09-11T00:57:07.828194+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.828194+0000","objects_scrubbed":3,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:41.823006+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":8319,"num_write":773,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"60'3033","reported_seq":3147,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:01.078438+0000","last_change":"2024-09-11T00:57:01.078438+0000","last_active":"2024-09-11T00:57:01.078438+0000","last_peered":"2024-09-11T00:57:01.078438+0000","last_clean":"2024-09-11T00:57:01.078438+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:57:01.078438+0000","last_undegraded":"2024-09-11T00:57:01.078438+0000","last_fullsized":"2024-09-11T00:57:01.078438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3033","last_scrub_stamp":"2024-09-11T00:57:01.078328+0000","last_deep_scrub":"60'3033","last_deep_scrub_stamp":"2024-09-11T00:57:01.078328+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.078328+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:42:37.346125+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16165,"num_write_kb":269490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"61'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:04.206248+0000","last_change":"2024-09-11T00:57:04.206248+0000","last_active":"2024-09-11T00:57:04.206248+0000","last_peered":"2024-09-11T00:57:04.206248+0000","last_clean":"2024-09-11T00:57:04.206248+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:57:04.206248+0000","last_undegraded":"2024-09-11T00:57:04.206248+0000","last_fullsized":"2024-09-11T00:57:04.206248+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-11T00:57:04.206146+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-11T00:57:04.206146+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.206146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:24:37.028054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"62'3995","reported_seq":4098,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:08.859063+0000","last_change":"2024-09-11T00:57:08.859063+0000","last_active":"2024-09-11T00:57:08.859063+0000","last_peered":"2024-09-11T00:57:08.859063+0000","last_clean":"2024-09-11T00:57:08.859063+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:57:08.859063+0000","last_undegraded":"2024-09-11T00:57:08.859063+0000","last_fullsized":"2024-09-11T00:57:08.859063+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3995","last_scrub_stamp":"2024-09-11T00:57:08.858975+0000","last_deep_scrub":"62'3995","last_deep_scrub_stamp":"2024-09-11T00:57:08.858975+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.858975+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:31:07.032208+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20665,"num_write_kb":324486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"62'22","reported_seq":138,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.046603+0000","last_change":"2024-09-11T00:56:54.046603+0000","last_active":"2024-09-11T00:56:54.046603+0000","last_peered":"2024-09-11T00:56:54.046603+0000","last_clean":"2024-09-11T00:56:54.046603+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:56:54.046603+0000","last_undegraded":"2024-09-11T00:56:54.046603+0000","last_fullsized":"2024-09-11T00:56:54.046603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'22","last_scrub_stamp":"2024-09-11T00:56:54.046518+0000","last_deep_scrub":"62'22","last_deep_scrub_stamp":"2024-09-11T00:56:54.046518+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.046518+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:50:58.895853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"60'3563","reported_seq":3682,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:02.119144+0000","last_change":"2024-09-11T00:57:02.119144+0000","last_active":"2024-09-11T00:57:02.119144+0000","last_peered":"2024-09-11T00:57:02.119144+0000","last_clean":"2024-09-11T00:57:02.119144+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:57:02.119144+0000","last_undegraded":"2024-09-11T00:57:02.119144+0000","last_fullsized":"2024-09-11T00:57:02.119144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3563","last_scrub_stamp":"2024-09-11T00:57:02.119042+0000","last_deep_scrub":"60'3563","last_deep_scrub_stamp":"2024-09-11T00:57:02.119042+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.119042+0000","objects_scrubbed":0,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:43:54.433498+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":44,"num_write":18834,"num_write_kb":321224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"62'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:05.165541+0000","last_change":"2024-09-11T00:57:05.165541+0000","last_active":"2024-09-11T00:57:05.165541+0000","last_peered":"2024-09-11T00:57:05.165541+0000","last_clean":"2024-09-11T00:57:05.165541+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:57:05.165541+0000","last_undegraded":"2024-09-11T00:57:05.165541+0000","last_fullsized":"2024-09-11T00:57:05.165541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3","last_scrub_stamp":"2024-09-11T00:57:05.165444+0000","last_deep_scrub":"62'3","last_deep_scrub_stamp":"2024-09-11T00:57:05.165444+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.165444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:10.208866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"62'3090","reported_seq":3193,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:01.418035+0000","last_change":"2024-09-11T00:57:01.418035+0000","last_active":"2024-09-11T00:57:01.418035+0000","last_peered":"2024-09-11T00:57:01.418035+0000","last_clean":"2024-09-11T00:57:01.418035+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:57:01.418035+0000","last_undegraded":"2024-09-11T00:57:01.418035+0000","last_fullsized":"2024-09-11T00:57:01.418035+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3090","last_scrub_stamp":"2024-09-11T00:57:01.417929+0000","last_deep_scrub":"62'3090","last_deep_scrub_stamp":"2024-09-11T00:57:01.417929+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.417929+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:58.184123+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15625,"num_write_kb":254958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"62'38","reported_seq":180,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.714721+0000","last_change":"2024-09-11T00:56:54.714721+0000","last_active":"2024-09-11T00:56:54.714721+0000","last_peered":"2024-09-11T00:56:54.714721+0000","last_clean":"2024-09-11T00:56:54.714721+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:56:54.714721+0000","last_undegraded":"2024-09-11T00:56:54.714721+0000","last_fullsized":"2024-09-11T00:56:54.714721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'38","last_scrub_stamp":"2024-09-11T00:56:54.714635+0000","last_deep_scrub":"62'38","last_deep_scrub_stamp":"2024-09-11T00:56:54.714635+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.714635+0000","objects_scrubbed":1,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:59:22.322810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"60'2040","reported_seq":2152,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:58.381316+0000","last_change":"2024-09-11T00:56:58.381316+0000","last_active":"2024-09-11T00:56:58.381316+0000","last_peered":"2024-09-11T00:56:58.381316+0000","last_clean":"2024-09-11T00:56:58.381316+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:56:58.381316+0000","last_undegraded":"2024-09-11T00:56:58.381316+0000","last_fullsized":"2024-09-11T00:56:58.381316+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2040","last_scrub_stamp":"2024-09-11T00:56:58.381233+0000","last_deep_scrub":"60'2040","last_deep_scrub_stamp":"2024-09-11T00:56:58.381233+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.381233+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:48:16.374087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":10218,"num_write_kb":171981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"62'717","reported_seq":846,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:07.176180+0000","last_change":"2024-09-11T00:57:07.176180+0000","last_active":"2024-09-11T00:57:07.176180+0000","last_peered":"2024-09-11T00:57:07.176180+0000","last_clean":"2024-09-11T00:57:07.176180+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:57:07.176180+0000","last_undegraded":"2024-09-11T00:57:07.176180+0000","last_fullsized":"2024-09-11T00:57:07.176180+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'717","last_scrub_stamp":"2024-09-11T00:57:07.176050+0000","last_deep_scrub":"62'717","last_deep_scrub_stamp":"2024-09-11T00:57:07.176050+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.176050+0000","objects_scrubbed":1,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:09.815079+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":73385,"num_write":707,"num_write_kb":4317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"62'2671","reported_seq":2774,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:08.143383+0000","last_change":"2024-09-11T00:57:08.143383+0000","last_active":"2024-09-11T00:57:08.143383+0000","last_peered":"2024-09-11T00:57:08.143383+0000","last_clean":"2024-09-11T00:57:08.143383+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:57:08.143383+0000","last_undegraded":"2024-09-11T00:57:08.143383+0000","last_fullsized":"2024-09-11T00:57:08.143383+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2671","last_scrub_stamp":"2024-09-11T00:57:08.143313+0000","last_deep_scrub":"62'2671","last_deep_scrub_stamp":"2024-09-11T00:57:08.143313+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.143313+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:29.149992+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":213871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"62'4","reported_seq":145,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:57.791494+0000","last_change":"2024-09-11T00:56:57.791494+0000","last_active":"2024-09-11T00:56:57.791494+0000","last_peered":"2024-09-11T00:56:57.791494+0000","last_clean":"2024-09-11T00:56:57.791494+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:56:57.791494+0000","last_undegraded":"2024-09-11T00:56:57.791494+0000","last_fullsized":"2024-09-11T00:56:57.791494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4","last_scrub_stamp":"2024-09-11T00:56:57.791406+0000","last_deep_scrub":"62'4","last_deep_scrub_stamp":"2024-09-11T00:56:57.791406+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.791406+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:19:14.259832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"60'3687","reported_seq":3798,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:04.235455+0000","last_change":"2024-09-11T00:57:04.235455+0000","last_active":"2024-09-11T00:57:04.235455+0000","last_peered":"2024-09-11T00:57:04.235455+0000","last_clean":"2024-09-11T00:57:04.235455+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:57:04.235455+0000","last_undegraded":"2024-09-11T00:57:04.235455+0000","last_fullsized":"2024-09-11T00:57:04.235455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3687","last_scrub_stamp":"2024-09-11T00:57:04.235390+0000","last_deep_scrub":"60'3687","last_deep_scrub_stamp":"2024-09-11T00:57:04.235390+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.235390+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:23.747635+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19525,"num_write_kb":315726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"62'140","reported_seq":245,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:06.125927+0000","last_change":"2024-09-11T00:57:06.125927+0000","last_active":"2024-09-11T00:57:06.125927+0000","last_peered":"2024-09-11T00:57:06.125927+0000","last_clean":"2024-09-11T00:57:06.125927+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:57:06.125927+0000","last_undegraded":"2024-09-11T00:57:06.125927+0000","last_fullsized":"2024-09-11T00:57:06.125927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'140","last_scrub_stamp":"2024-09-11T00:57:06.125841+0000","last_deep_scrub":"62'140","last_deep_scrub_stamp":"2024-09-11T00:57:06.125841+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.125841+0000","objects_scrubbed":1,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:36.011406+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1356353,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":1393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"62'3457","reported_seq":3553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.513475+0000","last_change":"2024-09-11T00:30:54.990000+0000","last_active":"2024-09-11T00:56:28.513475+0000","last_peered":"2024-09-11T00:56:28.513475+0000","last_clean":"2024-09-11T00:56:28.513475+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:56:28.513475+0000","last_undegraded":"2024-09-11T00:56:28.513475+0000","last_fullsized":"2024-09-11T00:56:28.513475+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3457,"log_dups_size":0,"ondisk_log_size":3457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:23:03.497994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18121,"num_write_kb":297725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"60'2670","reported_seq":2783,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:07.221647+0000","last_change":"2024-09-11T00:57:07.221647+0000","last_active":"2024-09-11T00:57:07.221647+0000","last_peered":"2024-09-11T00:57:07.221647+0000","last_clean":"2024-09-11T00:57:07.221647+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:57:07.221647+0000","last_undegraded":"2024-09-11T00:57:07.221647+0000","last_fullsized":"2024-09-11T00:57:07.221647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2670","last_scrub_stamp":"2024-09-11T00:57:07.221559+0000","last_deep_scrub":"60'2670","last_deep_scrub_stamp":"2024-09-11T00:57:07.221559+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.221559+0000","objects_scrubbed":0,"log_size":2670,"log_dups_size":0,"ondisk_log_size":2670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:39:34.086650+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14000,"num_write_kb":256678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"62'1370","reported_seq":1515,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:58.325639+0000","last_change":"2024-09-11T00:56:58.325639+0000","last_active":"2024-09-11T00:56:58.325639+0000","last_peered":"2024-09-11T00:56:58.325639+0000","last_clean":"2024-09-11T00:56:58.325639+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:56:58.325639+0000","last_undegraded":"2024-09-11T00:56:58.325639+0000","last_fullsized":"2024-09-11T00:56:58.325639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1370","last_scrub_stamp":"2024-09-11T00:56:58.325557+0000","last_deep_scrub":"62'1370","last_deep_scrub_stamp":"2024-09-11T00:56:58.325557+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.325557+0000","objects_scrubbed":2,"log_size":1370,"log_dups_size":0,"ondisk_log_size":1370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:22:26.289083+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":26260,"num_write":1345,"num_write_kb":4470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"61'2","reported_seq":121,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.068774+0000","last_change":"2024-09-11T00:30:53.986418+0000","last_active":"2024-09-11T00:56:28.068774+0000","last_peered":"2024-09-11T00:56:28.068774+0000","last_clean":"2024-09-11T00:56:28.068774+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:56:28.068774+0000","last_undegraded":"2024-09-11T00:56:28.068774+0000","last_fullsized":"2024-09-11T00:56:28.068774+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:20.086142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"62'3537","reported_seq":3641,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:08.245065+0000","last_change":"2024-09-11T00:57:08.245065+0000","last_active":"2024-09-11T00:57:08.245065+0000","last_peered":"2024-09-11T00:57:08.245065+0000","last_clean":"2024-09-11T00:57:08.245065+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:57:08.245065+0000","last_undegraded":"2024-09-11T00:57:08.245065+0000","last_fullsized":"2024-09-11T00:57:08.245065+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3537","last_scrub_stamp":"2024-09-11T00:57:08.244978+0000","last_deep_scrub":"62'3537","last_deep_scrub_stamp":"2024-09-11T00:57:08.244978+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.244978+0000","objects_scrubbed":0,"log_size":3537,"log_dups_size":0,"ondisk_log_size":3537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:00:43.862894+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18517,"num_write_kb":279095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"60'2630","reported_seq":2743,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:58.173597+0000","last_change":"2024-09-11T00:56:58.173597+0000","last_active":"2024-09-11T00:56:58.173597+0000","last_peered":"2024-09-11T00:56:58.173597+0000","last_clean":"2024-09-11T00:56:58.173597+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:56:58.173597+0000","last_undegraded":"2024-09-11T00:56:58.173597+0000","last_fullsized":"2024-09-11T00:56:58.173597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2630","last_scrub_stamp":"2024-09-11T00:56:58.173005+0000","last_deep_scrub":"60'2630","last_deep_scrub_stamp":"2024-09-11T00:56:58.173005+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.173005+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:32:13.917321+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":8,"num_write":13276,"num_write_kb":225109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"60'4","reported_seq":119,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:55.916202+0000","last_change":"2024-09-11T00:56:55.916202+0000","last_active":"2024-09-11T00:56:55.916202+0000","last_peered":"2024-09-11T00:56:55.916202+0000","last_clean":"2024-09-11T00:56:55.916202+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:56:55.916202+0000","last_undegraded":"2024-09-11T00:56:55.916202+0000","last_fullsized":"2024-09-11T00:56:55.916202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4","last_scrub_stamp":"2024-09-11T00:56:55.916108+0000","last_deep_scrub":"60'4","last_deep_scrub_stamp":"2024-09-11T00:56:55.916108+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.916108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:49:15.583153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"61'2","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:03.795761+0000","last_change":"2024-09-11T00:57:03.795761+0000","last_active":"2024-09-11T00:57:03.795761+0000","last_peered":"2024-09-11T00:57:03.795761+0000","last_clean":"2024-09-11T00:57:03.795761+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:57:03.795761+0000","last_undegraded":"2024-09-11T00:57:03.795761+0000","last_fullsized":"2024-09-11T00:57:03.795761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-11T00:57:03.795662+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-11T00:57:03.795662+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.795662+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:23.761978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"62'3470","reported_seq":3566,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.397582+0000","last_change":"2024-09-11T00:30:54.997519+0000","last_active":"2024-09-11T00:56:28.397582+0000","last_peered":"2024-09-11T00:56:28.397582+0000","last_clean":"2024-09-11T00:56:28.397582+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:56:28.397582+0000","last_undegraded":"2024-09-11T00:56:28.397582+0000","last_fullsized":"2024-09-11T00:56:28.397582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:32.624411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17553,"num_write_kb":279333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"60'2695","reported_seq":2810,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:59.863459+0000","last_change":"2024-09-11T00:56:59.863459+0000","last_active":"2024-09-11T00:56:59.863459+0000","last_peered":"2024-09-11T00:56:59.863459+0000","last_clean":"2024-09-11T00:56:59.863459+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:56:59.863459+0000","last_undegraded":"2024-09-11T00:56:59.863459+0000","last_fullsized":"2024-09-11T00:56:59.863459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2695","last_scrub_stamp":"2024-09-11T00:56:59.863365+0000","last_deep_scrub":"60'2695","last_deep_scrub_stamp":"2024-09-11T00:56:59.863365+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.863365+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:40.095008+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":14,"num_write":14032,"num_write_kb":244535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"60'3","reported_seq":118,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:53.850890+0000","last_change":"2024-09-11T00:56:53.850890+0000","last_active":"2024-09-11T00:56:53.850890+0000","last_peered":"2024-09-11T00:56:53.850890+0000","last_clean":"2024-09-11T00:56:53.850890+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:56:53.850890+0000","last_undegraded":"2024-09-11T00:56:53.850890+0000","last_fullsized":"2024-09-11T00:56:53.850890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3","last_scrub_stamp":"2024-09-11T00:56:53.850771+0000","last_deep_scrub":"60'3","last_deep_scrub_stamp":"2024-09-11T00:56:53.850771+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.850771+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:26.843309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"62'6","reported_seq":133,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:03.228171+0000","last_change":"2024-09-11T00:57:03.228171+0000","last_active":"2024-09-11T00:57:03.228171+0000","last_peered":"2024-09-11T00:57:03.228171+0000","last_clean":"2024-09-11T00:57:03.228171+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:57:03.228171+0000","last_undegraded":"2024-09-11T00:57:03.228171+0000","last_fullsized":"2024-09-11T00:57:03.228171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:57:03.228077+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:57:03.228077+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.228077+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:55:51.202975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"62'3622","reported_seq":3726,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:59.405701+0000","last_change":"2024-09-11T00:56:59.405701+0000","last_active":"2024-09-11T00:56:59.405701+0000","last_peered":"2024-09-11T00:56:59.405701+0000","last_clean":"2024-09-11T00:56:59.405701+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:56:59.405701+0000","last_undegraded":"2024-09-11T00:56:59.405701+0000","last_fullsized":"2024-09-11T00:56:59.405701+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3622","last_scrub_stamp":"2024-09-11T00:56:59.405598+0000","last_deep_scrub":"62'3622","last_deep_scrub_stamp":"2024-09-11T00:56:59.405598+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.405598+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:02:32.707334+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18608,"num_write_kb":291609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"60'2732","reported_seq":2846,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:03.280121+0000","last_change":"2024-09-11T00:57:03.280121+0000","last_active":"2024-09-11T00:57:03.280121+0000","last_peered":"2024-09-11T00:57:03.280121+0000","last_clean":"2024-09-11T00:57:03.280121+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:57:03.280121+0000","last_undegraded":"2024-09-11T00:57:03.280121+0000","last_fullsized":"2024-09-11T00:57:03.280121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2732","last_scrub_stamp":"2024-09-11T00:57:03.280040+0000","last_deep_scrub":"60'2732","last_deep_scrub_stamp":"2024-09-11T00:57:03.280040+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.280040+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:01:30.563785+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13936,"num_write_kb":248214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"60'3","reported_seq":118,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:50.663864+0000","last_change":"2024-09-11T00:56:50.663864+0000","last_active":"2024-09-11T00:56:50.663864+0000","last_peered":"2024-09-11T00:56:50.663864+0000","last_clean":"2024-09-11T00:56:50.663864+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:56:50.663864+0000","last_undegraded":"2024-09-11T00:56:50.663864+0000","last_fullsized":"2024-09-11T00:56:50.663864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3","last_scrub_stamp":"2024-09-11T00:56:50.663784+0000","last_deep_scrub":"60'3","last_deep_scrub_stamp":"2024-09-11T00:56:50.663784+0000","last_clean_scrub_stamp":"2024-09-11T00:56:50.663784+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:41:39.646786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"62'7","reported_seq":155,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:02.138476+0000","last_change":"2024-09-11T00:57:02.138476+0000","last_active":"2024-09-11T00:57:02.138476+0000","last_peered":"2024-09-11T00:57:02.138476+0000","last_clean":"2024-09-11T00:57:02.138476+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:57:02.138476+0000","last_undegraded":"2024-09-11T00:57:02.138476+0000","last_fullsized":"2024-09-11T00:57:02.138476+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'7","last_scrub_stamp":"2024-09-11T00:57:02.138384+0000","last_deep_scrub":"62'7","last_deep_scrub_stamp":"2024-09-11T00:57:02.138384+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.138384+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:56:14.449977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"62'3257","reported_seq":3364,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:05.816841+0000","last_change":"2024-09-11T00:57:05.816841+0000","last_active":"2024-09-11T00:57:05.816841+0000","last_peered":"2024-09-11T00:57:05.816841+0000","last_clean":"2024-09-11T00:57:05.816841+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:57:05.816841+0000","last_undegraded":"2024-09-11T00:57:05.816841+0000","last_fullsized":"2024-09-11T00:57:05.816841+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3257","last_scrub_stamp":"2024-09-11T00:57:05.816731+0000","last_deep_scrub":"62'3257","last_deep_scrub_stamp":"2024-09-11T00:57:05.816731+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.816731+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:41.183322+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17168,"num_write_kb":277608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'2688","reported_seq":2803,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:57.923506+0000","last_change":"2024-09-11T00:56:57.923506+0000","last_active":"2024-09-11T00:56:57.923506+0000","last_peered":"2024-09-11T00:56:57.923506+0000","last_clean":"2024-09-11T00:56:57.923506+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:56:57.923506+0000","last_undegraded":"2024-09-11T00:56:57.923506+0000","last_fullsized":"2024-09-11T00:56:57.923506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2688","last_scrub_stamp":"2024-09-11T00:56:57.923388+0000","last_deep_scrub":"60'2688","last_deep_scrub_stamp":"2024-09-11T00:56:57.923388+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.923388+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:11:19.979963+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13441,"num_write_kb":246089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"62'6","reported_seq":150,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:57.343381+0000","last_change":"2024-09-11T00:56:57.343381+0000","last_active":"2024-09-11T00:56:57.343381+0000","last_peered":"2024-09-11T00:56:57.343381+0000","last_clean":"2024-09-11T00:56:57.343381+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:56:57.343381+0000","last_undegraded":"2024-09-11T00:56:57.343381+0000","last_fullsized":"2024-09-11T00:56:57.343381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:56:57.343293+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:56:57.343293+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.343293+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:46:59.632126+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"62'65","reported_seq":253,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:01.140439+0000","last_change":"2024-09-11T00:57:01.140439+0000","last_active":"2024-09-11T00:57:01.140439+0000","last_peered":"2024-09-11T00:57:01.140439+0000","last_clean":"2024-09-11T00:57:01.140439+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:57:01.140439+0000","last_undegraded":"2024-09-11T00:57:01.140439+0000","last_fullsized":"2024-09-11T00:57:01.140439+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'65","last_scrub_stamp":"2024-09-11T00:57:01.140342+0000","last_deep_scrub":"62'65","last_deep_scrub_stamp":"2024-09-11T00:57:01.140342+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.140342+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:55.217199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":126,"num_write":62,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"62'3386","reported_seq":3484,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.422439+0000","last_change":"2024-09-11T00:30:54.994630+0000","last_active":"2024-09-11T00:56:28.422439+0000","last_peered":"2024-09-11T00:56:28.422439+0000","last_clean":"2024-09-11T00:56:28.422439+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:56:28.422439+0000","last_undegraded":"2024-09-11T00:56:28.422439+0000","last_fullsized":"2024-09-11T00:56:28.422439+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:47.372313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18005,"num_write_kb":276900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'3194","reported_seq":3307,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:02.888960+0000","last_change":"2024-09-11T00:57:02.888960+0000","last_active":"2024-09-11T00:57:02.888960+0000","last_peered":"2024-09-11T00:57:02.888960+0000","last_clean":"2024-09-11T00:57:02.888960+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:57:02.888960+0000","last_undegraded":"2024-09-11T00:57:02.888960+0000","last_fullsized":"2024-09-11T00:57:02.888960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3194","last_scrub_stamp":"2024-09-11T00:57:02.888869+0000","last_deep_scrub":"60'3194","last_deep_scrub_stamp":"2024-09-11T00:57:02.888869+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.888869+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:41.665898+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16397,"num_write_kb":262231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"62'521","reported_seq":650,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.897165+0000","last_change":"2024-09-11T00:56:54.897165+0000","last_active":"2024-09-11T00:56:54.897165+0000","last_peered":"2024-09-11T00:56:54.897165+0000","last_clean":"2024-09-11T00:56:54.897165+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:56:54.897165+0000","last_undegraded":"2024-09-11T00:56:54.897165+0000","last_fullsized":"2024-09-11T00:56:54.897165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'521","last_scrub_stamp":"2024-09-11T00:56:54.897074+0000","last_deep_scrub":"62'521","last_deep_scrub_stamp":"2024-09-11T00:56:54.897074+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.897074+0000","objects_scrubbed":1,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:32:57.026216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":41,"num_write":1157,"num_write_kb":372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"62'9","reported_seq":117,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:00.862066+0000","last_change":"2024-09-11T00:57:00.862066+0000","last_active":"2024-09-11T00:57:00.862066+0000","last_peered":"2024-09-11T00:57:00.862066+0000","last_clean":"2024-09-11T00:57:00.862066+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:57:00.862066+0000","last_undegraded":"2024-09-11T00:57:00.862066+0000","last_fullsized":"2024-09-11T00:57:00.862066+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-09-11T00:57:00.861964+0000","last_deep_scrub":"62'9","last_deep_scrub_stamp":"2024-09-11T00:57:00.861964+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.861964+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:54:37.779840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"61'3259","reported_seq":3365,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:04.850349+0000","last_change":"2024-09-11T00:57:04.850349+0000","last_active":"2024-09-11T00:57:04.850349+0000","last_peered":"2024-09-11T00:57:04.850349+0000","last_clean":"2024-09-11T00:57:04.850349+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:57:04.850349+0000","last_undegraded":"2024-09-11T00:57:04.850349+0000","last_fullsized":"2024-09-11T00:57:04.850349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3259","last_scrub_stamp":"2024-09-11T00:57:04.850220+0000","last_deep_scrub":"61'3259","last_deep_scrub_stamp":"2024-09-11T00:57:04.850220+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.850220+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:13.308965+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16779,"num_write_kb":276080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"60'3127","reported_seq":3241,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:58.834284+0000","last_change":"2024-09-11T00:56:58.834284+0000","last_active":"2024-09-11T00:56:58.834284+0000","last_peered":"2024-09-11T00:56:58.834284+0000","last_clean":"2024-09-11T00:56:58.834284+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:56:58.834284+0000","last_undegraded":"2024-09-11T00:56:58.834284+0000","last_fullsized":"2024-09-11T00:56:58.834284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3127","last_scrub_stamp":"2024-09-11T00:56:58.834193+0000","last_deep_scrub":"60'3127","last_deep_scrub_stamp":"2024-09-11T00:56:58.834193+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.834193+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:34.867337+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":12,"num_write":16580,"num_write_kb":258802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"60'4","reported_seq":119,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:55.312508+0000","last_change":"2024-09-11T00:56:55.312508+0000","last_active":"2024-09-11T00:56:55.312508+0000","last_peered":"2024-09-11T00:56:55.312508+0000","last_clean":"2024-09-11T00:56:55.312508+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:56:55.312508+0000","last_undegraded":"2024-09-11T00:56:55.312508+0000","last_fullsized":"2024-09-11T00:56:55.312508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4","last_scrub_stamp":"2024-09-11T00:56:55.312421+0000","last_deep_scrub":"60'4","last_deep_scrub_stamp":"2024-09-11T00:56:55.312421+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.312421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:38:09.069317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'1","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:05.696464+0000","last_change":"2024-09-11T00:57:05.696464+0000","last_active":"2024-09-11T00:57:05.696464+0000","last_peered":"2024-09-11T00:57:05.696464+0000","last_clean":"2024-09-11T00:57:05.696464+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:57:05.696464+0000","last_undegraded":"2024-09-11T00:57:05.696464+0000","last_fullsized":"2024-09-11T00:57:05.696464+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'1","last_scrub_stamp":"2024-09-11T00:57:05.696367+0000","last_deep_scrub":"30'1","last_deep_scrub_stamp":"2024-09-11T00:57:05.696367+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.696367+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:36.598846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"62'3801","reported_seq":3906,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:12.793863+0000","last_change":"2024-09-11T00:57:12.793863+0000","last_active":"2024-09-11T00:57:12.793863+0000","last_peered":"2024-09-11T00:57:12.793863+0000","last_clean":"2024-09-11T00:57:12.793863+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:57:12.793863+0000","last_undegraded":"2024-09-11T00:57:12.793863+0000","last_fullsized":"2024-09-11T00:57:12.793863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3801","last_scrub_stamp":"2024-09-11T00:57:12.793758+0000","last_deep_scrub":"62'3801","last_deep_scrub_stamp":"2024-09-11T00:57:12.793758+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.793758+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:10.927808+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20214,"num_write_kb":289874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"60'2498","reported_seq":2613,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.133953+0000","last_change":"2024-09-11T00:56:54.133953+0000","last_active":"2024-09-11T00:56:54.133953+0000","last_peered":"2024-09-11T00:56:54.133953+0000","last_clean":"2024-09-11T00:56:54.133953+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:56:54.133953+0000","last_undegraded":"2024-09-11T00:56:54.133953+0000","last_fullsized":"2024-09-11T00:56:54.133953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2498","last_scrub_stamp":"2024-09-11T00:56:54.133840+0000","last_deep_scrub":"60'2498","last_deep_scrub_stamp":"2024-09-11T00:56:54.133840+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.133840+0000","objects_scrubbed":0,"log_size":2498,"log_dups_size":0,"ondisk_log_size":2498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:46:40.030654+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":13564,"num_write_kb":230524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"60'5","reported_seq":120,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:56.110279+0000","last_change":"2024-09-11T00:56:56.110279+0000","last_active":"2024-09-11T00:56:56.110279+0000","last_peered":"2024-09-11T00:56:56.110279+0000","last_clean":"2024-09-11T00:56:56.110279+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:56:56.110279+0000","last_undegraded":"2024-09-11T00:56:56.110279+0000","last_fullsized":"2024-09-11T00:56:56.110279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'5","last_scrub_stamp":"2024-09-11T00:56:56.110182+0000","last_deep_scrub":"60'5","last_deep_scrub_stamp":"2024-09-11T00:56:56.110182+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.110182+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:12:40.133364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"62'5","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:04.666648+0000","last_change":"2024-09-11T00:57:04.666648+0000","last_active":"2024-09-11T00:57:04.666648+0000","last_peered":"2024-09-11T00:57:04.666648+0000","last_clean":"2024-09-11T00:57:04.666648+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:57:04.666648+0000","last_undegraded":"2024-09-11T00:57:04.666648+0000","last_fullsized":"2024-09-11T00:57:04.666648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2024-09-11T00:57:04.666519+0000","last_deep_scrub":"62'5","last_deep_scrub_stamp":"2024-09-11T00:57:04.666519+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.666519+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:00:53.386063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"62'3103","reported_seq":3209,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:06.187061+0000","last_change":"2024-09-11T00:57:06.187061+0000","last_active":"2024-09-11T00:57:06.187061+0000","last_peered":"2024-09-11T00:57:06.187061+0000","last_clean":"2024-09-11T00:57:06.187061+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:57:06.187061+0000","last_undegraded":"2024-09-11T00:57:06.187061+0000","last_fullsized":"2024-09-11T00:57:06.187061+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3103","last_scrub_stamp":"2024-09-11T00:57:06.186970+0000","last_deep_scrub":"62'3103","last_deep_scrub_stamp":"2024-09-11T00:57:06.186970+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.186970+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:43.692686+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16058,"num_write_kb":244544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"60'2543","reported_seq":2656,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:07.900193+0000","last_change":"2024-09-11T00:57:07.900193+0000","last_active":"2024-09-11T00:57:07.900193+0000","last_peered":"2024-09-11T00:57:07.900193+0000","last_clean":"2024-09-11T00:57:07.900193+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:57:07.900193+0000","last_undegraded":"2024-09-11T00:57:07.900193+0000","last_fullsized":"2024-09-11T00:57:07.900193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2543","last_scrub_stamp":"2024-09-11T00:57:07.900101+0000","last_deep_scrub":"60'2543","last_deep_scrub_stamp":"2024-09-11T00:57:07.900101+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.900101+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:06:17.326247+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":230458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"60'5","reported_seq":120,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.289576+0000","last_change":"2024-09-11T00:56:54.289576+0000","last_active":"2024-09-11T00:56:54.289576+0000","last_peered":"2024-09-11T00:56:54.289576+0000","last_clean":"2024-09-11T00:56:54.289576+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:56:54.289576+0000","last_undegraded":"2024-09-11T00:56:54.289576+0000","last_fullsized":"2024-09-11T00:56:54.289576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'5","last_scrub_stamp":"2024-09-11T00:56:54.289506+0000","last_deep_scrub":"60'5","last_deep_scrub_stamp":"2024-09-11T00:56:54.289506+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.289506+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:31:35.983150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"62'5","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:06.653146+0000","last_change":"2024-09-11T00:57:06.653146+0000","last_active":"2024-09-11T00:57:06.653146+0000","last_peered":"2024-09-11T00:57:06.653146+0000","last_clean":"2024-09-11T00:57:06.653146+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:57:06.653146+0000","last_undegraded":"2024-09-11T00:57:06.653146+0000","last_fullsized":"2024-09-11T00:57:06.653146+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2024-09-11T00:57:06.653051+0000","last_deep_scrub":"62'5","last_deep_scrub_stamp":"2024-09-11T00:57:06.653051+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.653051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:51.636784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"61'3199","reported_seq":3305,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:07.222332+0000","last_change":"2024-09-11T00:57:07.222332+0000","last_active":"2024-09-11T00:57:07.222332+0000","last_peered":"2024-09-11T00:57:07.222332+0000","last_clean":"2024-09-11T00:57:07.222332+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:57:07.222332+0000","last_undegraded":"2024-09-11T00:57:07.222332+0000","last_fullsized":"2024-09-11T00:57:07.222332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3199","last_scrub_stamp":"2024-09-11T00:57:07.222206+0000","last_deep_scrub":"61'3199","last_deep_scrub_stamp":"2024-09-11T00:57:07.222206+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.222206+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:38:55.432692+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15924,"num_write_kb":260293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"60'3298","reported_seq":3413,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:58.121159+0000","last_change":"2024-09-11T00:56:58.121159+0000","last_active":"2024-09-11T00:56:58.121159+0000","last_peered":"2024-09-11T00:56:58.121159+0000","last_clean":"2024-09-11T00:56:58.121159+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:56:58.121159+0000","last_undegraded":"2024-09-11T00:56:58.121159+0000","last_fullsized":"2024-09-11T00:56:58.121159+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3298","last_scrub_stamp":"2024-09-11T00:56:58.121061+0000","last_deep_scrub":"60'3298","last_deep_scrub_stamp":"2024-09-11T00:56:58.121061+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.121061+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:52:08.080517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17655,"num_write_kb":267324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"62'73","reported_seq":270,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:56.221341+0000","last_change":"2024-09-11T00:56:56.221341+0000","last_active":"2024-09-11T00:56:56.221341+0000","last_peered":"2024-09-11T00:56:56.221341+0000","last_clean":"2024-09-11T00:56:56.221341+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:56:56.221341+0000","last_undegraded":"2024-09-11T00:56:56.221341+0000","last_fullsized":"2024-09-11T00:56:56.221341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'73","last_scrub_stamp":"2024-09-11T00:56:56.221250+0000","last_deep_scrub":"62'73","last_deep_scrub_stamp":"2024-09-11T00:56:56.221250+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.221250+0000","objects_scrubbed":1,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:57:23.739180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":129,"num_write":68,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"62'6","reported_seq":156,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:07.697908+0000","last_change":"2024-09-11T00:57:07.697908+0000","last_active":"2024-09-11T00:57:07.697908+0000","last_peered":"2024-09-11T00:57:07.697908+0000","last_clean":"2024-09-11T00:57:07.697908+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:57:07.697908+0000","last_undegraded":"2024-09-11T00:57:07.697908+0000","last_fullsized":"2024-09-11T00:57:07.697908+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:57:07.697815+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:57:07.697815+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.697815+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:00.719170+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"62'2728","reported_seq":2825,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.520401+0000","last_change":"2024-09-11T00:30:54.991391+0000","last_active":"2024-09-11T00:56:28.520401+0000","last_peered":"2024-09-11T00:56:28.520401+0000","last_clean":"2024-09-11T00:56:28.520401+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:56:28.520401+0000","last_undegraded":"2024-09-11T00:56:28.520401+0000","last_fullsized":"2024-09-11T00:56:28.520401+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2728,"log_dups_size":0,"ondisk_log_size":2728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":237243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"60'2962","reported_seq":3076,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:59.838394+0000","last_change":"2024-09-11T00:56:59.838394+0000","last_active":"2024-09-11T00:56:59.838394+0000","last_peered":"2024-09-11T00:56:59.838394+0000","last_clean":"2024-09-11T00:56:59.838394+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:56:59.838394+0000","last_undegraded":"2024-09-11T00:56:59.838394+0000","last_fullsized":"2024-09-11T00:56:59.838394+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2962","last_scrub_stamp":"2024-09-11T00:56:59.838266+0000","last_deep_scrub":"60'2962","last_deep_scrub_stamp":"2024-09-11T00:56:59.838266+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.838266+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:38:39.202639+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15238,"num_write_kb":265720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"62'8","reported_seq":124,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:53.315719+0000","last_change":"2024-09-11T00:56:53.315719+0000","last_active":"2024-09-11T00:56:53.315719+0000","last_peered":"2024-09-11T00:56:53.315719+0000","last_clean":"2024-09-11T00:56:53.315719+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:56:53.315719+0000","last_undegraded":"2024-09-11T00:56:53.315719+0000","last_fullsized":"2024-09-11T00:56:53.315719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-09-11T00:56:53.315634+0000","last_deep_scrub":"62'8","last_deep_scrub_stamp":"2024-09-11T00:56:53.315634+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.315634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:27:19.909262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"62'500","reported_seq":628,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:11.156523+0000","last_change":"2024-09-11T00:57:11.156523+0000","last_active":"2024-09-11T00:57:11.156523+0000","last_peered":"2024-09-11T00:57:11.156523+0000","last_clean":"2024-09-11T00:57:11.156523+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:57:11.156523+0000","last_undegraded":"2024-09-11T00:57:11.156523+0000","last_fullsized":"2024-09-11T00:57:11.156523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'500","last_scrub_stamp":"2024-09-11T00:57:11.156436+0000","last_deep_scrub":"62'500","last_deep_scrub_stamp":"2024-09-11T00:57:11.156436+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.156436+0000","objects_scrubbed":1,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:33.248709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":1123,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"62'3325","reported_seq":3430,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:06.854834+0000","last_change":"2024-09-11T00:57:06.854834+0000","last_active":"2024-09-11T00:57:06.854834+0000","last_peered":"2024-09-11T00:57:06.854834+0000","last_clean":"2024-09-11T00:57:06.854834+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:57:06.854834+0000","last_undegraded":"2024-09-11T00:57:06.854834+0000","last_fullsized":"2024-09-11T00:57:06.854834+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3325","last_scrub_stamp":"2024-09-11T00:57:06.854742+0000","last_deep_scrub":"62'3325","last_deep_scrub_stamp":"2024-09-11T00:57:06.854742+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.854742+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:29:57.166335+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17451,"num_write_kb":278810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"60'3332","reported_seq":3446,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:11.777243+0000","last_change":"2024-09-11T00:57:11.777243+0000","last_active":"2024-09-11T00:57:11.777243+0000","last_peered":"2024-09-11T00:57:11.777243+0000","last_clean":"2024-09-11T00:57:11.777243+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:57:11.777243+0000","last_undegraded":"2024-09-11T00:57:11.777243+0000","last_fullsized":"2024-09-11T00:57:11.777243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3332","last_scrub_stamp":"2024-09-11T00:57:11.777113+0000","last_deep_scrub":"60'3332","last_deep_scrub_stamp":"2024-09-11T00:57:11.777113+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.777113+0000","objects_scrubbed":0,"log_size":3332,"log_dups_size":0,"ondisk_log_size":3332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:56:38.807453+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17164,"num_write_kb":285768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"31'1","reported_seq":116,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:56.308755+0000","last_change":"2024-09-11T00:56:56.308755+0000","last_active":"2024-09-11T00:56:56.308755+0000","last_peered":"2024-09-11T00:56:56.308755+0000","last_clean":"2024-09-11T00:56:56.308755+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:56:56.308755+0000","last_undegraded":"2024-09-11T00:56:56.308755+0000","last_fullsized":"2024-09-11T00:56:56.308755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'1","last_scrub_stamp":"2024-09-11T00:56:56.308650+0000","last_deep_scrub":"31'1","last_deep_scrub_stamp":"2024-09-11T00:56:56.308650+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.308650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:51:47.983215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"62'74","reported_seq":214,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:50.518182+0000","last_change":"2024-09-11T00:56:50.518182+0000","last_active":"2024-09-11T00:56:50.518182+0000","last_peered":"2024-09-11T00:56:50.518182+0000","last_clean":"2024-09-11T00:56:50.518182+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:56:50.518182+0000","last_undegraded":"2024-09-11T00:56:50.518182+0000","last_fullsized":"2024-09-11T00:56:50.518182+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'74","last_scrub_stamp":"2024-09-11T00:56:50.518060+0000","last_deep_scrub":"62'74","last_deep_scrub_stamp":"2024-09-11T00:56:50.518060+0000","last_clean_scrub_stamp":"2024-09-11T00:56:50.518060+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:18.176495+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"62'4","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:14.203141+0000","last_change":"2024-09-11T00:57:14.203141+0000","last_active":"2024-09-11T00:57:14.203141+0000","last_peered":"2024-09-11T00:57:14.203141+0000","last_clean":"2024-09-11T00:57:14.203141+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:57:14.203141+0000","last_undegraded":"2024-09-11T00:57:14.203141+0000","last_fullsized":"2024-09-11T00:57:14.203141+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4","last_scrub_stamp":"2024-09-11T00:57:14.203067+0000","last_deep_scrub":"62'4","last_deep_scrub_stamp":"2024-09-11T00:57:14.203067+0000","last_clean_scrub_stamp":"2024-09-11T00:57:14.203067+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:13:20.463200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"62'2541","reported_seq":2637,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.500843+0000","last_change":"2024-09-11T00:30:54.991222+0000","last_active":"2024-09-11T00:56:28.500843+0000","last_peered":"2024-09-11T00:56:28.500843+0000","last_clean":"2024-09-11T00:56:28.500843+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:56:28.500843+0000","last_undegraded":"2024-09-11T00:56:28.500843+0000","last_fullsized":"2024-09-11T00:56:28.500843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:43:36.383234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12611,"num_write_kb":206053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"60'2809","reported_seq":2925,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:02.322594+0000","last_change":"2024-09-11T00:57:02.322594+0000","last_active":"2024-09-11T00:57:02.322594+0000","last_peered":"2024-09-11T00:57:02.322594+0000","last_clean":"2024-09-11T00:57:02.322594+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:57:02.322594+0000","last_undegraded":"2024-09-11T00:57:02.322594+0000","last_fullsized":"2024-09-11T00:57:02.322594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2809","last_scrub_stamp":"2024-09-11T00:57:02.322480+0000","last_deep_scrub":"60'2809","last_deep_scrub_stamp":"2024-09-11T00:57:02.322480+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.322480+0000","objects_scrubbed":0,"log_size":2809,"log_dups_size":0,"ondisk_log_size":2809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:59.763288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":34,"num_write":15096,"num_write_kb":232439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"60'2","reported_seq":116,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:56.881566+0000","last_change":"2024-09-11T00:56:56.881566+0000","last_active":"2024-09-11T00:56:56.881566+0000","last_peered":"2024-09-11T00:56:56.881566+0000","last_clean":"2024-09-11T00:56:56.881566+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:56:56.881566+0000","last_undegraded":"2024-09-11T00:56:56.881566+0000","last_fullsized":"2024-09-11T00:56:56.881566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-11T00:56:56.881479+0000","last_deep_scrub":"60'2","last_deep_scrub_stamp":"2024-09-11T00:56:56.881479+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.881479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:18:20.964805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"62'4","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:03.185029+0000","last_change":"2024-09-11T00:57:03.185029+0000","last_active":"2024-09-11T00:57:03.185029+0000","last_peered":"2024-09-11T00:57:03.185029+0000","last_clean":"2024-09-11T00:57:03.185029+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:57:03.185029+0000","last_undegraded":"2024-09-11T00:57:03.185029+0000","last_fullsized":"2024-09-11T00:57:03.185029+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4","last_scrub_stamp":"2024-09-11T00:57:03.184956+0000","last_deep_scrub":"62'4","last_deep_scrub_stamp":"2024-09-11T00:57:03.184956+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.184956+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:54:31.364491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"62'3293","reported_seq":3398,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:10.632867+0000","last_change":"2024-09-11T00:57:10.632867+0000","last_active":"2024-09-11T00:57:10.632867+0000","last_peered":"2024-09-11T00:57:10.632867+0000","last_clean":"2024-09-11T00:57:10.632867+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:57:10.632867+0000","last_undegraded":"2024-09-11T00:57:10.632867+0000","last_fullsized":"2024-09-11T00:57:10.632867+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3293","last_scrub_stamp":"2024-09-11T00:57:10.632753+0000","last_deep_scrub":"62'3293","last_deep_scrub_stamp":"2024-09-11T00:57:10.632753+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.632753+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:52.728638+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17661,"num_write_kb":267414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"60'3034","reported_seq":3148,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:58.761933+0000","last_change":"2024-09-11T00:56:58.761933+0000","last_active":"2024-09-11T00:56:58.761933+0000","last_peered":"2024-09-11T00:56:58.761933+0000","last_clean":"2024-09-11T00:56:58.761933+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:56:58.761933+0000","last_undegraded":"2024-09-11T00:56:58.761933+0000","last_fullsized":"2024-09-11T00:56:58.761933+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3034","last_scrub_stamp":"2024-09-11T00:56:58.761839+0000","last_deep_scrub":"60'3034","last_deep_scrub_stamp":"2024-09-11T00:56:58.761839+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.761839+0000","objects_scrubbed":0,"log_size":3034,"log_dups_size":0,"ondisk_log_size":3034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:28:38.516070+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":15513,"num_write_kb":283129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"62'3","reported_seq":131,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:55.080078+0000","last_change":"2024-09-11T00:56:55.080078+0000","last_active":"2024-09-11T00:56:55.080078+0000","last_peered":"2024-09-11T00:56:55.080078+0000","last_clean":"2024-09-11T00:56:55.080078+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:56:55.080078+0000","last_undegraded":"2024-09-11T00:56:55.080078+0000","last_fullsized":"2024-09-11T00:56:55.080078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3","last_scrub_stamp":"2024-09-11T00:56:55.079993+0000","last_deep_scrub":"62'3","last_deep_scrub_stamp":"2024-09-11T00:56:55.079993+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.079993+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:52:56.783983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"62'1445","reported_seq":1596,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:10.142399+0000","last_change":"2024-09-11T00:57:10.142399+0000","last_active":"2024-09-11T00:57:10.142399+0000","last_peered":"2024-09-11T00:57:10.142399+0000","last_clean":"2024-09-11T00:57:10.142399+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:57:10.142399+0000","last_undegraded":"2024-09-11T00:57:10.142399+0000","last_fullsized":"2024-09-11T00:57:10.142399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1445","last_scrub_stamp":"2024-09-11T00:57:10.142315+0000","last_deep_scrub":"62'1445","last_deep_scrub_stamp":"2024-09-11T00:57:10.142315+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.142315+0000","objects_scrubbed":2,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:23:39.221194+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":36861,"num_write":1416,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'2904","reported_seq":3010,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:57.358960+0000","last_change":"2024-09-11T00:56:57.358857+0000","last_active":"2024-09-11T00:56:57.358960+0000","last_peered":"2024-09-11T00:56:57.358960+0000","last_clean":"2024-09-11T00:56:57.358960+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:56:57.358960+0000","last_undegraded":"2024-09-11T00:56:57.358960+0000","last_fullsized":"2024-09-11T00:56:57.358960+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2904","last_scrub_stamp":"2024-09-11T00:56:57.358761+0000","last_deep_scrub":"62'2904","last_deep_scrub_stamp":"2024-09-11T00:56:57.358761+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.358761+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:57.371165+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15203,"num_write_kb":241147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"60'3390","reported_seq":3502,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:56.792784+0000","last_change":"2024-09-11T00:56:56.792784+0000","last_active":"2024-09-11T00:56:56.792784+0000","last_peered":"2024-09-11T00:56:56.792784+0000","last_clean":"2024-09-11T00:56:56.792784+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:56:56.792784+0000","last_undegraded":"2024-09-11T00:56:56.792784+0000","last_fullsized":"2024-09-11T00:56:56.792784+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3390","last_scrub_stamp":"2024-09-11T00:56:56.792673+0000","last_deep_scrub":"60'3390","last_deep_scrub_stamp":"2024-09-11T00:56:56.792673+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.792673+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:21:25.347605+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17423,"num_write_kb":299173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"62'7","reported_seq":149,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:55.885497+0000","last_change":"2024-09-11T00:56:55.885497+0000","last_active":"2024-09-11T00:56:55.885497+0000","last_peered":"2024-09-11T00:56:55.885497+0000","last_clean":"2024-09-11T00:56:55.885497+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:56:55.885497+0000","last_undegraded":"2024-09-11T00:56:55.885497+0000","last_fullsized":"2024-09-11T00:56:55.885497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'7","last_scrub_stamp":"2024-09-11T00:56:55.885002+0000","last_deep_scrub":"62'7","last_deep_scrub_stamp":"2024-09-11T00:56:55.885002+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.885002+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:11.701806+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"62'3","reported_seq":102,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.493628+0000","last_change":"2024-09-11T00:30:53.985465+0000","last_active":"2024-09-11T00:56:28.493628+0000","last_peered":"2024-09-11T00:56:28.493628+0000","last_clean":"2024-09-11T00:56:28.493628+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:56:28.493628+0000","last_undegraded":"2024-09-11T00:56:28.493628+0000","last_fullsized":"2024-09-11T00:56:28.493628+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"62'3177","reported_seq":3288,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:09.641995+0000","last_change":"2024-09-11T00:57:09.641995+0000","last_active":"2024-09-11T00:57:09.641995+0000","last_peered":"2024-09-11T00:57:09.641995+0000","last_clean":"2024-09-11T00:57:09.641995+0000","last_became_active":"2024-09-11T00:31:26.119641+0000","last_became_peered":"2024-09-11T00:31:26.119641+0000","last_unstale":"2024-09-11T00:57:09.641995+0000","last_undegraded":"2024-09-11T00:57:09.641995+0000","last_fullsized":"2024-09-11T00:57:09.641995+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3177","last_scrub_stamp":"2024-09-11T00:57:09.641903+0000","last_deep_scrub":"62'3177","last_deep_scrub_stamp":"2024-09-11T00:57:09.641903+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.641903+0000","objects_scrubbed":0,"log_size":3177,"log_dups_size":0,"ondisk_log_size":3177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:25:43.992297+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16730,"num_write_kb":276023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"60'2906","reported_seq":3018,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:55.163838+0000","last_change":"2024-09-11T00:56:55.163838+0000","last_active":"2024-09-11T00:56:55.163838+0000","last_peered":"2024-09-11T00:56:55.163838+0000","last_clean":"2024-09-11T00:56:55.163838+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:56:55.163838+0000","last_undegraded":"2024-09-11T00:56:55.163838+0000","last_fullsized":"2024-09-11T00:56:55.163838+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2906","last_scrub_stamp":"2024-09-11T00:56:55.163688+0000","last_deep_scrub":"60'2906","last_deep_scrub_stamp":"2024-09-11T00:56:55.163688+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.163688+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:10.521002+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14712,"num_write_kb":255517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"62'6","reported_seq":134,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:51.684558+0000","last_change":"2024-09-11T00:56:51.684558+0000","last_active":"2024-09-11T00:56:51.684558+0000","last_peered":"2024-09-11T00:56:51.684558+0000","last_clean":"2024-09-11T00:56:51.684558+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:56:51.684558+0000","last_undegraded":"2024-09-11T00:56:51.684558+0000","last_fullsized":"2024-09-11T00:56:51.684558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:56:51.684468+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:56:51.684468+0000","last_clean_scrub_stamp":"2024-09-11T00:56:51.684468+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:43.127053+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"62'3","reported_seq":110,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:55.416696+0000","last_change":"2024-09-11T00:56:55.416696+0000","last_active":"2024-09-11T00:56:55.416696+0000","last_peered":"2024-09-11T00:56:55.416696+0000","last_clean":"2024-09-11T00:56:55.416696+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:56:55.416696+0000","last_undegraded":"2024-09-11T00:56:55.416696+0000","last_fullsized":"2024-09-11T00:56:55.416696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3","last_scrub_stamp":"2024-09-11T00:56:55.416614+0000","last_deep_scrub":"62'3","last_deep_scrub_stamp":"2024-09-11T00:56:55.416614+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.416614+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:26:10.782676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"62'2973","reported_seq":3063,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.532651+0000","last_change":"2024-09-11T00:31:26.119816+0000","last_active":"2024-09-11T00:56:28.532651+0000","last_peered":"2024-09-11T00:56:28.532651+0000","last_clean":"2024-09-11T00:56:28.532651+0000","last_became_active":"2024-09-11T00:31:26.119440+0000","last_became_peered":"2024-09-11T00:31:26.119440+0000","last_unstale":"2024-09-11T00:56:28.532651+0000","last_undegraded":"2024-09-11T00:56:28.532651+0000","last_fullsized":"2024-09-11T00:56:28.532651+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:21:45.707321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15629,"num_write_kb":242839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"60'3355","reported_seq":3467,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:05.188297+0000","last_change":"2024-09-11T00:57:05.188297+0000","last_active":"2024-09-11T00:57:05.188297+0000","last_peered":"2024-09-11T00:57:05.188297+0000","last_clean":"2024-09-11T00:57:05.188297+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:57:05.188297+0000","last_undegraded":"2024-09-11T00:57:05.188297+0000","last_fullsized":"2024-09-11T00:57:05.188297+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3355","last_scrub_stamp":"2024-09-11T00:57:05.188189+0000","last_deep_scrub":"60'3355","last_deep_scrub_stamp":"2024-09-11T00:57:05.188189+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.188189+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:47.673004+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18048,"num_write_kb":289456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"62'16","reported_seq":189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:56.935214+0000","last_change":"2024-09-11T00:56:56.935214+0000","last_active":"2024-09-11T00:56:56.935214+0000","last_peered":"2024-09-11T00:56:56.935214+0000","last_clean":"2024-09-11T00:56:56.935214+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:56:56.935214+0000","last_undegraded":"2024-09-11T00:56:56.935214+0000","last_fullsized":"2024-09-11T00:56:56.935214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2024-09-11T00:56:56.935120+0000","last_deep_scrub":"62'16","last_deep_scrub_stamp":"2024-09-11T00:56:56.935120+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.935120+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:37:57.590907+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":177,"num_read_kb":177,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"62'2","reported_seq":107,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:13.148750+0000","last_change":"2024-09-11T00:57:13.148750+0000","last_active":"2024-09-11T00:57:13.148750+0000","last_peered":"2024-09-11T00:57:13.148750+0000","last_clean":"2024-09-11T00:57:13.148750+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:57:13.148750+0000","last_undegraded":"2024-09-11T00:57:13.148750+0000","last_fullsized":"2024-09-11T00:57:13.148750+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-11T00:57:13.148668+0000","last_deep_scrub":"62'2","last_deep_scrub_stamp":"2024-09-11T00:57:13.148668+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.148668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:58.225052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"62'2678","reported_seq":2774,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.413635+0000","last_change":"2024-09-11T00:30:54.983771+0000","last_active":"2024-09-11T00:56:28.413635+0000","last_peered":"2024-09-11T00:56:28.413635+0000","last_clean":"2024-09-11T00:56:28.413635+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:56:28.413635+0000","last_undegraded":"2024-09-11T00:56:28.413635+0000","last_fullsized":"2024-09-11T00:56:28.413635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":2678,"log_dups_size":0,"ondisk_log_size":2678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:13.479314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13518,"num_write_kb":242590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"60'2791","reported_seq":2903,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:53.489186+0000","last_change":"2024-09-11T00:56:53.489186+0000","last_active":"2024-09-11T00:56:53.489186+0000","last_peered":"2024-09-11T00:56:53.489186+0000","last_clean":"2024-09-11T00:56:53.489186+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:56:53.489186+0000","last_undegraded":"2024-09-11T00:56:53.489186+0000","last_fullsized":"2024-09-11T00:56:53.489186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2791","last_scrub_stamp":"2024-09-11T00:56:53.489113+0000","last_deep_scrub":"60'2791","last_deep_scrub_stamp":"2024-09-11T00:56:53.489113+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.489113+0000","objects_scrubbed":0,"log_size":2791,"log_dups_size":0,"ondisk_log_size":2791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:13.547085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":254542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"60'6","reported_seq":120,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:52.719391+0000","last_change":"2024-09-11T00:56:52.719391+0000","last_active":"2024-09-11T00:56:52.719391+0000","last_peered":"2024-09-11T00:56:52.719391+0000","last_clean":"2024-09-11T00:56:52.719391+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:56:52.719391+0000","last_undegraded":"2024-09-11T00:56:52.719391+0000","last_fullsized":"2024-09-11T00:56:52.719391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'6","last_scrub_stamp":"2024-09-11T00:56:52.719331+0000","last_deep_scrub":"60'6","last_deep_scrub_stamp":"2024-09-11T00:56:52.719331+0000","last_clean_scrub_stamp":"2024-09-11T00:56:52.719331+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:18.018393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"62'3","reported_seq":108,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:12.140217+0000","last_change":"2024-09-11T00:57:12.140217+0000","last_active":"2024-09-11T00:57:12.140217+0000","last_peered":"2024-09-11T00:57:12.140217+0000","last_clean":"2024-09-11T00:57:12.140217+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:57:12.140217+0000","last_undegraded":"2024-09-11T00:57:12.140217+0000","last_fullsized":"2024-09-11T00:57:12.140217+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3","last_scrub_stamp":"2024-09-11T00:57:12.140125+0000","last_deep_scrub":"62'3","last_deep_scrub_stamp":"2024-09-11T00:57:12.140125+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.140125+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:17:12.627169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"62'3272","reported_seq":3367,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.066748+0000","last_change":"2024-09-11T00:30:54.999569+0000","last_active":"2024-09-11T00:56:28.066748+0000","last_peered":"2024-09-11T00:56:28.066748+0000","last_clean":"2024-09-11T00:56:28.066748+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:56:28.066748+0000","last_undegraded":"2024-09-11T00:56:28.066748+0000","last_fullsized":"2024-09-11T00:56:28.066748+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:20.902804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16692,"num_write_kb":275642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"60'3405","reported_seq":3519,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:06.186619+0000","last_change":"2024-09-11T00:57:06.186619+0000","last_active":"2024-09-11T00:57:06.186619+0000","last_peered":"2024-09-11T00:57:06.186619+0000","last_clean":"2024-09-11T00:57:06.186619+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:57:06.186619+0000","last_undegraded":"2024-09-11T00:57:06.186619+0000","last_fullsized":"2024-09-11T00:57:06.186619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3405","last_scrub_stamp":"2024-09-11T00:57:06.186523+0000","last_deep_scrub":"60'3405","last_deep_scrub_stamp":"2024-09-11T00:57:06.186523+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.186523+0000","objects_scrubbed":0,"log_size":3405,"log_dups_size":0,"ondisk_log_size":3405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:42.910335+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":17487,"num_write_kb":315618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"60'5","reported_seq":119,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:59.342382+0000","last_change":"2024-09-11T00:56:59.342382+0000","last_active":"2024-09-11T00:56:59.342382+0000","last_peered":"2024-09-11T00:56:59.342382+0000","last_clean":"2024-09-11T00:56:59.342382+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:56:59.342382+0000","last_undegraded":"2024-09-11T00:56:59.342382+0000","last_fullsized":"2024-09-11T00:56:59.342382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'5","last_scrub_stamp":"2024-09-11T00:56:59.342292+0000","last_deep_scrub":"60'5","last_deep_scrub_stamp":"2024-09-11T00:56:59.342292+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.342292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:16.379697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"62'371","reported_seq":584,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:08.679836+0000","last_change":"2024-09-11T00:57:08.679836+0000","last_active":"2024-09-11T00:57:08.679836+0000","last_peered":"2024-09-11T00:57:08.679836+0000","last_clean":"2024-09-11T00:57:08.679836+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:57:08.679836+0000","last_undegraded":"2024-09-11T00:57:08.679836+0000","last_fullsized":"2024-09-11T00:57:08.679836+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'371","last_scrub_stamp":"2024-09-11T00:57:08.679729+0000","last_deep_scrub":"62'371","last_deep_scrub_stamp":"2024-09-11T00:57:08.679729+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.679729+0000","objects_scrubbed":4,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:46.083044+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":191,"num_read_kb":78004,"num_write":360,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"62'4540","reported_seq":4645,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:06.776106+0000","last_change":"2024-09-11T00:57:06.776106+0000","last_active":"2024-09-11T00:57:06.776106+0000","last_peered":"2024-09-11T00:57:06.776106+0000","last_clean":"2024-09-11T00:57:06.776106+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:57:06.776106+0000","last_undegraded":"2024-09-11T00:57:06.776106+0000","last_fullsized":"2024-09-11T00:57:06.776106+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4540","last_scrub_stamp":"2024-09-11T00:57:06.776012+0000","last_deep_scrub":"62'4540","last_deep_scrub_stamp":"2024-09-11T00:57:06.776012+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.776012+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:58:40.459947+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23424,"num_write_kb":360436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"60'2652","reported_seq":2765,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:57.848088+0000","last_change":"2024-09-11T00:56:57.848088+0000","last_active":"2024-09-11T00:56:57.848088+0000","last_peered":"2024-09-11T00:56:57.848088+0000","last_clean":"2024-09-11T00:56:57.848088+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:56:57.848088+0000","last_undegraded":"2024-09-11T00:56:57.848088+0000","last_fullsized":"2024-09-11T00:56:57.848088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2652","last_scrub_stamp":"2024-09-11T00:56:57.847995+0000","last_deep_scrub":"60'2652","last_deep_scrub_stamp":"2024-09-11T00:56:57.847995+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.847995+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:35.515513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":13591,"num_write_kb":238653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"62'11","reported_seq":153,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:57.220239+0000","last_change":"2024-09-11T00:56:57.220239+0000","last_active":"2024-09-11T00:56:57.220239+0000","last_peered":"2024-09-11T00:56:57.220239+0000","last_clean":"2024-09-11T00:56:57.220239+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:56:57.220239+0000","last_undegraded":"2024-09-11T00:56:57.220239+0000","last_fullsized":"2024-09-11T00:56:57.220239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-09-11T00:56:57.220143+0000","last_deep_scrub":"62'11","last_deep_scrub_stamp":"2024-09-11T00:56:57.220143+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.220143+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:20.830740+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":112,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"62'37","reported_seq":197,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.985643+0000","last_change":"2024-09-11T00:30:53.984038+0000","last_active":"2024-09-11T00:56:28.985643+0000","last_peered":"2024-09-11T00:56:28.985643+0000","last_clean":"2024-09-11T00:56:28.985643+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:56:28.985643+0000","last_undegraded":"2024-09-11T00:56:28.985643+0000","last_fullsized":"2024-09-11T00:56:28.985643+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:03:23.792426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":36,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"62'4259","reported_seq":4356,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.548466+0000","last_change":"2024-09-11T00:30:54.989315+0000","last_active":"2024-09-11T00:56:28.548466+0000","last_peered":"2024-09-11T00:56:28.548466+0000","last_clean":"2024-09-11T00:56:28.548466+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:56:28.548466+0000","last_undegraded":"2024-09-11T00:56:28.548466+0000","last_fullsized":"2024-09-11T00:56:28.548466+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:35:06.225349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22192,"num_write_kb":330506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"60'2765","reported_seq":2876,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:55.743908+0000","last_change":"2024-09-11T00:56:55.743908+0000","last_active":"2024-09-11T00:56:55.743908+0000","last_peered":"2024-09-11T00:56:55.743908+0000","last_clean":"2024-09-11T00:56:55.743908+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:56:55.743908+0000","last_undegraded":"2024-09-11T00:56:55.743908+0000","last_fullsized":"2024-09-11T00:56:55.743908+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2765","last_scrub_stamp":"2024-09-11T00:56:55.743832+0000","last_deep_scrub":"60'2765","last_deep_scrub_stamp":"2024-09-11T00:56:55.743832+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.743832+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:34:46.345457+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14757,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"62'392","reported_seq":577,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:53.146103+0000","last_change":"2024-09-11T00:56:53.146103+0000","last_active":"2024-09-11T00:56:53.146103+0000","last_peered":"2024-09-11T00:56:53.146103+0000","last_clean":"2024-09-11T00:56:53.146103+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:56:53.146103+0000","last_undegraded":"2024-09-11T00:56:53.146103+0000","last_fullsized":"2024-09-11T00:56:53.146103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'392","last_scrub_stamp":"2024-09-11T00:56:53.146012+0000","last_deep_scrub":"62'392","last_deep_scrub_stamp":"2024-09-11T00:56:53.146012+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.146012+0000","objects_scrubbed":4,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:09:09.764335+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":127,"num_read_kb":56252,"num_write":381,"num_write_kb":4290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"62'5","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:13.739976+0000","last_change":"2024-09-11T00:57:13.739976+0000","last_active":"2024-09-11T00:57:13.739976+0000","last_peered":"2024-09-11T00:57:13.739976+0000","last_clean":"2024-09-11T00:57:13.739976+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:57:13.739976+0000","last_undegraded":"2024-09-11T00:57:13.739976+0000","last_fullsized":"2024-09-11T00:57:13.739976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2024-09-11T00:57:13.739888+0000","last_deep_scrub":"62'5","last_deep_scrub_stamp":"2024-09-11T00:57:13.739888+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.739888+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:13:54.348922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"62'3220","reported_seq":3324,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:11.666287+0000","last_change":"2024-09-11T00:57:11.666287+0000","last_active":"2024-09-11T00:57:11.666287+0000","last_peered":"2024-09-11T00:57:11.666287+0000","last_clean":"2024-09-11T00:57:11.666287+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:57:11.666287+0000","last_undegraded":"2024-09-11T00:57:11.666287+0000","last_fullsized":"2024-09-11T00:57:11.666287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3220","last_scrub_stamp":"2024-09-11T00:57:11.666200+0000","last_deep_scrub":"62'3220","last_deep_scrub_stamp":"2024-09-11T00:57:11.666200+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.666200+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:50.377437+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16450,"num_write_kb":262397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"60'3928","reported_seq":4041,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:08.260011+0000","last_change":"2024-09-11T00:57:08.260011+0000","last_active":"2024-09-11T00:57:08.260011+0000","last_peered":"2024-09-11T00:57:08.260011+0000","last_clean":"2024-09-11T00:57:08.260011+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:57:08.260011+0000","last_undegraded":"2024-09-11T00:57:08.260011+0000","last_fullsized":"2024-09-11T00:57:08.260011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3928","last_scrub_stamp":"2024-09-11T00:57:08.259909+0000","last_deep_scrub":"60'3928","last_deep_scrub_stamp":"2024-09-11T00:57:08.259909+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.259909+0000","objects_scrubbed":0,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:40:51.556710+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20599,"num_write_kb":340888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"62'44","reported_seq":199,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:55.186907+0000","last_change":"2024-09-11T00:56:55.186907+0000","last_active":"2024-09-11T00:56:55.186907+0000","last_peered":"2024-09-11T00:56:55.186907+0000","last_clean":"2024-09-11T00:56:55.186907+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:56:55.186907+0000","last_undegraded":"2024-09-11T00:56:55.186907+0000","last_fullsized":"2024-09-11T00:56:55.186907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'44","last_scrub_stamp":"2024-09-11T00:56:55.186811+0000","last_deep_scrub":"62'44","last_deep_scrub_stamp":"2024-09-11T00:56:55.186811+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.186811+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:25:04.079708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":42,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"62'10","reported_seq":158,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:04.217240+0000","last_change":"2024-09-11T00:57:04.217240+0000","last_active":"2024-09-11T00:57:04.217240+0000","last_peered":"2024-09-11T00:57:04.217240+0000","last_clean":"2024-09-11T00:57:04.217240+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:57:04.217240+0000","last_undegraded":"2024-09-11T00:57:04.217240+0000","last_fullsized":"2024-09-11T00:57:04.217240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-09-11T00:57:04.217171+0000","last_deep_scrub":"62'10","last_deep_scrub_stamp":"2024-09-11T00:57:04.217171+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.217171+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:46.011379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"62'4111","reported_seq":4207,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.522560+0000","last_change":"2024-09-11T00:30:54.990323+0000","last_active":"2024-09-11T00:56:28.522560+0000","last_peered":"2024-09-11T00:56:28.522560+0000","last_clean":"2024-09-11T00:56:28.522560+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:56:28.522560+0000","last_undegraded":"2024-09-11T00:56:28.522560+0000","last_fullsized":"2024-09-11T00:56:28.522560+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:08.836987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21746,"num_write_kb":328824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"60'2863","reported_seq":2980,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:59.109857+0000","last_change":"2024-09-11T00:56:59.109857+0000","last_active":"2024-09-11T00:56:59.109857+0000","last_peered":"2024-09-11T00:56:59.109857+0000","last_clean":"2024-09-11T00:56:59.109857+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:56:59.109857+0000","last_undegraded":"2024-09-11T00:56:59.109857+0000","last_fullsized":"2024-09-11T00:56:59.109857+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2863","last_scrub_stamp":"2024-09-11T00:56:59.109727+0000","last_deep_scrub":"60'2863","last_deep_scrub_stamp":"2024-09-11T00:56:59.109727+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.109727+0000","objects_scrubbed":0,"log_size":2863,"log_dups_size":0,"ondisk_log_size":2863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:59:01.225049+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":46,"num_write":14990,"num_write_kb":244331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"60'2","reported_seq":115,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:53.729230+0000","last_change":"2024-09-11T00:56:53.729230+0000","last_active":"2024-09-11T00:56:53.729230+0000","last_peered":"2024-09-11T00:56:53.729230+0000","last_clean":"2024-09-11T00:56:53.729230+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:56:53.729230+0000","last_undegraded":"2024-09-11T00:56:53.729230+0000","last_fullsized":"2024-09-11T00:56:53.729230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-11T00:56:53.729158+0000","last_deep_scrub":"60'2","last_deep_scrub_stamp":"2024-09-11T00:56:53.729158+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.729158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:06.778420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"62'16","reported_seq":215,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:02.192950+0000","last_change":"2024-09-11T00:57:02.192950+0000","last_active":"2024-09-11T00:57:02.192950+0000","last_peered":"2024-09-11T00:57:02.192950+0000","last_clean":"2024-09-11T00:57:02.192950+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:57:02.192950+0000","last_undegraded":"2024-09-11T00:57:02.192950+0000","last_fullsized":"2024-09-11T00:57:02.192950+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2024-09-11T00:57:02.192854+0000","last_deep_scrub":"62'16","last_deep_scrub_stamp":"2024-09-11T00:57:02.192854+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.192854+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:47.360278+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":279,"num_read_kb":279,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"62'3641","reported_seq":3745,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:09.114247+0000","last_change":"2024-09-11T00:57:09.114247+0000","last_active":"2024-09-11T00:57:09.114247+0000","last_peered":"2024-09-11T00:57:09.114247+0000","last_clean":"2024-09-11T00:57:09.114247+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:57:09.114247+0000","last_undegraded":"2024-09-11T00:57:09.114247+0000","last_fullsized":"2024-09-11T00:57:09.114247+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3641","last_scrub_stamp":"2024-09-11T00:57:09.114139+0000","last_deep_scrub":"62'3641","last_deep_scrub_stamp":"2024-09-11T00:57:09.114139+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.114139+0000","objects_scrubbed":0,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:03:56.218479+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19470,"num_write_kb":282742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"60'2896","reported_seq":3011,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:57.096342+0000","last_change":"2024-09-11T00:56:57.096342+0000","last_active":"2024-09-11T00:56:57.096342+0000","last_peered":"2024-09-11T00:56:57.096342+0000","last_clean":"2024-09-11T00:56:57.096342+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:56:57.096342+0000","last_undegraded":"2024-09-11T00:56:57.096342+0000","last_fullsized":"2024-09-11T00:56:57.096342+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2896","last_scrub_stamp":"2024-09-11T00:56:57.096241+0000","last_deep_scrub":"60'2896","last_deep_scrub_stamp":"2024-09-11T00:56:57.096241+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.096241+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:40.956202+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":22,"num_write":14816,"num_write_kb":255894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"62'5","reported_seq":118,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:51.508473+0000","last_change":"2024-09-11T00:56:51.508473+0000","last_active":"2024-09-11T00:56:51.508473+0000","last_peered":"2024-09-11T00:56:51.508473+0000","last_clean":"2024-09-11T00:56:51.508473+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:56:51.508473+0000","last_undegraded":"2024-09-11T00:56:51.508473+0000","last_fullsized":"2024-09-11T00:56:51.508473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2024-09-11T00:56:51.508373+0000","last_deep_scrub":"62'5","last_deep_scrub_stamp":"2024-09-11T00:56:51.508373+0000","last_clean_scrub_stamp":"2024-09-11T00:56:51.508373+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:38:34.105569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"62'6","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:01.907184+0000","last_change":"2024-09-11T00:57:01.907184+0000","last_active":"2024-09-11T00:57:01.907184+0000","last_peered":"2024-09-11T00:57:01.907184+0000","last_clean":"2024-09-11T00:57:01.907184+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:57:01.907184+0000","last_undegraded":"2024-09-11T00:57:01.907184+0000","last_fullsized":"2024-09-11T00:57:01.907184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:57:01.907071+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:57:01.907071+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.907071+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:30.767604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"62'3862","reported_seq":3969,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:11.152893+0000","last_change":"2024-09-11T00:57:11.152893+0000","last_active":"2024-09-11T00:57:11.152893+0000","last_peered":"2024-09-11T00:57:11.152893+0000","last_clean":"2024-09-11T00:57:11.152893+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:57:11.152893+0000","last_undegraded":"2024-09-11T00:57:11.152893+0000","last_fullsized":"2024-09-11T00:57:11.152893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3862","last_scrub_stamp":"2024-09-11T00:57:11.152457+0000","last_deep_scrub":"62'3862","last_deep_scrub_stamp":"2024-09-11T00:57:11.152457+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.152457+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:28.632709+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20250,"num_write_kb":318736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"62'29","reported_seq":129,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.529839+0000","last_change":"2024-09-11T00:30:53.982909+0000","last_active":"2024-09-11T00:56:28.529839+0000","last_peered":"2024-09-11T00:56:28.529839+0000","last_clean":"2024-09-11T00:56:28.529839+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:56:28.529839+0000","last_undegraded":"2024-09-11T00:56:28.529839+0000","last_fullsized":"2024-09-11T00:56:28.529839+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"60'3526","reported_seq":3638,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:58.901661+0000","last_change":"2024-09-11T00:56:58.901661+0000","last_active":"2024-09-11T00:56:58.901661+0000","last_peered":"2024-09-11T00:56:58.901661+0000","last_clean":"2024-09-11T00:56:58.901661+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:56:58.901661+0000","last_undegraded":"2024-09-11T00:56:58.901661+0000","last_fullsized":"2024-09-11T00:56:58.901661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3526","last_scrub_stamp":"2024-09-11T00:56:58.901567+0000","last_deep_scrub":"60'3526","last_deep_scrub_stamp":"2024-09-11T00:56:58.901567+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.901567+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:42:44.166766+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":18407,"num_write_kb":311214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"60'3","reported_seq":116,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:00.333194+0000","last_change":"2024-09-11T00:57:00.333194+0000","last_active":"2024-09-11T00:57:00.333194+0000","last_peered":"2024-09-11T00:57:00.333194+0000","last_clean":"2024-09-11T00:57:00.333194+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:57:00.333194+0000","last_undegraded":"2024-09-11T00:57:00.333194+0000","last_fullsized":"2024-09-11T00:57:00.333194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3","last_scrub_stamp":"2024-09-11T00:57:00.333106+0000","last_deep_scrub":"60'3","last_deep_scrub_stamp":"2024-09-11T00:57:00.333106+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.333106+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:05:56.262921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"61'3026","reported_seq":3124,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.053243+0000","last_change":"2024-09-11T00:30:54.978970+0000","last_active":"2024-09-11T00:56:28.053243+0000","last_peered":"2024-09-11T00:56:28.053243+0000","last_clean":"2024-09-11T00:56:28.053243+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:56:28.053243+0000","last_undegraded":"2024-09-11T00:56:28.053243+0000","last_fullsized":"2024-09-11T00:56:28.053243+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16026,"num_write_kb":256702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"61'972","reported_seq":1122,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.065625+0000","last_change":"2024-09-11T00:30:53.972165+0000","last_active":"2024-09-11T00:56:28.065625+0000","last_peered":"2024-09-11T00:56:28.065625+0000","last_clean":"2024-09-11T00:56:28.065625+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:56:28.065625+0000","last_undegraded":"2024-09-11T00:56:28.065625+0000","last_fullsized":"2024-09-11T00:56:28.065625+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:52.946095+0000","last_clean_scrub_stamp":"2024-09-11T00:30:52.946095+0000","objects_scrubbed":0,"log_size":972,"log_dups_size":0,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:18.713711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":7829,"num_write":949,"num_write_kb":4657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"60'3393","reported_seq":3506,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:00.073349+0000","last_change":"2024-09-11T00:57:00.073349+0000","last_active":"2024-09-11T00:57:00.073349+0000","last_peered":"2024-09-11T00:57:00.073349+0000","last_clean":"2024-09-11T00:57:00.073349+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:57:00.073349+0000","last_undegraded":"2024-09-11T00:57:00.073349+0000","last_fullsized":"2024-09-11T00:57:00.073349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3393","last_scrub_stamp":"2024-09-11T00:57:00.073221+0000","last_deep_scrub":"60'3393","last_deep_scrub_stamp":"2024-09-11T00:57:00.073221+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.073221+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:30.921492+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":17621,"num_write_kb":291741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"60'3","reported_seq":116,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.888596+0000","last_change":"2024-09-11T00:56:54.888596+0000","last_active":"2024-09-11T00:56:54.888596+0000","last_peered":"2024-09-11T00:56:54.888596+0000","last_clean":"2024-09-11T00:56:54.888596+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:56:54.888596+0000","last_undegraded":"2024-09-11T00:56:54.888596+0000","last_fullsized":"2024-09-11T00:56:54.888596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3","last_scrub_stamp":"2024-09-11T00:56:54.888498+0000","last_deep_scrub":"60'3","last_deep_scrub_stamp":"2024-09-11T00:56:54.888498+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.888498+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:49.517568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"62'3945","reported_seq":4048,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:09.826760+0000","last_change":"2024-09-11T00:57:09.826760+0000","last_active":"2024-09-11T00:57:09.826760+0000","last_peered":"2024-09-11T00:57:09.826760+0000","last_clean":"2024-09-11T00:57:09.826760+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:57:09.826760+0000","last_undegraded":"2024-09-11T00:57:09.826760+0000","last_fullsized":"2024-09-11T00:57:09.826760+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3945","last_scrub_stamp":"2024-09-11T00:57:09.826667+0000","last_deep_scrub":"62'3945","last_deep_scrub_stamp":"2024-09-11T00:57:09.826667+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.826667+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:24:55.731572+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20523,"num_write_kb":340217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"62'101","reported_seq":216,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:01.372311+0000","last_change":"2024-09-11T00:57:01.372311+0000","last_active":"2024-09-11T00:57:01.372311+0000","last_peered":"2024-09-11T00:57:01.372311+0000","last_clean":"2024-09-11T00:57:01.372311+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:57:01.372311+0000","last_undegraded":"2024-09-11T00:57:01.372311+0000","last_fullsized":"2024-09-11T00:57:01.372311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'101","last_scrub_stamp":"2024-09-11T00:57:01.372215+0000","last_deep_scrub":"62'101","last_deep_scrub_stamp":"2024-09-11T00:57:01.372215+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.372215+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:56.553328+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":768543,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":728,"num_write":83,"num_write_kb":803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"60'2411","reported_seq":2524,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:04.862489+0000","last_change":"2024-09-11T00:57:04.862489+0000","last_active":"2024-09-11T00:57:04.862489+0000","last_peered":"2024-09-11T00:57:04.862489+0000","last_clean":"2024-09-11T00:57:04.862489+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:57:04.862489+0000","last_undegraded":"2024-09-11T00:57:04.862489+0000","last_fullsized":"2024-09-11T00:57:04.862489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2411","last_scrub_stamp":"2024-09-11T00:57:04.862400+0000","last_deep_scrub":"60'2411","last_deep_scrub_stamp":"2024-09-11T00:57:04.862400+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.862400+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:46:44.628630+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":8,"num_write":12211,"num_write_kb":212619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"62'4","reported_seq":151,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:03.872845+0000","last_change":"2024-09-11T00:57:03.872845+0000","last_active":"2024-09-11T00:57:03.872845+0000","last_peered":"2024-09-11T00:57:03.872845+0000","last_clean":"2024-09-11T00:57:03.872845+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:57:03.872845+0000","last_undegraded":"2024-09-11T00:57:03.872845+0000","last_fullsized":"2024-09-11T00:57:03.872845+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4","last_scrub_stamp":"2024-09-11T00:57:03.872751+0000","last_deep_scrub":"62'4","last_deep_scrub_stamp":"2024-09-11T00:57:03.872751+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.872751+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:22:30.452252+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"62'2354","reported_seq":2457,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:10.117938+0000","last_change":"2024-09-11T00:57:10.117938+0000","last_active":"2024-09-11T00:57:10.117938+0000","last_peered":"2024-09-11T00:57:10.117938+0000","last_clean":"2024-09-11T00:57:10.117938+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:57:10.117938+0000","last_undegraded":"2024-09-11T00:57:10.117938+0000","last_fullsized":"2024-09-11T00:57:10.117938+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2354","last_scrub_stamp":"2024-09-11T00:57:10.117848+0000","last_deep_scrub":"62'2354","last_deep_scrub_stamp":"2024-09-11T00:57:10.117848+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.117848+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:25:12.751798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":205009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"62'677","reported_seq":806,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:57.175764+0000","last_change":"2024-09-11T00:56:57.175764+0000","last_active":"2024-09-11T00:56:57.175764+0000","last_peered":"2024-09-11T00:56:57.175764+0000","last_clean":"2024-09-11T00:56:57.175764+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:56:57.175764+0000","last_undegraded":"2024-09-11T00:56:57.175764+0000","last_fullsized":"2024-09-11T00:56:57.175764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'677","last_scrub_stamp":"2024-09-11T00:56:57.175675+0000","last_deep_scrub":"62'677","last_deep_scrub_stamp":"2024-09-11T00:56:57.175675+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.175675+0000","objects_scrubbed":1,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:14:07.621591+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":51443,"num_write":670,"num_write_kb":4291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"60'2936","reported_seq":3047,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:52.479641+0000","last_change":"2024-09-11T00:56:52.479641+0000","last_active":"2024-09-11T00:56:52.479641+0000","last_peered":"2024-09-11T00:56:52.479641+0000","last_clean":"2024-09-11T00:56:52.479641+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:56:52.479641+0000","last_undegraded":"2024-09-11T00:56:52.479641+0000","last_fullsized":"2024-09-11T00:56:52.479641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2936","last_scrub_stamp":"2024-09-11T00:56:52.479567+0000","last_deep_scrub":"60'2936","last_deep_scrub_stamp":"2024-09-11T00:56:52.479567+0000","last_clean_scrub_stamp":"2024-09-11T00:56:52.479567+0000","objects_scrubbed":0,"log_size":2936,"log_dups_size":0,"ondisk_log_size":2936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:12.543004+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":241742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"62'46","reported_seq":193,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:05.162624+0000","last_change":"2024-09-11T00:57:05.162624+0000","last_active":"2024-09-11T00:57:05.162624+0000","last_peered":"2024-09-11T00:57:05.162624+0000","last_clean":"2024-09-11T00:57:05.162624+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:57:05.162624+0000","last_undegraded":"2024-09-11T00:57:05.162624+0000","last_fullsized":"2024-09-11T00:57:05.162624+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'46","last_scrub_stamp":"2024-09-11T00:57:05.162538+0000","last_deep_scrub":"62'46","last_deep_scrub_stamp":"2024-09-11T00:57:05.162538+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.162538+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:40:01.325638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"62'3441","reported_seq":3532,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.489542+0000","last_change":"2024-09-11T00:31:26.326200+0000","last_active":"2024-09-11T00:56:28.489542+0000","last_peered":"2024-09-11T00:56:28.489542+0000","last_clean":"2024-09-11T00:56:28.489542+0000","last_became_active":"2024-09-11T00:31:26.325895+0000","last_became_peered":"2024-09-11T00:31:26.325895+0000","last_unstale":"2024-09-11T00:56:28.489542+0000","last_undegraded":"2024-09-11T00:56:28.489542+0000","last_fullsized":"2024-09-11T00:56:28.489542+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:39:08.753699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18163,"num_write_kb":273760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.20","version":"60'3848","reported_seq":3961,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:56.132108+0000","last_change":"2024-09-11T00:56:56.132108+0000","last_active":"2024-09-11T00:56:56.132108+0000","last_peered":"2024-09-11T00:56:56.132108+0000","last_clean":"2024-09-11T00:56:56.132108+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:56:56.132108+0000","last_undegraded":"2024-09-11T00:56:56.132108+0000","last_fullsized":"2024-09-11T00:56:56.132108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3848","last_scrub_stamp":"2024-09-11T00:56:56.132027+0000","last_deep_scrub":"60'3848","last_deep_scrub_stamp":"2024-09-11T00:56:56.132027+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.132027+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:12.456914+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20029,"num_write_kb":325938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"62'4287","reported_seq":4385,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.529257+0000","last_change":"2024-09-11T00:30:54.987108+0000","last_active":"2024-09-11T00:56:28.529257+0000","last_peered":"2024-09-11T00:56:28.529257+0000","last_clean":"2024-09-11T00:56:28.529257+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:56:28.529257+0000","last_undegraded":"2024-09-11T00:56:28.529257+0000","last_fullsized":"2024-09-11T00:56:28.529257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22520,"num_write_kb":336110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"60'2962","reported_seq":3075,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:59.151950+0000","last_change":"2024-09-11T00:56:59.151950+0000","last_active":"2024-09-11T00:56:59.151950+0000","last_peered":"2024-09-11T00:56:59.151950+0000","last_clean":"2024-09-11T00:56:59.151950+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:56:59.151950+0000","last_undegraded":"2024-09-11T00:56:59.151950+0000","last_fullsized":"2024-09-11T00:56:59.151950+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2962","last_scrub_stamp":"2024-09-11T00:56:59.151852+0000","last_deep_scrub":"60'2962","last_deep_scrub_stamp":"2024-09-11T00:56:59.151852+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.151852+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:39:00.023857+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15172,"num_write_kb":277710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"62'2334","reported_seq":2439,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:12.106848+0000","last_change":"2024-09-11T00:57:12.106848+0000","last_active":"2024-09-11T00:57:12.106848+0000","last_peered":"2024-09-11T00:57:12.106848+0000","last_clean":"2024-09-11T00:57:12.106848+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:57:12.106848+0000","last_undegraded":"2024-09-11T00:57:12.106848+0000","last_fullsized":"2024-09-11T00:57:12.106848+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2334","last_scrub_stamp":"2024-09-11T00:57:12.106749+0000","last_deep_scrub":"62'2334","last_deep_scrub_stamp":"2024-09-11T00:57:12.106749+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.106749+0000","objects_scrubbed":0,"log_size":2334,"log_dups_size":0,"ondisk_log_size":2334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:36.419261+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11753,"num_write_kb":206776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"60'4175","reported_seq":4290,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:09.259473+0000","last_change":"2024-09-11T00:57:09.259473+0000","last_active":"2024-09-11T00:57:09.259473+0000","last_peered":"2024-09-11T00:57:09.259473+0000","last_clean":"2024-09-11T00:57:09.259473+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:57:09.259473+0000","last_undegraded":"2024-09-11T00:57:09.259473+0000","last_fullsized":"2024-09-11T00:57:09.259473+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4175","last_scrub_stamp":"2024-09-11T00:57:09.259376+0000","last_deep_scrub":"60'4175","last_deep_scrub_stamp":"2024-09-11T00:57:09.259376+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.259376+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:54:35.346143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22002,"num_write_kb":346058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"62'3428","reported_seq":3533,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:14.190431+0000","last_change":"2024-09-11T00:57:14.190431+0000","last_active":"2024-09-11T00:57:14.190431+0000","last_peered":"2024-09-11T00:57:14.190431+0000","last_clean":"2024-09-11T00:57:14.190431+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:57:14.190431+0000","last_undegraded":"2024-09-11T00:57:14.190431+0000","last_fullsized":"2024-09-11T00:57:14.190431+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3428","last_scrub_stamp":"2024-09-11T00:57:14.190350+0000","last_deep_scrub":"62'3428","last_deep_scrub_stamp":"2024-09-11T00:57:14.190350+0000","last_clean_scrub_stamp":"2024-09-11T00:57:14.190350+0000","objects_scrubbed":0,"log_size":3428,"log_dups_size":0,"ondisk_log_size":3428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:00.568104+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17996,"num_write_kb":289213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"60'2543","reported_seq":2656,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:03.137279+0000","last_change":"2024-09-11T00:57:03.137279+0000","last_active":"2024-09-11T00:57:03.137279+0000","last_peered":"2024-09-11T00:57:03.137279+0000","last_clean":"2024-09-11T00:57:03.137279+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:57:03.137279+0000","last_undegraded":"2024-09-11T00:57:03.137279+0000","last_fullsized":"2024-09-11T00:57:03.137279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2543","last_scrub_stamp":"2024-09-11T00:57:03.137186+0000","last_deep_scrub":"60'2543","last_deep_scrub_stamp":"2024-09-11T00:57:03.137186+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.137186+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:34:50.050183+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13221,"num_write_kb":224922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"62'3390","reported_seq":3495,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:08.901854+0000","last_change":"2024-09-11T00:57:08.901854+0000","last_active":"2024-09-11T00:57:08.901854+0000","last_peered":"2024-09-11T00:57:08.901854+0000","last_clean":"2024-09-11T00:57:08.901854+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:57:08.901854+0000","last_undegraded":"2024-09-11T00:57:08.901854+0000","last_fullsized":"2024-09-11T00:57:08.901854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3390","last_scrub_stamp":"2024-09-11T00:57:08.901731+0000","last_deep_scrub":"62'3390","last_deep_scrub_stamp":"2024-09-11T00:57:08.901731+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.901731+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:26.911111+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17320,"num_write_kb":290458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"60'2853","reported_seq":2966,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:00.791064+0000","last_change":"2024-09-11T00:57:00.791064+0000","last_active":"2024-09-11T00:57:00.791064+0000","last_peered":"2024-09-11T00:57:00.791064+0000","last_clean":"2024-09-11T00:57:00.791064+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:57:00.791064+0000","last_undegraded":"2024-09-11T00:57:00.791064+0000","last_fullsized":"2024-09-11T00:57:00.791064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2853","last_scrub_stamp":"2024-09-11T00:57:00.790968+0000","last_deep_scrub":"60'2853","last_deep_scrub_stamp":"2024-09-11T00:57:00.790968+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.790968+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:28:17.766768+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14682,"num_write_kb":238952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"62'2740","reported_seq":2845,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:12.871414+0000","last_change":"2024-09-11T00:57:12.871414+0000","last_active":"2024-09-11T00:57:12.871414+0000","last_peered":"2024-09-11T00:57:12.871414+0000","last_clean":"2024-09-11T00:57:12.871414+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:57:12.871414+0000","last_undegraded":"2024-09-11T00:57:12.871414+0000","last_fullsized":"2024-09-11T00:57:12.871414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2740","last_scrub_stamp":"2024-09-11T00:57:12.871324+0000","last_deep_scrub":"62'2740","last_deep_scrub_stamp":"2024-09-11T00:57:12.871324+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.871324+0000","objects_scrubbed":0,"log_size":2740,"log_dups_size":0,"ondisk_log_size":2740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:19:09.405525+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14066,"num_write_kb":224154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"60'3660","reported_seq":3773,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:01.790608+0000","last_change":"2024-09-11T00:57:01.790608+0000","last_active":"2024-09-11T00:57:01.790608+0000","last_peered":"2024-09-11T00:57:01.790608+0000","last_clean":"2024-09-11T00:57:01.790608+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:57:01.790608+0000","last_undegraded":"2024-09-11T00:57:01.790608+0000","last_fullsized":"2024-09-11T00:57:01.790608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3660","last_scrub_stamp":"2024-09-11T00:57:01.790501+0000","last_deep_scrub":"60'3660","last_deep_scrub_stamp":"2024-09-11T00:57:01.790501+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.790501+0000","objects_scrubbed":0,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:21.173261+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19466,"num_write_kb":307301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"62'4363","reported_seq":4460,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.478746+0000","last_change":"2024-09-11T00:30:54.987532+0000","last_active":"2024-09-11T00:56:28.478746+0000","last_peered":"2024-09-11T00:56:28.478746+0000","last_clean":"2024-09-11T00:56:28.478746+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:56:28.478746+0000","last_undegraded":"2024-09-11T00:56:28.478746+0000","last_fullsized":"2024-09-11T00:56:28.478746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:49.666119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23064,"num_write_kb":350422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"60'2921","reported_seq":3034,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:58.088203+0000","last_change":"2024-09-11T00:56:58.088203+0000","last_active":"2024-09-11T00:56:58.088203+0000","last_peered":"2024-09-11T00:56:58.088203+0000","last_clean":"2024-09-11T00:56:58.088203+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:56:58.088203+0000","last_undegraded":"2024-09-11T00:56:58.088203+0000","last_fullsized":"2024-09-11T00:56:58.088203+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2921","last_scrub_stamp":"2024-09-11T00:56:58.088108+0000","last_deep_scrub":"60'2921","last_deep_scrub_stamp":"2024-09-11T00:56:58.088108+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.088108+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:21.480415+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15417,"num_write_kb":250464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"62'3030","reported_seq":3135,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:13.088347+0000","last_change":"2024-09-11T00:57:13.088347+0000","last_active":"2024-09-11T00:57:13.088347+0000","last_peered":"2024-09-11T00:57:13.088347+0000","last_clean":"2024-09-11T00:57:13.088347+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:57:13.088347+0000","last_undegraded":"2024-09-11T00:57:13.088347+0000","last_fullsized":"2024-09-11T00:57:13.088347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3030","last_scrub_stamp":"2024-09-11T00:57:13.088252+0000","last_deep_scrub":"62'3030","last_deep_scrub_stamp":"2024-09-11T00:57:13.088252+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.088252+0000","objects_scrubbed":0,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:59.168108+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15510,"num_write_kb":262754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"60'3321","reported_seq":3436,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:04.121253+0000","last_change":"2024-09-11T00:57:04.121253+0000","last_active":"2024-09-11T00:57:04.121253+0000","last_peered":"2024-09-11T00:57:04.121253+0000","last_clean":"2024-09-11T00:57:04.121253+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:57:04.121253+0000","last_undegraded":"2024-09-11T00:57:04.121253+0000","last_fullsized":"2024-09-11T00:57:04.121253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3321","last_scrub_stamp":"2024-09-11T00:57:04.121162+0000","last_deep_scrub":"60'3321","last_deep_scrub_stamp":"2024-09-11T00:57:04.121162+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.121162+0000","objects_scrubbed":0,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:04.945000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":25,"num_write":17209,"num_write_kb":314610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"62'2084","reported_seq":2189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:10.298132+0000","last_change":"2024-09-11T00:57:10.298132+0000","last_active":"2024-09-11T00:57:10.298132+0000","last_peered":"2024-09-11T00:57:10.298132+0000","last_clean":"2024-09-11T00:57:10.298132+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:57:10.298132+0000","last_undegraded":"2024-09-11T00:57:10.298132+0000","last_fullsized":"2024-09-11T00:57:10.298132+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2084","last_scrub_stamp":"2024-09-11T00:57:10.298028+0000","last_deep_scrub":"62'2084","last_deep_scrub_stamp":"2024-09-11T00:57:10.298028+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.298028+0000","objects_scrubbed":0,"log_size":2084,"log_dups_size":0,"ondisk_log_size":2084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:15.218052+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10636,"num_write_kb":173560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"60'2810","reported_seq":2923,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:10.297848+0000","last_change":"2024-09-11T00:57:10.297848+0000","last_active":"2024-09-11T00:57:10.297848+0000","last_peered":"2024-09-11T00:57:10.297848+0000","last_clean":"2024-09-11T00:57:10.297848+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:57:10.297848+0000","last_undegraded":"2024-09-11T00:57:10.297848+0000","last_fullsized":"2024-09-11T00:57:10.297848+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2810","last_scrub_stamp":"2024-09-11T00:57:10.297751+0000","last_deep_scrub":"60'2810","last_deep_scrub_stamp":"2024-09-11T00:57:10.297751+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.297751+0000","objects_scrubbed":0,"log_size":2810,"log_dups_size":0,"ondisk_log_size":2810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:32.109718+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":243007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"62'3697","reported_seq":3793,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.455458+0000","last_change":"2024-09-11T00:30:54.990835+0000","last_active":"2024-09-11T00:56:28.455458+0000","last_peered":"2024-09-11T00:56:28.455458+0000","last_clean":"2024-09-11T00:56:28.455458+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:56:28.455458+0000","last_undegraded":"2024-09-11T00:56:28.455458+0000","last_fullsized":"2024-09-11T00:56:28.455458+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:21.765867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18782,"num_write_kb":312790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"60'2824","reported_seq":2936,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:07.107360+0000","last_change":"2024-09-11T00:57:07.107360+0000","last_active":"2024-09-11T00:57:07.107360+0000","last_peered":"2024-09-11T00:57:07.107360+0000","last_clean":"2024-09-11T00:57:07.107360+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:57:07.107360+0000","last_undegraded":"2024-09-11T00:57:07.107360+0000","last_fullsized":"2024-09-11T00:57:07.107360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2824","last_scrub_stamp":"2024-09-11T00:57:07.107256+0000","last_deep_scrub":"60'2824","last_deep_scrub_stamp":"2024-09-11T00:57:07.107256+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.107256+0000","objects_scrubbed":0,"log_size":2824,"log_dups_size":0,"ondisk_log_size":2824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:24:35.809569+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14695,"num_write_kb":259410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"62'2917","reported_seq":3021,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:13.592153+0000","last_change":"2024-09-11T00:57:13.592153+0000","last_active":"2024-09-11T00:57:13.592153+0000","last_peered":"2024-09-11T00:57:13.592153+0000","last_clean":"2024-09-11T00:57:13.592153+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:57:13.592153+0000","last_undegraded":"2024-09-11T00:57:13.592153+0000","last_fullsized":"2024-09-11T00:57:13.592153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2917","last_scrub_stamp":"2024-09-11T00:57:13.592025+0000","last_deep_scrub":"62'2917","last_deep_scrub_stamp":"2024-09-11T00:57:13.592025+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.592025+0000","objects_scrubbed":0,"log_size":2917,"log_dups_size":0,"ondisk_log_size":2917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:16.008324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":242029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"60'3431","reported_seq":3543,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:56.373760+0000","last_change":"2024-09-11T00:56:56.373760+0000","last_active":"2024-09-11T00:56:56.373760+0000","last_peered":"2024-09-11T00:56:56.373760+0000","last_clean":"2024-09-11T00:56:56.373760+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:56:56.373760+0000","last_undegraded":"2024-09-11T00:56:56.373760+0000","last_fullsized":"2024-09-11T00:56:56.373760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3431","last_scrub_stamp":"2024-09-11T00:56:56.373653+0000","last_deep_scrub":"60'3431","last_deep_scrub_stamp":"2024-09-11T00:56:56.373653+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.373653+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:14.920519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18408,"num_write_kb":290787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"62'2945","reported_seq":3049,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:09.263328+0000","last_change":"2024-09-11T00:57:09.263328+0000","last_active":"2024-09-11T00:57:09.263328+0000","last_peered":"2024-09-11T00:57:09.263328+0000","last_clean":"2024-09-11T00:57:09.263328+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:57:09.263328+0000","last_undegraded":"2024-09-11T00:57:09.263328+0000","last_fullsized":"2024-09-11T00:57:09.263328+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2945","last_scrub_stamp":"2024-09-11T00:57:09.263243+0000","last_deep_scrub":"62'2945","last_deep_scrub_stamp":"2024-09-11T00:57:09.263243+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.263243+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:42.744739+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15229,"num_write_kb":249300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"60'2849","reported_seq":2963,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:10.776164+0000","last_change":"2024-09-11T00:57:10.776164+0000","last_active":"2024-09-11T00:57:10.776164+0000","last_peered":"2024-09-11T00:57:10.776164+0000","last_clean":"2024-09-11T00:57:10.776164+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:57:10.776164+0000","last_undegraded":"2024-09-11T00:57:10.776164+0000","last_fullsized":"2024-09-11T00:57:10.776164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2849","last_scrub_stamp":"2024-09-11T00:57:10.776069+0000","last_deep_scrub":"60'2849","last_deep_scrub_stamp":"2024-09-11T00:57:10.776069+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.776069+0000","objects_scrubbed":0,"log_size":2849,"log_dups_size":0,"ondisk_log_size":2849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:02.350395+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14625,"num_write_kb":263252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"62'4017","reported_seq":4113,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.477074+0000","last_change":"2024-09-11T00:30:54.996987+0000","last_active":"2024-09-11T00:56:28.477074+0000","last_peered":"2024-09-11T00:56:28.477074+0000","last_clean":"2024-09-11T00:56:28.477074+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:56:28.477074+0000","last_undegraded":"2024-09-11T00:56:28.477074+0000","last_fullsized":"2024-09-11T00:56:28.477074+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:17:38.213865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":329319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"60'2942","reported_seq":3054,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:01.797270+0000","last_change":"2024-09-11T00:57:01.797270+0000","last_active":"2024-09-11T00:57:01.797270+0000","last_peered":"2024-09-11T00:57:01.797270+0000","last_clean":"2024-09-11T00:57:01.797270+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:57:01.797270+0000","last_undegraded":"2024-09-11T00:57:01.797270+0000","last_fullsized":"2024-09-11T00:57:01.797270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2942","last_scrub_stamp":"2024-09-11T00:57:01.797171+0000","last_deep_scrub":"60'2942","last_deep_scrub_stamp":"2024-09-11T00:57:01.797171+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.797171+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:36:32.927031+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15436,"num_write_kb":246093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"62'3417","reported_seq":3522,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:13.889507+0000","last_change":"2024-09-11T00:57:13.889507+0000","last_active":"2024-09-11T00:57:13.889507+0000","last_peered":"2024-09-11T00:57:13.889507+0000","last_clean":"2024-09-11T00:57:13.889507+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:57:13.889507+0000","last_undegraded":"2024-09-11T00:57:13.889507+0000","last_fullsized":"2024-09-11T00:57:13.889507+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3417","last_scrub_stamp":"2024-09-11T00:57:13.889419+0000","last_deep_scrub":"62'3417","last_deep_scrub_stamp":"2024-09-11T00:57:13.889419+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.889419+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:27:52.118576+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18129,"num_write_kb":269216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"60'2746","reported_seq":2858,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:13.251217+0000","last_change":"2024-09-11T00:57:13.251217+0000","last_active":"2024-09-11T00:57:13.251217+0000","last_peered":"2024-09-11T00:57:13.251217+0000","last_clean":"2024-09-11T00:57:13.251217+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:57:13.251217+0000","last_undegraded":"2024-09-11T00:57:13.251217+0000","last_fullsized":"2024-09-11T00:57:13.251217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2746","last_scrub_stamp":"2024-09-11T00:57:13.251115+0000","last_deep_scrub":"60'2746","last_deep_scrub_stamp":"2024-09-11T00:57:13.251115+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.251115+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:31.529598+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13859,"num_write_kb":247924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"62'2570","reported_seq":2681,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:14.056466+0000","last_change":"2024-09-11T00:57:14.056466+0000","last_active":"2024-09-11T00:57:14.056466+0000","last_peered":"2024-09-11T00:57:14.056466+0000","last_clean":"2024-09-11T00:57:14.056466+0000","last_became_active":"2024-09-11T00:31:26.119436+0000","last_became_peered":"2024-09-11T00:31:26.119436+0000","last_unstale":"2024-09-11T00:57:14.056466+0000","last_undegraded":"2024-09-11T00:57:14.056466+0000","last_fullsized":"2024-09-11T00:57:14.056466+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2570","last_scrub_stamp":"2024-09-11T00:57:14.056377+0000","last_deep_scrub":"62'2570","last_deep_scrub_stamp":"2024-09-11T00:57:14.056377+0000","last_clean_scrub_stamp":"2024-09-11T00:57:14.056377+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:03.852666+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13205,"num_write_kb":216683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"60'2388","reported_seq":2502,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:00.186123+0000","last_change":"2024-09-11T00:57:00.186123+0000","last_active":"2024-09-11T00:57:00.186123+0000","last_peered":"2024-09-11T00:57:00.186123+0000","last_clean":"2024-09-11T00:57:00.186123+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:57:00.186123+0000","last_undegraded":"2024-09-11T00:57:00.186123+0000","last_fullsized":"2024-09-11T00:57:00.186123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2388","last_scrub_stamp":"2024-09-11T00:57:00.185990+0000","last_deep_scrub":"60'2388","last_deep_scrub_stamp":"2024-09-11T00:57:00.185990+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.185990+0000","objects_scrubbed":0,"log_size":2388,"log_dups_size":0,"ondisk_log_size":2388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:55.384651+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4,"num_write":12384,"num_write_kb":205064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"62'3381","reported_seq":3485,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:12.632325+0000","last_change":"2024-09-11T00:57:12.632325+0000","last_active":"2024-09-11T00:57:12.632325+0000","last_peered":"2024-09-11T00:57:12.632325+0000","last_clean":"2024-09-11T00:57:12.632325+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:57:12.632325+0000","last_undegraded":"2024-09-11T00:57:12.632325+0000","last_fullsized":"2024-09-11T00:57:12.632325+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3381","last_scrub_stamp":"2024-09-11T00:57:12.632220+0000","last_deep_scrub":"62'3381","last_deep_scrub_stamp":"2024-09-11T00:57:12.632220+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.632220+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:22:08.114108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17747,"num_write_kb":280034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"60'3936","reported_seq":4049,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:11.256534+0000","last_change":"2024-09-11T00:57:11.256534+0000","last_active":"2024-09-11T00:57:11.256534+0000","last_peered":"2024-09-11T00:57:11.256534+0000","last_clean":"2024-09-11T00:57:11.256534+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:57:11.256534+0000","last_undegraded":"2024-09-11T00:57:11.256534+0000","last_fullsized":"2024-09-11T00:57:11.256534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3936","last_scrub_stamp":"2024-09-11T00:57:11.256420+0000","last_deep_scrub":"60'3936","last_deep_scrub_stamp":"2024-09-11T00:57:11.256420+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.256420+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:46:09.333516+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":20459,"num_write_kb":335990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"62'3772","reported_seq":3867,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.454399+0000","last_change":"2024-09-11T00:30:54.989630+0000","last_active":"2024-09-11T00:56:28.454399+0000","last_peered":"2024-09-11T00:56:28.454399+0000","last_clean":"2024-09-11T00:56:28.454399+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:56:28.454399+0000","last_undegraded":"2024-09-11T00:56:28.454399+0000","last_fullsized":"2024-09-11T00:56:28.454399+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:29:58.980252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20201,"num_write_kb":302053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"60'2757","reported_seq":2869,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:12.280110+0000","last_change":"2024-09-11T00:57:12.280110+0000","last_active":"2024-09-11T00:57:12.280110+0000","last_peered":"2024-09-11T00:57:12.280110+0000","last_clean":"2024-09-11T00:57:12.280110+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:57:12.280110+0000","last_undegraded":"2024-09-11T00:57:12.280110+0000","last_fullsized":"2024-09-11T00:57:12.280110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2757","last_scrub_stamp":"2024-09-11T00:57:12.280012+0000","last_deep_scrub":"60'2757","last_deep_scrub_stamp":"2024-09-11T00:57:12.280012+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.280012+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:18:17.455521+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14203,"num_write_kb":241057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"62'4048","reported_seq":4151,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:03.439931+0000","last_change":"2024-09-11T00:57:03.439931+0000","last_active":"2024-09-11T00:57:03.439931+0000","last_peered":"2024-09-11T00:57:03.439931+0000","last_clean":"2024-09-11T00:57:03.439931+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:57:03.439931+0000","last_undegraded":"2024-09-11T00:57:03.439931+0000","last_fullsized":"2024-09-11T00:57:03.439931+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4048","last_scrub_stamp":"2024-09-11T00:57:03.439855+0000","last_deep_scrub":"62'4048","last_deep_scrub_stamp":"2024-09-11T00:57:03.439855+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.439855+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:07:30.557944+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21387,"num_write_kb":364007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"60'3572","reported_seq":3684,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:08.145697+0000","last_change":"2024-09-11T00:57:08.145697+0000","last_active":"2024-09-11T00:57:08.145697+0000","last_peered":"2024-09-11T00:57:08.145697+0000","last_clean":"2024-09-11T00:57:08.145697+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:57:08.145697+0000","last_undegraded":"2024-09-11T00:57:08.145697+0000","last_fullsized":"2024-09-11T00:57:08.145697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3572","last_scrub_stamp":"2024-09-11T00:57:08.145604+0000","last_deep_scrub":"60'3572","last_deep_scrub_stamp":"2024-09-11T00:57:08.145604+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.145604+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:30:37.316367+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18317,"num_write_kb":294510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"62'2987","reported_seq":3091,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:02.394571+0000","last_change":"2024-09-11T00:57:02.394571+0000","last_active":"2024-09-11T00:57:02.394571+0000","last_peered":"2024-09-11T00:57:02.394571+0000","last_clean":"2024-09-11T00:57:02.394571+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:57:02.394571+0000","last_undegraded":"2024-09-11T00:57:02.394571+0000","last_fullsized":"2024-09-11T00:57:02.394571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2987","last_scrub_stamp":"2024-09-11T00:57:02.394480+0000","last_deep_scrub":"62'2987","last_deep_scrub_stamp":"2024-09-11T00:57:02.394480+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.394480+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:35.080708+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15338,"num_write_kb":237611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"60'2929","reported_seq":3042,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:02.742713+0000","last_change":"2024-09-11T00:57:02.742713+0000","last_active":"2024-09-11T00:57:02.742713+0000","last_peered":"2024-09-11T00:57:02.742713+0000","last_clean":"2024-09-11T00:57:02.742713+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:57:02.742713+0000","last_undegraded":"2024-09-11T00:57:02.742713+0000","last_fullsized":"2024-09-11T00:57:02.742713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2929","last_scrub_stamp":"2024-09-11T00:57:02.742610+0000","last_deep_scrub":"60'2929","last_deep_scrub_stamp":"2024-09-11T00:57:02.742610+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.742610+0000","objects_scrubbed":0,"log_size":2929,"log_dups_size":0,"ondisk_log_size":2929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:17.529350+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":15867,"num_write_kb":268409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"62'3243","reported_seq":3338,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.433332+0000","last_change":"2024-09-11T00:30:54.999492+0000","last_active":"2024-09-11T00:56:28.433332+0000","last_peered":"2024-09-11T00:56:28.433332+0000","last_clean":"2024-09-11T00:56:28.433332+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:56:28.433332+0000","last_undegraded":"2024-09-11T00:56:28.433332+0000","last_fullsized":"2024-09-11T00:56:28.433332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:06.733917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16324,"num_write_kb":262051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"60'2913","reported_seq":3027,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:03.696860+0000","last_change":"2024-09-11T00:57:03.696860+0000","last_active":"2024-09-11T00:57:03.696860+0000","last_peered":"2024-09-11T00:57:03.696860+0000","last_clean":"2024-09-11T00:57:03.696860+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:57:03.696860+0000","last_undegraded":"2024-09-11T00:57:03.696860+0000","last_fullsized":"2024-09-11T00:57:03.696860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2913","last_scrub_stamp":"2024-09-11T00:57:03.696750+0000","last_deep_scrub":"60'2913","last_deep_scrub_stamp":"2024-09-11T00:57:03.696750+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.696750+0000","objects_scrubbed":0,"log_size":2913,"log_dups_size":0,"ondisk_log_size":2913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:23.920964+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15444,"num_write_kb":246103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"62'3344","reported_seq":3440,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.543938+0000","last_change":"2024-09-11T00:30:54.990549+0000","last_active":"2024-09-11T00:56:28.543938+0000","last_peered":"2024-09-11T00:56:28.543938+0000","last_clean":"2024-09-11T00:56:28.543938+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:56:28.543938+0000","last_undegraded":"2024-09-11T00:56:28.543938+0000","last_fullsized":"2024-09-11T00:56:28.543938+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3344,"log_dups_size":0,"ondisk_log_size":3344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:21:37.862599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17374,"num_write_kb":274437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"60'2266","reported_seq":2381,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:02.799588+0000","last_change":"2024-09-11T00:57:02.799588+0000","last_active":"2024-09-11T00:57:02.799588+0000","last_peered":"2024-09-11T00:57:02.799588+0000","last_clean":"2024-09-11T00:57:02.799588+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:57:02.799588+0000","last_undegraded":"2024-09-11T00:57:02.799588+0000","last_fullsized":"2024-09-11T00:57:02.799588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2266","last_scrub_stamp":"2024-09-11T00:57:02.799525+0000","last_deep_scrub":"60'2266","last_deep_scrub_stamp":"2024-09-11T00:57:02.799525+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.799525+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:30.562164+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":26,"num_write":11816,"num_write_kb":219351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"62'2714","reported_seq":2819,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:14.598669+0000","last_change":"2024-09-11T00:57:14.598669+0000","last_active":"2024-09-11T00:57:14.598669+0000","last_peered":"2024-09-11T00:57:14.598669+0000","last_clean":"2024-09-11T00:57:14.598669+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:57:14.598669+0000","last_undegraded":"2024-09-11T00:57:14.598669+0000","last_fullsized":"2024-09-11T00:57:14.598669+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2714","last_scrub_stamp":"2024-09-11T00:57:14.598569+0000","last_deep_scrub":"62'2714","last_deep_scrub_stamp":"2024-09-11T00:57:14.598569+0000","last_clean_scrub_stamp":"2024-09-11T00:57:14.598569+0000","objects_scrubbed":0,"log_size":2714,"log_dups_size":0,"ondisk_log_size":2714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:19:35.610839+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14634,"num_write_kb":230575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"60'3170","reported_seq":3267,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:01.228544+0000","last_change":"2024-09-11T00:57:01.228544+0000","last_active":"2024-09-11T00:57:01.228544+0000","last_peered":"2024-09-11T00:57:01.228544+0000","last_clean":"2024-09-11T00:57:01.228544+0000","last_became_active":"2024-09-11T00:31:26.117146+0000","last_became_peered":"2024-09-11T00:31:26.117146+0000","last_unstale":"2024-09-11T00:57:01.228544+0000","last_undegraded":"2024-09-11T00:57:01.228544+0000","last_fullsized":"2024-09-11T00:57:01.228544+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3170","last_scrub_stamp":"2024-09-11T00:57:01.228423+0000","last_deep_scrub":"60'3170","last_deep_scrub_stamp":"2024-09-11T00:57:01.228423+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.228423+0000","objects_scrubbed":0,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:30:52.030424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15922,"num_write_kb":268523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.30","version":"62'3135","reported_seq":3239,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:09.863390+0000","last_change":"2024-09-11T00:57:09.863390+0000","last_active":"2024-09-11T00:57:09.863390+0000","last_peered":"2024-09-11T00:57:09.863390+0000","last_clean":"2024-09-11T00:57:09.863390+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:57:09.863390+0000","last_undegraded":"2024-09-11T00:57:09.863390+0000","last_fullsized":"2024-09-11T00:57:09.863390+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3135","last_scrub_stamp":"2024-09-11T00:57:09.863276+0000","last_deep_scrub":"62'3135","last_deep_scrub_stamp":"2024-09-11T00:57:09.863276+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.863276+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:30:50.702426+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16038,"num_write_kb":252613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"60'3300","reported_seq":3412,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:59.775630+0000","last_change":"2024-09-11T00:56:59.775630+0000","last_active":"2024-09-11T00:56:59.775630+0000","last_peered":"2024-09-11T00:56:59.775630+0000","last_clean":"2024-09-11T00:56:59.775630+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:56:59.775630+0000","last_undegraded":"2024-09-11T00:56:59.775630+0000","last_fullsized":"2024-09-11T00:56:59.775630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3300","last_scrub_stamp":"2024-09-11T00:56:59.775536+0000","last_deep_scrub":"60'3300","last_deep_scrub_stamp":"2024-09-11T00:56:59.775536+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.775536+0000","objects_scrubbed":0,"log_size":3300,"log_dups_size":0,"ondisk_log_size":3300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:47.113900+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":17861,"num_write_kb":288513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"62'3599","reported_seq":3703,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:11.859963+0000","last_change":"2024-09-11T00:57:11.859963+0000","last_active":"2024-09-11T00:57:11.859963+0000","last_peered":"2024-09-11T00:57:11.859963+0000","last_clean":"2024-09-11T00:57:11.859963+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:57:11.859963+0000","last_undegraded":"2024-09-11T00:57:11.859963+0000","last_fullsized":"2024-09-11T00:57:11.859963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3599","last_scrub_stamp":"2024-09-11T00:57:11.859874+0000","last_deep_scrub":"62'3599","last_deep_scrub_stamp":"2024-09-11T00:57:11.859874+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.859874+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:58.937955+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18686,"num_write_kb":304162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"60'4168","reported_seq":4280,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.445154+0000","last_change":"2024-09-11T00:56:54.445154+0000","last_active":"2024-09-11T00:56:54.445154+0000","last_peered":"2024-09-11T00:56:54.445154+0000","last_clean":"2024-09-11T00:56:54.445154+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:56:54.445154+0000","last_undegraded":"2024-09-11T00:56:54.445154+0000","last_fullsized":"2024-09-11T00:56:54.445154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4168","last_scrub_stamp":"2024-09-11T00:56:54.445072+0000","last_deep_scrub":"60'4168","last_deep_scrub_stamp":"2024-09-11T00:56:54.445072+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.445072+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:52.768326+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22110,"num_write_kb":362983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"62'3937","reported_seq":4034,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.544539+0000","last_change":"2024-09-11T00:30:55.000512+0000","last_active":"2024-09-11T00:56:28.544539+0000","last_peered":"2024-09-11T00:56:28.544539+0000","last_clean":"2024-09-11T00:56:28.544539+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:56:28.544539+0000","last_undegraded":"2024-09-11T00:56:28.544539+0000","last_fullsized":"2024-09-11T00:56:28.544539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:28.721530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21285,"num_write_kb":343312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"60'3475","reported_seq":3587,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:05.089640+0000","last_change":"2024-09-11T00:57:05.089640+0000","last_active":"2024-09-11T00:57:05.089640+0000","last_peered":"2024-09-11T00:57:05.089640+0000","last_clean":"2024-09-11T00:57:05.089640+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:57:05.089640+0000","last_undegraded":"2024-09-11T00:57:05.089640+0000","last_fullsized":"2024-09-11T00:57:05.089640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3475","last_scrub_stamp":"2024-09-11T00:57:05.089554+0000","last_deep_scrub":"60'3475","last_deep_scrub_stamp":"2024-09-11T00:57:05.089554+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.089554+0000","objects_scrubbed":0,"log_size":3475,"log_dups_size":0,"ondisk_log_size":3475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:14.079385+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17645,"num_write_kb":295948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"62'3302","reported_seq":3399,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.512120+0000","last_change":"2024-09-11T00:30:54.983772+0000","last_active":"2024-09-11T00:56:28.512120+0000","last_peered":"2024-09-11T00:56:28.512120+0000","last_clean":"2024-09-11T00:56:28.512120+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:56:28.512120+0000","last_undegraded":"2024-09-11T00:56:28.512120+0000","last_fullsized":"2024-09-11T00:56:28.512120+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:06:37.282266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17558,"num_write_kb":271021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"60'3295","reported_seq":3407,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:00.766474+0000","last_change":"2024-09-11T00:57:00.766474+0000","last_active":"2024-09-11T00:57:00.766474+0000","last_peered":"2024-09-11T00:57:00.766474+0000","last_clean":"2024-09-11T00:57:00.766474+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:57:00.766474+0000","last_undegraded":"2024-09-11T00:57:00.766474+0000","last_fullsized":"2024-09-11T00:57:00.766474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3295","last_scrub_stamp":"2024-09-11T00:57:00.766374+0000","last_deep_scrub":"60'3295","last_deep_scrub_stamp":"2024-09-11T00:57:00.766374+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.766374+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:42:57.783884+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":25,"num_write":17271,"num_write_kb":265887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"61'3033","reported_seq":3131,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.048753+0000","last_change":"2024-09-11T00:30:54.992014+0000","last_active":"2024-09-11T00:56:28.048753+0000","last_peered":"2024-09-11T00:56:28.048753+0000","last_clean":"2024-09-11T00:56:28.048753+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:56:28.048753+0000","last_undegraded":"2024-09-11T00:56:28.048753+0000","last_fullsized":"2024-09-11T00:56:28.048753+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16092,"num_write_kb":252889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"60'2629","reported_seq":2741,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:59.077118+0000","last_change":"2024-09-11T00:56:59.077118+0000","last_active":"2024-09-11T00:56:59.077118+0000","last_peered":"2024-09-11T00:56:59.077118+0000","last_clean":"2024-09-11T00:56:59.077118+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:56:59.077118+0000","last_undegraded":"2024-09-11T00:56:59.077118+0000","last_fullsized":"2024-09-11T00:56:59.077118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2629","last_scrub_stamp":"2024-09-11T00:56:59.077028+0000","last_deep_scrub":"60'2629","last_deep_scrub_stamp":"2024-09-11T00:56:59.077028+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.077028+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:31.814418+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":13758,"num_write_kb":227101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"62'3227","reported_seq":3332,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:04.462923+0000","last_change":"2024-09-11T00:57:04.462923+0000","last_active":"2024-09-11T00:57:04.462923+0000","last_peered":"2024-09-11T00:57:04.462923+0000","last_clean":"2024-09-11T00:57:04.462923+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:57:04.462923+0000","last_undegraded":"2024-09-11T00:57:04.462923+0000","last_fullsized":"2024-09-11T00:57:04.462923+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3227","last_scrub_stamp":"2024-09-11T00:57:04.462848+0000","last_deep_scrub":"62'3227","last_deep_scrub_stamp":"2024-09-11T00:57:04.462848+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.462848+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:58:17.891741+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16396,"num_write_kb":258131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"60'3415","reported_seq":3528,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:10.894054+0000","last_change":"2024-09-11T00:57:10.894054+0000","last_active":"2024-09-11T00:57:10.894054+0000","last_peered":"2024-09-11T00:57:10.894054+0000","last_clean":"2024-09-11T00:57:10.894054+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:57:10.894054+0000","last_undegraded":"2024-09-11T00:57:10.894054+0000","last_fullsized":"2024-09-11T00:57:10.894054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3415","last_scrub_stamp":"2024-09-11T00:57:10.893962+0000","last_deep_scrub":"60'3415","last_deep_scrub_stamp":"2024-09-11T00:57:10.893962+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.893962+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:57:50.507574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":17180,"num_write_kb":310287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"62'2771","reported_seq":2874,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:11.315886+0000","last_change":"2024-09-11T00:57:11.315886+0000","last_active":"2024-09-11T00:57:11.315886+0000","last_peered":"2024-09-11T00:57:11.315886+0000","last_clean":"2024-09-11T00:57:11.315886+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:57:11.315886+0000","last_undegraded":"2024-09-11T00:57:11.315886+0000","last_fullsized":"2024-09-11T00:57:11.315886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2771","last_scrub_stamp":"2024-09-11T00:57:11.315776+0000","last_deep_scrub":"62'2771","last_deep_scrub_stamp":"2024-09-11T00:57:11.315776+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.315776+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:53:59.952427+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14317,"num_write_kb":221149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"60'2741","reported_seq":2855,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:00.094883+0000","last_change":"2024-09-11T00:57:00.094883+0000","last_active":"2024-09-11T00:57:00.094883+0000","last_peered":"2024-09-11T00:57:00.094883+0000","last_clean":"2024-09-11T00:57:00.094883+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:57:00.094883+0000","last_undegraded":"2024-09-11T00:57:00.094883+0000","last_fullsized":"2024-09-11T00:57:00.094883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2741","last_scrub_stamp":"2024-09-11T00:57:00.094785+0000","last_deep_scrub":"60'2741","last_deep_scrub_stamp":"2024-09-11T00:57:00.094785+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.094785+0000","objects_scrubbed":0,"log_size":2741,"log_dups_size":0,"ondisk_log_size":2741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:11:33.596446+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14755,"num_write_kb":255665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"62'3704","reported_seq":3799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.547303+0000","last_change":"2024-09-11T00:30:54.984995+0000","last_active":"2024-09-11T00:56:28.547303+0000","last_peered":"2024-09-11T00:56:28.547303+0000","last_clean":"2024-09-11T00:56:28.547303+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:56:28.547303+0000","last_undegraded":"2024-09-11T00:56:28.547303+0000","last_fullsized":"2024-09-11T00:56:28.547303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:07:53.617639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18483,"num_write_kb":307447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"60'3035","reported_seq":3148,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:06.063094+0000","last_change":"2024-09-11T00:57:06.063094+0000","last_active":"2024-09-11T00:57:06.063094+0000","last_peered":"2024-09-11T00:57:06.063094+0000","last_clean":"2024-09-11T00:57:06.063094+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:57:06.063094+0000","last_undegraded":"2024-09-11T00:57:06.063094+0000","last_fullsized":"2024-09-11T00:57:06.063094+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3035","last_scrub_stamp":"2024-09-11T00:57:06.062988+0000","last_deep_scrub":"60'3035","last_deep_scrub_stamp":"2024-09-11T00:57:06.062988+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.062988+0000","objects_scrubbed":0,"log_size":3035,"log_dups_size":0,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:14:10.554592+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15470,"num_write_kb":262623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"62'3351","reported_seq":3447,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.075584+0000","last_change":"2024-09-11T00:30:54.975612+0000","last_active":"2024-09-11T00:56:28.075584+0000","last_peered":"2024-09-11T00:56:28.075584+0000","last_clean":"2024-09-11T00:56:28.075584+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:56:28.075584+0000","last_undegraded":"2024-09-11T00:56:28.075584+0000","last_fullsized":"2024-09-11T00:56:28.075584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:21.027643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17034,"num_write_kb":277088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"60'4106","reported_seq":4221,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:09.161054+0000","last_change":"2024-09-11T00:57:09.161054+0000","last_active":"2024-09-11T00:57:09.161054+0000","last_peered":"2024-09-11T00:57:09.161054+0000","last_clean":"2024-09-11T00:57:09.161054+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:57:09.161054+0000","last_undegraded":"2024-09-11T00:57:09.161054+0000","last_fullsized":"2024-09-11T00:57:09.161054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4106","last_scrub_stamp":"2024-09-11T00:57:09.160958+0000","last_deep_scrub":"60'4106","last_deep_scrub_stamp":"2024-09-11T00:57:09.160958+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.160958+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:57:11.457246+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22024,"num_write_kb":354504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"62'3391","reported_seq":3486,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:28.526593+0000","last_change":"2024-09-11T00:30:54.989817+0000","last_active":"2024-09-11T00:56:28.526593+0000","last_peered":"2024-09-11T00:56:28.526593+0000","last_clean":"2024-09-11T00:56:28.526593+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:56:28.526593+0000","last_undegraded":"2024-09-11T00:56:28.526593+0000","last_fullsized":"2024-09-11T00:56:28.526593+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T00:30:53.953658+0000","last_clean_scrub_stamp":"2024-09-11T00:30:53.953658+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:40:11.937779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18265,"num_write_kb":274031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"60'2946","reported_seq":3061,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:05.857764+0000","last_change":"2024-09-11T00:57:05.857764+0000","last_active":"2024-09-11T00:57:05.857764+0000","last_peered":"2024-09-11T00:57:05.857764+0000","last_clean":"2024-09-11T00:57:05.857764+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:57:05.857764+0000","last_undegraded":"2024-09-11T00:57:05.857764+0000","last_fullsized":"2024-09-11T00:57:05.857764+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2946","last_scrub_stamp":"2024-09-11T00:57:05.857658+0000","last_deep_scrub":"60'2946","last_deep_scrub_stamp":"2024-09-11T00:57:05.857658+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.857658+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:12:07.015151+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":17,"num_write":15520,"num_write_kb":262805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1107571,"num_write_kb":17543218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212907,"ondisk_log_size":212907,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18223220,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1558,"num_read_kb":197403,"num_write":4817,"num_write_kb":19773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36511744,"data_stored":36446440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4431,"ondisk_log_size":4431,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":381,"num_write":1025649,"num_write_kb":17211858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196678,"ondisk_log_size":196678,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":17635410,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1062,"num_read_kb":143929,"num_write":4568,"num_write_kb":19046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35332096,"data_stored":35270820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4141,"ondisk_log_size":4141,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509763,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487152,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":486270,"kb_avail":93884688,"statfs":{"total":96636764160,"available":96137920512,"internally_reserved":0,"allocated":884736,"data_stored":496637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":497940938},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509765,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668164,"kb_used_data":10292,"kb_used_omap":1,"kb_used_meta":657854,"kb_avail":93703676,"statfs":{"total":96636764160,"available":95952564224,"internally_reserved":0,"allocated":10539008,"data_stored":10153943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":673642950},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509764,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685684,"kb_used_data":17828,"kb_used_omap":1,"kb_used_meta":667838,"kb_avail":93686156,"statfs":{"total":96636764160,"available":95934623744,"internally_reserved":0,"allocated":18255872,"data_stored":17864749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683866565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509763,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722280,"kb_used_data":13912,"kb_used_omap":1,"kb_used_meta":708350,"kb_avail":93649560,"statfs":{"total":96636764160,"available":95897149440,"internally_reserved":0,"allocated":14245888,"data_stored":13849304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":725350856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509764,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737056,"kb_used_data":14480,"kb_used_omap":1,"kb_used_meta":722558,"kb_avail":93634784,"statfs":{"total":96636764160,"available":95882018816,"internally_reserved":0,"allocated":14827520,"data_stored":14436462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739899845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509764,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598360,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":596990,"kb_avail":93773480,"statfs":{"total":96636764160,"available":96024043520,"internally_reserved":0,"allocated":1384448,"data_stored":998818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":611318213},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509765,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741960,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":740414,"kb_avail":93629880,"statfs":{"total":96636764160,"available":95876997120,"internally_reserved":0,"allocated":1564672,"data_stored":1176999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":758184389},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542468,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739432,"kb_used_data":17176,"kb_used_omap":1,"kb_used_meta":722238,"kb_avail":93632408,"statfs":{"total":96636764160,"available":95879585792,"internally_reserved":0,"allocated":17588224,"data_stored":17187564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739572165},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":778240,"data_stored":768599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9162752,"data_stored":9157733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4282577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5648384,"data_stored":5638872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":4283261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5562368,"data_stored":5551329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:57:16.016 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.016 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.016 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.016 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.017 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.017 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.017 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.017 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.018 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.018 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.018 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.018 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.018 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.019 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.019 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.019 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-09-11T00:30:52.946095+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.019 INFO:tasks.ceph:pgid 5.26 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.019 INFO:tasks.ceph:pgid 5.27 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.019 INFO:tasks.ceph:pgid 5.20 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.020 INFO:tasks.ceph:pgid 5.2f last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.020 INFO:tasks.ceph:pgid 5.2a last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.020 INFO:tasks.ceph:pgid 5.36 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.020 INFO:tasks.ceph:pgid 5.35 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.020 INFO:tasks.ceph:pgid 5.32 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.021 INFO:tasks.ceph:pgid 5.3e last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.021 INFO:tasks.ceph:pgid 5.3f last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.021 INFO:tasks.ceph:pgid 5.3c last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.021 INFO:tasks.ceph:pgid 5.3b last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.021 INFO:tasks.ceph:pgid 5.38 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.021 INFO:tasks.ceph:pgid 5.39 last_scrub_stamp 2024-09-11T00:30:53.953658+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=30, tm_sec=53, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=0, tm_min=56, tm_sec=48, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T00:57:16.021 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T00:57:19.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.417+0000 7f79bee00640 1 -- 172.21.15.31:0/3049892801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b8072480 msgr2=0x7f79b81511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.417+0000 7f79bee00640 1 --2- 172.21.15.31:0/3049892801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b8072480 0x7f79b81511c0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f79b80c2040 tx=0x7f79a802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.417+0000 7f79bee00640 1 -- 172.21.15.31:0/3049892801 shutdown_connections 2024-09-11T00:57:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.417+0000 7f79bee00640 1 --2- 172.21.15.31:0/3049892801 >> v2:172.21.15.204:3300/0 conn(0x7f79b8074080 0x7f79b80c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.417+0000 7f79bee00640 1 --2- 172.21.15.31:0/3049892801 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79b80c2410 0x7f79b8151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.417+0000 7f79bee00640 1 --2- 172.21.15.31:0/3049892801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b8072480 0x7f79b81511c0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.417+0000 7f79bee00640 1 -- 172.21.15.31:0/3049892801 >> 172.21.15.31:0/3049892801 conn(0x7f79b80a9750 msgr2=0x7f79b80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.417+0000 7f79bee00640 1 -- 172.21.15.31:0/3049892801 shutdown_connections 2024-09-11T00:57:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.417+0000 7f79bee00640 1 -- 172.21.15.31:0/3049892801 wait complete. 2024-09-11T00:57:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.418+0000 7f79bee00640 1 Processor -- start 2024-09-11T00:57:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.419+0000 7f79bee00640 1 -- start start 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.419+0000 7f79bee00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79b8072480 0x7f79b81535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.419+0000 7f79bee00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79b8074080 0x7f79b81ef290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.419+0000 7f79bee00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b80c2410 0x7f79b81526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bee00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f79b80729e0 con 0x7f79b80c2410 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bee00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f79b8072860 con 0x7f79b8072480 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bee00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f79b8153f20 con 0x7f79b8074080 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b80c2410 0x7f79b81526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79b7e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79b8072480 0x7f79b81535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b80c2410 0x7f79b81526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53372/0 (socket says 172.21.15.31:53372) 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bce00640 1 -- 172.21.15.31:0/3433452534 learned_addr learned my addr 172.21.15.31:0/3433452534 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79af400640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79b8074080 0x7f79b81ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bce00640 1 -- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79b8074080 msgr2=0x7f79b81ef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bce00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79b8074080 0x7f79b81ef290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bce00640 1 -- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79b8072480 msgr2=0x7f79b81535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bce00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79b8072480 0x7f79b81535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.420+0000 7f79bce00640 1 -- 172.21.15.31:0/3433452534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b81508c0 con 0x7f79b80c2410 2024-09-11T00:57:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.421+0000 7f79bce00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b80c2410 0x7f79b81526d0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f79b000cce0 tx=0x7f79b000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.421+0000 7f79b4c00640 1 -- 172.21.15.31:0/3433452534 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b000f450 con 0x7f79b80c2410 2024-09-11T00:57:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.421+0000 7f79b4c00640 1 -- 172.21.15.31:0/3433452534 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79b000f6e0 con 0x7f79b80c2410 2024-09-11T00:57:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.421+0000 7f79b4c00640 1 -- 172.21.15.31:0/3433452534 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b000fb30 con 0x7f79b80c2410 2024-09-11T00:57:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.421+0000 7f79bee00640 1 -- 172.21.15.31:0/3433452534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b81505e0 con 0x7f79b80c2410 2024-09-11T00:57:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.421+0000 7f79bee00640 1 -- 172.21.15.31:0/3433452534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b806e580 con 0x7f79b80c2410 2024-09-11T00:57:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.422+0000 7f79bee00640 1 -- 172.21.15.31:0/3433452534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b81511c0 con 0x7f79b80c2410 2024-09-11T00:57:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.423+0000 7f79b4c00640 1 -- 172.21.15.31:0/3433452534 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f79b000fcd0 con 0x7f79b80c2410 2024-09-11T00:57:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.424+0000 7f79b4c00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7994077fa0 0x7f799407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.424+0000 7f79b4c00640 1 -- 172.21.15.31:0/3433452534 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f79b009b6c0 con 0x7f79b80c2410 2024-09-11T00:57:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.425+0000 7f79b7e00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7994077fa0 0x7f799407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:19.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.426+0000 7f79b7e00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7994077fa0 0x7f799407a460 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f79a8033000 tx=0x7f79a8004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:19.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.429+0000 7f79b4c00640 1 -- 172.21.15.31:0/3433452534 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f79b0014030 con 0x7f79b80c2410 2024-09-11T00:57:19.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.660+0000 7f79bee00640 1 -- 172.21.15.31:0/3433452534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79b814bd70 con 0x7f79b80c2410 2024-09-11T00:57:19.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.661+0000 7f79b4c00640 1 -- 172.21.15.31:0/3433452534 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+6545 (secure 0 0 0) 0x7f79b0062bd0 con 0x7f79b80c2410 2024-09-11T00:57:19.663 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:19.663 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":186,"btime":"2024-09-11T00:56:41:092005+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:31.055051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7672,"qdb_cluster":[7672]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:19.663 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 186 2024-09-11T00:57:19.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.663+0000 7f79bee00640 1 -- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7994077fa0 msgr2=0x7f799407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:19.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.663+0000 7f79bee00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7994077fa0 0x7f799407a460 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f79a8033000 tx=0x7f79a8004730 comp rx=0 tx=0).stop 2024-09-11T00:57:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.663+0000 7f79bee00640 1 -- 172.21.15.31:0/3433452534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b80c2410 msgr2=0x7f79b81526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.663+0000 7f79bee00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b80c2410 0x7f79b81526d0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f79b000cce0 tx=0x7f79b000a620 comp rx=0 tx=0).stop 2024-09-11T00:57:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.663+0000 7f79bee00640 1 -- 172.21.15.31:0/3433452534 shutdown_connections 2024-09-11T00:57:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.663+0000 7f79bee00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7994077fa0 0x7f799407a460 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.663+0000 7f79bee00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79b80c2410 0x7f79b81526d0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.663+0000 7f79bee00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79b8074080 0x7f79b81ef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.663+0000 7f79bee00640 1 --2- 172.21.15.31:0/3433452534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79b8072480 0x7f79b81535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.663+0000 7f79bee00640 1 -- 172.21.15.31:0/3433452534 >> 172.21.15.31:0/3433452534 conn(0x7f79b80a9750 msgr2=0x7f79b80a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.664+0000 7f79bee00640 1 -- 172.21.15.31:0/3433452534 shutdown_connections 2024-09-11T00:57:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.664+0000 7f79bee00640 1 -- 172.21.15.31:0/3433452534 wait complete. 2024-09-11T00:57:19.681 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-11T00:57:19.681 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:57:19.681 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:57:19.701 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:57:19.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:19.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.860+0000 7f49f7000640 1 -- 172.21.15.31:0/1581178243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f006fda0 msgr2=0x7f49f006db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:19.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.861+0000 7f49f7000640 1 --2- 172.21.15.31:0/1581178243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f006fda0 0x7f49f006db00 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009830 tx=0x7f49e802a3f0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.861+0000 7f49f7000640 1 -- 172.21.15.31:0/1581178243 shutdown_connections 2024-09-11T00:57:19.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.861+0000 7f49f7000640 1 --2- 172.21.15.31:0/1581178243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f006fda0 0x7f49f006db00 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.861+0000 7f49f7000640 1 --2- 172.21.15.31:0/1581178243 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49f0071100 0x7f49f00bd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.861+0000 7f49f7000640 1 -- 172.21.15.31:0/1581178243 >> 172.21.15.31:0/1581178243 conn(0x7f49f00c4d20 msgr2=0x7f49f00c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:19.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.861+0000 7f49f7000640 1 -- 172.21.15.31:0/1581178243 shutdown_connections 2024-09-11T00:57:19.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.861+0000 7f49f7000640 1 -- 172.21.15.31:0/1581178243 wait complete. 2024-09-11T00:57:19.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.862+0000 7f49f7000640 1 Processor -- start 2024-09-11T00:57:19.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.862+0000 7f49f7000640 1 -- start start 2024-09-11T00:57:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.863+0000 7f49f7000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49f006fda0 0x7f49f014abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.863+0000 7f49f7000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f0071100 0x7f49f00ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.863+0000 7f49f7000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49f014cb80 0x7f49f01e0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.863+0000 7f49f7000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49f014d0e0 con 0x7f49f0071100 2024-09-11T00:57:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.863+0000 7f49f7000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f49f014cf60 con 0x7f49f014cb80 2024-09-11T00:57:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.863+0000 7f49f7000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f49f00701f0 con 0x7f49f006fda0 2024-09-11T00:57:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.863+0000 7f49f5200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f0071100 0x7f49f00ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.864+0000 7f49f5200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f0071100 0x7f49f00ab1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53398/0 (socket says 172.21.15.31:53398) 2024-09-11T00:57:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.864+0000 7f49f5200640 1 -- 172.21.15.31:0/3088367110 learned_addr learned my addr 172.21.15.31:0/3088367110 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.864+0000 7f49f5c00640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49f006fda0 0x7f49f014abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.864+0000 7f49f6600640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49f014cb80 0x7f49f01e0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.864+0000 7f49f5200640 1 -- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49f006fda0 msgr2=0x7f49f014abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.864+0000 7f49f5200640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49f006fda0 0x7f49f014abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.864+0000 7f49f5200640 1 -- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49f014cb80 msgr2=0x7f49f01e0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.864+0000 7f49f5200640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49f014cb80 0x7f49f01e0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.864+0000 7f49f5200640 1 -- 172.21.15.31:0/3088367110 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f014b0f0 con 0x7f49f0071100 2024-09-11T00:57:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.864+0000 7f49f5200640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f0071100 0x7f49f00ab1c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00d980 tx=0x7f49dc00de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.865+0000 7f49ed600640 1 -- 172.21.15.31:0/3088367110 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc017030 con 0x7f49f0071100 2024-09-11T00:57:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.865+0000 7f49ed600640 1 -- 172.21.15.31:0/3088367110 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49dc009b20 con 0x7f49f0071100 2024-09-11T00:57:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.865+0000 7f49f7000640 1 -- 172.21.15.31:0/3088367110 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49f00ae0d0 con 0x7f49f0071100 2024-09-11T00:57:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.865+0000 7f49ed600640 1 -- 172.21.15.31:0/3088367110 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc004020 con 0x7f49f0071100 2024-09-11T00:57:19.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.865+0000 7f49f7000640 1 -- 172.21.15.31:0/3088367110 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49f01e1aa0 con 0x7f49f0071100 2024-09-11T00:57:19.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.866+0000 7f49f7000640 1 -- 172.21.15.31:0/3088367110 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49f00bd240 con 0x7f49f0071100 2024-09-11T00:57:19.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.867+0000 7f49ed600640 1 -- 172.21.15.31:0/3088367110 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f49dc0041c0 con 0x7f49f0071100 2024-09-11T00:57:19.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.868+0000 7f49ed600640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49d0077fa0 0x7f49d007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:19.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.868+0000 7f49ed600640 1 -- 172.21.15.31:0/3088367110 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f49dc09ae10 con 0x7f49f0071100 2024-09-11T00:57:19.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.868+0000 7f49f5c00640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49d0077fa0 0x7f49d007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:19.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.869+0000 7f49f5c00640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49d0077fa0 0x7f49d007a460 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f49e40020f0 tx=0x7f49e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:19.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:19.872+0000 7f49ed600640 1 -- 172.21.15.31:0/3088367110 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f49dc09e030 con 0x7f49f0071100 2024-09-11T00:57:20.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.106+0000 7f49f7000640 1 -- 172.21.15.31:0/3088367110 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49f00c41d0 con 0x7f49f0071100 2024-09-11T00:57:20.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.107+0000 7f49ed600640 1 -- 172.21.15.31:0/3088367110 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+6545 (secure 0 0 0) 0x7f49dc062970 con 0x7f49f0071100 2024-09-11T00:57:20.109 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:20.109 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":186,"btime":"2024-09-11T00:56:41:092005+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:31.055051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7672,"qdb_cluster":[7672]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:20.109 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 186 2024-09-11T00:57:20.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 -- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49d0077fa0 msgr2=0x7f49d007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49d0077fa0 0x7f49d007a460 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f49e40020f0 tx=0x7f49e4009040 comp rx=0 tx=0).stop 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 -- 172.21.15.31:0/3088367110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f0071100 msgr2=0x7f49f00ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f0071100 0x7f49f00ab1c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00d980 tx=0x7f49dc00de40 comp rx=0 tx=0).stop 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 -- 172.21.15.31:0/3088367110 shutdown_connections 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f49d0077fa0 0x7f49d007a460 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f49f014cb80 0x7f49f01e0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f0071100 0x7f49f00ab1c0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 --2- 172.21.15.31:0/3088367110 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f49f006fda0 0x7f49f014abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 -- 172.21.15.31:0/3088367110 >> 172.21.15.31:0/3088367110 conn(0x7f49f00c4d20 msgr2=0x7f49f00746a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 -- 172.21.15.31:0/3088367110 shutdown_connections 2024-09-11T00:57:20.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:20.109+0000 7f49f7000640 1 -- 172.21.15.31:0/3088367110 wait complete. 2024-09-11T00:57:20.127 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:57:22.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.301+0000 7f075bc00640 1 -- 172.21.15.31:0/3863516671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0754156ad0 msgr2=0x7f0754153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.301+0000 7f075bc00640 1 --2- 172.21.15.31:0/3863516671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0754156ad0 0x7f0754153480 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0740005fd0 tx=0x7f074002a5c0 comp rx=0 tx=0).stop 2024-09-11T00:57:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.302+0000 7f075bc00640 1 -- 172.21.15.31:0/3863516671 shutdown_connections 2024-09-11T00:57:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.302+0000 7f075bc00640 1 --2- 172.21.15.31:0/3863516671 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f07541539c0 0x7f0754159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.302+0000 7f075bc00640 1 --2- 172.21.15.31:0/3863516671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0754156ad0 0x7f0754153480 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.302+0000 7f075bc00640 1 -- 172.21.15.31:0/3863516671 >> 172.21.15.31:0/3863516671 conn(0x7f07540a9750 msgr2=0x7f07540a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.302+0000 7f075bc00640 1 -- 172.21.15.31:0/3863516671 shutdown_connections 2024-09-11T00:57:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.302+0000 7f075bc00640 1 -- 172.21.15.31:0/3863516671 wait complete. 2024-09-11T00:57:22.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.303+0000 7f075bc00640 1 Processor -- start 2024-09-11T00:57:22.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.303+0000 7f075bc00640 1 -- start start 2024-09-11T00:57:22.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.304+0000 7f075bc00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07540c5b10 0x7f07541e06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:22.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.304+0000 7f075bc00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07541539c0 0x7f07541e0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:22.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.304+0000 7f075bc00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0754156ad0 0x7f07541557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:22.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.304+0000 7f075bc00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0754156f20 con 0x7f07540c5b10 2024-09-11T00:57:22.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.304+0000 7f075bc00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f07540c7320 con 0x7f07541539c0 2024-09-11T00:57:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.304+0000 7f075bc00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f07540c3950 con 0x7f0754156ad0 2024-09-11T00:57:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.304+0000 7f0759200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07540c5b10 0x7f07541e06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.305+0000 7f0759200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07540c5b10 0x7f07541e06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53406/0 (socket says 172.21.15.31:53406) 2024-09-11T00:57:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.305+0000 7f0759200640 1 -- 172.21.15.31:0/3667042932 learned_addr learned my addr 172.21.15.31:0/3667042932 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.305+0000 7f0759c00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0754156ad0 0x7f07541557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.305+0000 7f0753e00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07541539c0 0x7f07541e0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.305+0000 7f0759200640 1 -- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0754156ad0 msgr2=0x7f07541557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.305+0000 7f0759200640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0754156ad0 0x7f07541557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.305+0000 7f0759200640 1 -- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07541539c0 msgr2=0x7f07541e0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.305+0000 7f0759200640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07541539c0 0x7f07541e0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.305+0000 7f0759200640 1 -- 172.21.15.31:0/3667042932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075414cf80 con 0x7f07540c5b10 2024-09-11T00:57:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.305+0000 7f0759200640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07540c5b10 0x7f07541e06e0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f074800c450 tx=0x7f074800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.306+0000 7f0750c00640 1 -- 172.21.15.31:0/3667042932 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07480052d0 con 0x7f07540c5b10 2024-09-11T00:57:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.306+0000 7f0750c00640 1 -- 172.21.15.31:0/3667042932 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0748005470 con 0x7f07540c5b10 2024-09-11T00:57:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.306+0000 7f075bc00640 1 -- 172.21.15.31:0/3667042932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0754154830 con 0x7f07540c5b10 2024-09-11T00:57:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.306+0000 7f0750c00640 1 -- 172.21.15.31:0/3667042932 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748012870 con 0x7f07540c5b10 2024-09-11T00:57:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.306+0000 7f075bc00640 1 -- 172.21.15.31:0/3667042932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07541e1420 con 0x7f07540c5b10 2024-09-11T00:57:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.307+0000 7f075bc00640 1 -- 172.21.15.31:0/3667042932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0754159670 con 0x7f07540c5b10 2024-09-11T00:57:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.308+0000 7f0750c00640 1 -- 172.21.15.31:0/3667042932 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f07480035f0 con 0x7f07540c5b10 2024-09-11T00:57:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.309+0000 7f0750c00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0734077fa0 0x7f073407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.309+0000 7f0750c00640 1 -- 172.21.15.31:0/3667042932 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f074809a770 con 0x7f07540c5b10 2024-09-11T00:57:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.310+0000 7f0753e00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0734077fa0 0x7f073407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:22.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.310+0000 7f0753e00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0734077fa0 0x7f073407a460 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0740033000 tx=0x7f0740004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:22.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.313+0000 7f0750c00640 1 -- 172.21.15.31:0/3667042932 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f0748061c00 con 0x7f07540c5b10 2024-09-11T00:57:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.545+0000 7f075bc00640 1 -- 172.21.15.31:0/3667042932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07541e26d0 con 0x7f07540c5b10 2024-09-11T00:57:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.545+0000 7f0750c00640 1 -- 172.21.15.31:0/3667042932 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+6545 (secure 0 0 0) 0x7f0748066630 con 0x7f07540c5b10 2024-09-11T00:57:22.547 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:22.547 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":186,"btime":"2024-09-11T00:56:41:092005+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:31.055051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7672,"qdb_cluster":[7672]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 186 2024-09-11T00:57:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.547+0000 7f075bc00640 1 -- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0734077fa0 msgr2=0x7f073407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.547+0000 7f075bc00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0734077fa0 0x7f073407a460 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0740033000 tx=0x7f0740004730 comp rx=0 tx=0).stop 2024-09-11T00:57:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.547+0000 7f075bc00640 1 -- 172.21.15.31:0/3667042932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07540c5b10 msgr2=0x7f07541e06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.547+0000 7f075bc00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07540c5b10 0x7f07541e06e0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f074800c450 tx=0x7f074800c910 comp rx=0 tx=0).stop 2024-09-11T00:57:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.547+0000 7f075bc00640 1 -- 172.21.15.31:0/3667042932 shutdown_connections 2024-09-11T00:57:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.547+0000 7f075bc00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f0734077fa0 0x7f073407a460 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.547+0000 7f075bc00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f0754156ad0 0x7f07541557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.547+0000 7f075bc00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07541539c0 0x7f07541e0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.547+0000 7f075bc00640 1 --2- 172.21.15.31:0/3667042932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f07540c5b10 0x7f07541e06e0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.547+0000 7f075bc00640 1 -- 172.21.15.31:0/3667042932 >> 172.21.15.31:0/3667042932 conn(0x7f07540a9750 msgr2=0x7f0754150430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.548+0000 7f075bc00640 1 -- 172.21.15.31:0/3667042932 shutdown_connections 2024-09-11T00:57:22.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:22.548+0000 7f075bc00640 1 -- 172.21.15.31:0/3667042932 wait complete. 2024-09-11T00:57:22.566 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:57:24.567 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.736+0000 7f3584a00640 1 -- 172.21.15.31:0/1420905199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3580151600 msgr2=0x7f35800c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.736+0000 7f3584a00640 1 --2- 172.21.15.31:0/1420905199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3580151600 0x7f35800c61b0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3578005fd0 tx=0x7f357802a780 comp rx=0 tx=0).stop 2024-09-11T00:57:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.736+0000 7f3584a00640 1 -- 172.21.15.31:0/1420905199 shutdown_connections 2024-09-11T00:57:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.736+0000 7f3584a00640 1 --2- 172.21.15.31:0/1420905199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3580151600 0x7f35800c61b0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.736+0000 7f3584a00640 1 --2- 172.21.15.31:0/1420905199 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35800a6e20 0x7f35801510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.736+0000 7f3584a00640 1 -- 172.21.15.31:0/1420905199 >> 172.21.15.31:0/1420905199 conn(0x7f358006caf0 msgr2=0x7f358006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.736+0000 7f3584a00640 1 -- 172.21.15.31:0/1420905199 shutdown_connections 2024-09-11T00:57:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.736+0000 7f3584a00640 1 -- 172.21.15.31:0/1420905199 wait complete. 2024-09-11T00:57:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.737+0000 7f3584a00640 1 Processor -- start 2024-09-11T00:57:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.738+0000 7f3584a00640 1 -- start start 2024-09-11T00:57:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.738+0000 7f3584a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35800a6e20 0x7f3580155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.738+0000 7f3584a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35800ac2c0 0x7f35801ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.738+0000 7f3584a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3580151600 0x7f3580147100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.738+0000 7f3584a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f35800acf30 con 0x7f3580151600 2024-09-11T00:57:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.738+0000 7f3584a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f35800acdb0 con 0x7f35800a6e20 2024-09-11T00:57:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.738+0000 7f3584a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f35800c66f0 con 0x7f35800ac2c0 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f357f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3580151600 0x7f3580147100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f357f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3580151600 0x7f3580147100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53412/0 (socket says 172.21.15.31:53412) 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f357f200640 1 -- 172.21.15.31:0/1881461141 learned_addr learned my addr 172.21.15.31:0/1881461141 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f3575e00640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35800ac2c0 0x7f35801ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f357e800640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35800a6e20 0x7f3580155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f357f200640 1 -- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35800ac2c0 msgr2=0x7f35801ef0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f357f200640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35800ac2c0 0x7f35801ef0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f357f200640 1 -- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35800a6e20 msgr2=0x7f3580155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f357f200640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35800a6e20 0x7f3580155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f357f200640 1 -- 172.21.15.31:0/1881461141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35800725b0 con 0x7f3580151600 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.739+0000 7f3575e00640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35800ac2c0 0x7f35801ef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:24.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.740+0000 7f357f200640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3580151600 0x7f3580147100 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f3578034040 tx=0x7f3578003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:24.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.740+0000 7f3577400640 1 -- 172.21.15.31:0/1881461141 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3578040070 con 0x7f3580151600 2024-09-11T00:57:24.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.740+0000 7f3577400640 1 -- 172.21.15.31:0/1881461141 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3578004340 con 0x7f3580151600 2024-09-11T00:57:24.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.740+0000 7f3577400640 1 -- 172.21.15.31:0/1881461141 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3578004750 con 0x7f3580151600 2024-09-11T00:57:24.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.740+0000 7f3584a00640 1 -- 172.21.15.31:0/1881461141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35801efa20 con 0x7f3580151600 2024-09-11T00:57:24.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.740+0000 7f3584a00640 1 -- 172.21.15.31:0/1881461141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35800aa9b0 con 0x7f3580151600 2024-09-11T00:57:24.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.741+0000 7f3584a00640 1 -- 172.21.15.31:0/1881461141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35801510c0 con 0x7f3580151600 2024-09-11T00:57:24.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.742+0000 7f3577400640 1 -- 172.21.15.31:0/1881461141 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f357803c420 con 0x7f3580151600 2024-09-11T00:57:24.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.744+0000 7f3577400640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f355c077f70 0x7f355c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:24.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.744+0000 7f3577400640 1 -- 172.21.15.31:0/1881461141 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f35780bc1a0 con 0x7f3580151600 2024-09-11T00:57:24.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.744+0000 7f357e800640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f355c077f70 0x7f355c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:24.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.745+0000 7f357e800640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f355c077f70 0x7f355c07a430 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f35700016b0 tx=0x7f3570009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:24.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.748+0000 7f3577400640 1 -- 172.21.15.31:0/1881461141 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3578083d00 con 0x7f3580151600 2024-09-11T00:57:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.979+0000 7f3584a00640 1 -- 172.21.15.31:0/1881461141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3580150940 con 0x7f3580151600 2024-09-11T00:57:24.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.980+0000 7f3577400640 1 -- 172.21.15.31:0/1881461141 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+6545 (secure 0 0 0) 0x7f3578083600 con 0x7f3580151600 2024-09-11T00:57:24.983 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:24.983 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":186,"btime":"2024-09-11T00:56:41:092005+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:31.055051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7672,"qdb_cluster":[7672]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:24.983 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 186 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 -- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f355c077f70 msgr2=0x7f355c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f355c077f70 0x7f355c07a430 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f35700016b0 tx=0x7f3570009040 comp rx=0 tx=0).stop 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 -- 172.21.15.31:0/1881461141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3580151600 msgr2=0x7f3580147100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3580151600 0x7f3580147100 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f3578034040 tx=0x7f3578003c10 comp rx=0 tx=0).stop 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 -- 172.21.15.31:0/1881461141 shutdown_connections 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f355c077f70 0x7f355c07a430 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3580151600 0x7f3580147100 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f35800ac2c0 0x7f35801ef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 --2- 172.21.15.31:0/1881461141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35800a6e20 0x7f3580155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 -- 172.21.15.31:0/1881461141 >> 172.21.15.31:0/1881461141 conn(0x7f358006caf0 msgr2=0x7f358014ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 -- 172.21.15.31:0/1881461141 shutdown_connections 2024-09-11T00:57:24.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:24.983+0000 7f3584a00640 1 -- 172.21.15.31:0/1881461141 wait complete. 2024-09-11T00:57:25.002 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:57:27.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:27.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.175+0000 7fa5a9600640 1 -- 172.21.15.31:0/3054921067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5a41518b0 msgr2=0x7fa5a40c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:27.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.175+0000 7fa5a9600640 1 --2- 172.21.15.31:0/3054921067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5a41518b0 0x7fa5a40c3600 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009ca0 tx=0x7fa59c02ab50 comp rx=0 tx=0).stop 2024-09-11T00:57:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.175+0000 7fa5a9600640 1 -- 172.21.15.31:0/3054921067 shutdown_connections 2024-09-11T00:57:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.175+0000 7fa5a9600640 1 --2- 172.21.15.31:0/3054921067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5a41518b0 0x7fa5a40c3600 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.175+0000 7fa5a9600640 1 --2- 172.21.15.31:0/3054921067 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5a40c23a0 0x7fa5a40c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.175+0000 7fa5a9600640 1 -- 172.21.15.31:0/3054921067 >> 172.21.15.31:0/3054921067 conn(0x7fa5a40a9750 msgr2=0x7fa5a40a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.175+0000 7fa5a9600640 1 -- 172.21.15.31:0/3054921067 shutdown_connections 2024-09-11T00:57:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.175+0000 7fa5a9600640 1 -- 172.21.15.31:0/3054921067 wait complete. 2024-09-11T00:57:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.177+0000 7fa5a9600640 1 Processor -- start 2024-09-11T00:57:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.177+0000 7fa5a9600640 1 -- start start 2024-09-11T00:57:27.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa5a9600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5a4072480 0x7fa5a41530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:27.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa5a9600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a40c23a0 0x7fa5a406f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:27.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa5a9600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5a41518b0 0x7fa5a414bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa5a9600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a4072860 con 0x7fa5a41518b0 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa5a9600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a4074290 con 0x7fa5a40c23a0 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa5a9600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fa5a40729e0 con 0x7fa5a4072480 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa5a3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5a41518b0 0x7fa5a414bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa5a3400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5a41518b0 0x7fa5a414bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53434/0 (socket says 172.21.15.31:53434) 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa5a3400640 1 -- 172.21.15.31:0/45622289 learned_addr learned my addr 172.21.15.31:0/45622289 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa59a000640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a40c23a0 0x7fa5a406f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.178+0000 7fa5a2a00640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5a4072480 0x7fa5a41530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa5a3400640 1 -- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5a4072480 msgr2=0x7fa5a41530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa5a3400640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5a4072480 0x7fa5a41530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa5a3400640 1 -- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a40c23a0 msgr2=0x7fa5a406f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:27.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa5a3400640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a40c23a0 0x7fa5a406f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:27.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa5a3400640 1 -- 172.21.15.31:0/45622289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a414f1a0 con 0x7fa5a41518b0 2024-09-11T00:57:27.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa5a2a00640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5a4072480 0x7fa5a41530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:27.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa5a3400640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5a41518b0 0x7fa5a414bcc0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009ca0 tx=0x7fa59c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa59b400640 1 -- 172.21.15.31:0/45622289 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c044070 con 0x7fa5a41518b0 2024-09-11T00:57:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa59b400640 1 -- 172.21.15.31:0/45622289 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa59c004040 con 0x7fa5a41518b0 2024-09-11T00:57:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa5a9600640 1 -- 172.21.15.31:0/45622289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a40c7070 con 0x7fa5a41518b0 2024-09-11T00:57:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa59b400640 1 -- 172.21.15.31:0/45622289 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59c004430 con 0x7fa5a41518b0 2024-09-11T00:57:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.179+0000 7fa5a9600640 1 -- 172.21.15.31:0/45622289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a406f3b0 con 0x7fa5a41518b0 2024-09-11T00:57:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.180+0000 7fa5a9600640 1 -- 172.21.15.31:0/45622289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a40c7590 con 0x7fa5a41518b0 2024-09-11T00:57:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.182+0000 7fa59b400640 1 -- 172.21.15.31:0/45622289 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fa59c0045d0 con 0x7fa5a41518b0 2024-09-11T00:57:27.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.183+0000 7fa59b400640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa580077fa0 0x7fa58007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:27.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.183+0000 7fa59b400640 1 -- 172.21.15.31:0/45622289 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fa59c0bb200 con 0x7fa5a41518b0 2024-09-11T00:57:27.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.183+0000 7fa5a2a00640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa580077fa0 0x7fa58007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:27.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.184+0000 7fa5a2a00640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa580077fa0 0x7fa58007a460 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa594007fd0 tx=0x7fa594009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.187+0000 7fa59b400640 1 -- 172.21.15.31:0/45622289 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fa5a40c7590 con 0x7fa5a41518b0 2024-09-11T00:57:27.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.418+0000 7fa5a9600640 1 -- 172.21.15.31:0/45622289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5a40c3cb0 con 0x7fa5a41518b0 2024-09-11T00:57:27.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.419+0000 7fa59b400640 1 -- 172.21.15.31:0/45622289 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+6545 (secure 0 0 0) 0x7fa59c032030 con 0x7fa5a41518b0 2024-09-11T00:57:27.421 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:27.422 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":186,"btime":"2024-09-11T00:56:41:092005+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:31.055051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7672,"qdb_cluster":[7672]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:27.422 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 186 2024-09-11T00:57:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 -- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa580077fa0 msgr2=0x7fa58007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa580077fa0 0x7fa58007a460 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa594007fd0 tx=0x7fa594009040 comp rx=0 tx=0).stop 2024-09-11T00:57:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 -- 172.21.15.31:0/45622289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5a41518b0 msgr2=0x7fa5a414bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5a41518b0 0x7fa5a414bcc0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009ca0 tx=0x7fa59c003ce0 comp rx=0 tx=0).stop 2024-09-11T00:57:27.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 -- 172.21.15.31:0/45622289 shutdown_connections 2024-09-11T00:57:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fa580077fa0 0x7fa58007a460 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa5a41518b0 0x7fa5a414bcc0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a40c23a0 0x7fa5a406f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 --2- 172.21.15.31:0/45622289 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fa5a4072480 0x7fa5a41530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 -- 172.21.15.31:0/45622289 >> 172.21.15.31:0/45622289 conn(0x7fa5a40a9750 msgr2=0x7fa5a4150630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 -- 172.21.15.31:0/45622289 shutdown_connections 2024-09-11T00:57:27.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:27.421+0000 7fa5a9600640 1 -- 172.21.15.31:0/45622289 wait complete. 2024-09-11T00:57:27.440 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:57:29.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:29.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.610+0000 7f38a7800640 1 -- 172.21.15.31:0/495249686 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38a00c3f20 msgr2=0x7f38a00c2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:29.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.610+0000 7f38a7800640 1 --2- 172.21.15.31:0/495249686 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38a00c3f20 0x7f38a00c2ab0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f3898005fd0 tx=0x7f389802a8b0 comp rx=0 tx=0).stop 2024-09-11T00:57:29.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.610+0000 7f38a7800640 1 -- 172.21.15.31:0/495249686 shutdown_connections 2024-09-11T00:57:29.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.610+0000 7f38a7800640 1 --2- 172.21.15.31:0/495249686 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38a00c3f20 0x7f38a00c2ab0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:29.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.610+0000 7f38a7800640 1 -- 172.21.15.31:0/495249686 >> 172.21.15.31:0/495249686 conn(0x7f38a00a9750 msgr2=0x7f38a00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:29.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.610+0000 7f38a7800640 1 -- 172.21.15.31:0/495249686 shutdown_connections 2024-09-11T00:57:29.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.610+0000 7f38a7800640 1 -- 172.21.15.31:0/495249686 wait complete. 2024-09-11T00:57:29.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.611+0000 7f38a7800640 1 Processor -- start 2024-09-11T00:57:29.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.612+0000 7f38a7800640 1 -- start start 2024-09-11T00:57:29.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.612+0000 7f38a7800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38a006ff40 0x7f38a00c1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:29.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.612+0000 7f38a7800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f38a00c3f20 0x7f38a01e92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:29.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.612+0000 7f38a7800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a01532c0 0x7f38a01ec090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:29.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.612+0000 7f38a7800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38a00c7650 con 0x7f38a01532c0 2024-09-11T00:57:29.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.612+0000 7f38a7800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f38a006f340 con 0x7f38a006ff40 2024-09-11T00:57:29.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.612+0000 7f38a7800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f38a00c77d0 con 0x7f38a00c3f20 2024-09-11T00:57:29.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f38a5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a01532c0 0x7f38a01ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:29.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f38a5800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a01532c0 0x7f38a01ec090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42052/0 (socket says 172.21.15.31:42052) 2024-09-11T00:57:29.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f38a5800640 1 -- 172.21.15.31:0/4248892018 learned_addr learned my addr 172.21.15.31:0/4248892018 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:29.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f38a4e00640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38a006ff40 0x7f38a00c1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:29.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f3897e00640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f38a00c3f20 0x7f38a01e92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:29.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f38a5800640 1 -- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f38a00c3f20 msgr2=0x7f38a01e92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:29.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f38a5800640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f38a00c3f20 0x7f38a01e92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f38a5800640 1 -- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38a006ff40 msgr2=0x7f38a00c1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f38a5800640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38a006ff40 0x7f38a00c1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f38a5800640 1 -- 172.21.15.31:0/4248892018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a0155760 con 0x7f38a01532c0 2024-09-11T00:57:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.613+0000 7f38a4e00640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38a006ff40 0x7f38a00c1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:57:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.614+0000 7f38a5800640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a01532c0 0x7f38a01ec090 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f3898009cd0 tx=0x7f38980316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.614+0000 7f389d600640 1 -- 172.21.15.31:0/4248892018 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3898040070 con 0x7f38a01532c0 2024-09-11T00:57:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.614+0000 7f389d600640 1 -- 172.21.15.31:0/4248892018 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3898004150 con 0x7f38a01532c0 2024-09-11T00:57:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.614+0000 7f389d600640 1 -- 172.21.15.31:0/4248892018 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38980045a0 con 0x7f38a01532c0 2024-09-11T00:57:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.614+0000 7f38a7800640 1 -- 172.21.15.31:0/4248892018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a00ac240 con 0x7f38a01532c0 2024-09-11T00:57:29.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.614+0000 7f38a7800640 1 -- 172.21.15.31:0/4248892018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a00ac800 con 0x7f38a01532c0 2024-09-11T00:57:29.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.615+0000 7f38a7800640 1 -- 172.21.15.31:0/4248892018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38a00a7350 con 0x7f38a01532c0 2024-09-11T00:57:29.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.616+0000 7f389d600640 1 -- 172.21.15.31:0/4248892018 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f389803c420 con 0x7f38a01532c0 2024-09-11T00:57:29.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.617+0000 7f389d600640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f387c077f70 0x7f387c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:29.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.617+0000 7f389d600640 1 -- 172.21.15.31:0/4248892018 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f38980bc4a0 con 0x7f38a01532c0 2024-09-11T00:57:29.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.618+0000 7f38a4e00640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f387c077f70 0x7f387c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:29.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.619+0000 7f38a4e00640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f387c077f70 0x7f387c07a430 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f38a0153f70 tx=0x7f3890009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:29.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.622+0000 7f389d600640 1 -- 172.21.15.31:0/4248892018 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f38980839b0 con 0x7f38a01532c0 2024-09-11T00:57:29.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.856+0000 7f38a7800640 1 -- 172.21.15.31:0/4248892018 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38a0051400 con 0x7f38a01532c0 2024-09-11T00:57:29.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.856+0000 7f389d600640 1 -- 172.21.15.31:0/4248892018 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+6545 (secure 0 0 0) 0x7f38980883e0 con 0x7f38a01532c0 2024-09-11T00:57:29.859 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:29.859 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":186,"btime":"2024-09-11T00:56:41:092005+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:31.055051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7672,"qdb_cluster":[7672]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:29.860 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 186 2024-09-11T00:57:29.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 -- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f387c077f70 msgr2=0x7f387c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f387c077f70 0x7f387c07a430 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f38a0153f70 tx=0x7f3890009040 comp rx=0 tx=0).stop 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 -- 172.21.15.31:0/4248892018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a01532c0 msgr2=0x7f38a01ec090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a01532c0 0x7f38a01ec090 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f3898009cd0 tx=0x7f38980316f0 comp rx=0 tx=0).stop 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 -- 172.21.15.31:0/4248892018 shutdown_connections 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f387c077f70 0x7f387c07a430 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38a01532c0 0x7f38a01ec090 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f38a00c3f20 0x7f38a01e92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 --2- 172.21.15.31:0/4248892018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f38a006ff40 0x7f38a00c1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 -- 172.21.15.31:0/4248892018 >> 172.21.15.31:0/4248892018 conn(0x7f38a00a9750 msgr2=0x7f38a0073840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 -- 172.21.15.31:0/4248892018 shutdown_connections 2024-09-11T00:57:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:29.859+0000 7f38a7800640 1 -- 172.21.15.31:0/4248892018 wait complete. 2024-09-11T00:57:29.878 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:57:31.815 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:57:31.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:57:31.879 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.042+0000 7f4abb000640 1 -- 172.21.15.31:0/3399601399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ab4156350 msgr2=0x7f4ab4156800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.042+0000 7f4abb000640 1 --2- 172.21.15.31:0/3399601399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ab4156350 0x7f4ab4156800 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4aac005fd0 tx=0x7f4aac02a8a0 comp rx=0 tx=0).stop 2024-09-11T00:57:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.042+0000 7f4abb000640 1 -- 172.21.15.31:0/3399601399 shutdown_connections 2024-09-11T00:57:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.042+0000 7f4abb000640 1 --2- 172.21.15.31:0/3399601399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ab4156350 0x7f4ab4156800 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.042+0000 7f4abb000640 1 --2- 172.21.15.31:0/3399601399 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ab40c5aa0 0x7f4ab40c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.042+0000 7f4abb000640 1 -- 172.21.15.31:0/3399601399 >> 172.21.15.31:0/3399601399 conn(0x7f4ab406caf0 msgr2=0x7f4ab406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.042+0000 7f4abb000640 1 -- 172.21.15.31:0/3399601399 shutdown_connections 2024-09-11T00:57:32.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.042+0000 7f4abb000640 1 -- 172.21.15.31:0/3399601399 wait complete. 2024-09-11T00:57:32.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.043+0000 7f4abb000640 1 Processor -- start 2024-09-11T00:57:32.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.044+0000 7f4abb000640 1 -- start start 2024-09-11T00:57:32.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.044+0000 7f4abb000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ab40c5aa0 0x7f4ab41e0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:32.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.044+0000 7f4abb000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40c6420 0x7f4ab414fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:32.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.044+0000 7f4abb000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ab4156350 0x7f4ab414cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:32.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.044+0000 7f4abb000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab40c2e80 con 0x7f4ab40c6420 2024-09-11T00:57:32.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.044+0000 7f4abb000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab40c2ba0 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.044+0000 7f4abb000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f4ab4156d40 con 0x7f4ab4156350 2024-09-11T00:57:32.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.045+0000 7f4ab9c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ab40c5aa0 0x7f4ab41e0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:32.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.045+0000 7f4ab9200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40c6420 0x7f4ab414fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:32.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.045+0000 7f4ab9c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ab40c5aa0 0x7f4ab41e0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.31:48614/0 (socket says 172.21.15.31:48614) 2024-09-11T00:57:32.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.045+0000 7f4ab9c00640 1 -- 172.21.15.31:0/4112152168 learned_addr learned my addr 172.21.15.31:0/4112152168 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:32.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.045+0000 7f4aba600640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ab4156350 0x7f4ab414cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:32.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.045+0000 7f4ab9c00640 1 -- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ab4156350 msgr2=0x7f4ab414cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:32.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.045+0000 7f4ab9c00640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ab4156350 0x7f4ab414cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:32.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.045+0000 7f4ab9c00640 1 -- 172.21.15.31:0/4112152168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40c6420 msgr2=0x7f4ab414fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:32.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.045+0000 7f4ab9c00640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40c6420 0x7f4ab414fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:32.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.045+0000 7f4ab9c00640 1 -- 172.21.15.31:0/4112152168 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab41e1e10 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.046+0000 7f4ab9c00640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ab40c5aa0 0x7f4ab41e0990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800e9c0 tx=0x7f4aa800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:32.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.046+0000 7f4ab1600640 1 -- 172.21.15.31:0/4112152168 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa800ccc0 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.046+0000 7f4ab1600640 1 -- 172.21.15.31:0/4112152168 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4aa8005020 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.046+0000 7f4ab1600640 1 -- 172.21.15.31:0/4112152168 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa800ce60 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.046+0000 7f4abb000640 1 -- 172.21.15.31:0/4112152168 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab41df820 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.046+0000 7f4abb000640 1 -- 172.21.15.31:0/4112152168 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab40aa760 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.047+0000 7f4abb000640 1 -- 172.21.15.31:0/4112152168 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab40c5e80 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.048+0000 7f4ab1600640 1 -- 172.21.15.31:0/4112152168 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4aa8005450 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.049+0000 7f4ab1600640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a94077f70 0x7f4a9407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:32.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.050+0000 7f4ab1600640 1 -- 172.21.15.31:0/4112152168 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7f4aa809b7c0 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.050+0000 7f4ab9200640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a94077f70 0x7f4a9407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:32.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.050+0000 7f4ab9200640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a94077f70 0x7f4a9407a430 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4ab40ac830 tx=0x7f4aa0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:32.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.053+0000 7f4ab1600640 1 -- 172.21.15.31:0/4112152168 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f4aa8062d50 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.283+0000 7f4abb000640 1 -- 172.21.15.31:0/4112152168 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ab41dfb90 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.284+0000 7f4ab1600640 1 -- 172.21.15.31:0/4112152168 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+6545 (secure 0 0 0) 0x7f4aa8067780 con 0x7f4ab40c5aa0 2024-09-11T00:57:32.286 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:32.287 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":186,"btime":"2024-09-11T00:56:41:092005+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":7824,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:56:31.055051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7672":{"gid":7672,"name":"d","rank":0,"incarnation":182,"state":"up:active","state_seq":4,"addr":"172.21.15.204:6837/1329541959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":1329541959},{"type":"v1","addr":"172.21.15.204:6837","nonce":1329541959}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7672,"qdb_cluster":[7672]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:32.287 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 186 2024-09-11T00:57:32.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.286+0000 7f4abb000640 1 -- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a94077f70 msgr2=0x7f4a9407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:32.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.286+0000 7f4abb000640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a94077f70 0x7f4a9407a430 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4ab40ac830 tx=0x7f4aa0009040 comp rx=0 tx=0).stop 2024-09-11T00:57:32.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.286+0000 7f4abb000640 1 -- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ab40c5aa0 msgr2=0x7f4ab41e0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:32.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.286+0000 7f4abb000640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ab40c5aa0 0x7f4ab41e0990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800e9c0 tx=0x7f4aa800ee80 comp rx=0 tx=0).stop 2024-09-11T00:57:32.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.286+0000 7f4abb000640 1 -- 172.21.15.31:0/4112152168 shutdown_connections 2024-09-11T00:57:32.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.286+0000 7f4abb000640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f4a94077f70 0x7f4a9407a430 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:32.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.286+0000 7f4abb000640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f4ab4156350 0x7f4ab414cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:32.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.286+0000 7f4abb000640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ab40c6420 0x7f4ab414fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:32.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.286+0000 7f4abb000640 1 --2- 172.21.15.31:0/4112152168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4ab40c5aa0 0x7f4ab41e0990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:32.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.286+0000 7f4abb000640 1 -- 172.21.15.31:0/4112152168 >> 172.21.15.31:0/4112152168 conn(0x7f4ab406caf0 msgr2=0x7f4ab40a7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:32.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.287+0000 7f4abb000640 1 -- 172.21.15.31:0/4112152168 shutdown_connections 2024-09-11T00:57:32.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:32.287+0000 7f4abb000640 1 -- 172.21.15.31:0/4112152168 wait complete. 2024-09-11T00:57:32.306 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-11T00:57:34.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:34.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.478+0000 7fde16a00640 1 -- 172.21.15.31:0/2521229352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde100c23a0 msgr2=0x7fde100c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:34.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.478+0000 7fde16a00640 1 --2- 172.21.15.31:0/2521229352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde100c23a0 0x7fde100c30c0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fddfc005fd0 tx=0x7fddfc02a5c0 comp rx=0 tx=0).stop 2024-09-11T00:57:34.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.478+0000 7fde16a00640 1 -- 172.21.15.31:0/2521229352 shutdown_connections 2024-09-11T00:57:34.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.478+0000 7fde16a00640 1 --2- 172.21.15.31:0/2521229352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde101518b0 0x7fde100c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:34.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.478+0000 7fde16a00640 1 --2- 172.21.15.31:0/2521229352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde100c23a0 0x7fde100c30c0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:34.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.478+0000 7fde16a00640 1 -- 172.21.15.31:0/2521229352 >> 172.21.15.31:0/2521229352 conn(0x7fde100a9750 msgr2=0x7fde100a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:34.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.478+0000 7fde16a00640 1 -- 172.21.15.31:0/2521229352 shutdown_connections 2024-09-11T00:57:34.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.478+0000 7fde16a00640 1 -- 172.21.15.31:0/2521229352 wait complete. 2024-09-11T00:57:34.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.480+0000 7fde16a00640 1 Processor -- start 2024-09-11T00:57:34.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.480+0000 7fde16a00640 1 -- start start 2024-09-11T00:57:34.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde16a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde10072480 0x7fde100c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:34.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde16a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde100c23a0 0x7fde101ef790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:34.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde16a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde101518b0 0x7fde101e0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:34.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde16a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fde10074200 con 0x7fde100c23a0 2024-09-11T00:57:34.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde16a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fde10074080 con 0x7fde10072480 2024-09-11T00:57:34.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde16a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fde10072860 con 0x7fde101518b0 2024-09-11T00:57:34.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde0f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde100c23a0 0x7fde101ef790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:34.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde0f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde100c23a0 0x7fde101ef790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42078/0 (socket says 172.21.15.31:42078) 2024-09-11T00:57:34.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde0f400640 1 -- 172.21.15.31:0/3390272976 learned_addr learned my addr 172.21.15.31:0/3390272976 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:34.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde0fe00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde10072480 0x7fde100c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:34.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.481+0000 7fde14a00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde101518b0 0x7fde101e0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.482+0000 7fde0f400640 1 -- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde101518b0 msgr2=0x7fde101e0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.482+0000 7fde0f400640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde101518b0 0x7fde101e0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.482+0000 7fde0f400640 1 -- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde10072480 msgr2=0x7fde100c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.482+0000 7fde0f400640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde10072480 0x7fde100c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.482+0000 7fde0f400640 1 -- 172.21.15.31:0/3390272976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde100c77c0 con 0x7fde100c23a0 2024-09-11T00:57:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.482+0000 7fde0f400640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde100c23a0 0x7fde101ef790 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009850 tx=0x7fddfc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.482+0000 7fde07e00640 1 -- 172.21.15.31:0/3390272976 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc03f070 con 0x7fde100c23a0 2024-09-11T00:57:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.482+0000 7fde07e00640 1 -- 172.21.15.31:0/3390272976 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fddfc0047a0 con 0x7fde100c23a0 2024-09-11T00:57:34.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.482+0000 7fde07e00640 1 -- 172.21.15.31:0/3390272976 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc035030 con 0x7fde100c23a0 2024-09-11T00:57:34.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.482+0000 7fde16a00640 1 -- 172.21.15.31:0/3390272976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde101efcd0 con 0x7fde100c23a0 2024-09-11T00:57:34.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.483+0000 7fde16a00640 1 -- 172.21.15.31:0/3390272976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde10153870 con 0x7fde100c23a0 2024-09-11T00:57:34.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.484+0000 7fde16a00640 1 -- 172.21.15.31:0/3390272976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde10077240 con 0x7fde100c23a0 2024-09-11T00:57:34.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.485+0000 7fde07e00640 1 -- 172.21.15.31:0/3390272976 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fddfc004b90 con 0x7fde100c23a0 2024-09-11T00:57:34.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.486+0000 7fde07e00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fddf0077fa0 0x7fddf007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:34.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.486+0000 7fde07e00640 1 -- 172.21.15.31:0/3390272976 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9676+0+0 (secure 0 0 0) 0x7fddfc0bbdd0 con 0x7fde100c23a0 2024-09-11T00:57:34.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.486+0000 7fde0fe00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fddf0077fa0 0x7fddf007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:34.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.487+0000 7fde0fe00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fddf0077fa0 0x7fddf007a460 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fde00004f60 tx=0x7fde00007e30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:34.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.490+0000 7fde07e00640 1 -- 172.21.15.31:0/3390272976 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fde10077240 con 0x7fde100c23a0 2024-09-11T00:57:34.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.814+0000 7fde16a00640 1 -- 172.21.15.31:0/3390272976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde10072d90 con 0x7fde100c23a0 2024-09-11T00:57:34.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.815+0000 7fde07e00640 1 -- 172.21.15.31:0/3390272976 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+5709 (secure 0 0 0) 0x7fde10072d90 con 0x7fde100c23a0 2024-09-11T00:57:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:34.818 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":187,"btime":"2024-09-11T00:57:34:594243+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:57:34.594239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7824},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7824":{"gid":7824,"name":"a","rank":0,"incarnation":187,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:34.818 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 187 2024-09-11T00:57:34.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 -- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fddf0077fa0 msgr2=0x7fddf007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:34.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fddf0077fa0 0x7fddf007a460 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fde00004f60 tx=0x7fde00007e30 comp rx=0 tx=0).stop 2024-09-11T00:57:34.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 -- 172.21.15.31:0/3390272976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde100c23a0 msgr2=0x7fde101ef790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:34.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde100c23a0 0x7fde101ef790 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009850 tx=0x7fddfc0038a0 comp rx=0 tx=0).stop 2024-09-11T00:57:34.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 -- 172.21.15.31:0/3390272976 shutdown_connections 2024-09-11T00:57:34.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fddf0077fa0 0x7fddf007a460 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:34.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fde101518b0 0x7fde101e0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:34.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde100c23a0 0x7fde101ef790 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:34.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 --2- 172.21.15.31:0/3390272976 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde10072480 0x7fde100c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:34.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 -- 172.21.15.31:0/3390272976 >> 172.21.15.31:0/3390272976 conn(0x7fde100a9750 msgr2=0x7fde100c30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:34.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 -- 172.21.15.31:0/3390272976 shutdown_connections 2024-09-11T00:57:34.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.817+0000 7fde16a00640 1 -- 172.21.15.31:0/3390272976 wait complete. 2024-09-11T00:57:34.838 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-11T00:57:34.838 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-11T00:57:34.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.999+0000 7f246f200640 1 -- 172.21.15.31:0/697180777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f244c003aa0 msgr2=0x7f244c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.999+0000 7f246f200640 1 --2- 172.21.15.31:0/697180777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f244c003aa0 0x7f244c003ec0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f2458006f90 tx=0x7f244c005830 comp rx=0 tx=0).stop 2024-09-11T00:57:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.999+0000 7f246d200640 1 -- 172.21.15.31:0/697180777 reap_dead start 2024-09-11T00:57:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.999+0000 7f246f200640 1 -- 172.21.15.31:0/697180777 shutdown_connections 2024-09-11T00:57:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.999+0000 7f246f200640 1 -- 172.21.15.31:0/697180777 >> 172.21.15.31:0/697180777 conn(0x7f24680a9750 msgr2=0x7f24680a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.999+0000 7f246f200640 1 -- 172.21.15.31:0/697180777 shutdown_connections 2024-09-11T00:57:35.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:34.999+0000 7f246f200640 1 -- 172.21.15.31:0/697180777 wait complete. 2024-09-11T00:57:35.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.002+0000 7f246f200640 1 Processor -- start 2024-09-11T00:57:35.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.002+0000 7f246f200640 1 -- start start 2024-09-11T00:57:35.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.003+0000 7f246f200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f244c003aa0 0x7f24680a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:35.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.003+0000 7f246f200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f246814caf0 0x7f24680a8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.003+0000 7f246f200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24681ef370 0x7f24681ef7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.003+0000 7f246f200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2468156f20 con 0x7f246814caf0 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.003+0000 7f246f200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f24680c7320 con 0x7f24681ef370 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.003+0000 7f246f200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f24680c3950 con 0x7f244c003aa0 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f245f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f246814caf0 0x7f24680a8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f245f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f246814caf0 0x7f24680a8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42108/0 (socket says 172.21.15.31:42108) 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f245f400640 1 -- 172.21.15.31:0/1740319510 learned_addr learned my addr 172.21.15.31:0/1740319510 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f2467e00640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f244c003aa0 0x7f24680a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f246d200640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24681ef370 0x7f24681ef7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f245f400640 1 -- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f244c003aa0 msgr2=0x7f24680a8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f245f400640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f244c003aa0 0x7f24680a8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:35.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f245f400640 1 -- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24681ef370 msgr2=0x7f24681ef7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f245f400640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24681ef370 0x7f24681ef7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f245f400640 1 -- 172.21.15.31:0/1740319510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2468155150 con 0x7f246814caf0 2024-09-11T00:57:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f2467e00640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f244c003aa0 0x7f24680a8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f246d200640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24681ef370 0x7f24681ef7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.004+0000 7f245f400640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f246814caf0 0x7f24680a8370 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f24500119e0 tx=0x7f2450011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.005+0000 7f2464c00640 1 -- 172.21.15.31:0/1740319510 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245001f070 con 0x7f246814caf0 2024-09-11T00:57:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.005+0000 7f2464c00640 1 -- 172.21.15.31:0/1740319510 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2450012450 con 0x7f246814caf0 2024-09-11T00:57:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.005+0000 7f246f200640 1 -- 172.21.15.31:0/1740319510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2468155430 con 0x7f246814caf0 2024-09-11T00:57:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.005+0000 7f2464c00640 1 -- 172.21.15.31:0/1740319510 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2450012800 con 0x7f246814caf0 2024-09-11T00:57:35.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.005+0000 7f246f200640 1 -- 172.21.15.31:0/1740319510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24680a88b0 con 0x7f246814caf0 2024-09-11T00:57:35.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.006+0000 7f246f200640 1 -- 172.21.15.31:0/1740319510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2468077240 con 0x7f246814caf0 2024-09-11T00:57:35.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.007+0000 7f2464c00640 1 -- 172.21.15.31:0/1740319510 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f24500129a0 con 0x7f246814caf0 2024-09-11T00:57:35.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.009+0000 7f2464c00640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2444077f70 0x7f244407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:35.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.009+0000 7f2464c00640 1 -- 172.21.15.31:0/1740319510 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9762+0+0 (secure 0 0 0) 0x7f245009bc00 con 0x7f246814caf0 2024-09-11T00:57:35.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.009+0000 7f2467e00640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2444077f70 0x7f244407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:35.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.010+0000 7f2467e00640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2444077f70 0x7f244407a430 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2458033000 tx=0x7f2468071e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:35.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.012+0000 7f2464c00640 1 -- 172.21.15.31:0/1740319510 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2468077240 con 0x7f246814caf0 2024-09-11T00:57:35.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.263+0000 7f246f200640 1 -- 172.21.15.31:0/1740319510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2468153480 con 0x7f246814caf0 2024-09-11T00:57:35.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.264+0000 7f2464c00640 1 -- 172.21.15.31:0/1740319510 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+5709 (secure 0 0 0) 0x7f2450063020 con 0x7f246814caf0 2024-09-11T00:57:35.267 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:35.267 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":187,"btime":"2024-09-11T00:57:34:594243+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:57:34.594239+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7824},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7824":{"gid":7824,"name":"a","rank":0,"incarnation":187,"state":"up:replay","state_seq":1,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:35.267 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 187 2024-09-11T00:57:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 -- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2444077f70 msgr2=0x7f244407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2444077f70 0x7f244407a430 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2458033000 tx=0x7f2468071e90 comp rx=0 tx=0).stop 2024-09-11T00:57:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 -- 172.21.15.31:0/1740319510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f246814caf0 msgr2=0x7f24680a8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f246814caf0 0x7f24680a8370 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f24500119e0 tx=0x7f2450011ea0 comp rx=0 tx=0).stop 2024-09-11T00:57:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 -- 172.21.15.31:0/1740319510 shutdown_connections 2024-09-11T00:57:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2444077f70 0x7f244407a430 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24681ef370 0x7f24681ef7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f246814caf0 0x7f24680a8370 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:35.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 --2- 172.21.15.31:0/1740319510 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f244c003aa0 0x7f24680a8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:35.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 -- 172.21.15.31:0/1740319510 >> 172.21.15.31:0/1740319510 conn(0x7f24680a9750 msgr2=0x7f24681500c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:35.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 -- 172.21.15.31:0/1740319510 shutdown_connections 2024-09-11T00:57:35.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:35.266+0000 7f246f200640 1 -- 172.21.15.31:0/1740319510 wait complete. 2024-09-11T00:57:35.285 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-11T00:57:34:594243+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 187, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:45.342170+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 187, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7824": { "addr": "172.21.15.31:6833/1834111946", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6832", "nonce": 1834111946, "type": "v2" }, { "addr": "172.21.15.31:6833", "nonce": 1834111946, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7824, "incarnation": 187, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-11T00:57:34.594239+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7824 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-11T00:30:54.411975+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 180, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6593": { "addr": "172.21.15.204:6835/3620025884", "addrs": { "addrvec": [ { "addr": "172.21.15.204:6834", "nonce": 3620025884, "type": "v2" }, { "addr": "172.21.15.204:6835", "nonce": 3620025884, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6593, "incarnation": 177, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-11T00:56:17.960286+0000", "qdb_cluster": [ 6593 ], "qdb_leader": 6593, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6593 } } } ], "standbys": [ { "addr": "172.21.15.31:6835/2337200751", "addrs": { "addrvec": [ { "addr": "172.21.15.31:6834", "nonce": 2337200751, "type": "v2" }, { "addr": "172.21.15.31:6835", "nonce": 2337200751, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 166, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6503, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-11T00:57:36.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T00:57:36.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.190+0000 7fd822c00640 1 -- 172.21.15.31:0/491630691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd800004800 msgr2=0x7fd800004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:36.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.190+0000 7fd822c00640 1 --2- 172.21.15.31:0/491630691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd800004800 0x7fd800004c80 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fd804009b50 tx=0x7fd800005830 comp rx=0 tx=0).stop 2024-09-11T00:57:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.190+0000 7fd822c00640 1 -- 172.21.15.31:0/491630691 shutdown_connections 2024-09-11T00:57:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.190+0000 7fd822c00640 1 --2- 172.21.15.31:0/491630691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd800004800 0x7fd800004c80 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.190+0000 7fd822c00640 1 -- 172.21.15.31:0/491630691 >> 172.21.15.31:0/491630691 conn(0x7fd81c0a9750 msgr2=0x7fd81c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.190+0000 7fd822c00640 1 -- 172.21.15.31:0/491630691 shutdown_connections 2024-09-11T00:57:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.190+0000 7fd822c00640 1 -- 172.21.15.31:0/491630691 wait complete. 2024-09-11T00:57:36.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.192+0000 7fd822c00640 1 Processor -- start 2024-09-11T00:57:36.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.192+0000 7fd822c00640 1 -- start start 2024-09-11T00:57:36.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd822c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd800004800 0x7fd81c1ef810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd822c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd81c0a7900 0x7fd81c1ddc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd822c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd81c151d30 0x7fd81c142b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd822c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c0a7140 con 0x7fd81c151d30 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd822c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd81c0c23a0 con 0x7fd800004800 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd822c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd81c072860 con 0x7fd81c0a7900 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd820c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd81c151d30 0x7fd81c142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd820c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd81c151d30 0x7fd81c142b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42130/0 (socket says 172.21.15.31:42130) 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd820c00640 1 -- 172.21.15.31:0/2400819290 learned_addr learned my addr 172.21.15.31:0/2400819290 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd813400640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd81c0a7900 0x7fd81c1ddc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.193+0000 7fd81be00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd800004800 0x7fd81c1ef810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:36.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd820c00640 1 -- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd81c0a7900 msgr2=0x7fd81c1ddc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd820c00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd81c0a7900 0x7fd81c1ddc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd820c00640 1 -- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd800004800 msgr2=0x7fd81c1ef810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd820c00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd800004800 0x7fd81c1ef810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd820c00640 1 -- 172.21.15.31:0/2400819290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81c0a85a0 con 0x7fd81c151d30 2024-09-11T00:57:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd81be00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd800004800 0x7fd81c1ef810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:36.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd820c00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd81c151d30 0x7fd81c142b70 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fd814004fd0 tx=0x7fd8140019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd818c00640 1 -- 172.21.15.31:0/2400819290 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814011870 con 0x7fd81c151d30 2024-09-11T00:57:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd818c00640 1 -- 172.21.15.31:0/2400819290 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd814017070 con 0x7fd81c151d30 2024-09-11T00:57:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd818c00640 1 -- 172.21.15.31:0/2400819290 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814011dc0 con 0x7fd81c151d30 2024-09-11T00:57:36.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.194+0000 7fd822c00640 1 -- 172.21.15.31:0/2400819290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd81c0a7f60 con 0x7fd81c151d30 2024-09-11T00:57:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.195+0000 7fd822c00640 1 -- 172.21.15.31:0/2400819290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd81c1de150 con 0x7fd81c151d30 2024-09-11T00:57:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.196+0000 7fd822c00640 1 -- 172.21.15.31:0/2400819290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd81c0c2fa0 con 0x7fd81c151d30 2024-09-11T00:57:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.197+0000 7fd818c00640 1 -- 172.21.15.31:0/2400819290 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd81401c460 con 0x7fd81c151d30 2024-09-11T00:57:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.198+0000 7fd818c00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd7f8077f70 0x7fd7f807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:36.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.198+0000 7fd818c00640 1 -- 172.21.15.31:0/2400819290 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9762+0+0 (secure 0 0 0) 0x7fd81409b460 con 0x7fd81c151d30 2024-09-11T00:57:36.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.198+0000 7fd81be00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd7f8077f70 0x7fd7f807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:36.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.199+0000 7fd81be00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd7f8077f70 0x7fd7f807a430 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd80c001860 tx=0x7fd80c001020 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:36.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.202+0000 7fd818c00640 1 -- 172.21.15.31:0/2400819290 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd814062ed0 con 0x7fd81c151d30 2024-09-11T00:57:36.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.394+0000 7fd822c00640 1 -- 172.21.15.31:0/2400819290 --> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd81c0a6f20 con 0x7fd7f8077f70 2024-09-11T00:57:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.402+0000 7fd818c00640 1 -- 172.21.15.31:0/2400819290 <== mgr.4100 v2:172.21.15.204:6800/2097467593 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+495069 (secure 0 0 0) 0x7fd81c0a6f20 con 0x7fd7f8077f70 2024-09-11T00:57:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-09-11T00:57:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.408+0000 7fd822c00640 1 -- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd7f8077f70 msgr2=0x7fd7f807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.408+0000 7fd822c00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd7f8077f70 0x7fd7f807a430 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd80c001860 tx=0x7fd80c001020 comp rx=0 tx=0).stop 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.408+0000 7fd822c00640 1 -- 172.21.15.31:0/2400819290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd81c151d30 msgr2=0x7fd81c142b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.408+0000 7fd822c00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd81c151d30 0x7fd81c142b70 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fd814004fd0 tx=0x7fd8140019b0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.409+0000 7fd822c00640 1 -- 172.21.15.31:0/2400819290 shutdown_connections 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.409+0000 7fd822c00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd7f8077f70 0x7fd7f807a430 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.409+0000 7fd822c00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd81c151d30 0x7fd81c142b70 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.409+0000 7fd822c00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd81c0a7900 0x7fd81c1ddc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.409+0000 7fd822c00640 1 --2- 172.21.15.31:0/2400819290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd800004800 0x7fd81c1ef810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.409+0000 7fd822c00640 1 -- 172.21.15.31:0/2400819290 >> 172.21.15.31:0/2400819290 conn(0x7fd81c0a9750 msgr2=0x7fd81c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.409+0000 7fd822c00640 1 -- 172.21.15.31:0/2400819290 shutdown_connections 2024-09-11T00:57:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.409+0000 7fd822c00640 1 -- 172.21.15.31:0/2400819290 wait complete. 2024-09-11T00:57:36.427 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":867,"stamp":"2024-09-11T00:57:34.858947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36448998,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":2790,"num_read_kb":354813,"num_write":2142706,"num_write_kb":34795523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":418237,"ondisk_log_size":418237,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5380108,"kb_used_data":77448,"kb_used_omap":12,"kb_used_meta":5302515,"kb_avail":749594612,"statfs":{"total":773094113280,"available":767584882688,"internally_reserved":0,"allocated":79306752,"data_stored":76174524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":5429775921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":13045,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003242"},"pg_stats":[{"pgid":"5.18","version":"62'3612","reported_seq":3719,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.611525+0000","last_change":"2024-09-11T00:57:00.452987+0000","last_active":"2024-09-11T00:57:34.611525+0000","last_peered":"2024-09-11T00:57:34.611525+0000","last_clean":"2024-09-11T00:57:34.611525+0000","last_became_active":"2024-09-11T00:30:54.983140+0000","last_became_peered":"2024-09-11T00:30:54.983140+0000","last_unstale":"2024-09-11T00:57:34.611525+0000","last_undegraded":"2024-09-11T00:57:34.611525+0000","last_fullsized":"2024-09-11T00:57:34.611525+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3612","last_scrub_stamp":"2024-09-11T00:57:00.452887+0000","last_deep_scrub":"62'3612","last_deep_scrub_stamp":"2024-09-11T00:57:00.452887+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.452887+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:10:50.430948+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18811,"num_write_kb":284357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"62'791","reported_seq":939,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:07.828297+0000","last_change":"2024-09-11T00:57:07.828297+0000","last_active":"2024-09-11T00:57:07.828297+0000","last_peered":"2024-09-11T00:57:07.828297+0000","last_clean":"2024-09-11T00:57:07.828297+0000","last_became_active":"2024-09-11T00:30:44.922264+0000","last_became_peered":"2024-09-11T00:30:44.922264+0000","last_unstale":"2024-09-11T00:57:07.828297+0000","last_undegraded":"2024-09-11T00:57:07.828297+0000","last_fullsized":"2024-09-11T00:57:07.828297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'791","last_scrub_stamp":"2024-09-11T00:57:07.828194+0000","last_deep_scrub":"62'791","last_deep_scrub_stamp":"2024-09-11T00:57:07.828194+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.828194+0000","objects_scrubbed":3,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:41.823006+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":8319,"num_write":773,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"60'3033","reported_seq":3149,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607678+0000","last_change":"2024-09-11T00:57:01.078438+0000","last_active":"2024-09-11T00:57:34.607678+0000","last_peered":"2024-09-11T00:57:34.607678+0000","last_clean":"2024-09-11T00:57:34.607678+0000","last_became_active":"2024-09-11T00:30:45.938325+0000","last_became_peered":"2024-09-11T00:30:45.938325+0000","last_unstale":"2024-09-11T00:57:34.607678+0000","last_undegraded":"2024-09-11T00:57:34.607678+0000","last_fullsized":"2024-09-11T00:57:34.607678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3033","last_scrub_stamp":"2024-09-11T00:57:01.078328+0000","last_deep_scrub":"60'3033","last_deep_scrub_stamp":"2024-09-11T00:57:01.078328+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.078328+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:42:37.346125+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16165,"num_write_kb":269490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"61'3","reported_seq":109,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:04.206349+0000","last_change":"2024-09-11T00:57:04.206248+0000","last_active":"2024-09-11T00:57:04.206349+0000","last_peered":"2024-09-11T00:57:04.206349+0000","last_clean":"2024-09-11T00:57:04.206349+0000","last_became_active":"2024-09-11T00:30:53.984011+0000","last_became_peered":"2024-09-11T00:30:53.984011+0000","last_unstale":"2024-09-11T00:57:04.206349+0000","last_undegraded":"2024-09-11T00:57:04.206349+0000","last_fullsized":"2024-09-11T00:57:04.206349+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-11T00:57:04.206146+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-11T00:57:04.206146+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.206146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:24:37.028054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"62'3995","reported_seq":4099,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:08.859131+0000","last_change":"2024-09-11T00:57:08.859063+0000","last_active":"2024-09-11T00:57:08.859131+0000","last_peered":"2024-09-11T00:57:08.859131+0000","last_clean":"2024-09-11T00:57:08.859131+0000","last_became_active":"2024-09-11T00:30:54.986356+0000","last_became_peered":"2024-09-11T00:30:54.986356+0000","last_unstale":"2024-09-11T00:57:08.859131+0000","last_undegraded":"2024-09-11T00:57:08.859131+0000","last_fullsized":"2024-09-11T00:57:08.859131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3995","last_scrub_stamp":"2024-09-11T00:57:08.858975+0000","last_deep_scrub":"62'3995","last_deep_scrub_stamp":"2024-09-11T00:57:08.858975+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.858975+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:31:07.032208+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20665,"num_write_kb":324486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"62'22","reported_seq":140,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.608155+0000","last_change":"2024-09-11T00:56:54.046603+0000","last_active":"2024-09-11T00:57:34.608155+0000","last_peered":"2024-09-11T00:57:34.608155+0000","last_clean":"2024-09-11T00:57:34.608155+0000","last_became_active":"2024-09-11T00:30:44.924124+0000","last_became_peered":"2024-09-11T00:30:44.924124+0000","last_unstale":"2024-09-11T00:57:34.608155+0000","last_undegraded":"2024-09-11T00:57:34.608155+0000","last_fullsized":"2024-09-11T00:57:34.608155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'22","last_scrub_stamp":"2024-09-11T00:56:54.046518+0000","last_deep_scrub":"62'22","last_deep_scrub_stamp":"2024-09-11T00:56:54.046518+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.046518+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:50:58.895853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"60'3563","reported_seq":3684,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607226+0000","last_change":"2024-09-11T00:57:02.119144+0000","last_active":"2024-09-11T00:57:34.607226+0000","last_peered":"2024-09-11T00:57:34.607226+0000","last_clean":"2024-09-11T00:57:34.607226+0000","last_became_active":"2024-09-11T00:30:45.933531+0000","last_became_peered":"2024-09-11T00:30:45.933531+0000","last_unstale":"2024-09-11T00:57:34.607226+0000","last_undegraded":"2024-09-11T00:57:34.607226+0000","last_fullsized":"2024-09-11T00:57:34.607226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3563","last_scrub_stamp":"2024-09-11T00:57:02.119042+0000","last_deep_scrub":"60'3563","last_deep_scrub_stamp":"2024-09-11T00:57:02.119042+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.119042+0000","objects_scrubbed":0,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:43:54.433498+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":44,"num_write":18834,"num_write_kb":321224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"62'3","reported_seq":111,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.615923+0000","last_change":"2024-09-11T00:57:05.165541+0000","last_active":"2024-09-11T00:57:34.615923+0000","last_peered":"2024-09-11T00:57:34.615923+0000","last_clean":"2024-09-11T00:57:34.615923+0000","last_became_active":"2024-09-11T00:30:53.984702+0000","last_became_peered":"2024-09-11T00:30:53.984702+0000","last_unstale":"2024-09-11T00:57:34.615923+0000","last_undegraded":"2024-09-11T00:57:34.615923+0000","last_fullsized":"2024-09-11T00:57:34.615923+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3","last_scrub_stamp":"2024-09-11T00:57:05.165444+0000","last_deep_scrub":"62'3","last_deep_scrub_stamp":"2024-09-11T00:57:05.165444+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.165444+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:10.208866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"62'3090","reported_seq":3195,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.611863+0000","last_change":"2024-09-11T00:57:01.418035+0000","last_active":"2024-09-11T00:57:34.611863+0000","last_peered":"2024-09-11T00:57:34.611863+0000","last_clean":"2024-09-11T00:57:34.611863+0000","last_became_active":"2024-09-11T00:30:54.993260+0000","last_became_peered":"2024-09-11T00:30:54.993260+0000","last_unstale":"2024-09-11T00:57:34.611863+0000","last_undegraded":"2024-09-11T00:57:34.611863+0000","last_fullsized":"2024-09-11T00:57:34.611863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3090","last_scrub_stamp":"2024-09-11T00:57:01.417929+0000","last_deep_scrub":"62'3090","last_deep_scrub_stamp":"2024-09-11T00:57:01.417929+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.417929+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:58.184123+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15625,"num_write_kb":254958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"62'38","reported_seq":182,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.609207+0000","last_change":"2024-09-11T00:56:54.714721+0000","last_active":"2024-09-11T00:57:34.609207+0000","last_peered":"2024-09-11T00:57:34.609207+0000","last_clean":"2024-09-11T00:57:34.609207+0000","last_became_active":"2024-09-11T00:30:44.912500+0000","last_became_peered":"2024-09-11T00:30:44.912500+0000","last_unstale":"2024-09-11T00:57:34.609207+0000","last_undegraded":"2024-09-11T00:57:34.609207+0000","last_fullsized":"2024-09-11T00:57:34.609207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'38","last_scrub_stamp":"2024-09-11T00:56:54.714635+0000","last_deep_scrub":"62'38","last_deep_scrub_stamp":"2024-09-11T00:56:54.714635+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.714635+0000","objects_scrubbed":1,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:59:22.322810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"60'2040","reported_seq":2154,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.612953+0000","last_change":"2024-09-11T00:56:58.381316+0000","last_active":"2024-09-11T00:57:34.612953+0000","last_peered":"2024-09-11T00:57:34.612953+0000","last_clean":"2024-09-11T00:57:34.612953+0000","last_became_active":"2024-09-11T00:30:45.921043+0000","last_became_peered":"2024-09-11T00:30:45.921043+0000","last_unstale":"2024-09-11T00:57:34.612953+0000","last_undegraded":"2024-09-11T00:57:34.612953+0000","last_fullsized":"2024-09-11T00:57:34.612953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2040","last_scrub_stamp":"2024-09-11T00:56:58.381233+0000","last_deep_scrub":"60'2040","last_deep_scrub_stamp":"2024-09-11T00:56:58.381233+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.381233+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:48:16.374087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":10218,"num_write_kb":171981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"62'717","reported_seq":848,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616182+0000","last_change":"2024-09-11T00:57:07.176180+0000","last_active":"2024-09-11T00:57:34.616182+0000","last_peered":"2024-09-11T00:57:34.616182+0000","last_clean":"2024-09-11T00:57:34.616182+0000","last_became_active":"2024-09-11T00:30:53.973775+0000","last_became_peered":"2024-09-11T00:30:53.973775+0000","last_unstale":"2024-09-11T00:57:34.616182+0000","last_undegraded":"2024-09-11T00:57:34.616182+0000","last_fullsized":"2024-09-11T00:57:34.616182+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'717","last_scrub_stamp":"2024-09-11T00:57:07.176050+0000","last_deep_scrub":"62'717","last_deep_scrub_stamp":"2024-09-11T00:57:07.176050+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.176050+0000","objects_scrubbed":1,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:09.815079+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":73385,"num_write":707,"num_write_kb":4317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"62'2671","reported_seq":2776,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.615813+0000","last_change":"2024-09-11T00:57:08.143383+0000","last_active":"2024-09-11T00:57:34.615813+0000","last_peered":"2024-09-11T00:57:34.615813+0000","last_clean":"2024-09-11T00:57:34.615813+0000","last_became_active":"2024-09-11T00:30:54.984482+0000","last_became_peered":"2024-09-11T00:30:54.984482+0000","last_unstale":"2024-09-11T00:57:34.615813+0000","last_undegraded":"2024-09-11T00:57:34.615813+0000","last_fullsized":"2024-09-11T00:57:34.615813+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2671","last_scrub_stamp":"2024-09-11T00:57:08.143313+0000","last_deep_scrub":"62'2671","last_deep_scrub_stamp":"2024-09-11T00:57:08.143313+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.143313+0000","objects_scrubbed":0,"log_size":2671,"log_dups_size":0,"ondisk_log_size":2671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:29.149992+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":213871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"62'4","reported_seq":147,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.608740+0000","last_change":"2024-09-11T00:56:57.791494+0000","last_active":"2024-09-11T00:57:34.608740+0000","last_peered":"2024-09-11T00:57:34.608740+0000","last_clean":"2024-09-11T00:57:34.608740+0000","last_became_active":"2024-09-11T00:30:44.914945+0000","last_became_peered":"2024-09-11T00:30:44.914945+0000","last_unstale":"2024-09-11T00:57:34.608740+0000","last_undegraded":"2024-09-11T00:57:34.608740+0000","last_fullsized":"2024-09-11T00:57:34.608740+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4","last_scrub_stamp":"2024-09-11T00:56:57.791406+0000","last_deep_scrub":"62'4","last_deep_scrub_stamp":"2024-09-11T00:56:57.791406+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.791406+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:19:14.259832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"60'3687","reported_seq":3800,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606774+0000","last_change":"2024-09-11T00:57:04.235455+0000","last_active":"2024-09-11T00:57:34.606774+0000","last_peered":"2024-09-11T00:57:34.606774+0000","last_clean":"2024-09-11T00:57:34.606774+0000","last_became_active":"2024-09-11T00:30:45.923487+0000","last_became_peered":"2024-09-11T00:30:45.923487+0000","last_unstale":"2024-09-11T00:57:34.606774+0000","last_undegraded":"2024-09-11T00:57:34.606774+0000","last_fullsized":"2024-09-11T00:57:34.606774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3687","last_scrub_stamp":"2024-09-11T00:57:04.235390+0000","last_deep_scrub":"60'3687","last_deep_scrub_stamp":"2024-09-11T00:57:04.235390+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.235390+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:23.747635+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19525,"num_write_kb":315726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"62'140","reported_seq":247,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616118+0000","last_change":"2024-09-11T00:57:06.125927+0000","last_active":"2024-09-11T00:57:34.616118+0000","last_peered":"2024-09-11T00:57:34.616118+0000","last_clean":"2024-09-11T00:57:34.616118+0000","last_became_active":"2024-09-11T00:30:53.977364+0000","last_became_peered":"2024-09-11T00:30:53.977364+0000","last_unstale":"2024-09-11T00:57:34.616118+0000","last_undegraded":"2024-09-11T00:57:34.616118+0000","last_fullsized":"2024-09-11T00:57:34.616118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'140","last_scrub_stamp":"2024-09-11T00:57:06.125841+0000","last_deep_scrub":"62'140","last_deep_scrub_stamp":"2024-09-11T00:57:06.125841+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.125841+0000","objects_scrubbed":1,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:36.011406+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1356353,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":1393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"62'3457","reported_seq":3563,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605879+0000","last_change":"2024-09-11T00:57:15.141889+0000","last_active":"2024-09-11T00:57:34.605879+0000","last_peered":"2024-09-11T00:57:34.605879+0000","last_clean":"2024-09-11T00:57:34.605879+0000","last_became_active":"2024-09-11T00:30:54.989739+0000","last_became_peered":"2024-09-11T00:30:54.989739+0000","last_unstale":"2024-09-11T00:57:34.605879+0000","last_undegraded":"2024-09-11T00:57:34.605879+0000","last_fullsized":"2024-09-11T00:57:34.605879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3457","last_scrub_stamp":"2024-09-11T00:57:15.141770+0000","last_deep_scrub":"62'3457","last_deep_scrub_stamp":"2024-09-11T00:57:15.141770+0000","last_clean_scrub_stamp":"2024-09-11T00:57:15.141770+0000","objects_scrubbed":0,"log_size":3457,"log_dups_size":0,"ondisk_log_size":3457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:58:13.191311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18121,"num_write_kb":297725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"60'2670","reported_seq":2785,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605728+0000","last_change":"2024-09-11T00:57:07.221647+0000","last_active":"2024-09-11T00:57:34.605728+0000","last_peered":"2024-09-11T00:57:34.605728+0000","last_clean":"2024-09-11T00:57:34.605728+0000","last_became_active":"2024-09-11T00:30:45.930258+0000","last_became_peered":"2024-09-11T00:30:45.930258+0000","last_unstale":"2024-09-11T00:57:34.605728+0000","last_undegraded":"2024-09-11T00:57:34.605728+0000","last_fullsized":"2024-09-11T00:57:34.605728+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2670","last_scrub_stamp":"2024-09-11T00:57:07.221559+0000","last_deep_scrub":"60'2670","last_deep_scrub_stamp":"2024-09-11T00:57:07.221559+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.221559+0000","objects_scrubbed":0,"log_size":2670,"log_dups_size":0,"ondisk_log_size":2670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:39:34.086650+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14000,"num_write_kb":256678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"62'1370","reported_seq":1519,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605969+0000","last_change":"2024-09-11T00:56:58.325639+0000","last_active":"2024-09-11T00:57:34.605969+0000","last_peered":"2024-09-11T00:57:34.605969+0000","last_clean":"2024-09-11T00:57:34.605969+0000","last_became_active":"2024-09-11T00:30:44.923066+0000","last_became_peered":"2024-09-11T00:30:44.923066+0000","last_unstale":"2024-09-11T00:57:34.605969+0000","last_undegraded":"2024-09-11T00:57:34.605969+0000","last_fullsized":"2024-09-11T00:57:34.605969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1370","last_scrub_stamp":"2024-09-11T00:56:58.325557+0000","last_deep_scrub":"62'1370","last_deep_scrub_stamp":"2024-09-11T00:56:58.325557+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.325557+0000","objects_scrubbed":2,"log_size":1370,"log_dups_size":0,"ondisk_log_size":1370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:22:26.289083+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":30358,"num_write":1345,"num_write_kb":4470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"62'3","reported_seq":132,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605921+0000","last_change":"2024-09-11T00:57:16.268435+0000","last_active":"2024-09-11T00:57:34.605921+0000","last_peered":"2024-09-11T00:57:34.605921+0000","last_clean":"2024-09-11T00:57:34.605921+0000","last_became_active":"2024-09-11T00:30:53.984426+0000","last_became_peered":"2024-09-11T00:30:53.984426+0000","last_unstale":"2024-09-11T00:57:34.605921+0000","last_undegraded":"2024-09-11T00:57:34.605921+0000","last_fullsized":"2024-09-11T00:57:34.605921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3","last_scrub_stamp":"2024-09-11T00:57:16.268367+0000","last_deep_scrub":"62'3","last_deep_scrub_stamp":"2024-09-11T00:57:16.268367+0000","last_clean_scrub_stamp":"2024-09-11T00:57:16.268367+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:20:17.121180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"62'3537","reported_seq":3641,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:08.245065+0000","last_change":"2024-09-11T00:57:08.245065+0000","last_active":"2024-09-11T00:57:08.245065+0000","last_peered":"2024-09-11T00:57:08.245065+0000","last_clean":"2024-09-11T00:57:08.245065+0000","last_became_active":"2024-09-11T00:30:54.997118+0000","last_became_peered":"2024-09-11T00:30:54.997118+0000","last_unstale":"2024-09-11T00:57:08.245065+0000","last_undegraded":"2024-09-11T00:57:08.245065+0000","last_fullsized":"2024-09-11T00:57:08.245065+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3537","last_scrub_stamp":"2024-09-11T00:57:08.244978+0000","last_deep_scrub":"62'3537","last_deep_scrub_stamp":"2024-09-11T00:57:08.244978+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.244978+0000","objects_scrubbed":0,"log_size":3537,"log_dups_size":0,"ondisk_log_size":3537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:00:43.862894+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18517,"num_write_kb":279095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"60'2630","reported_seq":2743,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:58.173597+0000","last_change":"2024-09-11T00:56:58.173597+0000","last_active":"2024-09-11T00:56:58.173597+0000","last_peered":"2024-09-11T00:56:58.173597+0000","last_clean":"2024-09-11T00:56:58.173597+0000","last_became_active":"2024-09-11T00:30:45.924853+0000","last_became_peered":"2024-09-11T00:30:45.924853+0000","last_unstale":"2024-09-11T00:56:58.173597+0000","last_undegraded":"2024-09-11T00:56:58.173597+0000","last_fullsized":"2024-09-11T00:56:58.173597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2630","last_scrub_stamp":"2024-09-11T00:56:58.173005+0000","last_deep_scrub":"60'2630","last_deep_scrub_stamp":"2024-09-11T00:56:58.173005+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.173005+0000","objects_scrubbed":0,"log_size":2630,"log_dups_size":0,"ondisk_log_size":2630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:32:13.917321+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":8,"num_write":13276,"num_write_kb":225109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"60'4","reported_seq":121,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606147+0000","last_change":"2024-09-11T00:56:55.916202+0000","last_active":"2024-09-11T00:57:34.606147+0000","last_peered":"2024-09-11T00:57:34.606147+0000","last_clean":"2024-09-11T00:57:34.606147+0000","last_became_active":"2024-09-11T00:30:44.927005+0000","last_became_peered":"2024-09-11T00:30:44.927005+0000","last_unstale":"2024-09-11T00:57:34.606147+0000","last_undegraded":"2024-09-11T00:57:34.606147+0000","last_fullsized":"2024-09-11T00:57:34.606147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4","last_scrub_stamp":"2024-09-11T00:56:55.916108+0000","last_deep_scrub":"60'4","last_deep_scrub_stamp":"2024-09-11T00:56:55.916108+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.916108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:49:15.583153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"61'2","reported_seq":110,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.608118+0000","last_change":"2024-09-11T00:57:03.795761+0000","last_active":"2024-09-11T00:57:34.608118+0000","last_peered":"2024-09-11T00:57:34.608118+0000","last_clean":"2024-09-11T00:57:34.608118+0000","last_became_active":"2024-09-11T00:30:53.986557+0000","last_became_peered":"2024-09-11T00:30:53.986557+0000","last_unstale":"2024-09-11T00:57:34.608118+0000","last_undegraded":"2024-09-11T00:57:34.608118+0000","last_fullsized":"2024-09-11T00:57:34.608118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-11T00:57:03.795662+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-11T00:57:03.795662+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.795662+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:23.761978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"62'3470","reported_seq":3576,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605738+0000","last_change":"2024-09-11T00:57:21.309029+0000","last_active":"2024-09-11T00:57:34.605738+0000","last_peered":"2024-09-11T00:57:34.605738+0000","last_clean":"2024-09-11T00:57:34.605738+0000","last_became_active":"2024-09-11T00:30:54.997168+0000","last_became_peered":"2024-09-11T00:30:54.997168+0000","last_unstale":"2024-09-11T00:57:34.605738+0000","last_undegraded":"2024-09-11T00:57:34.605738+0000","last_fullsized":"2024-09-11T00:57:34.605738+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3470","last_scrub_stamp":"2024-09-11T00:57:21.308932+0000","last_deep_scrub":"62'3470","last_deep_scrub_stamp":"2024-09-11T00:57:21.308932+0000","last_clean_scrub_stamp":"2024-09-11T00:57:21.308932+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:21.974341+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17553,"num_write_kb":279333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"60'2695","reported_seq":2810,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:59.863459+0000","last_change":"2024-09-11T00:56:59.863459+0000","last_active":"2024-09-11T00:56:59.863459+0000","last_peered":"2024-09-11T00:56:59.863459+0000","last_clean":"2024-09-11T00:56:59.863459+0000","last_became_active":"2024-09-11T00:30:45.928764+0000","last_became_peered":"2024-09-11T00:30:45.928764+0000","last_unstale":"2024-09-11T00:56:59.863459+0000","last_undegraded":"2024-09-11T00:56:59.863459+0000","last_fullsized":"2024-09-11T00:56:59.863459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2695","last_scrub_stamp":"2024-09-11T00:56:59.863365+0000","last_deep_scrub":"60'2695","last_deep_scrub_stamp":"2024-09-11T00:56:59.863365+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.863365+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:47:40.095008+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":14,"num_write":14032,"num_write_kb":244535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"60'3","reported_seq":118,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:53.850890+0000","last_change":"2024-09-11T00:56:53.850890+0000","last_active":"2024-09-11T00:56:53.850890+0000","last_peered":"2024-09-11T00:56:53.850890+0000","last_clean":"2024-09-11T00:56:53.850890+0000","last_became_active":"2024-09-11T00:30:44.917108+0000","last_became_peered":"2024-09-11T00:30:44.917108+0000","last_unstale":"2024-09-11T00:56:53.850890+0000","last_undegraded":"2024-09-11T00:56:53.850890+0000","last_fullsized":"2024-09-11T00:56:53.850890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3","last_scrub_stamp":"2024-09-11T00:56:53.850771+0000","last_deep_scrub":"60'3","last_deep_scrub_stamp":"2024-09-11T00:56:53.850771+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.850771+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:26.843309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"62'6","reported_seq":133,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:03.228171+0000","last_change":"2024-09-11T00:57:03.228171+0000","last_active":"2024-09-11T00:57:03.228171+0000","last_peered":"2024-09-11T00:57:03.228171+0000","last_clean":"2024-09-11T00:57:03.228171+0000","last_became_active":"2024-09-11T00:30:53.990409+0000","last_became_peered":"2024-09-11T00:30:53.990409+0000","last_unstale":"2024-09-11T00:57:03.228171+0000","last_undegraded":"2024-09-11T00:57:03.228171+0000","last_fullsized":"2024-09-11T00:57:03.228171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:57:03.228077+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:57:03.228077+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.228077+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:55:51.202975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"62'3622","reported_seq":3728,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.612221+0000","last_change":"2024-09-11T00:56:59.405701+0000","last_active":"2024-09-11T00:57:34.612221+0000","last_peered":"2024-09-11T00:57:34.612221+0000","last_clean":"2024-09-11T00:57:34.612221+0000","last_became_active":"2024-09-11T00:30:54.982181+0000","last_became_peered":"2024-09-11T00:30:54.982181+0000","last_unstale":"2024-09-11T00:57:34.612221+0000","last_undegraded":"2024-09-11T00:57:34.612221+0000","last_fullsized":"2024-09-11T00:57:34.612221+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3622","last_scrub_stamp":"2024-09-11T00:56:59.405598+0000","last_deep_scrub":"62'3622","last_deep_scrub_stamp":"2024-09-11T00:56:59.405598+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.405598+0000","objects_scrubbed":0,"log_size":3622,"log_dups_size":0,"ondisk_log_size":3622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:02:32.707334+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18608,"num_write_kb":291609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"60'2732","reported_seq":2848,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605263+0000","last_change":"2024-09-11T00:57:03.280121+0000","last_active":"2024-09-11T00:57:34.605263+0000","last_peered":"2024-09-11T00:57:34.605263+0000","last_clean":"2024-09-11T00:57:34.605263+0000","last_became_active":"2024-09-11T00:30:45.931433+0000","last_became_peered":"2024-09-11T00:30:45.931433+0000","last_unstale":"2024-09-11T00:57:34.605263+0000","last_undegraded":"2024-09-11T00:57:34.605263+0000","last_fullsized":"2024-09-11T00:57:34.605263+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2732","last_scrub_stamp":"2024-09-11T00:57:03.280040+0000","last_deep_scrub":"60'2732","last_deep_scrub_stamp":"2024-09-11T00:57:03.280040+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.280040+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:01:30.563785+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13936,"num_write_kb":248214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"60'3","reported_seq":120,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604127+0000","last_change":"2024-09-11T00:56:50.663864+0000","last_active":"2024-09-11T00:57:34.604127+0000","last_peered":"2024-09-11T00:57:34.604127+0000","last_clean":"2024-09-11T00:57:34.604127+0000","last_became_active":"2024-09-11T00:30:44.920917+0000","last_became_peered":"2024-09-11T00:30:44.920917+0000","last_unstale":"2024-09-11T00:57:34.604127+0000","last_undegraded":"2024-09-11T00:57:34.604127+0000","last_fullsized":"2024-09-11T00:57:34.604127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3","last_scrub_stamp":"2024-09-11T00:56:50.663784+0000","last_deep_scrub":"60'3","last_deep_scrub_stamp":"2024-09-11T00:56:50.663784+0000","last_clean_scrub_stamp":"2024-09-11T00:56:50.663784+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:41:39.646786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"62'7","reported_seq":157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616536+0000","last_change":"2024-09-11T00:57:02.138476+0000","last_active":"2024-09-11T00:57:34.616536+0000","last_peered":"2024-09-11T00:57:34.616536+0000","last_clean":"2024-09-11T00:57:34.616536+0000","last_became_active":"2024-09-11T00:30:53.986733+0000","last_became_peered":"2024-09-11T00:30:53.986733+0000","last_unstale":"2024-09-11T00:57:34.616536+0000","last_undegraded":"2024-09-11T00:57:34.616536+0000","last_fullsized":"2024-09-11T00:57:34.616536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'7","last_scrub_stamp":"2024-09-11T00:57:02.138384+0000","last_deep_scrub":"62'7","last_deep_scrub_stamp":"2024-09-11T00:57:02.138384+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.138384+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:56:14.449977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"62'3257","reported_seq":3364,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:05.816841+0000","last_change":"2024-09-11T00:57:05.816841+0000","last_active":"2024-09-11T00:57:05.816841+0000","last_peered":"2024-09-11T00:57:05.816841+0000","last_clean":"2024-09-11T00:57:05.816841+0000","last_became_active":"2024-09-11T00:30:54.983184+0000","last_became_peered":"2024-09-11T00:30:54.983184+0000","last_unstale":"2024-09-11T00:57:05.816841+0000","last_undegraded":"2024-09-11T00:57:05.816841+0000","last_fullsized":"2024-09-11T00:57:05.816841+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3257","last_scrub_stamp":"2024-09-11T00:57:05.816731+0000","last_deep_scrub":"62'3257","last_deep_scrub_stamp":"2024-09-11T00:57:05.816731+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.816731+0000","objects_scrubbed":0,"log_size":3257,"log_dups_size":0,"ondisk_log_size":3257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:17:41.183322+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17168,"num_write_kb":277608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"60'2688","reported_seq":2803,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:57.923506+0000","last_change":"2024-09-11T00:56:57.923506+0000","last_active":"2024-09-11T00:56:57.923506+0000","last_peered":"2024-09-11T00:56:57.923506+0000","last_clean":"2024-09-11T00:56:57.923506+0000","last_became_active":"2024-09-11T00:30:45.923964+0000","last_became_peered":"2024-09-11T00:30:45.923964+0000","last_unstale":"2024-09-11T00:56:57.923506+0000","last_undegraded":"2024-09-11T00:56:57.923506+0000","last_fullsized":"2024-09-11T00:56:57.923506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2688","last_scrub_stamp":"2024-09-11T00:56:57.923388+0000","last_deep_scrub":"60'2688","last_deep_scrub_stamp":"2024-09-11T00:56:57.923388+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.923388+0000","objects_scrubbed":0,"log_size":2688,"log_dups_size":0,"ondisk_log_size":2688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:11:19.979963+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13441,"num_write_kb":246089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"62'6","reported_seq":152,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.610179+0000","last_change":"2024-09-11T00:56:57.343381+0000","last_active":"2024-09-11T00:57:34.610179+0000","last_peered":"2024-09-11T00:57:34.610179+0000","last_clean":"2024-09-11T00:57:34.610179+0000","last_became_active":"2024-09-11T00:30:44.929078+0000","last_became_peered":"2024-09-11T00:30:44.929078+0000","last_unstale":"2024-09-11T00:57:34.610179+0000","last_undegraded":"2024-09-11T00:57:34.610179+0000","last_fullsized":"2024-09-11T00:57:34.610179+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:56:57.343293+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:56:57.343293+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.343293+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:46:59.632126+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"62'65","reported_seq":255,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616080+0000","last_change":"2024-09-11T00:57:01.140439+0000","last_active":"2024-09-11T00:57:34.616080+0000","last_peered":"2024-09-11T00:57:34.616080+0000","last_clean":"2024-09-11T00:57:34.616080+0000","last_became_active":"2024-09-11T00:30:53.988223+0000","last_became_peered":"2024-09-11T00:30:53.988223+0000","last_unstale":"2024-09-11T00:57:34.616080+0000","last_undegraded":"2024-09-11T00:57:34.616080+0000","last_fullsized":"2024-09-11T00:57:34.616080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'65","last_scrub_stamp":"2024-09-11T00:57:01.140342+0000","last_deep_scrub":"62'65","last_deep_scrub_stamp":"2024-09-11T00:57:01.140342+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.140342+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:55.217199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":126,"num_write":62,"num_write_kb":126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"62'3386","reported_seq":3494,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604814+0000","last_change":"2024-09-11T00:57:19.298748+0000","last_active":"2024-09-11T00:57:34.604814+0000","last_peered":"2024-09-11T00:57:34.604814+0000","last_clean":"2024-09-11T00:57:34.604814+0000","last_became_active":"2024-09-11T00:30:54.994114+0000","last_became_peered":"2024-09-11T00:30:54.994114+0000","last_unstale":"2024-09-11T00:57:34.604814+0000","last_undegraded":"2024-09-11T00:57:34.604814+0000","last_fullsized":"2024-09-11T00:57:34.604814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3386","last_scrub_stamp":"2024-09-11T00:57:19.298632+0000","last_deep_scrub":"62'3386","last_deep_scrub_stamp":"2024-09-11T00:57:19.298632+0000","last_clean_scrub_stamp":"2024-09-11T00:57:19.298632+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:18:56.199454+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18005,"num_write_kb":276900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"60'3194","reported_seq":3307,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:02.888960+0000","last_change":"2024-09-11T00:57:02.888960+0000","last_active":"2024-09-11T00:57:02.888960+0000","last_peered":"2024-09-11T00:57:02.888960+0000","last_clean":"2024-09-11T00:57:02.888960+0000","last_became_active":"2024-09-11T00:30:45.925119+0000","last_became_peered":"2024-09-11T00:30:45.925119+0000","last_unstale":"2024-09-11T00:57:02.888960+0000","last_undegraded":"2024-09-11T00:57:02.888960+0000","last_fullsized":"2024-09-11T00:57:02.888960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3194","last_scrub_stamp":"2024-09-11T00:57:02.888869+0000","last_deep_scrub":"60'3194","last_deep_scrub_stamp":"2024-09-11T00:57:02.888869+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.888869+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:59:41.665898+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16397,"num_write_kb":262231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"62'521","reported_seq":650,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:54.897165+0000","last_change":"2024-09-11T00:56:54.897165+0000","last_active":"2024-09-11T00:56:54.897165+0000","last_peered":"2024-09-11T00:56:54.897165+0000","last_clean":"2024-09-11T00:56:54.897165+0000","last_became_active":"2024-09-11T00:30:44.917177+0000","last_became_peered":"2024-09-11T00:30:44.917177+0000","last_unstale":"2024-09-11T00:56:54.897165+0000","last_undegraded":"2024-09-11T00:56:54.897165+0000","last_fullsized":"2024-09-11T00:56:54.897165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'521","last_scrub_stamp":"2024-09-11T00:56:54.897074+0000","last_deep_scrub":"62'521","last_deep_scrub_stamp":"2024-09-11T00:56:54.897074+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.897074+0000","objects_scrubbed":1,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:32:57.026216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":41,"num_write":1157,"num_write_kb":372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"62'9","reported_seq":117,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:00.862066+0000","last_change":"2024-09-11T00:57:00.862066+0000","last_active":"2024-09-11T00:57:00.862066+0000","last_peered":"2024-09-11T00:57:00.862066+0000","last_clean":"2024-09-11T00:57:00.862066+0000","last_became_active":"2024-09-11T00:30:53.982234+0000","last_became_peered":"2024-09-11T00:30:53.982234+0000","last_unstale":"2024-09-11T00:57:00.862066+0000","last_undegraded":"2024-09-11T00:57:00.862066+0000","last_fullsized":"2024-09-11T00:57:00.862066+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-09-11T00:57:00.861964+0000","last_deep_scrub":"62'9","last_deep_scrub_stamp":"2024-09-11T00:57:00.861964+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.861964+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:54:37.779840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"61'3259","reported_seq":3367,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606212+0000","last_change":"2024-09-11T00:57:04.850349+0000","last_active":"2024-09-11T00:57:34.606212+0000","last_peered":"2024-09-11T00:57:34.606212+0000","last_clean":"2024-09-11T00:57:34.606212+0000","last_became_active":"2024-09-11T00:30:54.992883+0000","last_became_peered":"2024-09-11T00:30:54.992883+0000","last_unstale":"2024-09-11T00:57:34.606212+0000","last_undegraded":"2024-09-11T00:57:34.606212+0000","last_fullsized":"2024-09-11T00:57:34.606212+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3259","last_scrub_stamp":"2024-09-11T00:57:04.850220+0000","last_deep_scrub":"61'3259","last_deep_scrub_stamp":"2024-09-11T00:57:04.850220+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.850220+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:13.308965+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16779,"num_write_kb":276080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"60'3127","reported_seq":3243,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606930+0000","last_change":"2024-09-11T00:56:58.834284+0000","last_active":"2024-09-11T00:57:34.606930+0000","last_peered":"2024-09-11T00:57:34.606930+0000","last_clean":"2024-09-11T00:57:34.606930+0000","last_became_active":"2024-09-11T00:30:45.935643+0000","last_became_peered":"2024-09-11T00:30:45.935643+0000","last_unstale":"2024-09-11T00:57:34.606930+0000","last_undegraded":"2024-09-11T00:57:34.606930+0000","last_fullsized":"2024-09-11T00:57:34.606930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3127","last_scrub_stamp":"2024-09-11T00:56:58.834193+0000","last_deep_scrub":"60'3127","last_deep_scrub_stamp":"2024-09-11T00:56:58.834193+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.834193+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:34.867337+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":12,"num_write":16580,"num_write_kb":258802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"60'4","reported_seq":121,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.609530+0000","last_change":"2024-09-11T00:56:55.312508+0000","last_active":"2024-09-11T00:57:34.609530+0000","last_peered":"2024-09-11T00:57:34.609530+0000","last_clean":"2024-09-11T00:57:34.609530+0000","last_became_active":"2024-09-11T00:30:44.924658+0000","last_became_peered":"2024-09-11T00:30:44.924658+0000","last_unstale":"2024-09-11T00:57:34.609530+0000","last_undegraded":"2024-09-11T00:57:34.609530+0000","last_fullsized":"2024-09-11T00:57:34.609530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4","last_scrub_stamp":"2024-09-11T00:56:55.312421+0000","last_deep_scrub":"60'4","last_deep_scrub_stamp":"2024-09-11T00:56:55.312421+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.312421+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:38:09.069317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"30'1","reported_seq":110,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604805+0000","last_change":"2024-09-11T00:57:05.696464+0000","last_active":"2024-09-11T00:57:34.604805+0000","last_peered":"2024-09-11T00:57:34.604805+0000","last_clean":"2024-09-11T00:57:34.604805+0000","last_became_active":"2024-09-11T00:30:53.986533+0000","last_became_peered":"2024-09-11T00:30:53.986533+0000","last_unstale":"2024-09-11T00:57:34.604805+0000","last_undegraded":"2024-09-11T00:57:34.604805+0000","last_fullsized":"2024-09-11T00:57:34.604805+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'1","last_scrub_stamp":"2024-09-11T00:57:05.696367+0000","last_deep_scrub":"30'1","last_deep_scrub_stamp":"2024-09-11T00:57:05.696367+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.696367+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:36.598846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"62'3801","reported_seq":3906,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:12.793863+0000","last_change":"2024-09-11T00:57:12.793863+0000","last_active":"2024-09-11T00:57:12.793863+0000","last_peered":"2024-09-11T00:57:12.793863+0000","last_clean":"2024-09-11T00:57:12.793863+0000","last_became_active":"2024-09-11T00:30:54.986413+0000","last_became_peered":"2024-09-11T00:30:54.986413+0000","last_unstale":"2024-09-11T00:57:12.793863+0000","last_undegraded":"2024-09-11T00:57:12.793863+0000","last_fullsized":"2024-09-11T00:57:12.793863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3801","last_scrub_stamp":"2024-09-11T00:57:12.793758+0000","last_deep_scrub":"62'3801","last_deep_scrub_stamp":"2024-09-11T00:57:12.793758+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.793758+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:10.927808+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20214,"num_write_kb":289874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"60'2498","reported_seq":2615,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616601+0000","last_change":"2024-09-11T00:56:54.133953+0000","last_active":"2024-09-11T00:57:34.616601+0000","last_peered":"2024-09-11T00:57:34.616601+0000","last_clean":"2024-09-11T00:57:34.616601+0000","last_became_active":"2024-09-11T00:30:45.919907+0000","last_became_peered":"2024-09-11T00:30:45.919907+0000","last_unstale":"2024-09-11T00:57:34.616601+0000","last_undegraded":"2024-09-11T00:57:34.616601+0000","last_fullsized":"2024-09-11T00:57:34.616601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2498","last_scrub_stamp":"2024-09-11T00:56:54.133840+0000","last_deep_scrub":"60'2498","last_deep_scrub_stamp":"2024-09-11T00:56:54.133840+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.133840+0000","objects_scrubbed":0,"log_size":2498,"log_dups_size":0,"ondisk_log_size":2498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:46:40.030654+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":13564,"num_write_kb":230524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"60'5","reported_seq":122,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606516+0000","last_change":"2024-09-11T00:56:56.110279+0000","last_active":"2024-09-11T00:57:34.606516+0000","last_peered":"2024-09-11T00:57:34.606516+0000","last_clean":"2024-09-11T00:57:34.606516+0000","last_became_active":"2024-09-11T00:30:44.926366+0000","last_became_peered":"2024-09-11T00:30:44.926366+0000","last_unstale":"2024-09-11T00:57:34.606516+0000","last_undegraded":"2024-09-11T00:57:34.606516+0000","last_fullsized":"2024-09-11T00:57:34.606516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'5","last_scrub_stamp":"2024-09-11T00:56:56.110182+0000","last_deep_scrub":"60'5","last_deep_scrub_stamp":"2024-09-11T00:56:56.110182+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.110182+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:12:40.133364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"62'5","reported_seq":114,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.602712+0000","last_change":"2024-09-11T00:57:04.666648+0000","last_active":"2024-09-11T00:57:34.602712+0000","last_peered":"2024-09-11T00:57:34.602712+0000","last_clean":"2024-09-11T00:57:34.602712+0000","last_became_active":"2024-09-11T00:30:53.986571+0000","last_became_peered":"2024-09-11T00:30:53.986571+0000","last_unstale":"2024-09-11T00:57:34.602712+0000","last_undegraded":"2024-09-11T00:57:34.602712+0000","last_fullsized":"2024-09-11T00:57:34.602712+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2024-09-11T00:57:04.666519+0000","last_deep_scrub":"62'5","last_deep_scrub_stamp":"2024-09-11T00:57:04.666519+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.666519+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:00:53.386063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"62'3103","reported_seq":3209,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:06.187061+0000","last_change":"2024-09-11T00:57:06.187061+0000","last_active":"2024-09-11T00:57:06.187061+0000","last_peered":"2024-09-11T00:57:06.187061+0000","last_clean":"2024-09-11T00:57:06.187061+0000","last_became_active":"2024-09-11T00:30:55.000519+0000","last_became_peered":"2024-09-11T00:30:55.000519+0000","last_unstale":"2024-09-11T00:57:06.187061+0000","last_undegraded":"2024-09-11T00:57:06.187061+0000","last_fullsized":"2024-09-11T00:57:06.187061+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3103","last_scrub_stamp":"2024-09-11T00:57:06.186970+0000","last_deep_scrub":"62'3103","last_deep_scrub_stamp":"2024-09-11T00:57:06.186970+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.186970+0000","objects_scrubbed":0,"log_size":3103,"log_dups_size":0,"ondisk_log_size":3103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:43.692686+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16058,"num_write_kb":244544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"60'2543","reported_seq":2658,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607367+0000","last_change":"2024-09-11T00:57:07.900193+0000","last_active":"2024-09-11T00:57:34.607367+0000","last_peered":"2024-09-11T00:57:34.607367+0000","last_clean":"2024-09-11T00:57:34.607367+0000","last_became_active":"2024-09-11T00:30:45.937562+0000","last_became_peered":"2024-09-11T00:30:45.937562+0000","last_unstale":"2024-09-11T00:57:34.607367+0000","last_undegraded":"2024-09-11T00:57:34.607367+0000","last_fullsized":"2024-09-11T00:57:34.607367+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2543","last_scrub_stamp":"2024-09-11T00:57:07.900101+0000","last_deep_scrub":"60'2543","last_deep_scrub_stamp":"2024-09-11T00:57:07.900101+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.900101+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:06:17.326247+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":230458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"60'5","reported_seq":122,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606481+0000","last_change":"2024-09-11T00:56:54.289576+0000","last_active":"2024-09-11T00:57:34.606481+0000","last_peered":"2024-09-11T00:57:34.606481+0000","last_clean":"2024-09-11T00:57:34.606481+0000","last_became_active":"2024-09-11T00:30:44.923011+0000","last_became_peered":"2024-09-11T00:30:44.923011+0000","last_unstale":"2024-09-11T00:57:34.606481+0000","last_undegraded":"2024-09-11T00:57:34.606481+0000","last_fullsized":"2024-09-11T00:57:34.606481+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'5","last_scrub_stamp":"2024-09-11T00:56:54.289506+0000","last_deep_scrub":"60'5","last_deep_scrub_stamp":"2024-09-11T00:56:54.289506+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.289506+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:31:35.983150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"62'5","reported_seq":114,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606617+0000","last_change":"2024-09-11T00:57:06.653146+0000","last_active":"2024-09-11T00:57:34.606617+0000","last_peered":"2024-09-11T00:57:34.606617+0000","last_clean":"2024-09-11T00:57:34.606617+0000","last_became_active":"2024-09-11T00:30:53.977256+0000","last_became_peered":"2024-09-11T00:30:53.977256+0000","last_unstale":"2024-09-11T00:57:34.606617+0000","last_undegraded":"2024-09-11T00:57:34.606617+0000","last_fullsized":"2024-09-11T00:57:34.606617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2024-09-11T00:57:06.653051+0000","last_deep_scrub":"62'5","last_deep_scrub_stamp":"2024-09-11T00:57:06.653051+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.653051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:51.636784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"61'3199","reported_seq":3305,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:07.222332+0000","last_change":"2024-09-11T00:57:07.222332+0000","last_active":"2024-09-11T00:57:07.222332+0000","last_peered":"2024-09-11T00:57:07.222332+0000","last_clean":"2024-09-11T00:57:07.222332+0000","last_became_active":"2024-09-11T00:30:54.997320+0000","last_became_peered":"2024-09-11T00:30:54.997320+0000","last_unstale":"2024-09-11T00:57:07.222332+0000","last_undegraded":"2024-09-11T00:57:07.222332+0000","last_fullsized":"2024-09-11T00:57:07.222332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3199","last_scrub_stamp":"2024-09-11T00:57:07.222206+0000","last_deep_scrub":"61'3199","last_deep_scrub_stamp":"2024-09-11T00:57:07.222206+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.222206+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:38:55.432692+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15924,"num_write_kb":260293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"60'3298","reported_seq":3415,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604311+0000","last_change":"2024-09-11T00:56:58.121159+0000","last_active":"2024-09-11T00:57:34.604311+0000","last_peered":"2024-09-11T00:57:34.604311+0000","last_clean":"2024-09-11T00:57:34.604311+0000","last_became_active":"2024-09-11T00:30:45.926646+0000","last_became_peered":"2024-09-11T00:30:45.926646+0000","last_unstale":"2024-09-11T00:57:34.604311+0000","last_undegraded":"2024-09-11T00:57:34.604311+0000","last_fullsized":"2024-09-11T00:57:34.604311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3298","last_scrub_stamp":"2024-09-11T00:56:58.121061+0000","last_deep_scrub":"60'3298","last_deep_scrub_stamp":"2024-09-11T00:56:58.121061+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.121061+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:52:08.080517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17655,"num_write_kb":267324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"62'73","reported_seq":270,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:56.221341+0000","last_change":"2024-09-11T00:56:56.221341+0000","last_active":"2024-09-11T00:56:56.221341+0000","last_peered":"2024-09-11T00:56:56.221341+0000","last_clean":"2024-09-11T00:56:56.221341+0000","last_became_active":"2024-09-11T00:30:44.927545+0000","last_became_peered":"2024-09-11T00:30:44.927545+0000","last_unstale":"2024-09-11T00:56:56.221341+0000","last_undegraded":"2024-09-11T00:56:56.221341+0000","last_fullsized":"2024-09-11T00:56:56.221341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'73","last_scrub_stamp":"2024-09-11T00:56:56.221250+0000","last_deep_scrub":"62'73","last_deep_scrub_stamp":"2024-09-11T00:56:56.221250+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.221250+0000","objects_scrubbed":1,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:57:23.739180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88181,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":129,"num_write":68,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"62'6","reported_seq":158,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605433+0000","last_change":"2024-09-11T00:57:07.697908+0000","last_active":"2024-09-11T00:57:34.605433+0000","last_peered":"2024-09-11T00:57:34.605433+0000","last_clean":"2024-09-11T00:57:34.605433+0000","last_became_active":"2024-09-11T00:30:53.978755+0000","last_became_peered":"2024-09-11T00:30:53.978755+0000","last_unstale":"2024-09-11T00:57:34.605433+0000","last_undegraded":"2024-09-11T00:57:34.605433+0000","last_fullsized":"2024-09-11T00:57:34.605433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:57:07.697815+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:57:07.697815+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.697815+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:00.719170+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"62'2728","reported_seq":2835,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606310+0000","last_change":"2024-09-11T00:57:20.301278+0000","last_active":"2024-09-11T00:57:34.606310+0000","last_peered":"2024-09-11T00:57:34.606310+0000","last_clean":"2024-09-11T00:57:34.606310+0000","last_became_active":"2024-09-11T00:30:54.991137+0000","last_became_peered":"2024-09-11T00:30:54.991137+0000","last_unstale":"2024-09-11T00:57:34.606310+0000","last_undegraded":"2024-09-11T00:57:34.606310+0000","last_fullsized":"2024-09-11T00:57:34.606310+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2728","last_scrub_stamp":"2024-09-11T00:57:20.301203+0000","last_deep_scrub":"62'2728","last_deep_scrub_stamp":"2024-09-11T00:57:20.301203+0000","last_clean_scrub_stamp":"2024-09-11T00:57:20.301203+0000","objects_scrubbed":0,"log_size":2728,"log_dups_size":0,"ondisk_log_size":2728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:39:57.608601+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":237243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"60'2962","reported_seq":3078,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.602844+0000","last_change":"2024-09-11T00:56:59.838394+0000","last_active":"2024-09-11T00:57:34.602844+0000","last_peered":"2024-09-11T00:57:34.602844+0000","last_clean":"2024-09-11T00:57:34.602844+0000","last_became_active":"2024-09-11T00:30:45.916414+0000","last_became_peered":"2024-09-11T00:30:45.916414+0000","last_unstale":"2024-09-11T00:57:34.602844+0000","last_undegraded":"2024-09-11T00:57:34.602844+0000","last_fullsized":"2024-09-11T00:57:34.602844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2962","last_scrub_stamp":"2024-09-11T00:56:59.838266+0000","last_deep_scrub":"60'2962","last_deep_scrub_stamp":"2024-09-11T00:56:59.838266+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.838266+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:38:39.202639+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15238,"num_write_kb":265720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"62'8","reported_seq":126,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604739+0000","last_change":"2024-09-11T00:56:53.315719+0000","last_active":"2024-09-11T00:57:34.604739+0000","last_peered":"2024-09-11T00:57:34.604739+0000","last_clean":"2024-09-11T00:57:34.604739+0000","last_became_active":"2024-09-11T00:30:44.928261+0000","last_became_peered":"2024-09-11T00:30:44.928261+0000","last_unstale":"2024-09-11T00:57:34.604739+0000","last_undegraded":"2024-09-11T00:57:34.604739+0000","last_fullsized":"2024-09-11T00:57:34.604739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-09-11T00:56:53.315634+0000","last_deep_scrub":"62'8","last_deep_scrub_stamp":"2024-09-11T00:56:53.315634+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.315634+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:27:19.909262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"62'500","reported_seq":630,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.608099+0000","last_change":"2024-09-11T00:57:11.156523+0000","last_active":"2024-09-11T00:57:34.608099+0000","last_peered":"2024-09-11T00:57:34.608099+0000","last_clean":"2024-09-11T00:57:34.608099+0000","last_became_active":"2024-09-11T00:30:53.989013+0000","last_became_peered":"2024-09-11T00:30:53.989013+0000","last_unstale":"2024-09-11T00:57:34.608099+0000","last_undegraded":"2024-09-11T00:57:34.608099+0000","last_fullsized":"2024-09-11T00:57:34.608099+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'500","last_scrub_stamp":"2024-09-11T00:57:11.156436+0000","last_deep_scrub":"62'500","last_deep_scrub_stamp":"2024-09-11T00:57:11.156436+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.156436+0000","objects_scrubbed":1,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:55:33.248709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":55,"num_write":1123,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"62'3325","reported_seq":3432,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607279+0000","last_change":"2024-09-11T00:57:06.854834+0000","last_active":"2024-09-11T00:57:34.607279+0000","last_peered":"2024-09-11T00:57:34.607279+0000","last_clean":"2024-09-11T00:57:34.607279+0000","last_became_active":"2024-09-11T00:30:54.989802+0000","last_became_peered":"2024-09-11T00:30:54.989802+0000","last_unstale":"2024-09-11T00:57:34.607279+0000","last_undegraded":"2024-09-11T00:57:34.607279+0000","last_fullsized":"2024-09-11T00:57:34.607279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3325","last_scrub_stamp":"2024-09-11T00:57:06.854742+0000","last_deep_scrub":"62'3325","last_deep_scrub_stamp":"2024-09-11T00:57:06.854742+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.854742+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:29:57.166335+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17451,"num_write_kb":278810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"60'3332","reported_seq":3446,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:11.777243+0000","last_change":"2024-09-11T00:57:11.777243+0000","last_active":"2024-09-11T00:57:11.777243+0000","last_peered":"2024-09-11T00:57:11.777243+0000","last_clean":"2024-09-11T00:57:11.777243+0000","last_became_active":"2024-09-11T00:30:45.919385+0000","last_became_peered":"2024-09-11T00:30:45.919385+0000","last_unstale":"2024-09-11T00:57:11.777243+0000","last_undegraded":"2024-09-11T00:57:11.777243+0000","last_fullsized":"2024-09-11T00:57:11.777243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3332","last_scrub_stamp":"2024-09-11T00:57:11.777113+0000","last_deep_scrub":"60'3332","last_deep_scrub_stamp":"2024-09-11T00:57:11.777113+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.777113+0000","objects_scrubbed":0,"log_size":3332,"log_dups_size":0,"ondisk_log_size":3332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:56:38.807453+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17164,"num_write_kb":285768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"31'1","reported_seq":118,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605328+0000","last_change":"2024-09-11T00:56:56.308755+0000","last_active":"2024-09-11T00:57:34.605328+0000","last_peered":"2024-09-11T00:57:34.605328+0000","last_clean":"2024-09-11T00:57:34.605328+0000","last_became_active":"2024-09-11T00:30:44.929411+0000","last_became_peered":"2024-09-11T00:30:44.929411+0000","last_unstale":"2024-09-11T00:57:34.605328+0000","last_undegraded":"2024-09-11T00:57:34.605328+0000","last_fullsized":"2024-09-11T00:57:34.605328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'1","last_scrub_stamp":"2024-09-11T00:56:56.308650+0000","last_deep_scrub":"31'1","last_deep_scrub_stamp":"2024-09-11T00:56:56.308650+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.308650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:51:47.983215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"62'74","reported_seq":216,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.612172+0000","last_change":"2024-09-11T00:56:50.518182+0000","last_active":"2024-09-11T00:57:34.612172+0000","last_peered":"2024-09-11T00:57:34.612172+0000","last_clean":"2024-09-11T00:57:34.612172+0000","last_became_active":"2024-09-11T00:30:36.027598+0000","last_became_peered":"2024-09-11T00:30:36.027598+0000","last_unstale":"2024-09-11T00:57:34.612172+0000","last_undegraded":"2024-09-11T00:57:34.612172+0000","last_fullsized":"2024-09-11T00:57:34.612172+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'74","last_scrub_stamp":"2024-09-11T00:56:50.518060+0000","last_deep_scrub":"62'74","last_deep_scrub_stamp":"2024-09-11T00:56:50.518060+0000","last_clean_scrub_stamp":"2024-09-11T00:56:50.518060+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:13:18.176495+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"62'4","reported_seq":113,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606869+0000","last_change":"2024-09-11T00:57:14.203141+0000","last_active":"2024-09-11T00:57:34.606869+0000","last_peered":"2024-09-11T00:57:34.606869+0000","last_clean":"2024-09-11T00:57:34.606869+0000","last_became_active":"2024-09-11T00:30:53.981513+0000","last_became_peered":"2024-09-11T00:30:53.981513+0000","last_unstale":"2024-09-11T00:57:34.606869+0000","last_undegraded":"2024-09-11T00:57:34.606869+0000","last_fullsized":"2024-09-11T00:57:34.606869+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4","last_scrub_stamp":"2024-09-11T00:57:14.203067+0000","last_deep_scrub":"62'4","last_deep_scrub_stamp":"2024-09-11T00:57:14.203067+0000","last_clean_scrub_stamp":"2024-09-11T00:57:14.203067+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:13:20.463200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"62'2541","reported_seq":2647,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606299+0000","last_change":"2024-09-11T00:57:16.133359+0000","last_active":"2024-09-11T00:57:34.606299+0000","last_peered":"2024-09-11T00:57:34.606299+0000","last_clean":"2024-09-11T00:57:34.606299+0000","last_became_active":"2024-09-11T00:30:54.990512+0000","last_became_peered":"2024-09-11T00:30:54.990512+0000","last_unstale":"2024-09-11T00:57:34.606299+0000","last_undegraded":"2024-09-11T00:57:34.606299+0000","last_fullsized":"2024-09-11T00:57:34.606299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2541","last_scrub_stamp":"2024-09-11T00:57:16.133253+0000","last_deep_scrub":"62'2541","last_deep_scrub_stamp":"2024-09-11T00:57:16.133253+0000","last_clean_scrub_stamp":"2024-09-11T00:57:16.133253+0000","objects_scrubbed":0,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:26:17.663520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12611,"num_write_kb":206053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"60'2809","reported_seq":2927,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606997+0000","last_change":"2024-09-11T00:57:02.322594+0000","last_active":"2024-09-11T00:57:34.606997+0000","last_peered":"2024-09-11T00:57:34.606997+0000","last_clean":"2024-09-11T00:57:34.606997+0000","last_became_active":"2024-09-11T00:30:45.924184+0000","last_became_peered":"2024-09-11T00:30:45.924184+0000","last_unstale":"2024-09-11T00:57:34.606997+0000","last_undegraded":"2024-09-11T00:57:34.606997+0000","last_fullsized":"2024-09-11T00:57:34.606997+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2809","last_scrub_stamp":"2024-09-11T00:57:02.322480+0000","last_deep_scrub":"60'2809","last_deep_scrub_stamp":"2024-09-11T00:57:02.322480+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.322480+0000","objects_scrubbed":0,"log_size":2809,"log_dups_size":0,"ondisk_log_size":2809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:59.763288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":34,"num_write":15096,"num_write_kb":232439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"60'2","reported_seq":118,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607131+0000","last_change":"2024-09-11T00:56:56.881566+0000","last_active":"2024-09-11T00:57:34.607131+0000","last_peered":"2024-09-11T00:57:34.607131+0000","last_clean":"2024-09-11T00:57:34.607131+0000","last_became_active":"2024-09-11T00:30:44.915536+0000","last_became_peered":"2024-09-11T00:30:44.915536+0000","last_unstale":"2024-09-11T00:57:34.607131+0000","last_undegraded":"2024-09-11T00:57:34.607131+0000","last_fullsized":"2024-09-11T00:57:34.607131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-11T00:56:56.881479+0000","last_deep_scrub":"60'2","last_deep_scrub_stamp":"2024-09-11T00:56:56.881479+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.881479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:18:20.964805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"62'4","reported_seq":113,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616253+0000","last_change":"2024-09-11T00:57:03.185029+0000","last_active":"2024-09-11T00:57:34.616253+0000","last_peered":"2024-09-11T00:57:34.616253+0000","last_clean":"2024-09-11T00:57:34.616253+0000","last_became_active":"2024-09-11T00:30:53.988989+0000","last_became_peered":"2024-09-11T00:30:53.988989+0000","last_unstale":"2024-09-11T00:57:34.616253+0000","last_undegraded":"2024-09-11T00:57:34.616253+0000","last_fullsized":"2024-09-11T00:57:34.616253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4","last_scrub_stamp":"2024-09-11T00:57:03.184956+0000","last_deep_scrub":"62'4","last_deep_scrub_stamp":"2024-09-11T00:57:03.184956+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.184956+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:54:31.364491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"62'3293","reported_seq":3400,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.602248+0000","last_change":"2024-09-11T00:57:10.632867+0000","last_active":"2024-09-11T00:57:34.602248+0000","last_peered":"2024-09-11T00:57:34.602248+0000","last_clean":"2024-09-11T00:57:34.602248+0000","last_became_active":"2024-09-11T00:30:54.981411+0000","last_became_peered":"2024-09-11T00:30:54.981411+0000","last_unstale":"2024-09-11T00:57:34.602248+0000","last_undegraded":"2024-09-11T00:57:34.602248+0000","last_fullsized":"2024-09-11T00:57:34.602248+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3293","last_scrub_stamp":"2024-09-11T00:57:10.632753+0000","last_deep_scrub":"62'3293","last_deep_scrub_stamp":"2024-09-11T00:57:10.632753+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.632753+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:52.728638+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17661,"num_write_kb":267414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"60'3034","reported_seq":3150,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.608680+0000","last_change":"2024-09-11T00:56:58.761933+0000","last_active":"2024-09-11T00:57:34.608680+0000","last_peered":"2024-09-11T00:57:34.608680+0000","last_clean":"2024-09-11T00:57:34.608680+0000","last_became_active":"2024-09-11T00:30:45.926497+0000","last_became_peered":"2024-09-11T00:30:45.926497+0000","last_unstale":"2024-09-11T00:57:34.608680+0000","last_undegraded":"2024-09-11T00:57:34.608680+0000","last_fullsized":"2024-09-11T00:57:34.608680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3034","last_scrub_stamp":"2024-09-11T00:56:58.761839+0000","last_deep_scrub":"60'3034","last_deep_scrub_stamp":"2024-09-11T00:56:58.761839+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.761839+0000","objects_scrubbed":0,"log_size":3034,"log_dups_size":0,"ondisk_log_size":3034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:28:38.516070+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":15513,"num_write_kb":283129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"62'3","reported_seq":134,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607999+0000","last_change":"2024-09-11T00:56:55.080078+0000","last_active":"2024-09-11T00:57:34.607999+0000","last_peered":"2024-09-11T00:57:34.607999+0000","last_clean":"2024-09-11T00:57:34.607999+0000","last_became_active":"2024-09-11T00:30:44.924193+0000","last_became_peered":"2024-09-11T00:30:44.924193+0000","last_unstale":"2024-09-11T00:57:34.607999+0000","last_undegraded":"2024-09-11T00:57:34.607999+0000","last_fullsized":"2024-09-11T00:57:34.607999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3","last_scrub_stamp":"2024-09-11T00:56:55.079993+0000","last_deep_scrub":"62'3","last_deep_scrub_stamp":"2024-09-11T00:56:55.079993+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.079993+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:52:56.783983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"62'1445","reported_seq":1598,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606336+0000","last_change":"2024-09-11T00:57:10.142399+0000","last_active":"2024-09-11T00:57:34.606336+0000","last_peered":"2024-09-11T00:57:34.606336+0000","last_clean":"2024-09-11T00:57:34.606336+0000","last_became_active":"2024-09-11T00:30:53.983664+0000","last_became_peered":"2024-09-11T00:30:53.983664+0000","last_unstale":"2024-09-11T00:57:34.606336+0000","last_undegraded":"2024-09-11T00:57:34.606336+0000","last_fullsized":"2024-09-11T00:57:34.606336+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1445","last_scrub_stamp":"2024-09-11T00:57:10.142315+0000","last_deep_scrub":"62'1445","last_deep_scrub_stamp":"2024-09-11T00:57:10.142315+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.142315+0000","objects_scrubbed":2,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:23:39.221194+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":36861,"num_write":1416,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'2904","reported_seq":3012,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.611559+0000","last_change":"2024-09-11T00:56:57.358857+0000","last_active":"2024-09-11T00:57:34.611559+0000","last_peered":"2024-09-11T00:57:34.611559+0000","last_clean":"2024-09-11T00:57:34.611559+0000","last_became_active":"2024-09-11T00:30:54.996570+0000","last_became_peered":"2024-09-11T00:30:54.996570+0000","last_unstale":"2024-09-11T00:57:34.611559+0000","last_undegraded":"2024-09-11T00:57:34.611559+0000","last_fullsized":"2024-09-11T00:57:34.611559+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2904","last_scrub_stamp":"2024-09-11T00:56:57.358761+0000","last_deep_scrub":"62'2904","last_deep_scrub_stamp":"2024-09-11T00:56:57.358761+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.358761+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:57.371165+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15203,"num_write_kb":241147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"60'3390","reported_seq":3504,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.609859+0000","last_change":"2024-09-11T00:56:56.792784+0000","last_active":"2024-09-11T00:57:34.609859+0000","last_peered":"2024-09-11T00:57:34.609859+0000","last_clean":"2024-09-11T00:57:34.609859+0000","last_became_active":"2024-09-11T00:30:45.928480+0000","last_became_peered":"2024-09-11T00:30:45.928480+0000","last_unstale":"2024-09-11T00:57:34.609859+0000","last_undegraded":"2024-09-11T00:57:34.609859+0000","last_fullsized":"2024-09-11T00:57:34.609859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3390","last_scrub_stamp":"2024-09-11T00:56:56.792673+0000","last_deep_scrub":"60'3390","last_deep_scrub_stamp":"2024-09-11T00:56:56.792673+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.792673+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:21:25.347605+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17423,"num_write_kb":299173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"62'7","reported_seq":149,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:55.885497+0000","last_change":"2024-09-11T00:56:55.885497+0000","last_active":"2024-09-11T00:56:55.885497+0000","last_peered":"2024-09-11T00:56:55.885497+0000","last_clean":"2024-09-11T00:56:55.885497+0000","last_became_active":"2024-09-11T00:30:44.929558+0000","last_became_peered":"2024-09-11T00:30:44.929558+0000","last_unstale":"2024-09-11T00:56:55.885497+0000","last_undegraded":"2024-09-11T00:56:55.885497+0000","last_fullsized":"2024-09-11T00:56:55.885497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'7","last_scrub_stamp":"2024-09-11T00:56:55.885002+0000","last_deep_scrub":"62'7","last_deep_scrub_stamp":"2024-09-11T00:56:55.885002+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.885002+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:11.701806+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"62'3","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604223+0000","last_change":"2024-09-11T00:57:15.251849+0000","last_active":"2024-09-11T00:57:34.604223+0000","last_peered":"2024-09-11T00:57:34.604223+0000","last_clean":"2024-09-11T00:57:34.604223+0000","last_became_active":"2024-09-11T00:30:53.985217+0000","last_became_peered":"2024-09-11T00:30:53.985217+0000","last_unstale":"2024-09-11T00:57:34.604223+0000","last_undegraded":"2024-09-11T00:57:34.604223+0000","last_fullsized":"2024-09-11T00:57:34.604223+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3","last_scrub_stamp":"2024-09-11T00:57:15.251747+0000","last_deep_scrub":"62'3","last_deep_scrub_stamp":"2024-09-11T00:57:15.251747+0000","last_clean_scrub_stamp":"2024-09-11T00:57:15.251747+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:39.176639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"62'3177","reported_seq":3290,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.602839+0000","last_change":"2024-09-11T00:57:09.641995+0000","last_active":"2024-09-11T00:57:34.602839+0000","last_peered":"2024-09-11T00:57:34.602839+0000","last_clean":"2024-09-11T00:57:34.602839+0000","last_became_active":"2024-09-11T00:31:26.119641+0000","last_became_peered":"2024-09-11T00:31:26.119641+0000","last_unstale":"2024-09-11T00:57:34.602839+0000","last_undegraded":"2024-09-11T00:57:34.602839+0000","last_fullsized":"2024-09-11T00:57:34.602839+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3177","last_scrub_stamp":"2024-09-11T00:57:09.641903+0000","last_deep_scrub":"62'3177","last_deep_scrub_stamp":"2024-09-11T00:57:09.641903+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.641903+0000","objects_scrubbed":0,"log_size":3177,"log_dups_size":0,"ondisk_log_size":3177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:25:43.992297+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16730,"num_write_kb":276023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"60'2906","reported_seq":3020,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.617303+0000","last_change":"2024-09-11T00:56:55.163838+0000","last_active":"2024-09-11T00:57:34.617303+0000","last_peered":"2024-09-11T00:57:34.617303+0000","last_clean":"2024-09-11T00:57:34.617303+0000","last_became_active":"2024-09-11T00:30:45.923044+0000","last_became_peered":"2024-09-11T00:30:45.923044+0000","last_unstale":"2024-09-11T00:57:34.617303+0000","last_undegraded":"2024-09-11T00:57:34.617303+0000","last_fullsized":"2024-09-11T00:57:34.617303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2906","last_scrub_stamp":"2024-09-11T00:56:55.163688+0000","last_deep_scrub":"60'2906","last_deep_scrub_stamp":"2024-09-11T00:56:55.163688+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.163688+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:10.521002+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14712,"num_write_kb":255517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"62'6","reported_seq":137,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.610306+0000","last_change":"2024-09-11T00:56:51.684558+0000","last_active":"2024-09-11T00:57:34.610306+0000","last_peered":"2024-09-11T00:57:34.610306+0000","last_clean":"2024-09-11T00:57:34.610306+0000","last_became_active":"2024-09-11T00:30:44.927279+0000","last_became_peered":"2024-09-11T00:30:44.927279+0000","last_unstale":"2024-09-11T00:57:34.610306+0000","last_undegraded":"2024-09-11T00:57:34.610306+0000","last_fullsized":"2024-09-11T00:57:34.610306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:56:51.684468+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:56:51.684468+0000","last_clean_scrub_stamp":"2024-09-11T00:56:51.684468+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:55:43.127053+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"62'3","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.611969+0000","last_change":"2024-09-11T00:56:55.416696+0000","last_active":"2024-09-11T00:57:34.611969+0000","last_peered":"2024-09-11T00:57:34.611969+0000","last_clean":"2024-09-11T00:57:34.611969+0000","last_became_active":"2024-09-11T00:30:53.982802+0000","last_became_peered":"2024-09-11T00:30:53.982802+0000","last_unstale":"2024-09-11T00:57:34.611969+0000","last_undegraded":"2024-09-11T00:57:34.611969+0000","last_fullsized":"2024-09-11T00:57:34.611969+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3","last_scrub_stamp":"2024-09-11T00:56:55.416614+0000","last_deep_scrub":"62'3","last_deep_scrub_stamp":"2024-09-11T00:56:55.416614+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.416614+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:26:10.782676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"62'2973","reported_seq":3073,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.609779+0000","last_change":"2024-09-11T00:57:25.251427+0000","last_active":"2024-09-11T00:57:34.609779+0000","last_peered":"2024-09-11T00:57:34.609779+0000","last_clean":"2024-09-11T00:57:34.609779+0000","last_became_active":"2024-09-11T00:31:26.119440+0000","last_became_peered":"2024-09-11T00:31:26.119440+0000","last_unstale":"2024-09-11T00:57:34.609779+0000","last_undegraded":"2024-09-11T00:57:34.609779+0000","last_fullsized":"2024-09-11T00:57:34.609779+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2973","last_scrub_stamp":"2024-09-11T00:57:25.251349+0000","last_deep_scrub":"62'2973","last_deep_scrub_stamp":"2024-09-11T00:57:25.251349+0000","last_clean_scrub_stamp":"2024-09-11T00:57:25.251349+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:33:43.124760+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15629,"num_write_kb":242839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"60'3355","reported_seq":3469,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605238+0000","last_change":"2024-09-11T00:57:05.188297+0000","last_active":"2024-09-11T00:57:34.605238+0000","last_peered":"2024-09-11T00:57:34.605238+0000","last_clean":"2024-09-11T00:57:34.605238+0000","last_became_active":"2024-09-11T00:30:45.931043+0000","last_became_peered":"2024-09-11T00:30:45.931043+0000","last_unstale":"2024-09-11T00:57:34.605238+0000","last_undegraded":"2024-09-11T00:57:34.605238+0000","last_fullsized":"2024-09-11T00:57:34.605238+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3355","last_scrub_stamp":"2024-09-11T00:57:05.188189+0000","last_deep_scrub":"60'3355","last_deep_scrub_stamp":"2024-09-11T00:57:05.188189+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.188189+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:47.673004+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18048,"num_write_kb":289456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"62'16","reported_seq":189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:56.935214+0000","last_change":"2024-09-11T00:56:56.935214+0000","last_active":"2024-09-11T00:56:56.935214+0000","last_peered":"2024-09-11T00:56:56.935214+0000","last_clean":"2024-09-11T00:56:56.935214+0000","last_became_active":"2024-09-11T00:30:44.912746+0000","last_became_peered":"2024-09-11T00:30:44.912746+0000","last_unstale":"2024-09-11T00:56:56.935214+0000","last_undegraded":"2024-09-11T00:56:56.935214+0000","last_fullsized":"2024-09-11T00:56:56.935214+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2024-09-11T00:56:56.935120+0000","last_deep_scrub":"62'16","last_deep_scrub_stamp":"2024-09-11T00:56:56.935120+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.935120+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:37:57.590907+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":177,"num_read_kb":177,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"62'2","reported_seq":110,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607093+0000","last_change":"2024-09-11T00:57:13.148750+0000","last_active":"2024-09-11T00:57:34.607093+0000","last_peered":"2024-09-11T00:57:34.607093+0000","last_clean":"2024-09-11T00:57:34.607093+0000","last_became_active":"2024-09-11T00:30:53.980409+0000","last_became_peered":"2024-09-11T00:30:53.980409+0000","last_unstale":"2024-09-11T00:57:34.607093+0000","last_undegraded":"2024-09-11T00:57:34.607093+0000","last_fullsized":"2024-09-11T00:57:34.607093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2","last_scrub_stamp":"2024-09-11T00:57:13.148668+0000","last_deep_scrub":"62'2","last_deep_scrub_stamp":"2024-09-11T00:57:13.148668+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.148668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:58.225052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"62'2678","reported_seq":2785,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605364+0000","last_change":"2024-09-11T00:57:24.270189+0000","last_active":"2024-09-11T00:57:34.605364+0000","last_peered":"2024-09-11T00:57:34.605364+0000","last_clean":"2024-09-11T00:57:34.605364+0000","last_became_active":"2024-09-11T00:30:54.983487+0000","last_became_peered":"2024-09-11T00:30:54.983487+0000","last_unstale":"2024-09-11T00:57:34.605364+0000","last_undegraded":"2024-09-11T00:57:34.605364+0000","last_fullsized":"2024-09-11T00:57:34.605364+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2678","last_scrub_stamp":"2024-09-11T00:57:24.270093+0000","last_deep_scrub":"62'2678","last_deep_scrub_stamp":"2024-09-11T00:57:24.270093+0000","last_clean_scrub_stamp":"2024-09-11T00:57:24.270093+0000","objects_scrubbed":0,"log_size":2678,"log_dups_size":0,"ondisk_log_size":2678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:26:47.031268+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13518,"num_write_kb":242590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"60'2791","reported_seq":2905,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.612167+0000","last_change":"2024-09-11T00:56:53.489186+0000","last_active":"2024-09-11T00:57:34.612167+0000","last_peered":"2024-09-11T00:57:34.612167+0000","last_clean":"2024-09-11T00:57:34.612167+0000","last_became_active":"2024-09-11T00:30:45.935517+0000","last_became_peered":"2024-09-11T00:30:45.935517+0000","last_unstale":"2024-09-11T00:57:34.612167+0000","last_undegraded":"2024-09-11T00:57:34.612167+0000","last_fullsized":"2024-09-11T00:57:34.612167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2791","last_scrub_stamp":"2024-09-11T00:56:53.489113+0000","last_deep_scrub":"60'2791","last_deep_scrub_stamp":"2024-09-11T00:56:53.489113+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.489113+0000","objects_scrubbed":0,"log_size":2791,"log_dups_size":0,"ondisk_log_size":2791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:13.547085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":254542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"60'6","reported_seq":122,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.608618+0000","last_change":"2024-09-11T00:56:52.719391+0000","last_active":"2024-09-11T00:57:34.608618+0000","last_peered":"2024-09-11T00:57:34.608618+0000","last_clean":"2024-09-11T00:57:34.608618+0000","last_became_active":"2024-09-11T00:30:44.923817+0000","last_became_peered":"2024-09-11T00:30:44.923817+0000","last_unstale":"2024-09-11T00:57:34.608618+0000","last_undegraded":"2024-09-11T00:57:34.608618+0000","last_fullsized":"2024-09-11T00:57:34.608618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'6","last_scrub_stamp":"2024-09-11T00:56:52.719331+0000","last_deep_scrub":"60'6","last_deep_scrub_stamp":"2024-09-11T00:56:52.719331+0000","last_clean_scrub_stamp":"2024-09-11T00:56:52.719331+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:18.018393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"62'3","reported_seq":111,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606681+0000","last_change":"2024-09-11T00:57:12.140217+0000","last_active":"2024-09-11T00:57:34.606681+0000","last_peered":"2024-09-11T00:57:34.606681+0000","last_clean":"2024-09-11T00:57:34.606681+0000","last_became_active":"2024-09-11T00:30:53.987474+0000","last_became_peered":"2024-09-11T00:30:53.987474+0000","last_unstale":"2024-09-11T00:57:34.606681+0000","last_undegraded":"2024-09-11T00:57:34.606681+0000","last_fullsized":"2024-09-11T00:57:34.606681+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3","last_scrub_stamp":"2024-09-11T00:57:12.140125+0000","last_deep_scrub":"62'3","last_deep_scrub_stamp":"2024-09-11T00:57:12.140125+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.140125+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:17:12.627169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"62'3272","reported_seq":3378,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605540+0000","last_change":"2024-09-11T00:57:23.225032+0000","last_active":"2024-09-11T00:57:34.605540+0000","last_peered":"2024-09-11T00:57:34.605540+0000","last_clean":"2024-09-11T00:57:34.605540+0000","last_became_active":"2024-09-11T00:30:54.999302+0000","last_became_peered":"2024-09-11T00:30:54.999302+0000","last_unstale":"2024-09-11T00:57:34.605540+0000","last_undegraded":"2024-09-11T00:57:34.605540+0000","last_fullsized":"2024-09-11T00:57:34.605540+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3272","last_scrub_stamp":"2024-09-11T00:57:23.224940+0000","last_deep_scrub":"62'3272","last_deep_scrub_stamp":"2024-09-11T00:57:23.224940+0000","last_clean_scrub_stamp":"2024-09-11T00:57:23.224940+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:23.922632+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16692,"num_write_kb":275642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"60'3405","reported_seq":3521,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604872+0000","last_change":"2024-09-11T00:57:06.186619+0000","last_active":"2024-09-11T00:57:34.604872+0000","last_peered":"2024-09-11T00:57:34.604872+0000","last_clean":"2024-09-11T00:57:34.604872+0000","last_became_active":"2024-09-11T00:30:45.922658+0000","last_became_peered":"2024-09-11T00:30:45.922658+0000","last_unstale":"2024-09-11T00:57:34.604872+0000","last_undegraded":"2024-09-11T00:57:34.604872+0000","last_fullsized":"2024-09-11T00:57:34.604872+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3405","last_scrub_stamp":"2024-09-11T00:57:06.186523+0000","last_deep_scrub":"60'3405","last_deep_scrub_stamp":"2024-09-11T00:57:06.186523+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.186523+0000","objects_scrubbed":0,"log_size":3405,"log_dups_size":0,"ondisk_log_size":3405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:42.910335+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":17487,"num_write_kb":315618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"60'5","reported_seq":121,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606276+0000","last_change":"2024-09-11T00:56:59.342382+0000","last_active":"2024-09-11T00:57:34.606276+0000","last_peered":"2024-09-11T00:57:34.606276+0000","last_clean":"2024-09-11T00:57:34.606276+0000","last_became_active":"2024-09-11T00:30:44.919246+0000","last_became_peered":"2024-09-11T00:30:44.919246+0000","last_unstale":"2024-09-11T00:57:34.606276+0000","last_undegraded":"2024-09-11T00:57:34.606276+0000","last_fullsized":"2024-09-11T00:57:34.606276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'5","last_scrub_stamp":"2024-09-11T00:56:59.342292+0000","last_deep_scrub":"60'5","last_deep_scrub_stamp":"2024-09-11T00:56:59.342292+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.342292+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:16.379697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"62'371","reported_seq":586,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604533+0000","last_change":"2024-09-11T00:57:08.679836+0000","last_active":"2024-09-11T00:57:34.604533+0000","last_peered":"2024-09-11T00:57:34.604533+0000","last_clean":"2024-09-11T00:57:34.604533+0000","last_became_active":"2024-09-11T00:30:53.983774+0000","last_became_peered":"2024-09-11T00:30:53.983774+0000","last_unstale":"2024-09-11T00:57:34.604533+0000","last_undegraded":"2024-09-11T00:57:34.604533+0000","last_fullsized":"2024-09-11T00:57:34.604533+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'371","last_scrub_stamp":"2024-09-11T00:57:08.679729+0000","last_deep_scrub":"62'371","last_deep_scrub_stamp":"2024-09-11T00:57:08.679729+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.679729+0000","objects_scrubbed":4,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:46.083044+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":191,"num_read_kb":78004,"num_write":360,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"62'4540","reported_seq":4645,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:06.776106+0000","last_change":"2024-09-11T00:57:06.776106+0000","last_active":"2024-09-11T00:57:06.776106+0000","last_peered":"2024-09-11T00:57:06.776106+0000","last_clean":"2024-09-11T00:57:06.776106+0000","last_became_active":"2024-09-11T00:30:54.989371+0000","last_became_peered":"2024-09-11T00:30:54.989371+0000","last_unstale":"2024-09-11T00:57:06.776106+0000","last_undegraded":"2024-09-11T00:57:06.776106+0000","last_fullsized":"2024-09-11T00:57:06.776106+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4540","last_scrub_stamp":"2024-09-11T00:57:06.776012+0000","last_deep_scrub":"62'4540","last_deep_scrub_stamp":"2024-09-11T00:57:06.776012+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.776012+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:58:40.459947+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23424,"num_write_kb":360436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"60'2652","reported_seq":2767,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605475+0000","last_change":"2024-09-11T00:56:57.848088+0000","last_active":"2024-09-11T00:57:34.605475+0000","last_peered":"2024-09-11T00:57:34.605475+0000","last_clean":"2024-09-11T00:57:34.605475+0000","last_became_active":"2024-09-11T00:30:45.914458+0000","last_became_peered":"2024-09-11T00:30:45.914458+0000","last_unstale":"2024-09-11T00:57:34.605475+0000","last_undegraded":"2024-09-11T00:57:34.605475+0000","last_fullsized":"2024-09-11T00:57:34.605475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2652","last_scrub_stamp":"2024-09-11T00:56:57.847995+0000","last_deep_scrub":"60'2652","last_deep_scrub_stamp":"2024-09-11T00:56:57.847995+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.847995+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:35.515513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":13591,"num_write_kb":238653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"62'11","reported_seq":153,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:57.220239+0000","last_change":"2024-09-11T00:56:57.220239+0000","last_active":"2024-09-11T00:56:57.220239+0000","last_peered":"2024-09-11T00:56:57.220239+0000","last_clean":"2024-09-11T00:56:57.220239+0000","last_became_active":"2024-09-11T00:30:44.919720+0000","last_became_peered":"2024-09-11T00:30:44.919720+0000","last_unstale":"2024-09-11T00:56:57.220239+0000","last_undegraded":"2024-09-11T00:56:57.220239+0000","last_fullsized":"2024-09-11T00:56:57.220239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-09-11T00:56:57.220143+0000","last_deep_scrub":"62'11","last_deep_scrub_stamp":"2024-09-11T00:56:57.220143+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.220143+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:20.830740+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":112,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"62'39","reported_seq":210,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606738+0000","last_change":"2024-09-11T00:57:17.319224+0000","last_active":"2024-09-11T00:57:34.606738+0000","last_peered":"2024-09-11T00:57:34.606738+0000","last_clean":"2024-09-11T00:57:34.606738+0000","last_became_active":"2024-09-11T00:30:53.981824+0000","last_became_peered":"2024-09-11T00:30:53.981824+0000","last_unstale":"2024-09-11T00:57:34.606738+0000","last_undegraded":"2024-09-11T00:57:34.606738+0000","last_fullsized":"2024-09-11T00:57:34.606738+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'39","last_scrub_stamp":"2024-09-11T00:57:17.319113+0000","last_deep_scrub":"62'39","last_deep_scrub_stamp":"2024-09-11T00:57:17.319113+0000","last_clean_scrub_stamp":"2024-09-11T00:57:17.319113+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:08.728809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":36,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"62'4259","reported_seq":4364,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:15.823674+0000","last_change":"2024-09-11T00:57:15.823674+0000","last_active":"2024-09-11T00:57:15.823674+0000","last_peered":"2024-09-11T00:57:15.823674+0000","last_clean":"2024-09-11T00:57:15.823674+0000","last_became_active":"2024-09-11T00:30:54.988151+0000","last_became_peered":"2024-09-11T00:30:54.988151+0000","last_unstale":"2024-09-11T00:57:15.823674+0000","last_undegraded":"2024-09-11T00:57:15.823674+0000","last_fullsized":"2024-09-11T00:57:15.823674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4259","last_scrub_stamp":"2024-09-11T00:57:15.823579+0000","last_deep_scrub":"62'4259","last_deep_scrub_stamp":"2024-09-11T00:57:15.823579+0000","last_clean_scrub_stamp":"2024-09-11T00:57:15.823579+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:55:17.708669+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22192,"num_write_kb":330506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"60'2765","reported_seq":2879,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.609137+0000","last_change":"2024-09-11T00:56:55.743908+0000","last_active":"2024-09-11T00:57:34.609137+0000","last_peered":"2024-09-11T00:57:34.609137+0000","last_clean":"2024-09-11T00:57:34.609137+0000","last_became_active":"2024-09-11T00:30:45.927178+0000","last_became_peered":"2024-09-11T00:30:45.927178+0000","last_unstale":"2024-09-11T00:57:34.609137+0000","last_undegraded":"2024-09-11T00:57:34.609137+0000","last_fullsized":"2024-09-11T00:57:34.609137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2765","last_scrub_stamp":"2024-09-11T00:56:55.743832+0000","last_deep_scrub":"60'2765","last_deep_scrub_stamp":"2024-09-11T00:56:55.743832+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.743832+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:34:46.345457+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14757,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"62'392","reported_seq":582,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.618847+0000","last_change":"2024-09-11T00:56:53.146103+0000","last_active":"2024-09-11T00:57:34.618847+0000","last_peered":"2024-09-11T00:57:34.618847+0000","last_clean":"2024-09-11T00:57:34.618847+0000","last_became_active":"2024-09-11T00:30:44.923023+0000","last_became_peered":"2024-09-11T00:30:44.923023+0000","last_unstale":"2024-09-11T00:57:34.618847+0000","last_undegraded":"2024-09-11T00:57:34.618847+0000","last_fullsized":"2024-09-11T00:57:34.618847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'392","last_scrub_stamp":"2024-09-11T00:56:53.146012+0000","last_deep_scrub":"62'392","last_deep_scrub_stamp":"2024-09-11T00:56:53.146012+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.146012+0000","objects_scrubbed":4,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:09:09.764335+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":60350,"num_write":381,"num_write_kb":4290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"62'5","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:13.739976+0000","last_change":"2024-09-11T00:57:13.739976+0000","last_active":"2024-09-11T00:57:13.739976+0000","last_peered":"2024-09-11T00:57:13.739976+0000","last_clean":"2024-09-11T00:57:13.739976+0000","last_became_active":"2024-09-11T00:30:53.986747+0000","last_became_peered":"2024-09-11T00:30:53.986747+0000","last_unstale":"2024-09-11T00:57:13.739976+0000","last_undegraded":"2024-09-11T00:57:13.739976+0000","last_fullsized":"2024-09-11T00:57:13.739976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2024-09-11T00:57:13.739888+0000","last_deep_scrub":"62'5","last_deep_scrub_stamp":"2024-09-11T00:57:13.739888+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.739888+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:13:54.348922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"62'3220","reported_seq":3326,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604394+0000","last_change":"2024-09-11T00:57:11.666287+0000","last_active":"2024-09-11T00:57:34.604394+0000","last_peered":"2024-09-11T00:57:34.604394+0000","last_clean":"2024-09-11T00:57:34.604394+0000","last_became_active":"2024-09-11T00:30:54.985694+0000","last_became_peered":"2024-09-11T00:30:54.985694+0000","last_unstale":"2024-09-11T00:57:34.604394+0000","last_undegraded":"2024-09-11T00:57:34.604394+0000","last_fullsized":"2024-09-11T00:57:34.604394+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3220","last_scrub_stamp":"2024-09-11T00:57:11.666200+0000","last_deep_scrub":"62'3220","last_deep_scrub_stamp":"2024-09-11T00:57:11.666200+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.666200+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:36:50.377437+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16450,"num_write_kb":262397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"60'3928","reported_seq":4043,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605222+0000","last_change":"2024-09-11T00:57:08.260011+0000","last_active":"2024-09-11T00:57:34.605222+0000","last_peered":"2024-09-11T00:57:34.605222+0000","last_clean":"2024-09-11T00:57:34.605222+0000","last_became_active":"2024-09-11T00:30:45.940898+0000","last_became_peered":"2024-09-11T00:30:45.940898+0000","last_unstale":"2024-09-11T00:57:34.605222+0000","last_undegraded":"2024-09-11T00:57:34.605222+0000","last_fullsized":"2024-09-11T00:57:34.605222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3928","last_scrub_stamp":"2024-09-11T00:57:08.259909+0000","last_deep_scrub":"60'3928","last_deep_scrub_stamp":"2024-09-11T00:57:08.259909+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.259909+0000","objects_scrubbed":0,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:40:51.556710+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20599,"num_write_kb":340888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"62'44","reported_seq":200,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:55.186964+0000","last_change":"2024-09-11T00:56:55.186907+0000","last_active":"2024-09-11T00:56:55.186964+0000","last_peered":"2024-09-11T00:56:55.186964+0000","last_clean":"2024-09-11T00:56:55.186964+0000","last_became_active":"2024-09-11T00:30:44.917260+0000","last_became_peered":"2024-09-11T00:30:44.917260+0000","last_unstale":"2024-09-11T00:56:55.186964+0000","last_undegraded":"2024-09-11T00:56:55.186964+0000","last_fullsized":"2024-09-11T00:56:55.186964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'44","last_scrub_stamp":"2024-09-11T00:56:55.186811+0000","last_deep_scrub":"62'44","last_deep_scrub_stamp":"2024-09-11T00:56:55.186811+0000","last_clean_scrub_stamp":"2024-09-11T00:56:55.186811+0000","objects_scrubbed":2,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:25:04.079708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":42,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"62'10","reported_seq":160,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616302+0000","last_change":"2024-09-11T00:57:04.217240+0000","last_active":"2024-09-11T00:57:34.616302+0000","last_peered":"2024-09-11T00:57:34.616302+0000","last_clean":"2024-09-11T00:57:34.616302+0000","last_became_active":"2024-09-11T00:30:53.988237+0000","last_became_peered":"2024-09-11T00:30:53.988237+0000","last_unstale":"2024-09-11T00:57:34.616302+0000","last_undegraded":"2024-09-11T00:57:34.616302+0000","last_fullsized":"2024-09-11T00:57:34.616302+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-09-11T00:57:04.217171+0000","last_deep_scrub":"62'10","last_deep_scrub_stamp":"2024-09-11T00:57:04.217171+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.217171+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:47:46.011379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"62'4111","reported_seq":4217,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604810+0000","last_change":"2024-09-11T00:57:22.262654+0000","last_active":"2024-09-11T00:57:34.604810+0000","last_peered":"2024-09-11T00:57:34.604810+0000","last_clean":"2024-09-11T00:57:34.604810+0000","last_became_active":"2024-09-11T00:30:54.988863+0000","last_became_peered":"2024-09-11T00:30:54.988863+0000","last_unstale":"2024-09-11T00:57:34.604810+0000","last_undegraded":"2024-09-11T00:57:34.604810+0000","last_fullsized":"2024-09-11T00:57:34.604810+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4111","last_scrub_stamp":"2024-09-11T00:57:22.262573+0000","last_deep_scrub":"62'4111","last_deep_scrub_stamp":"2024-09-11T00:57:22.262573+0000","last_clean_scrub_stamp":"2024-09-11T00:57:22.262573+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:17:36.535351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21746,"num_write_kb":328824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"60'2863","reported_seq":2983,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607492+0000","last_change":"2024-09-11T00:56:59.109857+0000","last_active":"2024-09-11T00:57:34.607492+0000","last_peered":"2024-09-11T00:57:34.607492+0000","last_clean":"2024-09-11T00:57:34.607492+0000","last_became_active":"2024-09-11T00:30:45.943843+0000","last_became_peered":"2024-09-11T00:30:45.943843+0000","last_unstale":"2024-09-11T00:57:34.607492+0000","last_undegraded":"2024-09-11T00:57:34.607492+0000","last_fullsized":"2024-09-11T00:57:34.607492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2863","last_scrub_stamp":"2024-09-11T00:56:59.109727+0000","last_deep_scrub":"60'2863","last_deep_scrub_stamp":"2024-09-11T00:56:59.109727+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.109727+0000","objects_scrubbed":0,"log_size":2863,"log_dups_size":0,"ondisk_log_size":2863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:59:01.225049+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":46,"num_write":14990,"num_write_kb":244331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"60'2","reported_seq":118,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607640+0000","last_change":"2024-09-11T00:56:53.729230+0000","last_active":"2024-09-11T00:57:34.607640+0000","last_peered":"2024-09-11T00:57:34.607640+0000","last_clean":"2024-09-11T00:57:34.607640+0000","last_became_active":"2024-09-11T00:30:44.911696+0000","last_became_peered":"2024-09-11T00:30:44.911696+0000","last_unstale":"2024-09-11T00:57:34.607640+0000","last_undegraded":"2024-09-11T00:57:34.607640+0000","last_fullsized":"2024-09-11T00:57:34.607640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2","last_scrub_stamp":"2024-09-11T00:56:53.729158+0000","last_deep_scrub":"60'2","last_deep_scrub_stamp":"2024-09-11T00:56:53.729158+0000","last_clean_scrub_stamp":"2024-09-11T00:56:53.729158+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:23:06.778420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"62'16","reported_seq":215,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:02.192950+0000","last_change":"2024-09-11T00:57:02.192950+0000","last_active":"2024-09-11T00:57:02.192950+0000","last_peered":"2024-09-11T00:57:02.192950+0000","last_clean":"2024-09-11T00:57:02.192950+0000","last_became_active":"2024-09-11T00:30:53.986354+0000","last_became_peered":"2024-09-11T00:30:53.986354+0000","last_unstale":"2024-09-11T00:57:02.192950+0000","last_undegraded":"2024-09-11T00:57:02.192950+0000","last_fullsized":"2024-09-11T00:57:02.192950+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'16","last_scrub_stamp":"2024-09-11T00:57:02.192854+0000","last_deep_scrub":"62'16","last_deep_scrub_stamp":"2024-09-11T00:57:02.192854+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.192854+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:58:47.360278+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":279,"num_read_kb":279,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"62'3641","reported_seq":3747,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.615519+0000","last_change":"2024-09-11T00:57:09.114247+0000","last_active":"2024-09-11T00:57:34.615519+0000","last_peered":"2024-09-11T00:57:34.615519+0000","last_clean":"2024-09-11T00:57:34.615519+0000","last_became_active":"2024-09-11T00:30:54.982212+0000","last_became_peered":"2024-09-11T00:30:54.982212+0000","last_unstale":"2024-09-11T00:57:34.615519+0000","last_undegraded":"2024-09-11T00:57:34.615519+0000","last_fullsized":"2024-09-11T00:57:34.615519+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3641","last_scrub_stamp":"2024-09-11T00:57:09.114139+0000","last_deep_scrub":"62'3641","last_deep_scrub_stamp":"2024-09-11T00:57:09.114139+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.114139+0000","objects_scrubbed":0,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:03:56.218479+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19470,"num_write_kb":282742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"60'2896","reported_seq":3014,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616904+0000","last_change":"2024-09-11T00:56:57.096342+0000","last_active":"2024-09-11T00:57:34.616904+0000","last_peered":"2024-09-11T00:57:34.616904+0000","last_clean":"2024-09-11T00:57:34.616904+0000","last_became_active":"2024-09-11T00:30:45.944510+0000","last_became_peered":"2024-09-11T00:30:45.944510+0000","last_unstale":"2024-09-11T00:57:34.616904+0000","last_undegraded":"2024-09-11T00:57:34.616904+0000","last_fullsized":"2024-09-11T00:57:34.616904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2896","last_scrub_stamp":"2024-09-11T00:56:57.096241+0000","last_deep_scrub":"60'2896","last_deep_scrub_stamp":"2024-09-11T00:56:57.096241+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.096241+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:40.956202+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":22,"num_write":14816,"num_write_kb":255894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"62'5","reported_seq":121,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.612683+0000","last_change":"2024-09-11T00:56:51.508473+0000","last_active":"2024-09-11T00:57:34.612683+0000","last_peered":"2024-09-11T00:57:34.612683+0000","last_clean":"2024-09-11T00:57:34.612683+0000","last_became_active":"2024-09-11T00:30:44.913379+0000","last_became_peered":"2024-09-11T00:30:44.913379+0000","last_unstale":"2024-09-11T00:57:34.612683+0000","last_undegraded":"2024-09-11T00:57:34.612683+0000","last_fullsized":"2024-09-11T00:57:34.612683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2024-09-11T00:56:51.508373+0000","last_deep_scrub":"62'5","last_deep_scrub_stamp":"2024-09-11T00:56:51.508373+0000","last_clean_scrub_stamp":"2024-09-11T00:56:51.508373+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:38:34.105569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"62'6","reported_seq":112,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:01.907184+0000","last_change":"2024-09-11T00:57:01.907184+0000","last_active":"2024-09-11T00:57:01.907184+0000","last_peered":"2024-09-11T00:57:01.907184+0000","last_clean":"2024-09-11T00:57:01.907184+0000","last_became_active":"2024-09-11T00:30:53.981133+0000","last_became_peered":"2024-09-11T00:30:53.981133+0000","last_unstale":"2024-09-11T00:57:01.907184+0000","last_undegraded":"2024-09-11T00:57:01.907184+0000","last_fullsized":"2024-09-11T00:57:01.907184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'6","last_scrub_stamp":"2024-09-11T00:57:01.907071+0000","last_deep_scrub":"62'6","last_deep_scrub_stamp":"2024-09-11T00:57:01.907071+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.907071+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:30.767604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"62'3862","reported_seq":3971,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.617491+0000","last_change":"2024-09-11T00:57:11.152893+0000","last_active":"2024-09-11T00:57:34.617491+0000","last_peered":"2024-09-11T00:57:34.617491+0000","last_clean":"2024-09-11T00:57:34.617491+0000","last_became_active":"2024-09-11T00:30:54.984525+0000","last_became_peered":"2024-09-11T00:30:54.984525+0000","last_unstale":"2024-09-11T00:57:34.617491+0000","last_undegraded":"2024-09-11T00:57:34.617491+0000","last_fullsized":"2024-09-11T00:57:34.617491+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3862","last_scrub_stamp":"2024-09-11T00:57:11.152457+0000","last_deep_scrub":"62'3862","last_deep_scrub_stamp":"2024-09-11T00:57:11.152457+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.152457+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:47:28.632709+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20250,"num_write_kb":318736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"62'29","reported_seq":137,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:17.795231+0000","last_change":"2024-09-11T00:57:17.795231+0000","last_active":"2024-09-11T00:57:17.795231+0000","last_peered":"2024-09-11T00:57:17.795231+0000","last_clean":"2024-09-11T00:57:17.795231+0000","last_became_active":"2024-09-11T00:30:53.982610+0000","last_became_peered":"2024-09-11T00:30:53.982610+0000","last_unstale":"2024-09-11T00:57:17.795231+0000","last_undegraded":"2024-09-11T00:57:17.795231+0000","last_fullsized":"2024-09-11T00:57:17.795231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'29","last_scrub_stamp":"2024-09-11T00:57:17.795144+0000","last_deep_scrub":"62'29","last_deep_scrub_stamp":"2024-09-11T00:57:17.795144+0000","last_clean_scrub_stamp":"2024-09-11T00:57:17.795144+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:51:35.523384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"60'3526","reported_seq":3639,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:58.901714+0000","last_change":"2024-09-11T00:56:58.901661+0000","last_active":"2024-09-11T00:56:58.901714+0000","last_peered":"2024-09-11T00:56:58.901714+0000","last_clean":"2024-09-11T00:56:58.901714+0000","last_became_active":"2024-09-11T00:30:45.913606+0000","last_became_peered":"2024-09-11T00:30:45.913606+0000","last_unstale":"2024-09-11T00:56:58.901714+0000","last_undegraded":"2024-09-11T00:56:58.901714+0000","last_fullsized":"2024-09-11T00:56:58.901714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3526","last_scrub_stamp":"2024-09-11T00:56:58.901567+0000","last_deep_scrub":"60'3526","last_deep_scrub_stamp":"2024-09-11T00:56:58.901567+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.901567+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:42:44.166766+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":18407,"num_write_kb":311214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"60'3","reported_seq":119,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604677+0000","last_change":"2024-09-11T00:57:00.333194+0000","last_active":"2024-09-11T00:57:34.604677+0000","last_peered":"2024-09-11T00:57:34.604677+0000","last_clean":"2024-09-11T00:57:34.604677+0000","last_became_active":"2024-09-11T00:30:44.919606+0000","last_became_peered":"2024-09-11T00:30:44.919606+0000","last_unstale":"2024-09-11T00:57:34.604677+0000","last_undegraded":"2024-09-11T00:57:34.604677+0000","last_fullsized":"2024-09-11T00:57:34.604677+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3","last_scrub_stamp":"2024-09-11T00:57:00.333106+0000","last_deep_scrub":"60'3","last_deep_scrub_stamp":"2024-09-11T00:57:00.333106+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.333106+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:05:56.262921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"61'3026","reported_seq":3134,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605451+0000","last_change":"2024-09-11T00:57:15.584196+0000","last_active":"2024-09-11T00:57:34.605451+0000","last_peered":"2024-09-11T00:57:34.605451+0000","last_clean":"2024-09-11T00:57:34.605451+0000","last_became_active":"2024-09-11T00:30:54.978675+0000","last_became_peered":"2024-09-11T00:30:54.978675+0000","last_unstale":"2024-09-11T00:57:34.605451+0000","last_undegraded":"2024-09-11T00:57:34.605451+0000","last_fullsized":"2024-09-11T00:57:34.605451+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3026","last_scrub_stamp":"2024-09-11T00:57:15.584056+0000","last_deep_scrub":"61'3026","last_deep_scrub_stamp":"2024-09-11T00:57:15.584056+0000","last_clean_scrub_stamp":"2024-09-11T00:57:15.584056+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:11:17.797042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16026,"num_write_kb":256702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"62'975","reported_seq":1135,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.609542+0000","last_change":"2024-09-11T00:57:18.321461+0000","last_active":"2024-09-11T00:57:34.609542+0000","last_peered":"2024-09-11T00:57:34.609542+0000","last_clean":"2024-09-11T00:57:34.609542+0000","last_became_active":"2024-09-11T00:30:53.971829+0000","last_became_peered":"2024-09-11T00:30:53.971829+0000","last_unstale":"2024-09-11T00:57:34.609542+0000","last_undegraded":"2024-09-11T00:57:34.609542+0000","last_fullsized":"2024-09-11T00:57:34.609542+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'975","last_scrub_stamp":"2024-09-11T00:57:18.321373+0000","last_deep_scrub":"62'975","last_deep_scrub_stamp":"2024-09-11T00:57:18.321373+0000","last_clean_scrub_stamp":"2024-09-11T00:57:18.321373+0000","objects_scrubbed":3,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:39.111938+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":7829,"num_write":949,"num_write_kb":4657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"60'3393","reported_seq":3509,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604797+0000","last_change":"2024-09-11T00:57:00.073349+0000","last_active":"2024-09-11T00:57:34.604797+0000","last_peered":"2024-09-11T00:57:34.604797+0000","last_clean":"2024-09-11T00:57:34.604797+0000","last_became_active":"2024-09-11T00:30:45.931706+0000","last_became_peered":"2024-09-11T00:30:45.931706+0000","last_unstale":"2024-09-11T00:57:34.604797+0000","last_undegraded":"2024-09-11T00:57:34.604797+0000","last_fullsized":"2024-09-11T00:57:34.604797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3393","last_scrub_stamp":"2024-09-11T00:57:00.073221+0000","last_deep_scrub":"60'3393","last_deep_scrub_stamp":"2024-09-11T00:57:00.073221+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.073221+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:02:30.921492+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":17621,"num_write_kb":291741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"60'3","reported_seq":119,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.603259+0000","last_change":"2024-09-11T00:56:54.888596+0000","last_active":"2024-09-11T00:57:34.603259+0000","last_peered":"2024-09-11T00:57:34.603259+0000","last_clean":"2024-09-11T00:57:34.603259+0000","last_became_active":"2024-09-11T00:30:44.914107+0000","last_became_peered":"2024-09-11T00:30:44.914107+0000","last_unstale":"2024-09-11T00:57:34.603259+0000","last_undegraded":"2024-09-11T00:57:34.603259+0000","last_fullsized":"2024-09-11T00:57:34.603259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3","last_scrub_stamp":"2024-09-11T00:56:54.888498+0000","last_deep_scrub":"60'3","last_deep_scrub_stamp":"2024-09-11T00:56:54.888498+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.888498+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:35:49.517568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"62'3945","reported_seq":4048,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:09.826760+0000","last_change":"2024-09-11T00:57:09.826760+0000","last_active":"2024-09-11T00:57:09.826760+0000","last_peered":"2024-09-11T00:57:09.826760+0000","last_clean":"2024-09-11T00:57:09.826760+0000","last_became_active":"2024-09-11T00:30:54.997409+0000","last_became_peered":"2024-09-11T00:30:54.997409+0000","last_unstale":"2024-09-11T00:57:09.826760+0000","last_undegraded":"2024-09-11T00:57:09.826760+0000","last_fullsized":"2024-09-11T00:57:09.826760+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3945","last_scrub_stamp":"2024-09-11T00:57:09.826667+0000","last_deep_scrub":"62'3945","last_deep_scrub_stamp":"2024-09-11T00:57:09.826667+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.826667+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:24:55.731572+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20523,"num_write_kb":340217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"62'101","reported_seq":220,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.618101+0000","last_change":"2024-09-11T00:57:01.372311+0000","last_active":"2024-09-11T00:57:34.618101+0000","last_peered":"2024-09-11T00:57:34.618101+0000","last_clean":"2024-09-11T00:57:34.618101+0000","last_became_active":"2024-09-11T00:30:44.927069+0000","last_became_peered":"2024-09-11T00:30:44.927069+0000","last_unstale":"2024-09-11T00:57:34.618101+0000","last_undegraded":"2024-09-11T00:57:34.618101+0000","last_fullsized":"2024-09-11T00:57:34.618101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'101","last_scrub_stamp":"2024-09-11T00:57:01.372215+0000","last_deep_scrub":"62'101","last_deep_scrub_stamp":"2024-09-11T00:57:01.372215+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.372215+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:23:56.553328+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":768543,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1479,"num_write":83,"num_write_kb":803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"60'2411","reported_seq":2524,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:04.862489+0000","last_change":"2024-09-11T00:57:04.862489+0000","last_active":"2024-09-11T00:57:04.862489+0000","last_peered":"2024-09-11T00:57:04.862489+0000","last_clean":"2024-09-11T00:57:04.862489+0000","last_became_active":"2024-09-11T00:30:45.919432+0000","last_became_peered":"2024-09-11T00:30:45.919432+0000","last_unstale":"2024-09-11T00:57:04.862489+0000","last_undegraded":"2024-09-11T00:57:04.862489+0000","last_fullsized":"2024-09-11T00:57:04.862489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2411","last_scrub_stamp":"2024-09-11T00:57:04.862400+0000","last_deep_scrub":"60'2411","last_deep_scrub_stamp":"2024-09-11T00:57:04.862400+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.862400+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:46:44.628630+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":8,"num_write":12211,"num_write_kb":212619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"62'4","reported_seq":151,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:03.872845+0000","last_change":"2024-09-11T00:57:03.872845+0000","last_active":"2024-09-11T00:57:03.872845+0000","last_peered":"2024-09-11T00:57:03.872845+0000","last_clean":"2024-09-11T00:57:03.872845+0000","last_became_active":"2024-09-11T00:30:53.982965+0000","last_became_peered":"2024-09-11T00:30:53.982965+0000","last_unstale":"2024-09-11T00:57:03.872845+0000","last_undegraded":"2024-09-11T00:57:03.872845+0000","last_fullsized":"2024-09-11T00:57:03.872845+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4","last_scrub_stamp":"2024-09-11T00:57:03.872751+0000","last_deep_scrub":"62'4","last_deep_scrub_stamp":"2024-09-11T00:57:03.872751+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.872751+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:22:30.452252+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"62'2354","reported_seq":2459,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616773+0000","last_change":"2024-09-11T00:57:10.117938+0000","last_active":"2024-09-11T00:57:34.616773+0000","last_peered":"2024-09-11T00:57:34.616773+0000","last_clean":"2024-09-11T00:57:34.616773+0000","last_became_active":"2024-09-11T00:30:54.998133+0000","last_became_peered":"2024-09-11T00:30:54.998133+0000","last_unstale":"2024-09-11T00:57:34.616773+0000","last_undegraded":"2024-09-11T00:57:34.616773+0000","last_fullsized":"2024-09-11T00:57:34.616773+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2354","last_scrub_stamp":"2024-09-11T00:57:10.117848+0000","last_deep_scrub":"62'2354","last_deep_scrub_stamp":"2024-09-11T00:57:10.117848+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.117848+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:25:12.751798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":205009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"62'677","reported_seq":809,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606744+0000","last_change":"2024-09-11T00:56:57.175764+0000","last_active":"2024-09-11T00:57:34.606744+0000","last_peered":"2024-09-11T00:57:34.606744+0000","last_clean":"2024-09-11T00:57:34.606744+0000","last_became_active":"2024-09-11T00:30:44.924102+0000","last_became_peered":"2024-09-11T00:30:44.924102+0000","last_unstale":"2024-09-11T00:57:34.606744+0000","last_undegraded":"2024-09-11T00:57:34.606744+0000","last_fullsized":"2024-09-11T00:57:34.606744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'677","last_scrub_stamp":"2024-09-11T00:56:57.175675+0000","last_deep_scrub":"62'677","last_deep_scrub_stamp":"2024-09-11T00:56:57.175675+0000","last_clean_scrub_stamp":"2024-09-11T00:56:57.175675+0000","objects_scrubbed":1,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:14:07.621591+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":55539,"num_write":670,"num_write_kb":4291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"60'2936","reported_seq":3049,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.612772+0000","last_change":"2024-09-11T00:56:52.479641+0000","last_active":"2024-09-11T00:57:34.612772+0000","last_peered":"2024-09-11T00:57:34.612772+0000","last_clean":"2024-09-11T00:57:34.612772+0000","last_became_active":"2024-09-11T00:30:45.935557+0000","last_became_peered":"2024-09-11T00:30:45.935557+0000","last_unstale":"2024-09-11T00:57:34.612772+0000","last_undegraded":"2024-09-11T00:57:34.612772+0000","last_fullsized":"2024-09-11T00:57:34.612772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2936","last_scrub_stamp":"2024-09-11T00:56:52.479567+0000","last_deep_scrub":"60'2936","last_deep_scrub_stamp":"2024-09-11T00:56:52.479567+0000","last_clean_scrub_stamp":"2024-09-11T00:56:52.479567+0000","objects_scrubbed":0,"log_size":2936,"log_dups_size":0,"ondisk_log_size":2936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:15:12.543004+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":241742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"62'46","reported_seq":193,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:05.162624+0000","last_change":"2024-09-11T00:57:05.162624+0000","last_active":"2024-09-11T00:57:05.162624+0000","last_peered":"2024-09-11T00:57:05.162624+0000","last_clean":"2024-09-11T00:57:05.162624+0000","last_became_active":"2024-09-11T00:30:53.987361+0000","last_became_peered":"2024-09-11T00:30:53.987361+0000","last_unstale":"2024-09-11T00:57:05.162624+0000","last_undegraded":"2024-09-11T00:57:05.162624+0000","last_fullsized":"2024-09-11T00:57:05.162624+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'46","last_scrub_stamp":"2024-09-11T00:57:05.162538+0000","last_deep_scrub":"62'46","last_deep_scrub_stamp":"2024-09-11T00:57:05.162538+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.162538+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:40:01.325638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"62'3441","reported_seq":3543,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607229+0000","last_change":"2024-09-11T00:57:17.087969+0000","last_active":"2024-09-11T00:57:34.607229+0000","last_peered":"2024-09-11T00:57:34.607229+0000","last_clean":"2024-09-11T00:57:34.607229+0000","last_became_active":"2024-09-11T00:31:26.325895+0000","last_became_peered":"2024-09-11T00:31:26.325895+0000","last_unstale":"2024-09-11T00:57:34.607229+0000","last_undegraded":"2024-09-11T00:57:34.607229+0000","last_fullsized":"2024-09-11T00:57:34.607229+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3441","last_scrub_stamp":"2024-09-11T00:57:17.087871+0000","last_deep_scrub":"62'3441","last_deep_scrub_stamp":"2024-09-11T00:57:17.087871+0000","last_clean_scrub_stamp":"2024-09-11T00:57:17.087871+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:40:45.052345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18163,"num_write_kb":273760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.20","version":"60'3848","reported_seq":3963,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616063+0000","last_change":"2024-09-11T00:56:56.132108+0000","last_active":"2024-09-11T00:57:34.616063+0000","last_peered":"2024-09-11T00:57:34.616063+0000","last_clean":"2024-09-11T00:57:34.616063+0000","last_became_active":"2024-09-11T00:30:45.921262+0000","last_became_peered":"2024-09-11T00:30:45.921262+0000","last_unstale":"2024-09-11T00:57:34.616063+0000","last_undegraded":"2024-09-11T00:57:34.616063+0000","last_fullsized":"2024-09-11T00:57:34.616063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3848","last_scrub_stamp":"2024-09-11T00:56:56.132027+0000","last_deep_scrub":"60'3848","last_deep_scrub_stamp":"2024-09-11T00:56:56.132027+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.132027+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:53:12.456914+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20029,"num_write_kb":325938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"62'4287","reported_seq":4393,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:14.776915+0000","last_change":"2024-09-11T00:57:14.776915+0000","last_active":"2024-09-11T00:57:14.776915+0000","last_peered":"2024-09-11T00:57:14.776915+0000","last_clean":"2024-09-11T00:57:14.776915+0000","last_became_active":"2024-09-11T00:30:54.986666+0000","last_became_peered":"2024-09-11T00:30:54.986666+0000","last_unstale":"2024-09-11T00:57:14.776915+0000","last_undegraded":"2024-09-11T00:57:14.776915+0000","last_fullsized":"2024-09-11T00:57:14.776915+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4287","last_scrub_stamp":"2024-09-11T00:57:14.776825+0000","last_deep_scrub":"62'4287","last_deep_scrub_stamp":"2024-09-11T00:57:14.776825+0000","last_clean_scrub_stamp":"2024-09-11T00:57:14.776825+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:51.245498+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22520,"num_write_kb":336110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"60'2962","reported_seq":3075,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:56:59.151950+0000","last_change":"2024-09-11T00:56:59.151950+0000","last_active":"2024-09-11T00:56:59.151950+0000","last_peered":"2024-09-11T00:56:59.151950+0000","last_clean":"2024-09-11T00:56:59.151950+0000","last_became_active":"2024-09-11T00:30:45.919584+0000","last_became_peered":"2024-09-11T00:30:45.919584+0000","last_unstale":"2024-09-11T00:56:59.151950+0000","last_undegraded":"2024-09-11T00:56:59.151950+0000","last_fullsized":"2024-09-11T00:56:59.151950+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2962","last_scrub_stamp":"2024-09-11T00:56:59.151852+0000","last_deep_scrub":"60'2962","last_deep_scrub_stamp":"2024-09-11T00:56:59.151852+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.151852+0000","objects_scrubbed":0,"log_size":2962,"log_dups_size":0,"ondisk_log_size":2962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:39:00.023857+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15172,"num_write_kb":277710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"62'2334","reported_seq":2441,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616611+0000","last_change":"2024-09-11T00:57:12.106848+0000","last_active":"2024-09-11T00:57:34.616611+0000","last_peered":"2024-09-11T00:57:34.616611+0000","last_clean":"2024-09-11T00:57:34.616611+0000","last_became_active":"2024-09-11T00:30:54.978829+0000","last_became_peered":"2024-09-11T00:30:54.978829+0000","last_unstale":"2024-09-11T00:57:34.616611+0000","last_undegraded":"2024-09-11T00:57:34.616611+0000","last_fullsized":"2024-09-11T00:57:34.616611+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2334","last_scrub_stamp":"2024-09-11T00:57:12.106749+0000","last_deep_scrub":"62'2334","last_deep_scrub_stamp":"2024-09-11T00:57:12.106749+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.106749+0000","objects_scrubbed":0,"log_size":2334,"log_dups_size":0,"ondisk_log_size":2334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:36.419261+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11753,"num_write_kb":206776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"60'4175","reported_seq":4292,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605175+0000","last_change":"2024-09-11T00:57:09.259473+0000","last_active":"2024-09-11T00:57:34.605175+0000","last_peered":"2024-09-11T00:57:34.605175+0000","last_clean":"2024-09-11T00:57:34.605175+0000","last_became_active":"2024-09-11T00:30:45.934903+0000","last_became_peered":"2024-09-11T00:30:45.934903+0000","last_unstale":"2024-09-11T00:57:34.605175+0000","last_undegraded":"2024-09-11T00:57:34.605175+0000","last_fullsized":"2024-09-11T00:57:34.605175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4175","last_scrub_stamp":"2024-09-11T00:57:09.259376+0000","last_deep_scrub":"60'4175","last_deep_scrub_stamp":"2024-09-11T00:57:09.259376+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.259376+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:54:35.346143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22002,"num_write_kb":346058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"62'3428","reported_seq":3535,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605915+0000","last_change":"2024-09-11T00:57:14.190431+0000","last_active":"2024-09-11T00:57:34.605915+0000","last_peered":"2024-09-11T00:57:34.605915+0000","last_clean":"2024-09-11T00:57:34.605915+0000","last_became_active":"2024-09-11T00:30:54.988062+0000","last_became_peered":"2024-09-11T00:30:54.988062+0000","last_unstale":"2024-09-11T00:57:34.605915+0000","last_undegraded":"2024-09-11T00:57:34.605915+0000","last_fullsized":"2024-09-11T00:57:34.605915+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3428","last_scrub_stamp":"2024-09-11T00:57:14.190350+0000","last_deep_scrub":"62'3428","last_deep_scrub_stamp":"2024-09-11T00:57:14.190350+0000","last_clean_scrub_stamp":"2024-09-11T00:57:14.190350+0000","objects_scrubbed":0,"log_size":3428,"log_dups_size":0,"ondisk_log_size":3428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:00.568104+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17996,"num_write_kb":289213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"60'2543","reported_seq":2658,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606630+0000","last_change":"2024-09-11T00:57:03.137279+0000","last_active":"2024-09-11T00:57:34.606630+0000","last_peered":"2024-09-11T00:57:34.606630+0000","last_clean":"2024-09-11T00:57:34.606630+0000","last_became_active":"2024-09-11T00:30:45.925596+0000","last_became_peered":"2024-09-11T00:30:45.925596+0000","last_unstale":"2024-09-11T00:57:34.606630+0000","last_undegraded":"2024-09-11T00:57:34.606630+0000","last_fullsized":"2024-09-11T00:57:34.606630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2543","last_scrub_stamp":"2024-09-11T00:57:03.137186+0000","last_deep_scrub":"60'2543","last_deep_scrub_stamp":"2024-09-11T00:57:03.137186+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.137186+0000","objects_scrubbed":0,"log_size":2543,"log_dups_size":0,"ondisk_log_size":2543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:34:50.050183+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13221,"num_write_kb":224922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"62'3390","reported_seq":3497,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606126+0000","last_change":"2024-09-11T00:57:08.901854+0000","last_active":"2024-09-11T00:57:34.606126+0000","last_peered":"2024-09-11T00:57:34.606126+0000","last_clean":"2024-09-11T00:57:34.606126+0000","last_became_active":"2024-09-11T00:30:54.980903+0000","last_became_peered":"2024-09-11T00:30:54.980903+0000","last_unstale":"2024-09-11T00:57:34.606126+0000","last_undegraded":"2024-09-11T00:57:34.606126+0000","last_fullsized":"2024-09-11T00:57:34.606126+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3390","last_scrub_stamp":"2024-09-11T00:57:08.901731+0000","last_deep_scrub":"62'3390","last_deep_scrub_stamp":"2024-09-11T00:57:08.901731+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.901731+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:15:26.911111+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17320,"num_write_kb":290458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"60'2853","reported_seq":2968,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606753+0000","last_change":"2024-09-11T00:57:00.791064+0000","last_active":"2024-09-11T00:57:34.606753+0000","last_peered":"2024-09-11T00:57:34.606753+0000","last_clean":"2024-09-11T00:57:34.606753+0000","last_became_active":"2024-09-11T00:30:45.929879+0000","last_became_peered":"2024-09-11T00:30:45.929879+0000","last_unstale":"2024-09-11T00:57:34.606753+0000","last_undegraded":"2024-09-11T00:57:34.606753+0000","last_fullsized":"2024-09-11T00:57:34.606753+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2853","last_scrub_stamp":"2024-09-11T00:57:00.790968+0000","last_deep_scrub":"60'2853","last_deep_scrub_stamp":"2024-09-11T00:57:00.790968+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.790968+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:28:17.766768+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14682,"num_write_kb":238952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"62'2740","reported_seq":2847,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606645+0000","last_change":"2024-09-11T00:57:12.871414+0000","last_active":"2024-09-11T00:57:34.606645+0000","last_peered":"2024-09-11T00:57:34.606645+0000","last_clean":"2024-09-11T00:57:34.606645+0000","last_became_active":"2024-09-11T00:30:54.991835+0000","last_became_peered":"2024-09-11T00:30:54.991835+0000","last_unstale":"2024-09-11T00:57:34.606645+0000","last_undegraded":"2024-09-11T00:57:34.606645+0000","last_fullsized":"2024-09-11T00:57:34.606645+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2740","last_scrub_stamp":"2024-09-11T00:57:12.871324+0000","last_deep_scrub":"62'2740","last_deep_scrub_stamp":"2024-09-11T00:57:12.871324+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.871324+0000","objects_scrubbed":0,"log_size":2740,"log_dups_size":0,"ondisk_log_size":2740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:19:09.405525+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14066,"num_write_kb":224154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"60'3660","reported_seq":3775,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605335+0000","last_change":"2024-09-11T00:57:01.790608+0000","last_active":"2024-09-11T00:57:34.605335+0000","last_peered":"2024-09-11T00:57:34.605335+0000","last_clean":"2024-09-11T00:57:34.605335+0000","last_became_active":"2024-09-11T00:30:45.928557+0000","last_became_peered":"2024-09-11T00:30:45.928557+0000","last_unstale":"2024-09-11T00:57:34.605335+0000","last_undegraded":"2024-09-11T00:57:34.605335+0000","last_fullsized":"2024-09-11T00:57:34.605335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3660","last_scrub_stamp":"2024-09-11T00:57:01.790501+0000","last_deep_scrub":"60'3660","last_deep_scrub_stamp":"2024-09-11T00:57:01.790501+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.790501+0000","objects_scrubbed":0,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:08:21.173261+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19466,"num_write_kb":307301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"62'4363","reported_seq":4470,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604188+0000","last_change":"2024-09-11T00:57:28.203184+0000","last_active":"2024-09-11T00:57:34.604188+0000","last_peered":"2024-09-11T00:57:34.604188+0000","last_clean":"2024-09-11T00:57:34.604188+0000","last_became_active":"2024-09-11T00:30:54.987241+0000","last_became_peered":"2024-09-11T00:30:54.987241+0000","last_unstale":"2024-09-11T00:57:34.604188+0000","last_undegraded":"2024-09-11T00:57:34.604188+0000","last_fullsized":"2024-09-11T00:57:34.604188+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4363","last_scrub_stamp":"2024-09-11T00:57:28.203081+0000","last_deep_scrub":"62'4363","last_deep_scrub_stamp":"2024-09-11T00:57:28.203081+0000","last_clean_scrub_stamp":"2024-09-11T00:57:28.203081+0000","objects_scrubbed":0,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:48:32.585406+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23064,"num_write_kb":350422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"60'2921","reported_seq":3036,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.617642+0000","last_change":"2024-09-11T00:56:58.088203+0000","last_active":"2024-09-11T00:57:34.617642+0000","last_peered":"2024-09-11T00:57:34.617642+0000","last_clean":"2024-09-11T00:57:34.617642+0000","last_became_active":"2024-09-11T00:30:45.944572+0000","last_became_peered":"2024-09-11T00:30:45.944572+0000","last_unstale":"2024-09-11T00:57:34.617642+0000","last_undegraded":"2024-09-11T00:57:34.617642+0000","last_fullsized":"2024-09-11T00:57:34.617642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2921","last_scrub_stamp":"2024-09-11T00:56:58.088108+0000","last_deep_scrub":"60'2921","last_deep_scrub_stamp":"2024-09-11T00:56:58.088108+0000","last_clean_scrub_stamp":"2024-09-11T00:56:58.088108+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:21.480415+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15417,"num_write_kb":250464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"62'3030","reported_seq":3137,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.615599+0000","last_change":"2024-09-11T00:57:13.088347+0000","last_active":"2024-09-11T00:57:34.615599+0000","last_peered":"2024-09-11T00:57:34.615599+0000","last_clean":"2024-09-11T00:57:34.615599+0000","last_became_active":"2024-09-11T00:30:54.983978+0000","last_became_peered":"2024-09-11T00:30:54.983978+0000","last_unstale":"2024-09-11T00:57:34.615599+0000","last_undegraded":"2024-09-11T00:57:34.615599+0000","last_fullsized":"2024-09-11T00:57:34.615599+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3030","last_scrub_stamp":"2024-09-11T00:57:13.088252+0000","last_deep_scrub":"62'3030","last_deep_scrub_stamp":"2024-09-11T00:57:13.088252+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.088252+0000","objects_scrubbed":0,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:56:59.168108+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15510,"num_write_kb":262754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"60'3321","reported_seq":3438,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606817+0000","last_change":"2024-09-11T00:57:04.121253+0000","last_active":"2024-09-11T00:57:34.606817+0000","last_peered":"2024-09-11T00:57:34.606817+0000","last_clean":"2024-09-11T00:57:34.606817+0000","last_became_active":"2024-09-11T00:30:45.933980+0000","last_became_peered":"2024-09-11T00:30:45.933980+0000","last_unstale":"2024-09-11T00:57:34.606817+0000","last_undegraded":"2024-09-11T00:57:34.606817+0000","last_fullsized":"2024-09-11T00:57:34.606817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3321","last_scrub_stamp":"2024-09-11T00:57:04.121162+0000","last_deep_scrub":"60'3321","last_deep_scrub_stamp":"2024-09-11T00:57:04.121162+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.121162+0000","objects_scrubbed":0,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:04.945000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":25,"num_write":17209,"num_write_kb":314610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"62'2084","reported_seq":2189,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:10.298132+0000","last_change":"2024-09-11T00:57:10.298132+0000","last_active":"2024-09-11T00:57:10.298132+0000","last_peered":"2024-09-11T00:57:10.298132+0000","last_clean":"2024-09-11T00:57:10.298132+0000","last_became_active":"2024-09-11T00:30:54.990893+0000","last_became_peered":"2024-09-11T00:30:54.990893+0000","last_unstale":"2024-09-11T00:57:10.298132+0000","last_undegraded":"2024-09-11T00:57:10.298132+0000","last_fullsized":"2024-09-11T00:57:10.298132+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2084","last_scrub_stamp":"2024-09-11T00:57:10.298028+0000","last_deep_scrub":"62'2084","last_deep_scrub_stamp":"2024-09-11T00:57:10.298028+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.298028+0000","objects_scrubbed":0,"log_size":2084,"log_dups_size":0,"ondisk_log_size":2084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:04:15.218052+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10636,"num_write_kb":173560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"60'2810","reported_seq":2925,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604497+0000","last_change":"2024-09-11T00:57:10.297848+0000","last_active":"2024-09-11T00:57:34.604497+0000","last_peered":"2024-09-11T00:57:34.604497+0000","last_clean":"2024-09-11T00:57:34.604497+0000","last_became_active":"2024-09-11T00:30:45.938183+0000","last_became_peered":"2024-09-11T00:30:45.938183+0000","last_unstale":"2024-09-11T00:57:34.604497+0000","last_undegraded":"2024-09-11T00:57:34.604497+0000","last_fullsized":"2024-09-11T00:57:34.604497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2810","last_scrub_stamp":"2024-09-11T00:57:10.297751+0000","last_deep_scrub":"60'2810","last_deep_scrub_stamp":"2024-09-11T00:57:10.297751+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.297751+0000","objects_scrubbed":0,"log_size":2810,"log_dups_size":0,"ondisk_log_size":2810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:04:32.109718+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":243007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"62'3697","reported_seq":3803,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607002+0000","last_change":"2024-09-11T00:57:26.207083+0000","last_active":"2024-09-11T00:57:34.607002+0000","last_peered":"2024-09-11T00:57:34.607002+0000","last_clean":"2024-09-11T00:57:34.607002+0000","last_became_active":"2024-09-11T00:30:54.988938+0000","last_became_peered":"2024-09-11T00:30:54.988938+0000","last_unstale":"2024-09-11T00:57:34.607002+0000","last_undegraded":"2024-09-11T00:57:34.607002+0000","last_fullsized":"2024-09-11T00:57:34.607002+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3697","last_scrub_stamp":"2024-09-11T00:57:26.206980+0000","last_deep_scrub":"62'3697","last_deep_scrub_stamp":"2024-09-11T00:57:26.206980+0000","last_clean_scrub_stamp":"2024-09-11T00:57:26.206980+0000","objects_scrubbed":0,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:25.489380+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18782,"num_write_kb":312790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"60'2824","reported_seq":2938,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605822+0000","last_change":"2024-09-11T00:57:07.107360+0000","last_active":"2024-09-11T00:57:34.605822+0000","last_peered":"2024-09-11T00:57:34.605822+0000","last_clean":"2024-09-11T00:57:34.605822+0000","last_became_active":"2024-09-11T00:30:45.940751+0000","last_became_peered":"2024-09-11T00:30:45.940751+0000","last_unstale":"2024-09-11T00:57:34.605822+0000","last_undegraded":"2024-09-11T00:57:34.605822+0000","last_fullsized":"2024-09-11T00:57:34.605822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2824","last_scrub_stamp":"2024-09-11T00:57:07.107256+0000","last_deep_scrub":"60'2824","last_deep_scrub_stamp":"2024-09-11T00:57:07.107256+0000","last_clean_scrub_stamp":"2024-09-11T00:57:07.107256+0000","objects_scrubbed":0,"log_size":2824,"log_dups_size":0,"ondisk_log_size":2824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:24:35.809569+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14695,"num_write_kb":259410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"62'2917","reported_seq":3023,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604478+0000","last_change":"2024-09-11T00:57:13.592153+0000","last_active":"2024-09-11T00:57:34.604478+0000","last_peered":"2024-09-11T00:57:34.604478+0000","last_clean":"2024-09-11T00:57:34.604478+0000","last_became_active":"2024-09-11T00:30:54.981480+0000","last_became_peered":"2024-09-11T00:30:54.981480+0000","last_unstale":"2024-09-11T00:57:34.604478+0000","last_undegraded":"2024-09-11T00:57:34.604478+0000","last_fullsized":"2024-09-11T00:57:34.604478+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2917","last_scrub_stamp":"2024-09-11T00:57:13.592025+0000","last_deep_scrub":"62'2917","last_deep_scrub_stamp":"2024-09-11T00:57:13.592025+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.592025+0000","objects_scrubbed":0,"log_size":2917,"log_dups_size":0,"ondisk_log_size":2917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:16.008324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":242029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"60'3431","reported_seq":3545,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.612347+0000","last_change":"2024-09-11T00:56:56.373760+0000","last_active":"2024-09-11T00:57:34.612347+0000","last_peered":"2024-09-11T00:57:34.612347+0000","last_clean":"2024-09-11T00:57:34.612347+0000","last_became_active":"2024-09-11T00:30:45.919063+0000","last_became_peered":"2024-09-11T00:30:45.919063+0000","last_unstale":"2024-09-11T00:57:34.612347+0000","last_undegraded":"2024-09-11T00:57:34.612347+0000","last_fullsized":"2024-09-11T00:57:34.612347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3431","last_scrub_stamp":"2024-09-11T00:56:56.373653+0000","last_deep_scrub":"60'3431","last_deep_scrub_stamp":"2024-09-11T00:56:56.373653+0000","last_clean_scrub_stamp":"2024-09-11T00:56:56.373653+0000","objects_scrubbed":0,"log_size":3431,"log_dups_size":0,"ondisk_log_size":3431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:14.920519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18408,"num_write_kb":290787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"62'2945","reported_seq":3049,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:09.263328+0000","last_change":"2024-09-11T00:57:09.263328+0000","last_active":"2024-09-11T00:57:09.263328+0000","last_peered":"2024-09-11T00:57:09.263328+0000","last_clean":"2024-09-11T00:57:09.263328+0000","last_became_active":"2024-09-11T00:30:54.991605+0000","last_became_peered":"2024-09-11T00:30:54.991605+0000","last_unstale":"2024-09-11T00:57:09.263328+0000","last_undegraded":"2024-09-11T00:57:09.263328+0000","last_fullsized":"2024-09-11T00:57:09.263328+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2945","last_scrub_stamp":"2024-09-11T00:57:09.263243+0000","last_deep_scrub":"62'2945","last_deep_scrub_stamp":"2024-09-11T00:57:09.263243+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.263243+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":0,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:42.744739+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15229,"num_write_kb":249300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"60'2849","reported_seq":2963,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:10.776164+0000","last_change":"2024-09-11T00:57:10.776164+0000","last_active":"2024-09-11T00:57:10.776164+0000","last_peered":"2024-09-11T00:57:10.776164+0000","last_clean":"2024-09-11T00:57:10.776164+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:57:10.776164+0000","last_undegraded":"2024-09-11T00:57:10.776164+0000","last_fullsized":"2024-09-11T00:57:10.776164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2849","last_scrub_stamp":"2024-09-11T00:57:10.776069+0000","last_deep_scrub":"60'2849","last_deep_scrub_stamp":"2024-09-11T00:57:10.776069+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.776069+0000","objects_scrubbed":0,"log_size":2849,"log_dups_size":0,"ondisk_log_size":2849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:02.350395+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14625,"num_write_kb":263252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"62'4017","reported_seq":4123,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.615641+0000","last_change":"2024-09-11T00:57:17.088883+0000","last_active":"2024-09-11T00:57:34.615641+0000","last_peered":"2024-09-11T00:57:34.615641+0000","last_clean":"2024-09-11T00:57:34.615641+0000","last_became_active":"2024-09-11T00:30:54.996695+0000","last_became_peered":"2024-09-11T00:30:54.996695+0000","last_unstale":"2024-09-11T00:57:34.615641+0000","last_undegraded":"2024-09-11T00:57:34.615641+0000","last_fullsized":"2024-09-11T00:57:34.615641+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4017","last_scrub_stamp":"2024-09-11T00:57:17.088799+0000","last_deep_scrub":"62'4017","last_deep_scrub_stamp":"2024-09-11T00:57:17.088799+0000","last_clean_scrub_stamp":"2024-09-11T00:57:17.088799+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:35:40.830920+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":329319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"60'2942","reported_seq":3056,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.608236+0000","last_change":"2024-09-11T00:57:01.797270+0000","last_active":"2024-09-11T00:57:34.608236+0000","last_peered":"2024-09-11T00:57:34.608236+0000","last_clean":"2024-09-11T00:57:34.608236+0000","last_became_active":"2024-09-11T00:30:45.928064+0000","last_became_peered":"2024-09-11T00:30:45.928064+0000","last_unstale":"2024-09-11T00:57:34.608236+0000","last_undegraded":"2024-09-11T00:57:34.608236+0000","last_fullsized":"2024-09-11T00:57:34.608236+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2942","last_scrub_stamp":"2024-09-11T00:57:01.797171+0000","last_deep_scrub":"60'2942","last_deep_scrub_stamp":"2024-09-11T00:57:01.797171+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.797171+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:36:32.927031+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15436,"num_write_kb":246093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"62'3417","reported_seq":3524,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606690+0000","last_change":"2024-09-11T00:57:13.889507+0000","last_active":"2024-09-11T00:57:34.606690+0000","last_peered":"2024-09-11T00:57:34.606690+0000","last_clean":"2024-09-11T00:57:34.606690+0000","last_became_active":"2024-09-11T00:30:54.985049+0000","last_became_peered":"2024-09-11T00:30:54.985049+0000","last_unstale":"2024-09-11T00:57:34.606690+0000","last_undegraded":"2024-09-11T00:57:34.606690+0000","last_fullsized":"2024-09-11T00:57:34.606690+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3417","last_scrub_stamp":"2024-09-11T00:57:13.889419+0000","last_deep_scrub":"62'3417","last_deep_scrub_stamp":"2024-09-11T00:57:13.889419+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.889419+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:27:52.118576+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18129,"num_write_kb":269216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"60'2746","reported_seq":2860,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.610316+0000","last_change":"2024-09-11T00:57:13.251217+0000","last_active":"2024-09-11T00:57:34.610316+0000","last_peered":"2024-09-11T00:57:34.610316+0000","last_clean":"2024-09-11T00:57:34.610316+0000","last_became_active":"2024-09-11T00:30:45.940137+0000","last_became_peered":"2024-09-11T00:30:45.940137+0000","last_unstale":"2024-09-11T00:57:34.610316+0000","last_undegraded":"2024-09-11T00:57:34.610316+0000","last_fullsized":"2024-09-11T00:57:34.610316+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2746","last_scrub_stamp":"2024-09-11T00:57:13.251115+0000","last_deep_scrub":"60'2746","last_deep_scrub_stamp":"2024-09-11T00:57:13.251115+0000","last_clean_scrub_stamp":"2024-09-11T00:57:13.251115+0000","objects_scrubbed":0,"log_size":2746,"log_dups_size":0,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:48:31.529598+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13859,"num_write_kb":247924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"62'2570","reported_seq":2683,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.615519+0000","last_change":"2024-09-11T00:57:14.056466+0000","last_active":"2024-09-11T00:57:34.615519+0000","last_peered":"2024-09-11T00:57:34.615519+0000","last_clean":"2024-09-11T00:57:34.615519+0000","last_became_active":"2024-09-11T00:31:26.119436+0000","last_became_peered":"2024-09-11T00:31:26.119436+0000","last_unstale":"2024-09-11T00:57:34.615519+0000","last_undegraded":"2024-09-11T00:57:34.615519+0000","last_fullsized":"2024-09-11T00:57:34.615519+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2570","last_scrub_stamp":"2024-09-11T00:57:14.056377+0000","last_deep_scrub":"62'2570","last_deep_scrub_stamp":"2024-09-11T00:57:14.056377+0000","last_clean_scrub_stamp":"2024-09-11T00:57:14.056377+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:12:03.852666+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13205,"num_write_kb":216683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"60'2388","reported_seq":2502,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:00.186123+0000","last_change":"2024-09-11T00:57:00.186123+0000","last_active":"2024-09-11T00:57:00.186123+0000","last_peered":"2024-09-11T00:57:00.186123+0000","last_clean":"2024-09-11T00:57:00.186123+0000","last_became_active":"2024-09-11T00:30:45.926041+0000","last_became_peered":"2024-09-11T00:30:45.926041+0000","last_unstale":"2024-09-11T00:57:00.186123+0000","last_undegraded":"2024-09-11T00:57:00.186123+0000","last_fullsized":"2024-09-11T00:57:00.186123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2388","last_scrub_stamp":"2024-09-11T00:57:00.185990+0000","last_deep_scrub":"60'2388","last_deep_scrub_stamp":"2024-09-11T00:57:00.185990+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.185990+0000","objects_scrubbed":0,"log_size":2388,"log_dups_size":0,"ondisk_log_size":2388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:00:55.384651+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4,"num_write":12384,"num_write_kb":205064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"62'3381","reported_seq":3487,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.602349+0000","last_change":"2024-09-11T00:57:12.632325+0000","last_active":"2024-09-11T00:57:34.602349+0000","last_peered":"2024-09-11T00:57:34.602349+0000","last_clean":"2024-09-11T00:57:34.602349+0000","last_became_active":"2024-09-11T00:30:54.992659+0000","last_became_peered":"2024-09-11T00:30:54.992659+0000","last_unstale":"2024-09-11T00:57:34.602349+0000","last_undegraded":"2024-09-11T00:57:34.602349+0000","last_fullsized":"2024-09-11T00:57:34.602349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3381","last_scrub_stamp":"2024-09-11T00:57:12.632220+0000","last_deep_scrub":"62'3381","last_deep_scrub_stamp":"2024-09-11T00:57:12.632220+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.632220+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:22:08.114108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17747,"num_write_kb":280034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"60'3936","reported_seq":4051,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.609882+0000","last_change":"2024-09-11T00:57:11.256534+0000","last_active":"2024-09-11T00:57:34.609882+0000","last_peered":"2024-09-11T00:57:34.609882+0000","last_clean":"2024-09-11T00:57:34.609882+0000","last_became_active":"2024-09-11T00:30:45.925294+0000","last_became_peered":"2024-09-11T00:30:45.925294+0000","last_unstale":"2024-09-11T00:57:34.609882+0000","last_undegraded":"2024-09-11T00:57:34.609882+0000","last_fullsized":"2024-09-11T00:57:34.609882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3936","last_scrub_stamp":"2024-09-11T00:57:11.256420+0000","last_deep_scrub":"60'3936","last_deep_scrub_stamp":"2024-09-11T00:57:11.256420+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.256420+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:46:09.333516+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":20459,"num_write_kb":335990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"62'3772","reported_seq":3876,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:18.789002+0000","last_change":"2024-09-11T00:57:18.788938+0000","last_active":"2024-09-11T00:57:18.789002+0000","last_peered":"2024-09-11T00:57:18.789002+0000","last_clean":"2024-09-11T00:57:18.789002+0000","last_became_active":"2024-09-11T00:30:54.989105+0000","last_became_peered":"2024-09-11T00:30:54.989105+0000","last_unstale":"2024-09-11T00:57:18.789002+0000","last_undegraded":"2024-09-11T00:57:18.789002+0000","last_fullsized":"2024-09-11T00:57:18.789002+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3772","last_scrub_stamp":"2024-09-11T00:57:18.788843+0000","last_deep_scrub":"62'3772","last_deep_scrub_stamp":"2024-09-11T00:57:18.788843+0000","last_clean_scrub_stamp":"2024-09-11T00:57:18.788843+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:47:35.637718+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20201,"num_write_kb":302053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"60'2757","reported_seq":2871,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604298+0000","last_change":"2024-09-11T00:57:12.280110+0000","last_active":"2024-09-11T00:57:34.604298+0000","last_peered":"2024-09-11T00:57:34.604298+0000","last_clean":"2024-09-11T00:57:34.604298+0000","last_became_active":"2024-09-11T00:30:45.936423+0000","last_became_peered":"2024-09-11T00:30:45.936423+0000","last_unstale":"2024-09-11T00:57:34.604298+0000","last_undegraded":"2024-09-11T00:57:34.604298+0000","last_fullsized":"2024-09-11T00:57:34.604298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2757","last_scrub_stamp":"2024-09-11T00:57:12.280012+0000","last_deep_scrub":"60'2757","last_deep_scrub_stamp":"2024-09-11T00:57:12.280012+0000","last_clean_scrub_stamp":"2024-09-11T00:57:12.280012+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:18:17.455521+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14203,"num_write_kb":241057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"62'4048","reported_seq":4154,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.611825+0000","last_change":"2024-09-11T00:57:03.439931+0000","last_active":"2024-09-11T00:57:34.611825+0000","last_peered":"2024-09-11T00:57:34.611825+0000","last_clean":"2024-09-11T00:57:34.611825+0000","last_became_active":"2024-09-11T00:30:54.993147+0000","last_became_peered":"2024-09-11T00:30:54.993147+0000","last_unstale":"2024-09-11T00:57:34.611825+0000","last_undegraded":"2024-09-11T00:57:34.611825+0000","last_fullsized":"2024-09-11T00:57:34.611825+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'4048","last_scrub_stamp":"2024-09-11T00:57:03.439855+0000","last_deep_scrub":"62'4048","last_deep_scrub_stamp":"2024-09-11T00:57:03.439855+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.439855+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:07:30.557944+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21387,"num_write_kb":364007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"60'3572","reported_seq":3686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605717+0000","last_change":"2024-09-11T00:57:08.145697+0000","last_active":"2024-09-11T00:57:34.605717+0000","last_peered":"2024-09-11T00:57:34.605717+0000","last_clean":"2024-09-11T00:57:34.605717+0000","last_became_active":"2024-09-11T00:30:45.933024+0000","last_became_peered":"2024-09-11T00:30:45.933024+0000","last_unstale":"2024-09-11T00:57:34.605717+0000","last_undegraded":"2024-09-11T00:57:34.605717+0000","last_fullsized":"2024-09-11T00:57:34.605717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3572","last_scrub_stamp":"2024-09-11T00:57:08.145604+0000","last_deep_scrub":"60'3572","last_deep_scrub_stamp":"2024-09-11T00:57:08.145604+0000","last_clean_scrub_stamp":"2024-09-11T00:57:08.145604+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:30:37.316367+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18317,"num_write_kb":294510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"62'2987","reported_seq":3094,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.612848+0000","last_change":"2024-09-11T00:57:02.394571+0000","last_active":"2024-09-11T00:57:34.612848+0000","last_peered":"2024-09-11T00:57:34.612848+0000","last_clean":"2024-09-11T00:57:34.612848+0000","last_became_active":"2024-09-11T00:30:54.985990+0000","last_became_peered":"2024-09-11T00:30:54.985990+0000","last_unstale":"2024-09-11T00:57:34.612848+0000","last_undegraded":"2024-09-11T00:57:34.612848+0000","last_fullsized":"2024-09-11T00:57:34.612848+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2987","last_scrub_stamp":"2024-09-11T00:57:02.394480+0000","last_deep_scrub":"62'2987","last_deep_scrub_stamp":"2024-09-11T00:57:02.394480+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.394480+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:36:35.080708+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15338,"num_write_kb":237611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"60'2929","reported_seq":3044,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604866+0000","last_change":"2024-09-11T00:57:02.742713+0000","last_active":"2024-09-11T00:57:34.604866+0000","last_peered":"2024-09-11T00:57:34.604866+0000","last_clean":"2024-09-11T00:57:34.604866+0000","last_became_active":"2024-09-11T00:30:45.920277+0000","last_became_peered":"2024-09-11T00:30:45.920277+0000","last_unstale":"2024-09-11T00:57:34.604866+0000","last_undegraded":"2024-09-11T00:57:34.604866+0000","last_fullsized":"2024-09-11T00:57:34.604866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2929","last_scrub_stamp":"2024-09-11T00:57:02.742610+0000","last_deep_scrub":"60'2929","last_deep_scrub_stamp":"2024-09-11T00:57:02.742610+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.742610+0000","objects_scrubbed":0,"log_size":2929,"log_dups_size":0,"ondisk_log_size":2929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:09:17.529350+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":15867,"num_write_kb":268409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"62'3243","reported_seq":3349,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605735+0000","last_change":"2024-09-11T00:57:27.225312+0000","last_active":"2024-09-11T00:57:34.605735+0000","last_peered":"2024-09-11T00:57:34.605735+0000","last_clean":"2024-09-11T00:57:34.605735+0000","last_became_active":"2024-09-11T00:30:54.999212+0000","last_became_peered":"2024-09-11T00:30:54.999212+0000","last_unstale":"2024-09-11T00:57:34.605735+0000","last_undegraded":"2024-09-11T00:57:34.605735+0000","last_fullsized":"2024-09-11T00:57:34.605735+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3243","last_scrub_stamp":"2024-09-11T00:57:27.225243+0000","last_deep_scrub":"62'3243","last_deep_scrub_stamp":"2024-09-11T00:57:27.225243+0000","last_clean_scrub_stamp":"2024-09-11T00:57:27.225243+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:19:43.982884+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16324,"num_write_kb":262051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"60'2913","reported_seq":3030,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605186+0000","last_change":"2024-09-11T00:57:03.696860+0000","last_active":"2024-09-11T00:57:34.605186+0000","last_peered":"2024-09-11T00:57:34.605186+0000","last_clean":"2024-09-11T00:57:34.605186+0000","last_became_active":"2024-09-11T00:30:45.929830+0000","last_became_peered":"2024-09-11T00:30:45.929830+0000","last_unstale":"2024-09-11T00:57:34.605186+0000","last_undegraded":"2024-09-11T00:57:34.605186+0000","last_fullsized":"2024-09-11T00:57:34.605186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2913","last_scrub_stamp":"2024-09-11T00:57:03.696750+0000","last_deep_scrub":"60'2913","last_deep_scrub_stamp":"2024-09-11T00:57:03.696750+0000","last_clean_scrub_stamp":"2024-09-11T00:57:03.696750+0000","objects_scrubbed":0,"log_size":2913,"log_dups_size":0,"ondisk_log_size":2913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:23.920964+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15444,"num_write_kb":246103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"62'3344","reported_seq":3448,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:16.777259+0000","last_change":"2024-09-11T00:57:16.777259+0000","last_active":"2024-09-11T00:57:16.777259+0000","last_peered":"2024-09-11T00:57:16.777259+0000","last_clean":"2024-09-11T00:57:16.777259+0000","last_became_active":"2024-09-11T00:30:54.989864+0000","last_became_peered":"2024-09-11T00:30:54.989864+0000","last_unstale":"2024-09-11T00:57:16.777259+0000","last_undegraded":"2024-09-11T00:57:16.777259+0000","last_fullsized":"2024-09-11T00:57:16.777259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3344","last_scrub_stamp":"2024-09-11T00:57:16.777157+0000","last_deep_scrub":"62'3344","last_deep_scrub_stamp":"2024-09-11T00:57:16.777157+0000","last_clean_scrub_stamp":"2024-09-11T00:57:16.777157+0000","objects_scrubbed":0,"log_size":3344,"log_dups_size":0,"ondisk_log_size":3344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:16:51.631918+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17374,"num_write_kb":274437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"60'2266","reported_seq":2384,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607770+0000","last_change":"2024-09-11T00:57:02.799588+0000","last_active":"2024-09-11T00:57:34.607770+0000","last_peered":"2024-09-11T00:57:34.607770+0000","last_clean":"2024-09-11T00:57:34.607770+0000","last_became_active":"2024-09-11T00:30:45.925888+0000","last_became_peered":"2024-09-11T00:30:45.925888+0000","last_unstale":"2024-09-11T00:57:34.607770+0000","last_undegraded":"2024-09-11T00:57:34.607770+0000","last_fullsized":"2024-09-11T00:57:34.607770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2266","last_scrub_stamp":"2024-09-11T00:57:02.799525+0000","last_deep_scrub":"60'2266","last_deep_scrub_stamp":"2024-09-11T00:57:02.799525+0000","last_clean_scrub_stamp":"2024-09-11T00:57:02.799525+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:30.562164+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":26,"num_write":11816,"num_write_kb":219351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"62'2714","reported_seq":2822,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605110+0000","last_change":"2024-09-11T00:57:14.598669+0000","last_active":"2024-09-11T00:57:34.605110+0000","last_peered":"2024-09-11T00:57:34.605110+0000","last_clean":"2024-09-11T00:57:34.605110+0000","last_became_active":"2024-09-11T00:30:54.994656+0000","last_became_peered":"2024-09-11T00:30:54.994656+0000","last_unstale":"2024-09-11T00:57:34.605110+0000","last_undegraded":"2024-09-11T00:57:34.605110+0000","last_fullsized":"2024-09-11T00:57:34.605110+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2714","last_scrub_stamp":"2024-09-11T00:57:14.598569+0000","last_deep_scrub":"62'2714","last_deep_scrub_stamp":"2024-09-11T00:57:14.598569+0000","last_clean_scrub_stamp":"2024-09-11T00:57:14.598569+0000","objects_scrubbed":0,"log_size":2714,"log_dups_size":0,"ondisk_log_size":2714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:19:35.610839+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14634,"num_write_kb":230575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"60'3170","reported_seq":3267,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:01.228544+0000","last_change":"2024-09-11T00:57:01.228544+0000","last_active":"2024-09-11T00:57:01.228544+0000","last_peered":"2024-09-11T00:57:01.228544+0000","last_clean":"2024-09-11T00:57:01.228544+0000","last_became_active":"2024-09-11T00:31:26.117146+0000","last_became_peered":"2024-09-11T00:31:26.117146+0000","last_unstale":"2024-09-11T00:57:01.228544+0000","last_undegraded":"2024-09-11T00:57:01.228544+0000","last_fullsized":"2024-09-11T00:57:01.228544+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3170","last_scrub_stamp":"2024-09-11T00:57:01.228423+0000","last_deep_scrub":"60'3170","last_deep_scrub_stamp":"2024-09-11T00:57:01.228423+0000","last_clean_scrub_stamp":"2024-09-11T00:57:01.228423+0000","objects_scrubbed":0,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:30:52.030424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15922,"num_write_kb":268523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.30","version":"62'3135","reported_seq":3241,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.603990+0000","last_change":"2024-09-11T00:57:09.863390+0000","last_active":"2024-09-11T00:57:34.603990+0000","last_peered":"2024-09-11T00:57:34.603990+0000","last_clean":"2024-09-11T00:57:34.603990+0000","last_became_active":"2024-09-11T00:30:54.984169+0000","last_became_peered":"2024-09-11T00:30:54.984169+0000","last_unstale":"2024-09-11T00:57:34.603990+0000","last_undegraded":"2024-09-11T00:57:34.603990+0000","last_fullsized":"2024-09-11T00:57:34.603990+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3135","last_scrub_stamp":"2024-09-11T00:57:09.863276+0000","last_deep_scrub":"62'3135","last_deep_scrub_stamp":"2024-09-11T00:57:09.863276+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.863276+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:30:50.702426+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16038,"num_write_kb":252613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"60'3300","reported_seq":3415,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607364+0000","last_change":"2024-09-11T00:56:59.775630+0000","last_active":"2024-09-11T00:57:34.607364+0000","last_peered":"2024-09-11T00:57:34.607364+0000","last_clean":"2024-09-11T00:57:34.607364+0000","last_became_active":"2024-09-11T00:30:45.930220+0000","last_became_peered":"2024-09-11T00:30:45.930220+0000","last_unstale":"2024-09-11T00:57:34.607364+0000","last_undegraded":"2024-09-11T00:57:34.607364+0000","last_fullsized":"2024-09-11T00:57:34.607364+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3300","last_scrub_stamp":"2024-09-11T00:56:59.775536+0000","last_deep_scrub":"60'3300","last_deep_scrub_stamp":"2024-09-11T00:56:59.775536+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.775536+0000","objects_scrubbed":0,"log_size":3300,"log_dups_size":0,"ondisk_log_size":3300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:48:47.113900+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":17861,"num_write_kb":288513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"62'3599","reported_seq":3705,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604066+0000","last_change":"2024-09-11T00:57:11.859963+0000","last_active":"2024-09-11T00:57:34.604066+0000","last_peered":"2024-09-11T00:57:34.604066+0000","last_clean":"2024-09-11T00:57:34.604066+0000","last_became_active":"2024-09-11T00:30:54.986193+0000","last_became_peered":"2024-09-11T00:30:54.986193+0000","last_unstale":"2024-09-11T00:57:34.604066+0000","last_undegraded":"2024-09-11T00:57:34.604066+0000","last_fullsized":"2024-09-11T00:57:34.604066+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3599","last_scrub_stamp":"2024-09-11T00:57:11.859874+0000","last_deep_scrub":"62'3599","last_deep_scrub_stamp":"2024-09-11T00:57:11.859874+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.859874+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:03:58.937955+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18686,"num_write_kb":304162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"60'4168","reported_seq":4283,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.612358+0000","last_change":"2024-09-11T00:56:54.445154+0000","last_active":"2024-09-11T00:57:34.612358+0000","last_peered":"2024-09-11T00:57:34.612358+0000","last_clean":"2024-09-11T00:57:34.612358+0000","last_became_active":"2024-09-11T00:30:45.913131+0000","last_became_peered":"2024-09-11T00:30:45.913131+0000","last_unstale":"2024-09-11T00:57:34.612358+0000","last_undegraded":"2024-09-11T00:57:34.612358+0000","last_fullsized":"2024-09-11T00:57:34.612358+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4168","last_scrub_stamp":"2024-09-11T00:56:54.445072+0000","last_deep_scrub":"60'4168","last_deep_scrub_stamp":"2024-09-11T00:56:54.445072+0000","last_clean_scrub_stamp":"2024-09-11T00:56:54.445072+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:22:52.768326+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22110,"num_write_kb":362983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"62'3937","reported_seq":4044,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607286+0000","last_change":"2024-09-11T00:57:19.036755+0000","last_active":"2024-09-11T00:57:34.607286+0000","last_peered":"2024-09-11T00:57:34.607286+0000","last_clean":"2024-09-11T00:57:34.607286+0000","last_became_active":"2024-09-11T00:30:55.000249+0000","last_became_peered":"2024-09-11T00:30:55.000249+0000","last_unstale":"2024-09-11T00:57:34.607286+0000","last_undegraded":"2024-09-11T00:57:34.607286+0000","last_fullsized":"2024-09-11T00:57:34.607286+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3937","last_scrub_stamp":"2024-09-11T00:57:19.036657+0000","last_deep_scrub":"62'3937","last_deep_scrub_stamp":"2024-09-11T00:57:19.036657+0000","last_clean_scrub_stamp":"2024-09-11T00:57:19.036657+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:17.282201+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21285,"num_write_kb":343312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"60'3475","reported_seq":3589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.604274+0000","last_change":"2024-09-11T00:57:05.089640+0000","last_active":"2024-09-11T00:57:34.604274+0000","last_peered":"2024-09-11T00:57:34.604274+0000","last_clean":"2024-09-11T00:57:34.604274+0000","last_became_active":"2024-09-11T00:30:45.931453+0000","last_became_peered":"2024-09-11T00:30:45.931453+0000","last_unstale":"2024-09-11T00:57:34.604274+0000","last_undegraded":"2024-09-11T00:57:34.604274+0000","last_fullsized":"2024-09-11T00:57:34.604274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3475","last_scrub_stamp":"2024-09-11T00:57:05.089554+0000","last_deep_scrub":"60'3475","last_deep_scrub_stamp":"2024-09-11T00:57:05.089554+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.089554+0000","objects_scrubbed":0,"log_size":3475,"log_dups_size":0,"ondisk_log_size":3475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:59:14.079385+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17645,"num_write_kb":295948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"62'3302","reported_seq":3409,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.617226+0000","last_change":"2024-09-11T00:57:18.097797+0000","last_active":"2024-09-11T00:57:34.617226+0000","last_peered":"2024-09-11T00:57:34.617226+0000","last_clean":"2024-09-11T00:57:34.617226+0000","last_became_active":"2024-09-11T00:30:54.982411+0000","last_became_peered":"2024-09-11T00:30:54.982411+0000","last_unstale":"2024-09-11T00:57:34.617226+0000","last_undegraded":"2024-09-11T00:57:34.617226+0000","last_fullsized":"2024-09-11T00:57:34.617226+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3302","last_scrub_stamp":"2024-09-11T00:57:18.097687+0000","last_deep_scrub":"62'3302","last_deep_scrub_stamp":"2024-09-11T00:57:18.097687+0000","last_clean_scrub_stamp":"2024-09-11T00:57:18.097687+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:37:14.784751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17558,"num_write_kb":271021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"60'3295","reported_seq":3409,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.608230+0000","last_change":"2024-09-11T00:57:00.766474+0000","last_active":"2024-09-11T00:57:34.608230+0000","last_peered":"2024-09-11T00:57:34.608230+0000","last_clean":"2024-09-11T00:57:34.608230+0000","last_became_active":"2024-09-11T00:30:45.926404+0000","last_became_peered":"2024-09-11T00:30:45.926404+0000","last_unstale":"2024-09-11T00:57:34.608230+0000","last_undegraded":"2024-09-11T00:57:34.608230+0000","last_fullsized":"2024-09-11T00:57:34.608230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3295","last_scrub_stamp":"2024-09-11T00:57:00.766374+0000","last_deep_scrub":"60'3295","last_deep_scrub_stamp":"2024-09-11T00:57:00.766374+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.766374+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:42:57.783884+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":25,"num_write":17271,"num_write_kb":265887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"61'3033","reported_seq":3139,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:19.762055+0000","last_change":"2024-09-11T00:57:19.762055+0000","last_active":"2024-09-11T00:57:19.762055+0000","last_peered":"2024-09-11T00:57:19.762055+0000","last_clean":"2024-09-11T00:57:19.762055+0000","last_became_active":"2024-09-11T00:30:54.991673+0000","last_became_peered":"2024-09-11T00:30:54.991673+0000","last_unstale":"2024-09-11T00:57:19.762055+0000","last_undegraded":"2024-09-11T00:57:19.762055+0000","last_fullsized":"2024-09-11T00:57:19.762055+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3033","last_scrub_stamp":"2024-09-11T00:57:19.761953+0000","last_deep_scrub":"61'3033","last_deep_scrub_stamp":"2024-09-11T00:57:19.761953+0000","last_clean_scrub_stamp":"2024-09-11T00:57:19.761953+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:29:56.631268+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16092,"num_write_kb":252889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"60'2629","reported_seq":2743,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616384+0000","last_change":"2024-09-11T00:56:59.077118+0000","last_active":"2024-09-11T00:57:34.616384+0000","last_peered":"2024-09-11T00:57:34.616384+0000","last_clean":"2024-09-11T00:57:34.616384+0000","last_became_active":"2024-09-11T00:30:45.928597+0000","last_became_peered":"2024-09-11T00:30:45.928597+0000","last_unstale":"2024-09-11T00:57:34.616384+0000","last_undegraded":"2024-09-11T00:57:34.616384+0000","last_fullsized":"2024-09-11T00:57:34.616384+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2629","last_scrub_stamp":"2024-09-11T00:56:59.077028+0000","last_deep_scrub":"60'2629","last_deep_scrub_stamp":"2024-09-11T00:56:59.077028+0000","last_clean_scrub_stamp":"2024-09-11T00:56:59.077028+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:31.814418+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":13758,"num_write_kb":227101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"62'3227","reported_seq":3334,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.612063+0000","last_change":"2024-09-11T00:57:04.462923+0000","last_active":"2024-09-11T00:57:34.612063+0000","last_peered":"2024-09-11T00:57:34.612063+0000","last_clean":"2024-09-11T00:57:34.612063+0000","last_became_active":"2024-09-11T00:30:54.990233+0000","last_became_peered":"2024-09-11T00:30:54.990233+0000","last_unstale":"2024-09-11T00:57:34.612063+0000","last_undegraded":"2024-09-11T00:57:34.612063+0000","last_fullsized":"2024-09-11T00:57:34.612063+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3227","last_scrub_stamp":"2024-09-11T00:57:04.462848+0000","last_deep_scrub":"62'3227","last_deep_scrub_stamp":"2024-09-11T00:57:04.462848+0000","last_clean_scrub_stamp":"2024-09-11T00:57:04.462848+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:58:17.891741+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16396,"num_write_kb":258131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"60'3415","reported_seq":3530,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607343+0000","last_change":"2024-09-11T00:57:10.894054+0000","last_active":"2024-09-11T00:57:34.607343+0000","last_peered":"2024-09-11T00:57:34.607343+0000","last_clean":"2024-09-11T00:57:34.607343+0000","last_became_active":"2024-09-11T00:30:45.933362+0000","last_became_peered":"2024-09-11T00:30:45.933362+0000","last_unstale":"2024-09-11T00:57:34.607343+0000","last_undegraded":"2024-09-11T00:57:34.607343+0000","last_fullsized":"2024-09-11T00:57:34.607343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3415","last_scrub_stamp":"2024-09-11T00:57:10.893962+0000","last_deep_scrub":"60'3415","last_deep_scrub_stamp":"2024-09-11T00:57:10.893962+0000","last_clean_scrub_stamp":"2024-09-11T00:57:10.893962+0000","objects_scrubbed":0,"log_size":3415,"log_dups_size":0,"ondisk_log_size":3415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:57:50.507574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":6,"num_write":17180,"num_write_kb":310287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"62'2771","reported_seq":2874,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T00:57:11.315886+0000","last_change":"2024-09-11T00:57:11.315886+0000","last_active":"2024-09-11T00:57:11.315886+0000","last_peered":"2024-09-11T00:57:11.315886+0000","last_clean":"2024-09-11T00:57:11.315886+0000","last_became_active":"2024-09-11T00:30:54.990497+0000","last_became_peered":"2024-09-11T00:30:54.990497+0000","last_unstale":"2024-09-11T00:57:11.315886+0000","last_undegraded":"2024-09-11T00:57:11.315886+0000","last_fullsized":"2024-09-11T00:57:11.315886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'2771","last_scrub_stamp":"2024-09-11T00:57:11.315776+0000","last_deep_scrub":"62'2771","last_deep_scrub_stamp":"2024-09-11T00:57:11.315776+0000","last_clean_scrub_stamp":"2024-09-11T00:57:11.315776+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:53:59.952427+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14317,"num_write_kb":221149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"60'2741","reported_seq":2857,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.617059+0000","last_change":"2024-09-11T00:57:00.094883+0000","last_active":"2024-09-11T00:57:34.617059+0000","last_peered":"2024-09-11T00:57:34.617059+0000","last_clean":"2024-09-11T00:57:34.617059+0000","last_became_active":"2024-09-11T00:30:45.930891+0000","last_became_peered":"2024-09-11T00:30:45.930891+0000","last_unstale":"2024-09-11T00:57:34.617059+0000","last_undegraded":"2024-09-11T00:57:34.617059+0000","last_fullsized":"2024-09-11T00:57:34.617059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2741","last_scrub_stamp":"2024-09-11T00:57:00.094785+0000","last_deep_scrub":"60'2741","last_deep_scrub_stamp":"2024-09-11T00:57:00.094785+0000","last_clean_scrub_stamp":"2024-09-11T00:57:00.094785+0000","objects_scrubbed":0,"log_size":2741,"log_dups_size":0,"ondisk_log_size":2741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:11:33.596446+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14755,"num_write_kb":255665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"62'3704","reported_seq":3809,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.605812+0000","last_change":"2024-09-11T00:57:18.081280+0000","last_active":"2024-09-11T00:57:34.605812+0000","last_peered":"2024-09-11T00:57:34.605812+0000","last_clean":"2024-09-11T00:57:34.605812+0000","last_became_active":"2024-09-11T00:30:54.984699+0000","last_became_peered":"2024-09-11T00:30:54.984699+0000","last_unstale":"2024-09-11T00:57:34.605812+0000","last_undegraded":"2024-09-11T00:57:34.605812+0000","last_fullsized":"2024-09-11T00:57:34.605812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3704","last_scrub_stamp":"2024-09-11T00:57:18.081174+0000","last_deep_scrub":"62'3704","last_deep_scrub_stamp":"2024-09-11T00:57:18.081174+0000","last_clean_scrub_stamp":"2024-09-11T00:57:18.081174+0000","objects_scrubbed":0,"log_size":3704,"log_dups_size":0,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:05:02.673786+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18483,"num_write_kb":307447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"60'3035","reported_seq":3150,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.606362+0000","last_change":"2024-09-11T00:57:06.063094+0000","last_active":"2024-09-11T00:57:34.606362+0000","last_peered":"2024-09-11T00:57:34.606362+0000","last_clean":"2024-09-11T00:57:34.606362+0000","last_became_active":"2024-09-11T00:30:45.939956+0000","last_became_peered":"2024-09-11T00:30:45.939956+0000","last_unstale":"2024-09-11T00:57:34.606362+0000","last_undegraded":"2024-09-11T00:57:34.606362+0000","last_fullsized":"2024-09-11T00:57:34.606362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'3035","last_scrub_stamp":"2024-09-11T00:57:06.062988+0000","last_deep_scrub":"60'3035","last_deep_scrub_stamp":"2024-09-11T00:57:06.062988+0000","last_clean_scrub_stamp":"2024-09-11T00:57:06.062988+0000","objects_scrubbed":0,"log_size":3035,"log_dups_size":0,"ondisk_log_size":3035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:14:10.554592+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15470,"num_write_kb":262623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"62'3351","reported_seq":3457,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.616045+0000","last_change":"2024-09-11T00:57:15.099297+0000","last_active":"2024-09-11T00:57:34.616045+0000","last_peered":"2024-09-11T00:57:34.616045+0000","last_clean":"2024-09-11T00:57:34.616045+0000","last_became_active":"2024-09-11T00:30:54.975228+0000","last_became_peered":"2024-09-11T00:30:54.975228+0000","last_unstale":"2024-09-11T00:57:34.616045+0000","last_undegraded":"2024-09-11T00:57:34.616045+0000","last_fullsized":"2024-09-11T00:57:34.616045+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3351","last_scrub_stamp":"2024-09-11T00:57:15.099191+0000","last_deep_scrub":"62'3351","last_deep_scrub_stamp":"2024-09-11T00:57:15.099191+0000","last_clean_scrub_stamp":"2024-09-11T00:57:15.099191+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:19:12.755943+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17034,"num_write_kb":277088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"60'4106","reported_seq":4223,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.607589+0000","last_change":"2024-09-11T00:57:09.161054+0000","last_active":"2024-09-11T00:57:34.607589+0000","last_peered":"2024-09-11T00:57:34.607589+0000","last_clean":"2024-09-11T00:57:34.607589+0000","last_became_active":"2024-09-11T00:30:45.937250+0000","last_became_peered":"2024-09-11T00:30:45.937250+0000","last_unstale":"2024-09-11T00:57:34.607589+0000","last_undegraded":"2024-09-11T00:57:34.607589+0000","last_fullsized":"2024-09-11T00:57:34.607589+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'4106","last_scrub_stamp":"2024-09-11T00:57:09.160958+0000","last_deep_scrub":"60'4106","last_deep_scrub_stamp":"2024-09-11T00:57:09.160958+0000","last_clean_scrub_stamp":"2024-09-11T00:57:09.160958+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:57:11.457246+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":11,"num_write":22024,"num_write_kb":354504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"62'3391","reported_seq":3496,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.615570+0000","last_change":"2024-09-11T00:57:16.092148+0000","last_active":"2024-09-11T00:57:34.615570+0000","last_peered":"2024-09-11T00:57:34.615570+0000","last_clean":"2024-09-11T00:57:34.615570+0000","last_became_active":"2024-09-11T00:30:54.989344+0000","last_became_peered":"2024-09-11T00:30:54.989344+0000","last_unstale":"2024-09-11T00:57:34.615570+0000","last_undegraded":"2024-09-11T00:57:34.615570+0000","last_fullsized":"2024-09-11T00:57:34.615570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'3391","last_scrub_stamp":"2024-09-11T00:57:16.092063+0000","last_deep_scrub":"62'3391","last_deep_scrub_stamp":"2024-09-11T00:57:16.092063+0000","last_clean_scrub_stamp":"2024-09-11T00:57:16.092063+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:34:30.457109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18265,"num_write_kb":274031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"60'2946","reported_seq":3063,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-11T00:57:34.609479+0000","last_change":"2024-09-11T00:57:05.857764+0000","last_active":"2024-09-11T00:57:34.609479+0000","last_peered":"2024-09-11T00:57:34.609479+0000","last_clean":"2024-09-11T00:57:34.609479+0000","last_became_active":"2024-09-11T00:30:45.941448+0000","last_became_peered":"2024-09-11T00:30:45.941448+0000","last_unstale":"2024-09-11T00:57:34.609479+0000","last_undegraded":"2024-09-11T00:57:34.609479+0000","last_fullsized":"2024-09-11T00:57:34.609479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'2946","last_scrub_stamp":"2024-09-11T00:57:05.857658+0000","last_deep_scrub":"60'2946","last_deep_scrub_stamp":"2024-09-11T00:57:05.857658+0000","last_clean_scrub_stamp":"2024-09-11T00:57:05.857658+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:12:07.015151+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":17,"num_write":15520,"num_write_kb":262805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1107571,"num_write_kb":17543218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212907,"ondisk_log_size":212907,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18223220,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1558,"num_read_kb":197403,"num_write":4817,"num_write_kb":19773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36511744,"data_stored":36446440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4437,"ondisk_log_size":4437,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":381,"num_write":1025649,"num_write_kb":17211858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196678,"ondisk_log_size":196678,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":17635410,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1073,"num_read_kb":156974,"num_write":4568,"num_write_kb":19046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35332096,"data_stored":35270820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4141,"ondisk_log_size":4141,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509767,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487152,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":486270,"kb_avail":93884688,"statfs":{"total":96636764160,"available":96137920512,"internally_reserved":0,"allocated":884736,"data_stored":496637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":497940938},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509769,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668184,"kb_used_data":10308,"kb_used_omap":1,"kb_used_meta":657854,"kb_avail":93703656,"statfs":{"total":96636764160,"available":95952543744,"internally_reserved":0,"allocated":10555392,"data_stored":10163991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":673642950},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509768,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685684,"kb_used_data":17828,"kb_used_omap":1,"kb_used_meta":667838,"kb_avail":93686156,"statfs":{"total":96636764160,"available":95934623744,"internally_reserved":0,"allocated":18255872,"data_stored":17864749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683866565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509767,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722280,"kb_used_data":13912,"kb_used_omap":1,"kb_used_meta":708350,"kb_avail":93649560,"statfs":{"total":96636764160,"available":95897149440,"internally_reserved":0,"allocated":14245888,"data_stored":13849304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":725350856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509768,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737056,"kb_used_data":14480,"kb_used_omap":1,"kb_used_meta":722558,"kb_avail":93634784,"statfs":{"total":96636764160,"available":95882018816,"internally_reserved":0,"allocated":14827520,"data_stored":14436462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739899845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509768,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598360,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":596990,"kb_avail":93773480,"statfs":{"total":96636764160,"available":96024043520,"internally_reserved":0,"allocated":1384448,"data_stored":998818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":611318213},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509769,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741960,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":740414,"kb_avail":93629880,"statfs":{"total":96636764160,"available":95876997120,"internally_reserved":0,"allocated":1564672,"data_stored":1176999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":758184389},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542472,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739432,"kb_used_data":17176,"kb_used_omap":1,"kb_used_meta":722238,"kb_avail":93632408,"statfs":{"total":96636764160,"available":95879585792,"internally_reserved":0,"allocated":17588224,"data_stored":17187564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":739572165},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":778240,"data_stored":768599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9162752,"data_stored":9157733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4282577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5648384,"data_stored":5638872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4300800,"data_stored":4283261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5562368,"data_stored":5551329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T00:57:36.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T00:57:36.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.622+0000 7f2bc0200640 1 -- 172.21.15.31:0/2143920969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb8153e90 msgr2=0x7f2bb8154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.622+0000 7f2bc0200640 1 --2- 172.21.15.31:0/2143920969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb8153e90 0x7f2bb8154340 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80c3a40 tx=0x7f2bb002a820 comp rx=0 tx=0).stop 2024-09-11T00:57:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.622+0000 7f2bc0200640 1 -- 172.21.15.31:0/2143920969 shutdown_connections 2024-09-11T00:57:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.622+0000 7f2bc0200640 1 --2- 172.21.15.31:0/2143920969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb8153e90 0x7f2bb8154340 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.622+0000 7f2bc0200640 1 --2- 172.21.15.31:0/2143920969 >> v2:172.21.15.204:3300/0 conn(0x7f2bb80c2330 0x7f2bb80c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.622+0000 7f2bc0200640 1 -- 172.21.15.31:0/2143920969 >> 172.21.15.31:0/2143920969 conn(0x7f2bb806d560 msgr2=0x7f2bb80c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.623+0000 7f2bc0200640 1 -- 172.21.15.31:0/2143920969 shutdown_connections 2024-09-11T00:57:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.623+0000 7f2bc0200640 1 -- 172.21.15.31:0/2143920969 wait complete. 2024-09-11T00:57:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.624+0000 7f2bc0200640 1 Processor -- start 2024-09-11T00:57:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.624+0000 7f2bc0200640 1 -- start start 2024-09-11T00:57:36.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.625+0000 7f2bc0200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bb80c2330 0x7f2bb814ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:36.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.625+0000 7f2bc0200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb814de40 0x7f2bb81579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.625+0000 7f2bc0200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bb8153e90 0x7f2bb8157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.625+0000 7f2bc0200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb806e200 con 0x7f2bb814de40 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.625+0000 7f2bc0200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb806e380 con 0x7f2bb80c2330 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.625+0000 7f2bc0200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2bb814e220 con 0x7f2bb8153e90 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.625+0000 7f2bbce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb814de40 0x7f2bb81579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.625+0000 7f2bbce00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb814de40 0x7f2bb81579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42152/0 (socket says 172.21.15.31:42152) 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.625+0000 7f2bbce00640 1 -- 172.21.15.31:0/542659644 learned_addr learned my addr 172.21.15.31:0/542659644 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bbd800640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bb80c2330 0x7f2bb814ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bbe200640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bb8153e90 0x7f2bb8157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bbce00640 1 -- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bb8153e90 msgr2=0x7f2bb8157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bbce00640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bb8153e90 0x7f2bb8157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bbce00640 1 -- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bb80c2330 msgr2=0x7f2bb814ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bbce00640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bb80c2330 0x7f2bb814ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bbce00640 1 -- 172.21.15.31:0/542659644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb8148d10 con 0x7f2bb814de40 2024-09-11T00:57:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bbce00640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb814de40 0x7f2bb81579d0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400d980 tx=0x7f2ba400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bb5600640 1 -- 172.21.15.31:0/542659644 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4017070 con 0x7f2bb814de40 2024-09-11T00:57:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bb5600640 1 -- 172.21.15.31:0/542659644 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ba4009d20 con 0x7f2bb814de40 2024-09-11T00:57:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bc0200640 1 -- 172.21.15.31:0/542659644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb81edb60 con 0x7f2bb814de40 2024-09-11T00:57:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bb5600640 1 -- 172.21.15.31:0/542659644 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba40106c0 con 0x7f2bb814de40 2024-09-11T00:57:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.626+0000 7f2bc0200640 1 -- 172.21.15.31:0/542659644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb81ed7a0 con 0x7f2bb814de40 2024-09-11T00:57:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.627+0000 7f2bc0200640 1 -- 172.21.15.31:0/542659644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bb8154a40 con 0x7f2bb814de40 2024-09-11T00:57:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.628+0000 7f2bb5600640 1 -- 172.21.15.31:0/542659644 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2ba4010860 con 0x7f2bb814de40 2024-09-11T00:57:36.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.629+0000 7f2bb5600640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b98077f70 0x7f2b9807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:36.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.630+0000 7f2bb5600640 1 -- 172.21.15.31:0/542659644 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9762+0+0 (secure 0 0 0) 0x7f2ba409b160 con 0x7f2bb814de40 2024-09-11T00:57:36.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.630+0000 7f2bbd800640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b98077f70 0x7f2b9807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:36.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.631+0000 7f2bbd800640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b98077f70 0x7f2b9807a430 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0016b0 tx=0x7f2bac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:36.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.633+0000 7f2bb5600640 1 -- 172.21.15.31:0/542659644 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2ba4062c60 con 0x7f2bb814de40 2024-09-11T00:57:36.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.829+0000 7f2bc0200640 1 -- 172.21.15.31:0/542659644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f2bb80c6a80 con 0x7f2bb814de40 2024-09-11T00:57:36.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.835+0000 7f2bb5600640 1 -- 172.21.15.31:0/542659644 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v32)=0 v32) ==== 123+0+0 (secure 0 0 0) 0x7f2ba4062560 con 0x7f2bb814de40 2024-09-11T00:57:36.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.836+0000 7f2bb5600640 1 -- 172.21.15.31:0/542659644 <== mon.0 v2:172.21.15.31:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f2ba4062070 con 0x7f2bb814de40 2024-09-11T00:57:36.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 -- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b98077f70 msgr2=0x7f2b9807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:36.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b98077f70 0x7f2b9807a430 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0016b0 tx=0x7f2bac009040 comp rx=0 tx=0).stop 2024-09-11T00:57:36.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 -- 172.21.15.31:0/542659644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb814de40 msgr2=0x7f2bb81579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:36.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb814de40 0x7f2bb81579d0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400d980 tx=0x7f2ba400de40 comp rx=0 tx=0).stop 2024-09-11T00:57:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 -- 172.21.15.31:0/542659644 shutdown_connections 2024-09-11T00:57:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2b98077f70 0x7f2b9807a430 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2bb8153e90 0x7f2bb8157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bb814de40 0x7f2bb81579d0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 --2- 172.21.15.31:0/542659644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bb80c2330 0x7f2bb814ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 -- 172.21.15.31:0/542659644 >> 172.21.15.31:0/542659644 conn(0x7f2bb806d560 msgr2=0x7f2bb8074190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 -- 172.21.15.31:0/542659644 shutdown_connections 2024-09-11T00:57:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:36.838+0000 7f2bc0200640 1 -- 172.21.15.31:0/542659644 wait complete. 2024-09-11T00:57:36.856 INFO:tasks.cephfs.filesystem:Destroying file system a and related pools 2024-09-11T00:57:36.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:37.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.014+0000 7f4875800640 1 -- 172.21.15.31:0/810176848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4870156ad0 msgr2=0x7f4870153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.014+0000 7f4875800640 1 --2- 172.21.15.31:0/810176848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4870156ad0 0x7f4870153480 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f48700c6d20 tx=0x7f485802a560 comp rx=0 tx=0).stop 2024-09-11T00:57:37.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.014+0000 7f4875800640 1 -- 172.21.15.31:0/810176848 shutdown_connections 2024-09-11T00:57:37.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.014+0000 7f4875800640 1 --2- 172.21.15.31:0/810176848 >> v2:172.21.15.204:3300/0 conn(0x7f48701539c0 0x7f4870159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.014+0000 7f4875800640 1 --2- 172.21.15.31:0/810176848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4870156ad0 0x7f4870153480 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.014+0000 7f4875800640 1 -- 172.21.15.31:0/810176848 >> 172.21.15.31:0/810176848 conn(0x7f48700a9750 msgr2=0x7f48700a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:37.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.014+0000 7f4875800640 1 -- 172.21.15.31:0/810176848 shutdown_connections 2024-09-11T00:57:37.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.014+0000 7f4875800640 1 -- 172.21.15.31:0/810176848 wait complete. 2024-09-11T00:57:37.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.016+0000 7f4875800640 1 Processor -- start 2024-09-11T00:57:37.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.016+0000 7f4875800640 1 -- start start 2024-09-11T00:57:37.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f4875800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48700c5b10 0x7f48701dfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f4875800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48701539c0 0x7f487014a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f4875800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4870156ad0 0x7f487014cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f4875800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4870156f20 con 0x7f48701539c0 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f4875800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f48700c7320 con 0x7f4870156ad0 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f4875800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f48700c3950 con 0x7f48700c5b10 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f4866000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48701539c0 0x7f487014a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f4866000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48701539c0 0x7f487014a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42162/0 (socket says 172.21.15.31:42162) 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f4866000640 1 -- 172.21.15.31:0/1528886478 learned_addr learned my addr 172.21.15.31:0/1528886478 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f486f400640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4870156ad0 0x7f487014cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.017+0000 7f486ea00640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48700c5b10 0x7f48701dfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4866000640 1 -- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48700c5b10 msgr2=0x7f48701dfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4866000640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48700c5b10 0x7f48701dfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4866000640 1 -- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4870156ad0 msgr2=0x7f487014cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4866000640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4870156ad0 0x7f487014cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4866000640 1 -- 172.21.15.31:0/1528886478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4870155390 con 0x7f48701539c0 2024-09-11T00:57:37.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f486f400640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4870156ad0 0x7f487014cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:37.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4866000640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48701539c0 0x7f487014a9a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f48580099c0 tx=0x7f4858003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:37.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4867400640 1 -- 172.21.15.31:0/1528886478 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485803e030 con 0x7f48701539c0 2024-09-11T00:57:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4867400640 1 -- 172.21.15.31:0/1528886478 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f4858004730 con 0x7f48701539c0 2024-09-11T00:57:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4875800640 1 -- 172.21.15.31:0/1528886478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48701ecd40 con 0x7f48701539c0 2024-09-11T00:57:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4867400640 1 -- 172.21.15.31:0/1528886478 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4858035040 con 0x7f48701539c0 2024-09-11T00:57:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.018+0000 7f4875800640 1 -- 172.21.15.31:0/1528886478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4870151170 con 0x7f48701539c0 2024-09-11T00:57:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.019+0000 7f4875800640 1 -- 172.21.15.31:0/1528886478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f487006d910 con 0x7f48701539c0 2024-09-11T00:57:37.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.021+0000 7f4867400640 1 -- 172.21.15.31:0/1528886478 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f4858004ba0 con 0x7f48701539c0 2024-09-11T00:57:37.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.021+0000 7f4867400640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f484c077f70 0x7f484c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.022+0000 7f4867400640 1 -- 172.21.15.31:0/1528886478 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9762+0+0 (secure 0 0 0) 0x7f48580bb640 con 0x7f48701539c0 2024-09-11T00:57:37.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.022+0000 7f486ea00640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f484c077f70 0x7f484c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.023+0000 7f486ea00640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f484c077f70 0x7f484c07a430 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4860004ea0 tx=0x7f4860009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:37.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.026+0000 7f4867400640 1 -- 172.21.15.31:0/1528886478 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f487006d910 con 0x7f48701539c0 2024-09-11T00:57:37.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.255+0000 7f4875800640 1 -- 172.21.15.31:0/1528886478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4870159670 con 0x7f48701539c0 2024-09-11T00:57:37.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.256+0000 7f4867400640 1 -- 172.21.15.31:0/1528886478 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+5710 (secure 0 0 0) 0x7f48580831a0 con 0x7f48701539c0 2024-09-11T00:57:37.259 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:37.259 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":189,"btime":"2024-09-11T00:57:36:607195+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:57:35.608933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7824},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7824":{"gid":7824,"name":"a","rank":0,"incarnation":187,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:37.259 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 189 2024-09-11T00:57:37.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 -- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f484c077f70 msgr2=0x7f484c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f484c077f70 0x7f484c07a430 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4860004ea0 tx=0x7f4860009040 comp rx=0 tx=0).stop 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 -- 172.21.15.31:0/1528886478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48701539c0 msgr2=0x7f487014a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48701539c0 0x7f487014a9a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f48580099c0 tx=0x7f4858003970 comp rx=0 tx=0).stop 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 -- 172.21.15.31:0/1528886478 shutdown_connections 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f484c077f70 0x7f484c07a430 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4870156ad0 0x7f487014cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48701539c0 0x7f487014a9a0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 --2- 172.21.15.31:0/1528886478 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f48700c5b10 0x7f48701dfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 -- 172.21.15.31:0/1528886478 >> 172.21.15.31:0/1528886478 conn(0x7f48700a9750 msgr2=0x7f487006f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 -- 172.21.15.31:0/1528886478 shutdown_connections 2024-09-11T00:57:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.259+0000 7f4875800640 1 -- 172.21.15.31:0/1528886478 wait complete. 2024-09-11T00:57:37.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:37.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:37.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.436+0000 7f6e25600640 1 -- 172.21.15.31:0/2225708325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2006f340 msgr2=0x7f6e200add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.436+0000 7f6e25600640 1 --2- 172.21.15.31:0/2225708325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2006f340 0x7f6e200add60 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f6e18005fd0 tx=0x7f6e1802a820 comp rx=0 tx=0).stop 2024-09-11T00:57:37.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.437+0000 7f6e25600640 1 -- 172.21.15.31:0/2225708325 shutdown_connections 2024-09-11T00:57:37.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.437+0000 7f6e25600640 1 --2- 172.21.15.31:0/2225708325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2006f340 0x7f6e200add60 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.437+0000 7f6e25600640 1 -- 172.21.15.31:0/2225708325 >> 172.21.15.31:0/2225708325 conn(0x7f6e200688e0 msgr2=0x7f6e20068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:37.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.437+0000 7f6e25600640 1 -- 172.21.15.31:0/2225708325 shutdown_connections 2024-09-11T00:57:37.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.437+0000 7f6e25600640 1 -- 172.21.15.31:0/2225708325 wait complete. 2024-09-11T00:57:37.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.438+0000 7f6e25600640 1 Processor -- start 2024-09-11T00:57:37.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.439+0000 7f6e25600640 1 -- start start 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.439+0000 7f6e25600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e2006f340 0x7f6e20154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.439+0000 7f6e25600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e200c5470 0x7f6e2014e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.439+0000 7f6e25600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2015a580 0x7f6e2014e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.439+0000 7f6e25600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6e200c5850 con 0x7f6e2015a580 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.439+0000 7f6e25600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2006fb40 con 0x7f6e2006f340 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.439+0000 7f6e25600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6e200c59d0 con 0x7f6e200c5470 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.440+0000 7f6e1f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2015a580 0x7f6e2014e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.440+0000 7f6e1f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2015a580 0x7f6e2014e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42186/0 (socket says 172.21.15.31:42186) 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.440+0000 7f6e1f400640 1 -- 172.21.15.31:0/956313598 learned_addr learned my addr 172.21.15.31:0/956313598 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.440+0000 7f6e1ea00640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e2006f340 0x7f6e20154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.440+0000 7f6e16000640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e200c5470 0x7f6e2014e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.440+0000 7f6e1f400640 1 -- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e200c5470 msgr2=0x7f6e2014e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.440+0000 7f6e1f400640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e200c5470 0x7f6e2014e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.440+0000 7f6e1f400640 1 -- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e2006f340 msgr2=0x7f6e20154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.440+0000 7f6e1f400640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e2006f340 0x7f6e20154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.440+0000 7f6e1f400640 1 -- 172.21.15.31:0/956313598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e20159310 con 0x7f6e2015a580 2024-09-11T00:57:37.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.441+0000 7f6e1f400640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2015a580 0x7f6e2014e640 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f6e18006010 tx=0x7f6e18032cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:37.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.441+0000 7f6e17400640 1 -- 172.21.15.31:0/956313598 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18037040 con 0x7f6e2015a580 2024-09-11T00:57:37.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.441+0000 7f6e17400640 1 -- 172.21.15.31:0/956313598 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f6e18042070 con 0x7f6e2015a580 2024-09-11T00:57:37.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.441+0000 7f6e25600640 1 -- 172.21.15.31:0/956313598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e201595d0 con 0x7f6e2015a580 2024-09-11T00:57:37.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.441+0000 7f6e17400640 1 -- 172.21.15.31:0/956313598 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e180042a0 con 0x7f6e2015a580 2024-09-11T00:57:37.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.441+0000 7f6e25600640 1 -- 172.21.15.31:0/956313598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e20157ac0 con 0x7f6e2015a580 2024-09-11T00:57:37.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.442+0000 7f6e25600640 1 -- 172.21.15.31:0/956313598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e200c4760 con 0x7f6e2015a580 2024-09-11T00:57:37.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.443+0000 7f6e17400640 1 -- 172.21.15.31:0/956313598 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6e1803a030 con 0x7f6e2015a580 2024-09-11T00:57:37.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.444+0000 7f6e17400640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6dfc077f70 0x7f6dfc07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.444+0000 7f6e17400640 1 -- 172.21.15.31:0/956313598 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9762+0+0 (secure 0 0 0) 0x7f6e180bb6c0 con 0x7f6e2015a580 2024-09-11T00:57:37.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.445+0000 7f6e1ea00640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6dfc077f70 0x7f6dfc07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.446+0000 7f6e1ea00640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6dfc077f70 0x7f6dfc07a430 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6e100046b0 tx=0x7f6e10007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:37.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.448+0000 7f6e17400640 1 -- 172.21.15.31:0/956313598 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6e18082b70 con 0x7f6e2015a580 2024-09-11T00:57:37.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.482+0000 7fb659400640 1 -- 172.21.15.31:0/1112404584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb654070360 msgr2=0x7fb6540712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.482+0000 7fb659400640 1 --2- 172.21.15.31:0/1112404584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb654070360 0x7fb6540712a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb640009900 tx=0x7fb64002a610 comp rx=0 tx=0).stop 2024-09-11T00:57:37.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.482+0000 7fb659400640 1 -- 172.21.15.31:0/1112404584 shutdown_connections 2024-09-11T00:57:37.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.482+0000 7fb659400640 1 --2- 172.21.15.31:0/1112404584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb654070360 0x7fb6540712a0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.482+0000 7fb659400640 1 --2- 172.21.15.31:0/1112404584 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb6540a7e40 0x7fb6540a8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.482+0000 7fb659400640 1 -- 172.21.15.31:0/1112404584 >> 172.21.15.31:0/1112404584 conn(0x7fb65406dc00 msgr2=0x7fb65406e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:37.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.482+0000 7fb659400640 1 -- 172.21.15.31:0/1112404584 shutdown_connections 2024-09-11T00:57:37.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.482+0000 7fb659400640 1 -- 172.21.15.31:0/1112404584 wait complete. 2024-09-11T00:57:37.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.484+0000 7fb659400640 1 Processor -- start 2024-09-11T00:57:37.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.484+0000 7fb659400640 1 -- start start 2024-09-11T00:57:37.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.485+0000 7fb659400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb654070360 0x7fb6541513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.485+0000 7fb659400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6540717e0 0x7fb6541e9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.485+0000 7fb659400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6540a7e40 0x7fb6541ea2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.485+0000 7fb659400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb654155490 con 0x7fb6540717e0 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.485+0000 7fb659400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb6540c2f30 con 0x7fb6540a7e40 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.485+0000 7fb659400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fb654071c90 con 0x7fb654070360 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.485+0000 7fb651e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6540717e0 0x7fb6541e9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.485+0000 7fb651e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6540717e0 0x7fb6541e9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42204/0 (socket says 172.21.15.31:42204) 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.485+0000 7fb651e00640 1 -- 172.21.15.31:0/3898358529 learned_addr learned my addr 172.21.15.31:0/3898358529 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.486+0000 7fb652800640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb654070360 0x7fb6541513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.486+0000 7fb653200640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6540a7e40 0x7fb6541ea2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.486+0000 7fb651e00640 1 -- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb654070360 msgr2=0x7fb6541513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.486+0000 7fb651e00640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb654070360 0x7fb6541513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.486+0000 7fb651e00640 1 -- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6540a7e40 msgr2=0x7fb6541ea2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.486+0000 7fb651e00640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6540a7e40 0x7fb6541ea2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.486+0000 7fb651e00640 1 -- 172.21.15.31:0/3898358529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6540c3520 con 0x7fb6540717e0 2024-09-11T00:57:37.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.486+0000 7fb652800640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb654070360 0x7fb6541513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:37.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.486+0000 7fb651e00640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6540717e0 0x7fb6541e9da0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fb640000c00 tx=0x7fb6400038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:37.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.487+0000 7fb64aa00640 1 -- 172.21.15.31:0/3898358529 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64003e030 con 0x7fb6540717e0 2024-09-11T00:57:37.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.487+0000 7fb64aa00640 1 -- 172.21.15.31:0/3898358529 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fb640003ca0 con 0x7fb6540717e0 2024-09-11T00:57:37.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.487+0000 7fb64aa00640 1 -- 172.21.15.31:0/3898358529 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640035040 con 0x7fb6540717e0 2024-09-11T00:57:37.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.487+0000 7fb659400640 1 -- 172.21.15.31:0/3898358529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6541ea9c0 con 0x7fb6540717e0 2024-09-11T00:57:37.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.487+0000 7fb659400640 1 -- 172.21.15.31:0/3898358529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb65414e270 con 0x7fb6540717e0 2024-09-11T00:57:37.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.488+0000 7fb659400640 1 -- 172.21.15.31:0/3898358529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6540712a0 con 0x7fb6540717e0 2024-09-11T00:57:37.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.490+0000 7fb64aa00640 1 -- 172.21.15.31:0/3898358529 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fb640004990 con 0x7fb6540717e0 2024-09-11T00:57:37.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.491+0000 7fb64aa00640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb634077fa0 0x7fb63407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.491+0000 7fb64aa00640 1 -- 172.21.15.31:0/3898358529 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9762+0+0 (secure 0 0 0) 0x7fb6400bb3e0 con 0x7fb6540717e0 2024-09-11T00:57:37.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.491+0000 7fb652800640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb634077fa0 0x7fb63407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.492+0000 7fb652800640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb634077fa0 0x7fb63407a460 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb6540c2d40 tx=0x7fb644005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:37.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.496+0000 7fb64aa00640 1 -- 172.21.15.31:0/3898358529 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fb640082ee0 con 0x7fb6540717e0 2024-09-11T00:57:37.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.689+0000 7f6e25600640 1 -- 172.21.15.31:0/956313598 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e201e15f0 con 0x7f6e2015a580 2024-09-11T00:57:37.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.690+0000 7f6e17400640 1 -- 172.21.15.31:0/956313598 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+5717 (secure 0 0 0) 0x7f6e180875a0 con 0x7f6e2015a580 2024-09-11T00:57:37.693 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:37.693 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":190,"btime":"2024-09-11T00:57:37:610203+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:57:37.610199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7824},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7824":{"gid":7824,"name":"a","rank":0,"incarnation":187,"state":"up:active","state_seq":17,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7824,"qdb_cluster":[7824]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:37.693 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 190 2024-09-11T00:57:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 -- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6dfc077f70 msgr2=0x7f6dfc07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6dfc077f70 0x7f6dfc07a430 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6e100046b0 tx=0x7f6e10007eb0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 -- 172.21.15.31:0/956313598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2015a580 msgr2=0x7f6e2014e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2015a580 0x7f6e2014e640 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f6e18006010 tx=0x7f6e18032cb0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 -- 172.21.15.31:0/956313598 shutdown_connections 2024-09-11T00:57:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6dfc077f70 0x7f6dfc07a430 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e2015a580 0x7f6e2014e640 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6e200c5470 0x7f6e2014e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 --2- 172.21.15.31:0/956313598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e2006f340 0x7f6e20154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 -- 172.21.15.31:0/956313598 >> 172.21.15.31:0/956313598 conn(0x7f6e200688e0 msgr2=0x7f6e20068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 -- 172.21.15.31:0/956313598 shutdown_connections 2024-09-11T00:57:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.693+0000 7f6e25600640 1 -- 172.21.15.31:0/956313598 wait complete. 2024-09-11T00:57:37.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:57:37.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.736+0000 7fb659400640 1 -- 172.21.15.31:0/3898358529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb654143610 con 0x7fb6540717e0 2024-09-11T00:57:37.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.736+0000 7fb64aa00640 1 -- 172.21.15.31:0/3898358529 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+5717 (secure 0 0 0) 0x7fb6400827e0 con 0x7fb6540717e0 2024-09-11T00:57:37.739 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:37.739 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":190,"btime":"2024-09-11T00:57:37:610203+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:45.342170+0000","modified":"2024-09-11T00:57:37.610199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7824},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7824":{"gid":7824,"name":"a","rank":0,"incarnation":187,"state":"up:active","state_seq":17,"addr":"172.21.15.31:6833/1834111946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":1834111946},{"type":"v1","addr":"172.21.15.31:6833","nonce":1834111946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7824,"qdb_cluster":[7824]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:37.740 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 190 2024-09-11T00:57:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 -- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb634077fa0 msgr2=0x7fb63407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb634077fa0 0x7fb63407a460 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb6540c2d40 tx=0x7fb644005f90 comp rx=0 tx=0).stop 2024-09-11T00:57:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 -- 172.21.15.31:0/3898358529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6540717e0 msgr2=0x7fb6541e9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6540717e0 0x7fb6541e9da0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fb640000c00 tx=0x7fb6400038a0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 -- 172.21.15.31:0/3898358529 shutdown_connections 2024-09-11T00:57:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fb634077fa0 0x7fb63407a460 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb6540a7e40 0x7fb6541ea2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb6540717e0 0x7fb6541e9da0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 --2- 172.21.15.31:0/3898358529 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fb654070360 0x7fb6541513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 -- 172.21.15.31:0/3898358529 >> 172.21.15.31:0/3898358529 conn(0x7fb65406dc00 msgr2=0x7fb6541e01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:37.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 -- 172.21.15.31:0/3898358529 shutdown_connections 2024-09-11T00:57:37.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.739+0000 7fb659400640 1 -- 172.21.15.31:0/3898358529 wait complete. 2024-09-11T00:57:37.758 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=7672 2024-09-11T00:57:37.758 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.d 2024-09-11T00:57:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.872+0000 7f2f15e00640 1 -- 172.21.15.31:0/261918174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f100c23a0 msgr2=0x7f2f101516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.872+0000 7f2f15e00640 1 --2- 172.21.15.31:0/261918174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f100c23a0 0x7f2f101516e0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f2f100c4400 tx=0x7f2ef802a5f0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.872+0000 7f2f15e00640 1 -- 172.21.15.31:0/261918174 shutdown_connections 2024-09-11T00:57:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.872+0000 7f2f15e00640 1 --2- 172.21.15.31:0/261918174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f100c23a0 0x7f2f101516e0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.872+0000 7f2f15e00640 1 --2- 172.21.15.31:0/261918174 >> v2:172.21.15.204:3300/0 conn(0x7f2f10072480 0x7f2f101511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.872+0000 7f2f15e00640 1 -- 172.21.15.31:0/261918174 >> 172.21.15.31:0/261918174 conn(0x7f2f100a9750 msgr2=0x7f2f100a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.872+0000 7f2f15e00640 1 -- 172.21.15.31:0/261918174 shutdown_connections 2024-09-11T00:57:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.872+0000 7f2f15e00640 1 -- 172.21.15.31:0/261918174 wait complete. 2024-09-11T00:57:37.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.874+0000 7f2f15e00640 1 Processor -- start 2024-09-11T00:57:37.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.874+0000 7f2f15e00640 1 -- start start 2024-09-11T00:57:37.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.874+0000 7f2f15e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f10072480 0x7f2f101ec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.874+0000 7f2f15e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f100c23a0 0x7f2f100c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.874+0000 7f2f15e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f10153f20 0x7f2f101df630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f15e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2f10074200 con 0x7f2f10153f20 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f15e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2f10074080 con 0x7f2f10072480 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f15e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f2f10072860 con 0x7f2f100c23a0 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f0f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f10153f20 0x7f2f101df630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f0f400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f10153f20 0x7f2f101df630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55330/0 (socket says 172.21.15.31:55330) 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f0f400640 1 -- 172.21.15.31:0/1546988912 learned_addr learned my addr 172.21.15.31:0/1546988912 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f0ea00640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f10072480 0x7f2f101ec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f06000640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f100c23a0 0x7f2f100c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f0f400640 1 -- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f100c23a0 msgr2=0x7f2f100c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f0f400640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f100c23a0 0x7f2f100c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f0f400640 1 -- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f10072480 msgr2=0x7f2f101ec5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:37.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f0f400640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f10072480 0x7f2f101ec5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:37.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.875+0000 7f2f0f400640 1 -- 172.21.15.31:0/1546988912 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f101ecaf0 con 0x7f2f10153f20 2024-09-11T00:57:37.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.876+0000 7f2f0f400640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f10153f20 0x7f2f101df630 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800b970 tx=0x7f2f0800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:37.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.876+0000 7f2f07e00640 1 -- 172.21.15.31:0/1546988912 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08019070 con 0x7f2f10153f20 2024-09-11T00:57:37.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.876+0000 7f2f07e00640 1 -- 172.21.15.31:0/1546988912 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f2f0800cb40 con 0x7f2f10153f20 2024-09-11T00:57:37.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.876+0000 7f2f07e00640 1 -- 172.21.15.31:0/1546988912 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08014440 con 0x7f2f10153f20 2024-09-11T00:57:37.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.876+0000 7f2f15e00640 1 -- 172.21.15.31:0/1546988912 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f101dee70 con 0x7f2f10153f20 2024-09-11T00:57:37.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.876+0000 7f2f15e00640 1 -- 172.21.15.31:0/1546988912 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f101dd7d0 con 0x7f2f10153f20 2024-09-11T00:57:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.877+0000 7f2f15e00640 1 -- 172.21.15.31:0/1546988912 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f100c4a10 con 0x7f2f10153f20 2024-09-11T00:57:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.878+0000 7f2f07e00640 1 -- 172.21.15.31:0/1546988912 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f2f0801e440 con 0x7f2f10153f20 2024-09-11T00:57:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.879+0000 7f2f07e00640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2eec077f70 0x7f2eec07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.879+0000 7f2f07e00640 1 -- 172.21.15.31:0/1546988912 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9762+0+0 (secure 0 0 0) 0x7f2f0809acc0 con 0x7f2f10153f20 2024-09-11T00:57:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.880+0000 7f2f0ea00640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2eec077f70 0x7f2eec07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.880+0000 7f2f0ea00640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2eec077f70 0x7f2eec07a430 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2f00005fd0 tx=0x7f2f00005850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:37.883+0000 7f2f07e00640 1 -- 172.21.15.31:0/1546988912 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f2f080a1080 con 0x7f2f10153f20 2024-09-11T00:57:38.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.077+0000 7f2f15e00640 1 -- 172.21.15.31:0/1546988912 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f1014bc30 con 0x7f2f10153f20 2024-09-11T00:57:38.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.079+0000 7f2f07e00640 1 -- 172.21.15.31:0/1546988912 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) ==== 74+0+25390 (secure 0 0 0) 0x7f2f080627c0 con 0x7f2f10153f20 2024-09-11T00:57:38.082 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:38.082 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":63,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:57:34.594057+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-11T00:30:43.375426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-11T00:30:44.303848+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.35","mappings":[{"from":0,"to":7}]},{"pgid":"5.8","mappings":[{"from":4,"to":5}]},{"pgid":"5.17","mappings":[{"from":4,"to":3}]},{"pgid":"5.26","mappings":[{"from":4,"to":5}]},{"pgid":"5.28","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.204:6836/1329541959":"2024-09-12T00:57:34.594017+0000","172.21.15.31:6832/3721481399":"2024-09-12T00:56:28.030054+0000","172.21.15.204:6835/4094068967":"2024-09-12T00:55:04.553768+0000","172.21.15.31:6833/2956367495":"2024-09-12T00:54:19.542253+0000","172.21.15.204:6837/2689217598":"2024-09-12T00:56:14.574459+0000","172.21.15.31:6834/1855324855":"2024-09-12T00:52:49.521133+0000","172.21.15.204:6835/1981755707":"2024-09-12T00:52:39.517968+0000","172.21.15.204:6834/98007221":"2024-09-12T00:41:49.383893+0000","172.21.15.31:6832/3881548149":"2024-09-12T00:40:39.371058+0000","172.21.15.31:6835/2163558126":"2024-09-12T00:48:14.453181+0000","172.21.15.204:6835/3506150555":"2024-09-12T00:40:24.367167+0000","172.21.15.204:6837/1278135080":"2024-09-12T00:38:14.342086+0000","172.21.15.31:6832/2956367495":"2024-09-12T00:54:19.542253+0000","172.21.15.31:6834/2909466339":"2024-09-12T00:43:22.735779+0000","172.21.15.204:6836/1278135080":"2024-09-12T00:38:14.342086+0000","172.21.15.204:6835/3444890210":"2024-09-12T00:37:16.369607+0000","172.21.15.204:6834/4094068967":"2024-09-12T00:55:04.553768+0000","172.21.15.204:6837/645869523":"2024-09-12T00:35:54.319031+0000","172.21.15.204:6837/4164016077":"2024-09-12T00:50:14.476294+0000","172.21.15.31:6834/2323279595":"2024-09-12T00:44:44.412976+0000","172.21.15.204:6837/1329541959":"2024-09-12T00:57:34.594017+0000","172.21.15.204:6834/3506150555":"2024-09-12T00:40:24.367167+0000","172.21.15.31:6833/4011734390":"2024-09-12T00:37:09.330669+0000","172.21.15.31:6834/3725011623":"2024-09-12T00:50:23.114454+0000","172.21.15.204:6837/3441932736":"2024-09-12T00:31:44.277804+0000","172.21.15.31:6833/3721481399":"2024-09-12T00:56:28.030054+0000","172.21.15.31:6833/2686855151":"2024-09-12T00:32:24.285279+0000","172.21.15.204:6837/1082799504":"2024-09-12T00:47:59.449917+0000","172.21.15.204:6834/1981755707":"2024-09-12T00:52:39.517968+0000","172.21.15.204:6836/645869523":"2024-09-12T00:35:54.319031+0000","172.21.15.31:6835/3925687919":"2024-09-12T00:41:09.376771+0000","172.21.15.204:6835/558470224":"2024-09-12T00:34:09.302673+0000","172.21.15.204:6835/1801149506":"2024-09-12T00:32:54.290770+0000","172.21.15.31:6833/4202305151":"2024-09-12T00:35:09.311954+0000","172.21.15.204:6836/3441932736":"2024-09-12T00:31:44.277804+0000","172.21.15.204:6836/1082799504":"2024-09-12T00:47:59.449917+0000","172.21.15.31:6833/3242133790":"2024-09-12T00:38:14.342154+0000","172.21.15.31:6834/3881548149":"2024-09-12T00:40:39.371058+0000","172.21.15.204:6834/1801149506":"2024-09-12T00:32:54.290770+0000","172.21.15.31:6835/4202305151":"2024-09-12T00:35:09.311954+0000","172.21.15.204:6836/2954547253":"2024-09-12T00:33:34.297314+0000","172.21.15.204:6834/1348312176":"2024-09-12T00:45:14.418207+0000","172.21.15.204:6834/3444890210":"2024-09-12T00:37:16.369607+0000","172.21.15.204:6836/2689217598":"2024-09-12T00:56:14.574459+0000","172.21.15.31:6833/3925687919":"2024-09-12T00:41:09.376771+0000","172.21.15.31:6835/4011734390":"2024-09-12T00:37:09.330669+0000","172.21.15.31:6835/1396366029":"2024-09-12T00:45:59.426330+0000","172.21.15.204:6837/2954547253":"2024-09-12T00:33:34.297314+0000","172.21.15.204:6834/558470224":"2024-09-12T00:34:09.302673+0000","172.21.15.31:6835/1855324855":"2024-09-12T00:52:49.521133+0000","172.21.15.204:6835/98007221":"2024-09-12T00:41:49.383893+0000","172.21.15.31:6834/1396366029":"2024-09-12T00:45:59.426330+0000","172.21.15.31:6832/2909466339":"2024-09-12T00:43:22.735779+0000","172.21.15.204:6837/2630216714":"2024-09-12T00:52:19.506830+0000","172.21.15.204:6834/1575320768":"2024-09-12T00:44:14.407953+0000","172.21.15.204:6836/2630216714":"2024-09-12T00:52:19.506830+0000","172.21.15.31:6835/2686855151":"2024-09-12T00:32:24.285279+0000","172.21.15.204:6835/1575320768":"2024-09-12T00:44:14.407953+0000","172.21.15.204:6835/1348312176":"2024-09-12T00:45:14.418207+0000","172.21.15.204:6837/3513641573":"2024-09-12T00:45:26.774916+0000","172.21.15.31:6833/4205320530":"2024-09-12T00:43:19.398942+0000","172.21.15.204:6834/2113744259":"2024-09-12T00:47:19.441133+0000","172.21.15.204:6835/2113744259":"2024-09-12T00:47:19.441133+0000","172.21.15.31:6835/2323279595":"2024-09-12T00:44:44.412976+0000","172.21.15.204:6836/3513641573":"2024-09-12T00:45:26.774916+0000","172.21.15.31:6834/2163558126":"2024-09-12T00:48:14.453181+0000","172.21.15.204:6834/634556689":"2024-09-12T00:49:34.468674+0000","172.21.15.204:6835/634556689":"2024-09-12T00:49:34.468674+0000","172.21.15.31:6835/3725011623":"2024-09-12T00:50:23.114454+0000","172.21.15.31:6835/3242133790":"2024-09-12T00:38:14.342154+0000","172.21.15.31:6835/4205320530":"2024-09-12T00:43:19.398942+0000","172.21.15.204:6836/4164016077":"2024-09-12T00:50:14.476294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:57:38.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.081+0000 7f2f15e00640 1 -- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2eec077f70 msgr2=0x7f2eec07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:38.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.081+0000 7f2f15e00640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2eec077f70 0x7f2eec07a430 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2f00005fd0 tx=0x7f2f00005850 comp rx=0 tx=0).stop 2024-09-11T00:57:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.081+0000 7f2f15e00640 1 -- 172.21.15.31:0/1546988912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f10153f20 msgr2=0x7f2f101df630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.081+0000 7f2f15e00640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f10153f20 0x7f2f101df630 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800b970 tx=0x7f2f0800be30 comp rx=0 tx=0).stop 2024-09-11T00:57:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.082+0000 7f2f15e00640 1 -- 172.21.15.31:0/1546988912 shutdown_connections 2024-09-11T00:57:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.082+0000 7f2f15e00640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f2eec077f70 0x7f2eec07a430 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.082+0000 7f2f15e00640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f10153f20 0x7f2f101df630 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.082+0000 7f2f15e00640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f2f100c23a0 0x7f2f100c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.082+0000 7f2f15e00640 1 --2- 172.21.15.31:0/1546988912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f10072480 0x7f2f101ec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.082+0000 7f2f15e00640 1 -- 172.21.15.31:0/1546988912 >> 172.21.15.31:0/1546988912 conn(0x7f2f100a9750 msgr2=0x7f2f1014ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.082+0000 7f2f15e00640 1 -- 172.21.15.31:0/1546988912 shutdown_connections 2024-09-11T00:57:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.082+0000 7f2f15e00640 1 -- 172.21.15.31:0/1546988912 wait complete. 2024-09-11T00:57:38.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail a 2024-09-11T00:57:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.262+0000 7f66d8000640 1 -- 172.21.15.31:0/3212681751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b8003c60 msgr2=0x7f66b8004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.262+0000 7f66d8000640 1 --2- 172.21.15.31:0/3212681751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b8003c60 0x7f66b8004060 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f66c4006f90 tx=0x7f66b8005830 comp rx=0 tx=0).stop 2024-09-11T00:57:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.262+0000 7f66d6000640 1 -- 172.21.15.31:0/3212681751 reap_dead start 2024-09-11T00:57:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.262+0000 7f66d8000640 1 -- 172.21.15.31:0/3212681751 shutdown_connections 2024-09-11T00:57:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.262+0000 7f66d8000640 1 -- 172.21.15.31:0/3212681751 >> 172.21.15.31:0/3212681751 conn(0x7f66d00a9750 msgr2=0x7f66d00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.262+0000 7f66d8000640 1 -- 172.21.15.31:0/3212681751 shutdown_connections 2024-09-11T00:57:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.262+0000 7f66d8000640 1 -- 172.21.15.31:0/3212681751 wait complete. 2024-09-11T00:57:38.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.264+0000 7f66d8000640 1 Processor -- start 2024-09-11T00:57:38.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.264+0000 7f66d8000640 1 -- start start 2024-09-11T00:57:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d8000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b8003c60 0x7f66d00a9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d8000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d01efa00 0x7f66d014ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d8000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d01ecb60 0x7f66d01ecfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d8000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66d00c7320 con 0x7f66b8003c60 2024-09-11T00:57:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d8000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f66d00c3950 con 0x7f66d01efa00 2024-09-11T00:57:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d8000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f66d0153b90 con 0x7f66d01ecb60 2024-09-11T00:57:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d5600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b8003c60 0x7f66d00a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d5600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b8003c60 0x7f66d00a9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55350/0 (socket says 172.21.15.31:55350) 2024-09-11T00:57:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d5600640 1 -- 172.21.15.31:0/3188526514 learned_addr learned my addr 172.21.15.31:0/3188526514 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:38.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d4c00640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d01efa00 0x7f66d014ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d6000640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d01ecb60 0x7f66d01ecfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d5600640 1 -- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d01ecb60 msgr2=0x7f66d01ecfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d5600640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d01ecb60 0x7f66d01ecfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d5600640 1 -- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d01efa00 msgr2=0x7f66d014ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d5600640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d01efa00 0x7f66d014ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d5600640 1 -- 172.21.15.31:0/3188526514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66d01550e0 con 0x7f66b8003c60 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.265+0000 7f66d4c00640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d01efa00 0x7f66d014ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.266+0000 7f66d5600640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b8003c60 0x7f66d00a9060 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f66c400c350 tx=0x7f66c40065d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.266+0000 7f66cd600640 1 -- 172.21.15.31:0/3188526514 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c4032030 con 0x7f66b8003c60 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.266+0000 7f66cd600640 1 -- 172.21.15.31:0/3188526514 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f66c4003c10 con 0x7f66b8003c60 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.266+0000 7f66cd600640 1 -- 172.21.15.31:0/3188526514 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c402e460 con 0x7f66b8003c60 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.266+0000 7f66d8000640 1 -- 172.21.15.31:0/3188526514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66d01553c0 con 0x7f66b8003c60 2024-09-11T00:57:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.266+0000 7f66d8000640 1 -- 172.21.15.31:0/3188526514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66d01ec640 con 0x7f66b8003c60 2024-09-11T00:57:38.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.267+0000 7f66d8000640 1 -- 172.21.15.31:0/3188526514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66d01ec450 con 0x7f66b8003c60 2024-09-11T00:57:38.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.268+0000 7f66cd600640 1 -- 172.21.15.31:0/3188526514 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f66c402e600 con 0x7f66b8003c60 2024-09-11T00:57:38.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.269+0000 7f66cd600640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b0077fa0 0x7f66b007a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:38.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.269+0000 7f66cd600640 1 -- 172.21.15.31:0/3188526514 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9762+0+0 (secure 0 0 0) 0x7f66c40bb820 con 0x7f66b8003c60 2024-09-11T00:57:38.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.270+0000 7f66d4c00640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b0077fa0 0x7f66b007a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:38.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.270+0000 7f66d4c00640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b0077fa0 0x7f66b007a460 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f66bc000f60 tx=0x7f66bc006090 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:38.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.273+0000 7f66cd600640 1 -- 172.21.15.31:0/3188526514 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f66c4047020 con 0x7f66b8003c60 2024-09-11T00:57:38.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.504+0000 7f66d8000640 1 -- 172.21.15.31:0/3188526514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "a"} v 0) -- 0x7f66d00c5ef0 con 0x7f66b8003c60 2024-09-11T00:57:38.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.507+0000 7f66cd600640 1 -- 172.21.15.31:0/3188526514 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "a"}]=0 a marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v191) ==== 151+0+0 (secure 0 0 0) 0x7f66c4083320 con 0x7f66b8003c60 2024-09-11T00:57:38.510 INFO:teuthology.orchestra.run.smithi031.stderr:a marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-11T00:57:38.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 -- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b0077fa0 msgr2=0x7f66b007a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:38.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b0077fa0 0x7f66b007a460 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f66bc000f60 tx=0x7f66bc006090 comp rx=0 tx=0).stop 2024-09-11T00:57:38.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 -- 172.21.15.31:0/3188526514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b8003c60 msgr2=0x7f66d00a9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:38.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b8003c60 0x7f66d00a9060 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f66c400c350 tx=0x7f66c40065d0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 -- 172.21.15.31:0/3188526514 shutdown_connections 2024-09-11T00:57:38.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f66b0077fa0 0x7f66b007a460 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f66d01ecb60 0x7f66d01ecfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66d01efa00 0x7f66d014ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 --2- 172.21.15.31:0/3188526514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b8003c60 0x7f66d00a9060 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 -- 172.21.15.31:0/3188526514 >> 172.21.15.31:0/3188526514 conn(0x7f66d00a9750 msgr2=0x7f66d014ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:38.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 -- 172.21.15.31:0/3188526514 shutdown_connections 2024-09-11T00:57:38.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.510+0000 7f66d8000640 1 -- 172.21.15.31:0/3188526514 wait complete. 2024-09-11T00:57:38.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm a --yes-i-really-mean-it 2024-09-11T00:57:38.586 INFO:tasks.ceph.mds.a.smithi031.stdout:starting mds.a at 2024-09-11T00:57:38.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.697+0000 7f3648800640 1 -- 172.21.15.31:0/2280673360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3640153e90 msgr2=0x7f3640154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:38.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.697+0000 7f3648800640 1 --2- 172.21.15.31:0/2280673360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3640153e90 0x7f3640154340 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f363c005fd0 tx=0x7f363c02aa00 comp rx=0 tx=0).stop 2024-09-11T00:57:38.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.697+0000 7f3648800640 1 -- 172.21.15.31:0/2280673360 shutdown_connections 2024-09-11T00:57:38.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.697+0000 7f3648800640 1 --2- 172.21.15.31:0/2280673360 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3640153e90 0x7f3640154340 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.697+0000 7f3648800640 1 --2- 172.21.15.31:0/2280673360 >> v2:172.21.15.204:3300/0 conn(0x7f36400c2330 0x7f36400c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.697+0000 7f3648800640 1 -- 172.21.15.31:0/2280673360 >> 172.21.15.31:0/2280673360 conn(0x7f364006d560 msgr2=0x7f36400c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:38.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.697+0000 7f3648800640 1 -- 172.21.15.31:0/2280673360 shutdown_connections 2024-09-11T00:57:38.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.697+0000 7f3648800640 1 -- 172.21.15.31:0/2280673360 wait complete. 2024-09-11T00:57:38.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.699+0000 7f3648800640 1 Processor -- start 2024-09-11T00:57:38.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.699+0000 7f3648800640 1 -- start start 2024-09-11T00:57:38.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.699+0000 7f3648800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36400c2330 0x7f364014b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:38.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.699+0000 7f3648800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f364014de40 0x7f36401ded30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:38.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.699+0000 7f3648800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3640153e90 0x7f36401ec300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:38.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.699+0000 7f3648800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f364006e380 con 0x7f36400c2330 2024-09-11T00:57:38.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.699+0000 7f3648800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f364006e200 con 0x7f364014de40 2024-09-11T00:57:38.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.699+0000 7f3648800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f364014e220 con 0x7f3640153e90 2024-09-11T00:57:38.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3645e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36400c2330 0x7f364014b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:38.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3646800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3640153e90 0x7f36401ec300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:38.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3645400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f364014de40 0x7f36401ded30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:38.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3646800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3640153e90 0x7f36401ec300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3301/0 says I am v2:172.21.15.31:57666/0 (socket says 172.21.15.31:57666) 2024-09-11T00:57:38.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3646800640 1 -- 172.21.15.31:0/1259048352 learned_addr learned my addr 172.21.15.31:0/1259048352 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:38.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3646800640 1 -- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f364014de40 msgr2=0x7f36401ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:38.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3646800640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f364014de40 0x7f36401ded30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3646800640 1 -- 172.21.15.31:0/1259048352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36400c2330 msgr2=0x7f364014b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:38.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3646800640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36400c2330 0x7f364014b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:38.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3646800640 1 -- 172.21.15.31:0/1259048352 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36401ee410 con 0x7f3640153e90 2024-09-11T00:57:38.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.700+0000 7f3645400640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f364014de40 0x7f36401ded30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T00:57:38.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.701+0000 7f3646800640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3640153e90 0x7f36401ec300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f363c009d90 tx=0x7f363c003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:38.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.701+0000 7f363a000640 1 -- 172.21.15.31:0/1259048352 <== mon.2 v2:172.21.15.204:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c033050 con 0x7f3640153e90 2024-09-11T00:57:38.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.701+0000 7f363a000640 1 -- 172.21.15.31:0/1259048352 <== mon.2 v2:172.21.15.204:3301/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f363c004560 con 0x7f3640153e90 2024-09-11T00:57:38.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.701+0000 7f363a000640 1 -- 172.21.15.31:0/1259048352 <== mon.2 v2:172.21.15.204:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c038450 con 0x7f3640153e90 2024-09-11T00:57:38.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.701+0000 7f3648800640 1 -- 172.21.15.31:0/1259048352 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36401eb540 con 0x7f3640153e90 2024-09-11T00:57:38.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.701+0000 7f3648800640 1 -- 172.21.15.31:0/1259048352 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f36401ebb00 con 0x7f3640153e90 2024-09-11T00:57:38.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.702+0000 7f3648800640 1 -- 172.21.15.31:0/1259048352 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36400c5520 con 0x7f3640153e90 2024-09-11T00:57:38.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.704+0000 7f363a000640 1 -- 172.21.15.31:0/1259048352 <== mon.2 v2:172.21.15.204:3301/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f363c040070 con 0x7f3640153e90 2024-09-11T00:57:38.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.705+0000 7f363a000640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3624077f70 0x7f362407a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:38.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.705+0000 7f363a000640 1 -- 172.21.15.31:0/1259048352 <== mon.2 v2:172.21.15.204:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 9848+0+0 (secure 0 0 0) 0x7f363c0bc2f0 con 0x7f3640153e90 2024-09-11T00:57:38.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.705+0000 7f3645e00640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3624077f70 0x7f362407a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:38.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.706+0000 7f3645e00640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3624077f70 0x7f362407a430 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f36340017c0 tx=0x7f3634009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:38.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.708+0000 7f363a000640 1 -- 172.21.15.31:0/1259048352 <== mon.2 v2:172.21.15.204:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f36400c5520 con 0x7f3640153e90 2024-09-11T00:57:38.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:38.941+0000 7f3648800640 1 -- 172.21.15.31:0/1259048352 --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "a", "yes_i_really_mean_it": true} v 0) -- 0x7f3640154a90 con 0x7f3640153e90 2024-09-11T00:57:39.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.516+0000 7f363a000640 1 -- 172.21.15.31:0/1259048352 <== mon.2 v2:172.21.15.204:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "a", "yes_i_really_mean_it": true}]=0 v192) ==== 99+0+0 (secure 0 0 0) 0x7f363c0098b0 con 0x7f3640153e90 2024-09-11T00:57:39.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.519+0000 7f3648800640 1 -- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3624077f70 msgr2=0x7f362407a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:39.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.519+0000 7f3648800640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3624077f70 0x7f362407a430 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f36340017c0 tx=0x7f3634009040 comp rx=0 tx=0).stop 2024-09-11T00:57:39.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.519+0000 7f3648800640 1 -- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3640153e90 msgr2=0x7f36401ec300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.519+0000 7f3648800640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3640153e90 0x7f36401ec300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f363c009d90 tx=0x7f363c003f90 comp rx=0 tx=0).stop 2024-09-11T00:57:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.519+0000 7f3648800640 1 -- 172.21.15.31:0/1259048352 shutdown_connections 2024-09-11T00:57:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.519+0000 7f3648800640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3624077f70 0x7f362407a430 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.519+0000 7f3648800640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3640153e90 0x7f36401ec300 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.519+0000 7f3648800640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f364014de40 0x7f36401ded30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.519+0000 7f3648800640 1 --2- 172.21.15.31:0/1259048352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36400c2330 0x7f364014b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.519+0000 7f3648800640 1 -- 172.21.15.31:0/1259048352 >> 172.21.15.31:0/1259048352 conn(0x7f364006d560 msgr2=0x7f3640152490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.520+0000 7f3648800640 1 -- 172.21.15.31:0/1259048352 shutdown_connections 2024-09-11T00:57:39.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.520+0000 7f3648800640 1 -- 172.21.15.31:0/1259048352 wait complete. 2024-09-11T00:57:39.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm a_metadata a_metadata --yes-i-really-really-mean-it 2024-09-11T00:57:39.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.704+0000 7f3a40a00640 1 -- 172.21.15.31:0/478320788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a3c072480 msgr2=0x7f3a3c1511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:39.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.704+0000 7f3a40a00640 1 --2- 172.21.15.31:0/478320788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a3c072480 0x7f3a3c1511a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009830 tx=0x7f3a2c02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:57:39.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.704+0000 7f3a40a00640 1 -- 172.21.15.31:0/478320788 shutdown_connections 2024-09-11T00:57:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.704+0000 7f3a40a00640 1 --2- 172.21.15.31:0/478320788 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3a3c153f20 0x7f3a3c1516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.704+0000 7f3a40a00640 1 --2- 172.21.15.31:0/478320788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a3c072480 0x7f3a3c1511a0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.704+0000 7f3a40a00640 1 -- 172.21.15.31:0/478320788 >> 172.21.15.31:0/478320788 conn(0x7f3a3c0a9750 msgr2=0x7f3a3c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.704+0000 7f3a40a00640 1 -- 172.21.15.31:0/478320788 shutdown_connections 2024-09-11T00:57:39.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.704+0000 7f3a40a00640 1 -- 172.21.15.31:0/478320788 wait complete. 2024-09-11T00:57:39.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.706+0000 7f3a40a00640 1 Processor -- start 2024-09-11T00:57:39.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.706+0000 7f3a40a00640 1 -- start start 2024-09-11T00:57:39.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a40a00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a3c072480 0x7f3a3c14bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a40a00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a3c0c23a0 0x7f3a3c1dd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a40a00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3a3c153f20 0x7f3a3c1ef6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a40a00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c074200 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a40a00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c074080 con 0x7f3a3c072480 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a40a00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f3a3c072860 con 0x7f3a3c153f20 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a31400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a3c0c23a0 0x7f3a3c1dd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a31400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a3c0c23a0 0x7f3a3c1dd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55398/0 (socket says 172.21.15.31:55398) 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a31400640 1 -- 172.21.15.31:0/3474868076 learned_addr learned my addr 172.21.15.31:0/3474868076 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a3a800640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3a3c153f20 0x7f3a3c1ef6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a39e00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a3c072480 0x7f3a3c14bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a31400640 1 -- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3a3c153f20 msgr2=0x7f3a3c1ef6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a31400640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3a3c153f20 0x7f3a3c1ef6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:39.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a31400640 1 -- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a3c072480 msgr2=0x7f3a3c14bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a31400640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a3c072480 0x7f3a3c14bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a31400640 1 -- 172.21.15.31:0/3474868076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3c14c440 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a39e00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a3c072480 0x7f3a3c14bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.707+0000 7f3a31400640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a3c0c23a0 0x7f3a3c1dd4f0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f3a2400c460 tx=0x7f3a2400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.708+0000 7f3a32a00640 1 -- 172.21.15.31:0/3474868076 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24016030 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.708+0000 7f3a32a00640 1 -- 172.21.15.31:0/3474868076 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f3a24006890 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.708+0000 7f3a32a00640 1 -- 172.21.15.31:0/3474868076 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24006d20 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.708+0000 7f3a40a00640 1 -- 172.21.15.31:0/3474868076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3c1ec1a0 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.708+0000 7f3a40a00640 1 -- 172.21.15.31:0/3474868076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a3c151ea0 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.709+0000 7f3a40a00640 1 -- 172.21.15.31:0/3474868076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3c0c3900 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.710+0000 7f3a32a00640 1 -- 172.21.15.31:0/3474868076 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f3a24023060 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.712+0000 7f3a32a00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3a18077fa0 0x7f3a1807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.712+0000 7f3a32a00640 1 -- 172.21.15.31:0/3474868076 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9848+0+0 (secure 0 0 0) 0x7f3a2409a8f0 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.712+0000 7f3a39e00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3a18077fa0 0x7f3a1807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:39.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.713+0000 7f3a39e00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3a18077fa0 0x7f3a1807a460 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c034000 tx=0x7f3a2c0046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:39.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.716+0000 7f3a32a00640 1 -- 172.21.15.31:0/3474868076 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f3a24062310 con 0x7f3a3c0c23a0 2024-09-11T00:57:39.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:39.913+0000 7f3a40a00640 1 -- 172.21.15.31:0/3474868076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "a_metadata", "pool2": "a_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3a3c0c1900 con 0x7f3a3c0c23a0 2024-09-11T00:57:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.516+0000 7f3a32a00640 1 -- 172.21.15.31:0/3474868076 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "a_metadata", "pool2": "a_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'a_metadata' removed v65) ==== 166+0+0 (secure 0 0 0) 0x7f3a24061c10 con 0x7f3a3c0c23a0 2024-09-11T00:57:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'a_metadata' removed 2024-09-11T00:57:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.519+0000 7f3a40a00640 1 -- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3a18077fa0 msgr2=0x7f3a1807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.519+0000 7f3a40a00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3a18077fa0 0x7f3a1807a460 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c034000 tx=0x7f3a2c0046c0 comp rx=0 tx=0).stop 2024-09-11T00:57:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.519+0000 7f3a40a00640 1 -- 172.21.15.31:0/3474868076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a3c0c23a0 msgr2=0x7f3a3c1dd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.519+0000 7f3a40a00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a3c0c23a0 0x7f3a3c1dd4f0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f3a2400c460 tx=0x7f3a2400c920 comp rx=0 tx=0).stop 2024-09-11T00:57:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.519+0000 7f3a40a00640 1 -- 172.21.15.31:0/3474868076 shutdown_connections 2024-09-11T00:57:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.519+0000 7f3a40a00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f3a18077fa0 0x7f3a1807a460 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.519+0000 7f3a40a00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f3a3c153f20 0x7f3a3c1ef6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.519+0000 7f3a40a00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a3c0c23a0 0x7f3a3c1dd4f0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.519+0000 7f3a40a00640 1 --2- 172.21.15.31:0/3474868076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a3c072480 0x7f3a3c14bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.519+0000 7f3a40a00640 1 -- 172.21.15.31:0/3474868076 >> 172.21.15.31:0/3474868076 conn(0x7f3a3c0a9750 msgr2=0x7f3a3c14c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.520+0000 7f3a40a00640 1 -- 172.21.15.31:0/3474868076 shutdown_connections 2024-09-11T00:57:40.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.520+0000 7f3a40a00640 1 -- 172.21.15.31:0/3474868076 wait complete. 2024-09-11T00:57:40.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm a_data a_data --yes-i-really-really-mean-it 2024-09-11T00:57:40.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.717+0000 7f5d87200640 1 -- 172.21.15.31:0/211265772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d880c9e90 msgr2=0x7f5d8806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:40.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.717+0000 7f5d87200640 1 --2- 172.21.15.31:0/211265772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d880c9e90 0x7f5d8806d960 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f5d80005fd0 tx=0x7f5d8002a820 comp rx=0 tx=0).stop 2024-09-11T00:57:40.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.717+0000 7f5d87200640 1 -- 172.21.15.31:0/211265772 shutdown_connections 2024-09-11T00:57:40.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.717+0000 7f5d87200640 1 --2- 172.21.15.31:0/211265772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d880c9e90 0x7f5d8806d960 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:40.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.717+0000 7f5d87200640 1 --2- 172.21.15.31:0/211265772 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d88159690 0x7f5d88159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:40.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.717+0000 7f5d87200640 1 -- 172.21.15.31:0/211265772 >> 172.21.15.31:0/211265772 conn(0x7f5d880ad810 msgr2=0x7f5d880adc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:40.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.717+0000 7f5d87200640 1 -- 172.21.15.31:0/211265772 shutdown_connections 2024-09-11T00:57:40.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.717+0000 7f5d87200640 1 -- 172.21.15.31:0/211265772 wait complete. 2024-09-11T00:57:40.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.718+0000 7f5d87200640 1 Processor -- start 2024-09-11T00:57:40.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.719+0000 7f5d87200640 1 -- start start 2024-09-11T00:57:40.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.719+0000 7f5d87200640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d880c9980 0x7f5d881511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:40.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.719+0000 7f5d87200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d880c9e90 0x7f5d88155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.719+0000 7f5d87200640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d88159690 0x7f5d880c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.719+0000 7f5d87200640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5d880b1420 con 0x7f5d880c9e90 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.719+0000 7f5d87200640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d880c6a30 con 0x7f5d88159690 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.719+0000 7f5d87200640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f5d880b15a0 con 0x7f5d880c9980 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d85400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d880c9e90 0x7f5d88155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d85400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d880c9e90 0x7f5d88155400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55402/0 (socket says 172.21.15.31:55402) 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d85400640 1 -- 172.21.15.31:0/2549515132 learned_addr learned my addr 172.21.15.31:0/2549515132 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d85e00640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d880c9980 0x7f5d881511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d86800640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d88159690 0x7f5d880c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d85400640 1 -- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d880c9980 msgr2=0x7f5d881511f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d85400640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d880c9980 0x7f5d881511f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d85400640 1 -- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d88159690 msgr2=0x7f5d880c41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:40.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d85400640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d88159690 0x7f5d880c41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:40.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d85400640 1 -- 172.21.15.31:0/2549515132 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d88153dd0 con 0x7f5d880c9e90 2024-09-11T00:57:40.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d86800640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d88159690 0x7f5d880c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:40.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.720+0000 7f5d85400640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d880c9e90 0x7f5d88155400 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5d7400c450 tx=0x7f5d7400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:40.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.721+0000 7f5d7e000640 1 -- 172.21.15.31:0/2549515132 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74016020 con 0x7f5d880c9e90 2024-09-11T00:57:40.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.721+0000 7f5d7e000640 1 -- 172.21.15.31:0/2549515132 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f5d74012830 con 0x7f5d880c9e90 2024-09-11T00:57:40.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.721+0000 7f5d7e000640 1 -- 172.21.15.31:0/2549515132 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74012c60 con 0x7f5d880c9e90 2024-09-11T00:57:40.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.721+0000 7f5d87200640 1 -- 172.21.15.31:0/2549515132 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d880c5830 con 0x7f5d880c9e90 2024-09-11T00:57:40.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.721+0000 7f5d87200640 1 -- 172.21.15.31:0/2549515132 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d88155940 con 0x7f5d880c9e90 2024-09-11T00:57:40.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.722+0000 7f5d87200640 1 -- 172.21.15.31:0/2549515132 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d8806e0b0 con 0x7f5d880c9e90 2024-09-11T00:57:40.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.723+0000 7f5d7e000640 1 -- 172.21.15.31:0/2549515132 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f5d7401b400 con 0x7f5d880c9e90 2024-09-11T00:57:40.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.724+0000 7f5d7e000640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5d68077f70 0x7f5d6807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:40.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.724+0000 7f5d7e000640 1 -- 172.21.15.31:0/2549515132 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9420+0+0 (secure 0 0 0) 0x7f5d7409adc0 con 0x7f5d880c9e90 2024-09-11T00:57:40.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.725+0000 7f5d85e00640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5d68077f70 0x7f5d6807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:40.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.725+0000 7f5d85e00640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5d68077f70 0x7f5d6807a430 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009b30 tx=0x7f5d78005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:40.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.729+0000 7f5d7e000640 1 -- 172.21.15.31:0/2549515132 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f5d74020020 con 0x7f5d880c9e90 2024-09-11T00:57:40.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:40.924+0000 7f5d87200640 1 -- 172.21.15.31:0/2549515132 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "a_data", "pool2": "a_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5d88159be0 con 0x7f5d880c9e90 2024-09-11T00:57:41.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.535+0000 7f5d7e000640 1 -- 172.21.15.31:0/2549515132 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "a_data", "pool2": "a_data", "yes_i_really_really_mean_it": true}]=0 pool 'a_data' removed v66) ==== 154+0+0 (secure 0 0 0) 0x7f5d74062a20 con 0x7f5d880c9e90 2024-09-11T00:57:41.537 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'a_data' removed 2024-09-11T00:57:41.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 -- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5d68077f70 msgr2=0x7f5d6807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:41.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5d68077f70 0x7f5d6807a430 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009b30 tx=0x7f5d78005f90 comp rx=0 tx=0).stop 2024-09-11T00:57:41.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 -- 172.21.15.31:0/2549515132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d880c9e90 msgr2=0x7f5d88155400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:41.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d880c9e90 0x7f5d88155400 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5d7400c450 tx=0x7f5d7400c910 comp rx=0 tx=0).stop 2024-09-11T00:57:41.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 -- 172.21.15.31:0/2549515132 shutdown_connections 2024-09-11T00:57:41.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f5d68077f70 0x7f5d6807a430 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d88159690 0x7f5d880c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d880c9e90 0x7f5d88155400 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 --2- 172.21.15.31:0/2549515132 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f5d880c9980 0x7f5d881511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 -- 172.21.15.31:0/2549515132 >> 172.21.15.31:0/2549515132 conn(0x7f5d880ad810 msgr2=0x7f5d88158be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:41.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.538+0000 7f5d87200640 1 -- 172.21.15.31:0/2549515132 shutdown_connections 2024-09-11T00:57:41.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.539+0000 7f5d87200640 1 -- 172.21.15.31:0/2549515132 wait complete. 2024-09-11T00:57:41.558 INFO:tasks.cephfs.filesystem:Destroying file system b and related pools 2024-09-11T00:57:41.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:41.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.719+0000 7fcfa0c00640 1 -- 172.21.15.31:0/2235897093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf9c156ad0 msgr2=0x7fcf9c153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:41.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.719+0000 7fcfa0c00640 1 --2- 172.21.15.31:0/2235897093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf9c156ad0 0x7fcf9c153480 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0c6d20 tx=0x7fcf8402a560 comp rx=0 tx=0).stop 2024-09-11T00:57:41.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.719+0000 7fcfa0c00640 1 -- 172.21.15.31:0/2235897093 shutdown_connections 2024-09-11T00:57:41.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.719+0000 7fcfa0c00640 1 --2- 172.21.15.31:0/2235897093 >> v2:172.21.15.204:3300/0 conn(0x7fcf9c1539c0 0x7fcf9c159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.719+0000 7fcfa0c00640 1 --2- 172.21.15.31:0/2235897093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf9c156ad0 0x7fcf9c153480 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.719+0000 7fcfa0c00640 1 -- 172.21.15.31:0/2235897093 >> 172.21.15.31:0/2235897093 conn(0x7fcf9c0a9750 msgr2=0x7fcf9c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:41.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.719+0000 7fcfa0c00640 1 -- 172.21.15.31:0/2235897093 shutdown_connections 2024-09-11T00:57:41.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.719+0000 7fcfa0c00640 1 -- 172.21.15.31:0/2235897093 wait complete. 2024-09-11T00:57:41.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.721+0000 7fcfa0c00640 1 Processor -- start 2024-09-11T00:57:41.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.721+0000 7fcfa0c00640 1 -- start start 2024-09-11T00:57:41.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcfa0c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf9c0c5b10 0x7fcf9c1ef870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:41.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcfa0c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf9c1539c0 0x7fcf9c151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:41.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcfa0c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf9c156ad0 0x7fcf9c152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:41.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcfa0c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9c156f20 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcfa0c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9c0c7320 con 0x7fcf9c1539c0 2024-09-11T00:57:41.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcfa0c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fcf9c0c3950 con 0x7fcf9c156ad0 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcf99e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf9c0c5b10 0x7fcf9c1ef870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcf99e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf9c0c5b10 0x7fcf9c1ef870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55428/0 (socket says 172.21.15.31:55428) 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcf99e00640 1 -- 172.21.15.31:0/36772096 learned_addr learned my addr 172.21.15.31:0/36772096 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcf9a800640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf9c156ad0 0x7fcf9c152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.722+0000 7fcf91400640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf9c1539c0 0x7fcf9c151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcf99e00640 1 -- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf9c156ad0 msgr2=0x7fcf9c152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcf99e00640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf9c156ad0 0x7fcf9c152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcf99e00640 1 -- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf9c1539c0 msgr2=0x7fcf9c151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcf99e00640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf9c1539c0 0x7fcf9c151c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcf99e00640 1 -- 172.21.15.31:0/36772096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf9c1f0260 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcf99e00640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf9c0c5b10 0x7fcf9c1ef870 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00c430 tx=0x7fcf8c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:41.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcf92a00640 1 -- 172.21.15.31:0/36772096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c011790 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcf92a00640 1 -- 172.21.15.31:0/36772096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fcf8c017070 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcf92a00640 1 -- 172.21.15.31:0/36772096 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c011c30 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcfa0c00640 1 -- 172.21.15.31:0/36772096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf9c14f2e0 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.723+0000 7fcfa0c00640 1 -- 172.21.15.31:0/36772096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf9c14d510 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.724+0000 7fcfa0c00640 1 -- 172.21.15.31:0/36772096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf9c159670 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.725+0000 7fcf92a00640 1 -- 172.21.15.31:0/36772096 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fcf8c01c460 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.726+0000 7fcf92a00640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf78077f70 0x7fcf7807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:41.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.727+0000 7fcf92a00640 1 -- 172.21.15.31:0/36772096 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 8987+0+0 (secure 0 0 0) 0x7fcf8c09a8b0 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.727+0000 7fcf91400640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf78077f70 0x7fcf7807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:41.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.728+0000 7fcf91400640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf78077f70 0x7fcf7807a430 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fcf84033000 tx=0x7fcf840046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:41.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.731+0000 7fcf92a00640 1 -- 172.21.15.31:0/36772096 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fcf8c0626c0 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.963+0000 7fcfa0c00640 1 -- 172.21.15.31:0/36772096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf9c0c1e70 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.964+0000 7fcf92a00640 1 -- 172.21.15.31:0/36772096 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) ==== 95+0+4399 (secure 0 0 0) 0x7fcf8c061fc0 con 0x7fcf9c0c5b10 2024-09-11T00:57:41.966 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:41.966 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":192,"btime":"2024-09-11T00:57:39:509458+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":8103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/22472818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":22472818},{"type":"v1","addr":"172.21.15.31:6833","nonce":22472818}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:41.966 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 192 2024-09-11T00:57:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.966+0000 7fcfa0c00640 1 -- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf78077f70 msgr2=0x7fcf7807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.966+0000 7fcfa0c00640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf78077f70 0x7fcf7807a430 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fcf84033000 tx=0x7fcf840046c0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.966+0000 7fcfa0c00640 1 -- 172.21.15.31:0/36772096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf9c0c5b10 msgr2=0x7fcf9c1ef870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.966+0000 7fcfa0c00640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf9c0c5b10 0x7fcf9c1ef870 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00c430 tx=0x7fcf8c00c8f0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.967+0000 7fcfa0c00640 1 -- 172.21.15.31:0/36772096 shutdown_connections 2024-09-11T00:57:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.967+0000 7fcfa0c00640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fcf78077f70 0x7fcf7807a430 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.967+0000 7fcfa0c00640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fcf9c156ad0 0x7fcf9c152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.967+0000 7fcfa0c00640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf9c1539c0 0x7fcf9c151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.967+0000 7fcfa0c00640 1 --2- 172.21.15.31:0/36772096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf9c0c5b10 0x7fcf9c1ef870 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.967+0000 7fcfa0c00640 1 -- 172.21.15.31:0/36772096 >> 172.21.15.31:0/36772096 conn(0x7fcf9c0a9750 msgr2=0x7fcf9c0c4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:41.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.967+0000 7fcfa0c00640 1 -- 172.21.15.31:0/36772096 shutdown_connections 2024-09-11T00:57:41.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:41.967+0000 7fcfa0c00640 1 -- 172.21.15.31:0/36772096 wait complete. 2024-09-11T00:57:41.985 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:42.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.148+0000 7f1982e00640 1 -- 172.21.15.31:0/51153136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19600051c0 msgr2=0x7f196000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:42.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.148+0000 7f1982e00640 1 --2- 172.21.15.31:0/51153136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19600051c0 0x7f196000b9c0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f1974009ed0 tx=0x7f1960005830 comp rx=0 tx=0).stop 2024-09-11T00:57:42.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.148+0000 7f1980e00640 1 -- 172.21.15.31:0/51153136 reap_dead start 2024-09-11T00:57:42.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.148+0000 7f1982e00640 1 -- 172.21.15.31:0/51153136 shutdown_connections 2024-09-11T00:57:42.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.148+0000 7f1982e00640 1 -- 172.21.15.31:0/51153136 >> 172.21.15.31:0/51153136 conn(0x7f197c0a9750 msgr2=0x7f197c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:42.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.148+0000 7f1982e00640 1 -- 172.21.15.31:0/51153136 shutdown_connections 2024-09-11T00:57:42.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.148+0000 7f1982e00640 1 -- 172.21.15.31:0/51153136 wait complete. 2024-09-11T00:57:42.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.150+0000 7f1982e00640 1 Processor -- start 2024-09-11T00:57:42.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.150+0000 7f1982e00640 1 -- start start 2024-09-11T00:57:42.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1982e00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19600051c0 0x7f197c0c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:42.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1982e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f197c06f4c0 0x7f197c1ed670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:42.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1982e00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f197c1edbb0 0x7f197c0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:42.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1982e00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f197c0c23a0 con 0x7f197c06f4c0 2024-09-11T00:57:42.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1982e00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f197c0a7140 con 0x7f19600051c0 2024-09-11T00:57:42.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1982e00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f197c0c36b0 con 0x7f197c1edbb0 2024-09-11T00:57:42.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1973400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f197c06f4c0 0x7f197c1ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:42.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1973400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f197c06f4c0 0x7f197c1ed670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55446/0 (socket says 172.21.15.31:55446) 2024-09-11T00:57:42.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1973400640 1 -- 172.21.15.31:0/113732949 learned_addr learned my addr 172.21.15.31:0/113732949 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:42.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f197be00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19600051c0 0x7f197c0c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1980e00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f197c1edbb0 0x7f197c0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1973400640 1 -- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f197c1edbb0 msgr2=0x7f197c0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1973400640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f197c1edbb0 0x7f197c0c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1973400640 1 -- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19600051c0 msgr2=0x7f197c0c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1973400640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19600051c0 0x7f197c0c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1973400640 1 -- 172.21.15.31:0/113732949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197c1f0660 con 0x7f197c06f4c0 2024-09-11T00:57:42.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.151+0000 7f1973400640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f197c06f4c0 0x7f197c1ed670 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f19640066b0 tx=0x7f19640039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.152+0000 7f1978c00640 1 -- 172.21.15.31:0/113732949 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1964015070 con 0x7f197c06f4c0 2024-09-11T00:57:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.152+0000 7f1978c00640 1 -- 172.21.15.31:0/113732949 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f1964011a30 con 0x7f197c06f4c0 2024-09-11T00:57:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.152+0000 7f1982e00640 1 -- 172.21.15.31:0/113732949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197c1ef8a0 con 0x7f197c06f4c0 2024-09-11T00:57:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.152+0000 7f1978c00640 1 -- 172.21.15.31:0/113732949 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196400f430 con 0x7f197c06f4c0 2024-09-11T00:57:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.152+0000 7f1982e00640 1 -- 172.21.15.31:0/113732949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197c1efe60 con 0x7f197c06f4c0 2024-09-11T00:57:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.153+0000 7f1982e00640 1 -- 172.21.15.31:0/113732949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f197c072900 con 0x7f197c06f4c0 2024-09-11T00:57:42.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.154+0000 7f1978c00640 1 -- 172.21.15.31:0/113732949 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f196400f5d0 con 0x7f197c06f4c0 2024-09-11T00:57:42.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.155+0000 7f1978c00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077fa0 0x7f195807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:42.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.155+0000 7f1978c00640 1 -- 172.21.15.31:0/113732949 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 8987+0+0 (secure 0 0 0) 0x7f196409b7a0 con 0x7f197c06f4c0 2024-09-11T00:57:42.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.155+0000 7f197be00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077fa0 0x7f195807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:42.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.156+0000 7f197be00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077fa0 0x7f195807a460 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f197c0aa790 tx=0x7f196c005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:42.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.159+0000 7f1978c00640 1 -- 172.21.15.31:0/113732949 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f1964062f40 con 0x7f197c06f4c0 2024-09-11T00:57:42.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.484+0000 7f1982e00640 1 -- 172.21.15.31:0/113732949 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f197c0a6f20 con 0x7f197c06f4c0 2024-09-11T00:57:42.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.484+0000 7f1978c00640 1 -- 172.21.15.31:0/113732949 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) ==== 95+0+4399 (secure 0 0 0) 0x7f1964067970 con 0x7f197c06f4c0 2024-09-11T00:57:42.486 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:42.486 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":192,"btime":"2024-09-11T00:57:39:509458+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":8103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/22472818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":22472818},{"type":"v1","addr":"172.21.15.31:6833","nonce":22472818}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-11T00:30:54.411975+0000","modified":"2024-09-11T00:56:17.960286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6593":{"gid":6593,"name":"b","rank":0,"incarnation":177,"state":"up:active","state_seq":19,"addr":"172.21.15.204:6835/3620025884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":3620025884},{"type":"v1","addr":"172.21.15.204:6835","nonce":3620025884}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6593,"qdb_cluster":[6593]},"id":2}]} 2024-09-11T00:57:42.486 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 192 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 -- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077fa0 msgr2=0x7f195807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077fa0 0x7f195807a460 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f197c0aa790 tx=0x7f196c005f90 comp rx=0 tx=0).stop 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 -- 172.21.15.31:0/113732949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f197c06f4c0 msgr2=0x7f197c1ed670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f197c06f4c0 0x7f197c1ed670 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f19640066b0 tx=0x7f19640039c0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 -- 172.21.15.31:0/113732949 shutdown_connections 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f1958077fa0 0x7f195807a460 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f197c1edbb0 0x7f197c0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f197c06f4c0 0x7f197c1ed670 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 --2- 172.21.15.31:0/113732949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19600051c0 0x7f197c0c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 -- 172.21.15.31:0/113732949 >> 172.21.15.31:0/113732949 conn(0x7f197c0a9750 msgr2=0x7f197c14e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:42.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 -- 172.21.15.31:0/113732949 shutdown_connections 2024-09-11T00:57:42.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.487+0000 7f1982e00640 1 -- 172.21.15.31:0/113732949 wait complete. 2024-09-11T00:57:42.505 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T00:57:42.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.663+0000 7fd26f600640 1 -- 172.21.15.31:0/17886536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268072480 msgr2=0x7fd2681511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:42.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.663+0000 7fd26f600640 1 --2- 172.21.15.31:0/17886536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268072480 0x7fd2681511a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009860 tx=0x7fd25c02a5a0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.663+0000 7fd26f600640 1 -- 172.21.15.31:0/17886536 shutdown_connections 2024-09-11T00:57:42.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.663+0000 7fd26f600640 1 --2- 172.21.15.31:0/17886536 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268072480 0x7fd2681511a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.663+0000 7fd26f600640 1 -- 172.21.15.31:0/17886536 >> 172.21.15.31:0/17886536 conn(0x7fd2680a9750 msgr2=0x7fd2680a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:42.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.663+0000 7fd26f600640 1 -- 172.21.15.31:0/17886536 shutdown_connections 2024-09-11T00:57:42.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.663+0000 7fd26f600640 1 -- 172.21.15.31:0/17886536 wait complete. 2024-09-11T00:57:42.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.664+0000 7fd26f600640 1 Processor -- start 2024-09-11T00:57:42.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.665+0000 7fd26f600640 1 -- start start 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.665+0000 7fd26f600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268072480 0x7fd2681ec9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26f600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680c23a0 0x7fd2680c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26f600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd268153f20 0x7fd2680c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26f600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd268074290 con 0x7fd268153f20 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26f600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd268074e30 con 0x7fd2680c23a0 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26f600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd268072860 con 0x7fd268072480 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd268153f20 0x7fd2680c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26d600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd268153f20 0x7fd2680c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55450/0 (socket says 172.21.15.31:55450) 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26d600640 1 -- 172.21.15.31:0/3837153632 learned_addr learned my addr 172.21.15.31:0/3837153632 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26cc00640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268072480 0x7fd2681ec9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd267e00640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680c23a0 0x7fd2680c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:42.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26d600640 1 -- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268072480 msgr2=0x7fd2681ec9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26d600640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268072480 0x7fd2681ec9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26d600640 1 -- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680c23a0 msgr2=0x7fd2680c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26d600640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680c23a0 0x7fd2680c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26d600640 1 -- 172.21.15.31:0/3837153632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2681ddf00 con 0x7fd268153f20 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.666+0000 7fd26cc00640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268072480 0x7fd2681ec9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.667+0000 7fd26d600640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd268153f20 0x7fd2680c6460 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd26000cac0 tx=0x7fd26000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.667+0000 7fd264c00640 1 -- 172.21.15.31:0/3837153632 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd260016020 con 0x7fd268153f20 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.667+0000 7fd264c00640 1 -- 172.21.15.31:0/3837153632 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fd26000ac50 con 0x7fd268153f20 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.667+0000 7fd26f600640 1 -- 172.21.15.31:0/3837153632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2681ebf70 con 0x7fd268153f20 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.667+0000 7fd264c00640 1 -- 172.21.15.31:0/3837153632 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26000ae60 con 0x7fd268153f20 2024-09-11T00:57:42.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.667+0000 7fd26f600640 1 -- 172.21.15.31:0/3837153632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2681f01d0 con 0x7fd268153f20 2024-09-11T00:57:42.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.668+0000 7fd26f600640 1 -- 172.21.15.31:0/3837153632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2681518f0 con 0x7fd268153f20 2024-09-11T00:57:42.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.669+0000 7fd264c00640 1 -- 172.21.15.31:0/3837153632 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd26001b400 con 0x7fd268153f20 2024-09-11T00:57:42.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.670+0000 7fd264c00640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd248077f70 0x7fd24807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:42.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.670+0000 7fd264c00640 1 -- 172.21.15.31:0/3837153632 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 8987+0+0 (secure 0 0 0) 0x7fd26009b400 con 0x7fd268153f20 2024-09-11T00:57:42.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.671+0000 7fd26cc00640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd248077f70 0x7fd24807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:42.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.672+0000 7fd26cc00640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd248077f70 0x7fd24807a430 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd25c034000 tx=0x7fd25c0046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.675+0000 7fd264c00640 1 -- 172.21.15.31:0/3837153632 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd260020020 con 0x7fd268153f20 2024-09-11T00:57:42.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.870+0000 7fd26f600640 1 -- 172.21.15.31:0/3837153632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2680008d0 con 0x7fd268153f20 2024-09-11T00:57:42.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.871+0000 7fd264c00640 1 -- 172.21.15.31:0/3837153632 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+21727 (secure 0 0 0) 0x7fd2681ebf70 con 0x7fd268153f20 2024-09-11T00:57:42.873 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:42.873 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":66,"fsid":"4a2c9b4f-c931-4bcb-847e-5521a1bafcf6","created":"2024-09-11T00:30:19.290206+0000","modified":"2024-09-11T00:57:41.526400+0000","last_up_change":"2024-09-11T00:30:33.773845+0000","last_in_change":"2024-09-11T00:30:23.801176+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T00:30:34.418004+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-11T00:30:52.399156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-11T00:30:53.364342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33da8841-7f3f-47d7-81aa-d0775619d938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6809","nonce":2913875274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6811","nonce":2913875274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6815","nonce":2913875274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2913875274},{"type":"v1","addr":"172.21.15.31:6813","nonce":2913875274}]},"public_addr":"172.21.15.31:6809/2913875274","cluster_addr":"172.21.15.31:6811/2913875274","heartbeat_back_addr":"172.21.15.31:6815/2913875274","heartbeat_front_addr":"172.21.15.31:6813/2913875274","state":["exists","up"]},{"osd":1,"uuid":"9d32c08c-6b40-4a83-8b5f-973af86789d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6801","nonce":3912540156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6803","nonce":3912540156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6807","nonce":3912540156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3912540156},{"type":"v1","addr":"172.21.15.31:6805","nonce":3912540156}]},"public_addr":"172.21.15.31:6801/3912540156","cluster_addr":"172.21.15.31:6803/3912540156","heartbeat_back_addr":"172.21.15.31:6807/3912540156","heartbeat_front_addr":"172.21.15.31:6805/3912540156","state":["exists","up"]},{"osd":2,"uuid":"16b0951a-976f-4612-8c55-8053d5a6c269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6817","nonce":2777449146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6819","nonce":2777449146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6823","nonce":2777449146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2777449146},{"type":"v1","addr":"172.21.15.31:6821","nonce":2777449146}]},"public_addr":"172.21.15.31:6817/2777449146","cluster_addr":"172.21.15.31:6819/2777449146","heartbeat_back_addr":"172.21.15.31:6823/2777449146","heartbeat_front_addr":"172.21.15.31:6821/2777449146","state":["exists","up"]},{"osd":3,"uuid":"530d98c7-9645-4a8b-b1ed-a63c60f70706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6825","nonce":4217184896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6827","nonce":4217184896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6831","nonce":4217184896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":4217184896},{"type":"v1","addr":"172.21.15.31:6829","nonce":4217184896}]},"public_addr":"172.21.15.31:6825/4217184896","cluster_addr":"172.21.15.31:6827/4217184896","heartbeat_back_addr":"172.21.15.31:6831/4217184896","heartbeat_front_addr":"172.21.15.31:6829/4217184896","state":["exists","up"]},{"osd":4,"uuid":"42fd8670-5d3a-4bb8-92b5-ff140b680b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6803","nonce":635888001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6805","nonce":635888001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6809","nonce":635888001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":635888001},{"type":"v1","addr":"172.21.15.204:6807","nonce":635888001}]},"public_addr":"172.21.15.204:6803/635888001","cluster_addr":"172.21.15.204:6805/635888001","heartbeat_back_addr":"172.21.15.204:6809/635888001","heartbeat_front_addr":"172.21.15.204:6807/635888001","state":["exists","up"]},{"osd":5,"uuid":"ce393e08-131f-4a1a-9977-c2d4cdc48d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6811","nonce":482649985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6813","nonce":482649985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6817","nonce":482649985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":482649985},{"type":"v1","addr":"172.21.15.204:6815","nonce":482649985}]},"public_addr":"172.21.15.204:6811/482649985","cluster_addr":"172.21.15.204:6813/482649985","heartbeat_back_addr":"172.21.15.204:6817/482649985","heartbeat_front_addr":"172.21.15.204:6815/482649985","state":["exists","up"]},{"osd":6,"uuid":"49eefb9a-2b2d-49a0-82c9-3cfdbac30336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6819","nonce":1935657097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6821","nonce":1935657097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6825","nonce":1935657097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1935657097},{"type":"v1","addr":"172.21.15.204:6823","nonce":1935657097}]},"public_addr":"172.21.15.204:6819/1935657097","cluster_addr":"172.21.15.204:6821/1935657097","heartbeat_back_addr":"172.21.15.204:6825/1935657097","heartbeat_front_addr":"172.21.15.204:6823/1935657097","state":["exists","up"]},{"osd":7,"uuid":"234a7330-98fe-4736-80b8-6ef614d792be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6827","nonce":276504597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6829","nonce":276504597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6833","nonce":276504597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":276504597},{"type":"v1","addr":"172.21.15.204:6831","nonce":276504597}]},"public_addr":"172.21.15.204:6827/276504597","cluster_addr":"172.21.15.204:6829/276504597","heartbeat_back_addr":"172.21.15.204:6833/276504597","heartbeat_front_addr":"172.21.15.204:6831/276504597","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.586275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.327819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.385293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.823041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:29.836299+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.044102+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.748657+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T00:30:30.701008+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.8","mappings":[{"from":4,"to":5}]},{"pgid":"5.17","mappings":[{"from":4,"to":3}]},{"pgid":"5.26","mappings":[{"from":4,"to":5}]},{"pgid":"5.28","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6833/1834111946":"2024-09-12T00:57:38.506780+0000","172.21.15.204:6836/1329541959":"2024-09-12T00:57:34.594017+0000","172.21.15.31:6832/3721481399":"2024-09-12T00:56:28.030054+0000","172.21.15.204:6835/4094068967":"2024-09-12T00:55:04.553768+0000","172.21.15.31:6833/2956367495":"2024-09-12T00:54:19.542253+0000","172.21.15.204:6837/2689217598":"2024-09-12T00:56:14.574459+0000","172.21.15.31:6834/1855324855":"2024-09-12T00:52:49.521133+0000","172.21.15.204:6835/1981755707":"2024-09-12T00:52:39.517968+0000","172.21.15.204:6834/98007221":"2024-09-12T00:41:49.383893+0000","172.21.15.31:6832/3881548149":"2024-09-12T00:40:39.371058+0000","172.21.15.31:6832/1834111946":"2024-09-12T00:57:38.506780+0000","172.21.15.31:6835/2163558126":"2024-09-12T00:48:14.453181+0000","172.21.15.204:6835/3506150555":"2024-09-12T00:40:24.367167+0000","172.21.15.204:6837/1278135080":"2024-09-12T00:38:14.342086+0000","172.21.15.31:6832/2956367495":"2024-09-12T00:54:19.542253+0000","172.21.15.31:6834/2909466339":"2024-09-12T00:43:22.735779+0000","172.21.15.204:6836/1278135080":"2024-09-12T00:38:14.342086+0000","172.21.15.204:6835/3444890210":"2024-09-12T00:37:16.369607+0000","172.21.15.204:6834/4094068967":"2024-09-12T00:55:04.553768+0000","172.21.15.204:6837/645869523":"2024-09-12T00:35:54.319031+0000","172.21.15.204:6837/4164016077":"2024-09-12T00:50:14.476294+0000","172.21.15.31:6834/2323279595":"2024-09-12T00:44:44.412976+0000","172.21.15.204:6837/1329541959":"2024-09-12T00:57:34.594017+0000","172.21.15.204:6834/3506150555":"2024-09-12T00:40:24.367167+0000","172.21.15.31:6833/4011734390":"2024-09-12T00:37:09.330669+0000","172.21.15.31:6834/3725011623":"2024-09-12T00:50:23.114454+0000","172.21.15.204:6837/3441932736":"2024-09-12T00:31:44.277804+0000","172.21.15.31:6833/3721481399":"2024-09-12T00:56:28.030054+0000","172.21.15.31:6833/2686855151":"2024-09-12T00:32:24.285279+0000","172.21.15.204:6837/1082799504":"2024-09-12T00:47:59.449917+0000","172.21.15.204:6834/1981755707":"2024-09-12T00:52:39.517968+0000","172.21.15.204:6836/645869523":"2024-09-12T00:35:54.319031+0000","172.21.15.31:6835/3925687919":"2024-09-12T00:41:09.376771+0000","172.21.15.204:6835/558470224":"2024-09-12T00:34:09.302673+0000","172.21.15.204:6835/1801149506":"2024-09-12T00:32:54.290770+0000","172.21.15.31:6833/4202305151":"2024-09-12T00:35:09.311954+0000","172.21.15.204:6836/3441932736":"2024-09-12T00:31:44.277804+0000","172.21.15.204:6836/1082799504":"2024-09-12T00:47:59.449917+0000","172.21.15.31:6833/3242133790":"2024-09-12T00:38:14.342154+0000","172.21.15.31:6834/3881548149":"2024-09-12T00:40:39.371058+0000","172.21.15.204:6834/1801149506":"2024-09-12T00:32:54.290770+0000","172.21.15.31:6835/4202305151":"2024-09-12T00:35:09.311954+0000","172.21.15.204:6836/2954547253":"2024-09-12T00:33:34.297314+0000","172.21.15.204:6834/1348312176":"2024-09-12T00:45:14.418207+0000","172.21.15.204:6834/3444890210":"2024-09-12T00:37:16.369607+0000","172.21.15.204:6836/2689217598":"2024-09-12T00:56:14.574459+0000","172.21.15.31:6833/3925687919":"2024-09-12T00:41:09.376771+0000","172.21.15.31:6835/4011734390":"2024-09-12T00:37:09.330669+0000","172.21.15.31:6835/1396366029":"2024-09-12T00:45:59.426330+0000","172.21.15.204:6837/2954547253":"2024-09-12T00:33:34.297314+0000","172.21.15.204:6834/558470224":"2024-09-12T00:34:09.302673+0000","172.21.15.31:6835/1855324855":"2024-09-12T00:52:49.521133+0000","172.21.15.204:6835/98007221":"2024-09-12T00:41:49.383893+0000","172.21.15.31:6834/1396366029":"2024-09-12T00:45:59.426330+0000","172.21.15.31:6832/2909466339":"2024-09-12T00:43:22.735779+0000","172.21.15.204:6837/2630216714":"2024-09-12T00:52:19.506830+0000","172.21.15.204:6834/1575320768":"2024-09-12T00:44:14.407953+0000","172.21.15.204:6836/2630216714":"2024-09-12T00:52:19.506830+0000","172.21.15.31:6835/2686855151":"2024-09-12T00:32:24.285279+0000","172.21.15.204:6835/1575320768":"2024-09-12T00:44:14.407953+0000","172.21.15.204:6835/1348312176":"2024-09-12T00:45:14.418207+0000","172.21.15.204:6837/3513641573":"2024-09-12T00:45:26.774916+0000","172.21.15.31:6833/4205320530":"2024-09-12T00:43:19.398942+0000","172.21.15.204:6834/2113744259":"2024-09-12T00:47:19.441133+0000","172.21.15.204:6835/2113744259":"2024-09-12T00:47:19.441133+0000","172.21.15.31:6835/2323279595":"2024-09-12T00:44:44.412976+0000","172.21.15.204:6836/3513641573":"2024-09-12T00:45:26.774916+0000","172.21.15.31:6834/2163558126":"2024-09-12T00:48:14.453181+0000","172.21.15.204:6834/634556689":"2024-09-12T00:49:34.468674+0000","172.21.15.204:6835/634556689":"2024-09-12T00:49:34.468674+0000","172.21.15.31:6835/3725011623":"2024-09-12T00:50:23.114454+0000","172.21.15.31:6835/3242133790":"2024-09-12T00:38:14.342154+0000","172.21.15.31:6835/4205320530":"2024-09-12T00:43:19.398942+0000","172.21.15.204:6836/4164016077":"2024-09-12T00:50:14.476294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 -- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd248077f70 msgr2=0x7fd24807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd248077f70 0x7fd24807a430 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd25c034000 tx=0x7fd25c0046c0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 -- 172.21.15.31:0/3837153632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd268153f20 msgr2=0x7fd2680c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd268153f20 0x7fd2680c6460 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd26000cac0 tx=0x7fd26000cf80 comp rx=0 tx=0).stop 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 -- 172.21.15.31:0/3837153632 shutdown_connections 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd248077f70 0x7fd24807a430 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd268153f20 0x7fd2680c6460 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2680c23a0 0x7fd2680c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 --2- 172.21.15.31:0/3837153632 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd268072480 0x7fd2681ec9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 -- 172.21.15.31:0/3837153632 >> 172.21.15.31:0/3837153632 conn(0x7fd2680a9750 msgr2=0x7fd26814a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:42.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 -- 172.21.15.31:0/3837153632 shutdown_connections 2024-09-11T00:57:42.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:42.874+0000 7fd26f600640 1 -- 172.21.15.31:0/3837153632 wait complete. 2024-09-11T00:57:42.891 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail b 2024-09-11T00:57:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.059+0000 7fd4b2c00640 1 -- 172.21.15.31:0/3516715321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4940052d0 msgr2=0x7fd49400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.059+0000 7fd4b2c00640 1 --2- 172.21.15.31:0/3516715321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4940052d0 0x7fd49400b9c0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4001f30 tx=0x7fd494005830 comp rx=0 tx=0).stop 2024-09-11T00:57:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.059+0000 7fd4b2200640 1 -- 172.21.15.31:0/3516715321 reap_dead start 2024-09-11T00:57:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.059+0000 7fd4b2c00640 1 -- 172.21.15.31:0/3516715321 shutdown_connections 2024-09-11T00:57:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.059+0000 7fd4b2c00640 1 -- 172.21.15.31:0/3516715321 >> 172.21.15.31:0/3516715321 conn(0x7fd4ac0c4d20 msgr2=0x7fd4ac0c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.059+0000 7fd4b2c00640 1 -- 172.21.15.31:0/3516715321 shutdown_connections 2024-09-11T00:57:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.059+0000 7fd4b2c00640 1 -- 172.21.15.31:0/3516715321 wait complete. 2024-09-11T00:57:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.061+0000 7fd4b2c00640 1 Processor -- start 2024-09-11T00:57:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.061+0000 7fd4b2c00640 1 -- start start 2024-09-11T00:57:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b2c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4940052d0 0x7fd4ac145210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b2c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4ac145e30 0x7fd4ac145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b2c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4ac152d90 0x7fd4ac153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b2c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac06e390 con 0x7fd4ac145e30 2024-09-11T00:57:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b2c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac06e210 con 0x7fd4ac152d90 2024-09-11T00:57:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b2c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fd4ac14cf60 con 0x7fd4940052d0 2024-09-11T00:57:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4ac145e30 0x7fd4ac145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b0e00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4ac145e30 0x7fd4ac145750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55480/0 (socket says 172.21.15.31:55480) 2024-09-11T00:57:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b0e00640 1 -- 172.21.15.31:0/2583746984 learned_addr learned my addr 172.21.15.31:0/2583746984 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b1800640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4940052d0 0x7fd4ac145210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:43.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b2200640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4ac152d90 0x7fd4ac153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:43.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b0e00640 1 -- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4940052d0 msgr2=0x7fd4ac145210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b0e00640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4940052d0 0x7fd4ac145210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b0e00640 1 -- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4ac152d90 msgr2=0x7fd4ac153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b0e00640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4ac152d90 0x7fd4ac153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.062+0000 7fd4b0e00640 1 -- 172.21.15.31:0/2583746984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ac152410 con 0x7fd4ac145e30 2024-09-11T00:57:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.063+0000 7fd4b0e00640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4ac145e30 0x7fd4ac145750 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fd4980119f0 tx=0x7fd498011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.063+0000 7fd4a9600640 1 -- 172.21.15.31:0/2583746984 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49801f070 con 0x7fd4ac145e30 2024-09-11T00:57:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.063+0000 7fd4a9600640 1 -- 172.21.15.31:0/2583746984 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fd49800fdf0 con 0x7fd4ac145e30 2024-09-11T00:57:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.063+0000 7fd4a9600640 1 -- 172.21.15.31:0/2583746984 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4980126b0 con 0x7fd4ac145e30 2024-09-11T00:57:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.063+0000 7fd4b2c00640 1 -- 172.21.15.31:0/2583746984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4ac151650 con 0x7fd4ac145e30 2024-09-11T00:57:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.063+0000 7fd4b2c00640 1 -- 172.21.15.31:0/2583746984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4ac151c10 con 0x7fd4ac145e30 2024-09-11T00:57:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.065+0000 7fd4b2c00640 1 -- 172.21.15.31:0/2583746984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4ac070450 con 0x7fd4ac145e30 2024-09-11T00:57:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.066+0000 7fd4a9600640 1 -- 172.21.15.31:0/2583746984 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fd498012850 con 0x7fd4ac145e30 2024-09-11T00:57:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.066+0000 7fd4a9600640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd48c077f70 0x7fd48c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.067+0000 7fd4a9600640 1 -- 172.21.15.31:0/2583746984 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 8987+0+0 (secure 0 0 0) 0x7fd49809ba10 con 0x7fd4ac145e30 2024-09-11T00:57:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.067+0000 7fd4b1800640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd48c077f70 0x7fd48c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.068+0000 7fd4b1800640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd48c077f70 0x7fd48c07a430 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0006bd0 tx=0x7fd4a0001040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.071+0000 7fd4a9600640 1 -- 172.21.15.31:0/2583746984 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fd4980630b0 con 0x7fd4ac145e30 2024-09-11T00:57:43.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.301+0000 7fd4b2c00640 1 -- 172.21.15.31:0/2583746984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "b"} v 0) -- 0x7fd4ac14c4a0 con 0x7fd4ac145e30 2024-09-11T00:57:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.304+0000 7fd4a9600640 1 -- 172.21.15.31:0/2583746984 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "b"}]=0 b marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v193) ==== 151+0+0 (secure 0 0 0) 0x7fd498067ae0 con 0x7fd4ac145e30 2024-09-11T00:57:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:b marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-11T00:57:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.306+0000 7fd4b2c00640 1 -- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd48c077f70 msgr2=0x7fd48c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.306+0000 7fd4b2c00640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd48c077f70 0x7fd48c07a430 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0006bd0 tx=0x7fd4a0001040 comp rx=0 tx=0).stop 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.306+0000 7fd4b2c00640 1 -- 172.21.15.31:0/2583746984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4ac145e30 msgr2=0x7fd4ac145750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.306+0000 7fd4b2c00640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4ac145e30 0x7fd4ac145750 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fd4980119f0 tx=0x7fd498011eb0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.306+0000 7fd4b2c00640 1 -- 172.21.15.31:0/2583746984 shutdown_connections 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.306+0000 7fd4b2c00640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fd48c077f70 0x7fd48c07a430 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.306+0000 7fd4b2c00640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4ac152d90 0x7fd4ac153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.306+0000 7fd4b2c00640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4ac145e30 0x7fd4ac145750 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.306+0000 7fd4b2c00640 1 --2- 172.21.15.31:0/2583746984 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fd4940052d0 0x7fd4ac145210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.306+0000 7fd4b2c00640 1 -- 172.21.15.31:0/2583746984 >> 172.21.15.31:0/2583746984 conn(0x7fd4ac0c4d20 msgr2=0x7fd4ac0737f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.307+0000 7fd4b2c00640 1 -- 172.21.15.31:0/2583746984 shutdown_connections 2024-09-11T00:57:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.307+0000 7fd4b2c00640 1 -- 172.21.15.31:0/2583746984 wait complete. 2024-09-11T00:57:43.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm b --yes-i-really-mean-it 2024-09-11T00:57:43.379 INFO:tasks.ceph.mds.b.smithi204.stdout:starting mds.b at 2024-09-11T00:57:43.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.490+0000 7f6b45000640 1 -- 172.21.15.31:0/3213130128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b40153e90 msgr2=0x7f6b40154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.490+0000 7f6b45000640 1 --2- 172.21.15.31:0/3213130128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b40153e90 0x7f6b40154320 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f6b4006dee0 tx=0x7f6b3802a9c0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.490+0000 7f6b45000640 1 -- 172.21.15.31:0/3213130128 shutdown_connections 2024-09-11T00:57:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.490+0000 7f6b45000640 1 --2- 172.21.15.31:0/3213130128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b40153e90 0x7f6b40154320 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.490+0000 7f6b45000640 1 --2- 172.21.15.31:0/3213130128 >> v2:172.21.15.204:3300/0 conn(0x7f6b400c2330 0x7f6b400c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.490+0000 7f6b45000640 1 --2- 172.21.15.31:0/3213130128 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b4014de40 0x7f6b400c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.490+0000 7f6b45000640 1 -- 172.21.15.31:0/3213130128 >> 172.21.15.31:0/3213130128 conn(0x7f6b4006d560 msgr2=0x7f6b400c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.490+0000 7f6b45000640 1 -- 172.21.15.31:0/3213130128 shutdown_connections 2024-09-11T00:57:43.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.490+0000 7f6b45000640 1 -- 172.21.15.31:0/3213130128 wait complete. 2024-09-11T00:57:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.492+0000 7f6b45000640 1 Processor -- start 2024-09-11T00:57:43.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.492+0000 7f6b45000640 1 -- start start 2024-09-11T00:57:43.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.493+0000 7f6b45000640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b400c2330 0x7f6b4014b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:43.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.493+0000 7f6b45000640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b4014de40 0x7f6b400c1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:43.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.493+0000 7f6b45000640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b40153e90 0x7f6b401ec210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:43.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.493+0000 7f6b45000640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4014e3a0 con 0x7f6b400c2330 2024-09-11T00:57:43.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.493+0000 7f6b45000640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4014e220 con 0x7f6b4014de40 2024-09-11T00:57:43.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.493+0000 7f6b45000640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f6b400c4e40 con 0x7f6b40153e90 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.493+0000 7f6b3de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b400c2330 0x7f6b4014b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.493+0000 7f6b3de00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b400c2330 0x7f6b4014b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55510/0 (socket says 172.21.15.31:55510) 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.493+0000 7f6b3de00640 1 -- 172.21.15.31:0/1449197562 learned_addr learned my addr 172.21.15.31:0/1449197562 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b3e800640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b40153e90 0x7f6b401ec210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b35400640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b4014de40 0x7f6b400c1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b3de00640 1 -- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b40153e90 msgr2=0x7f6b401ec210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b3de00640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b40153e90 0x7f6b401ec210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b3de00640 1 -- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b4014de40 msgr2=0x7f6b400c1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b3de00640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b4014de40 0x7f6b400c1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b3de00640 1 -- 172.21.15.31:0/1449197562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b40156b20 con 0x7f6b400c2330 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b3de00640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b400c2330 0x7f6b4014b1d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000d770 tx=0x7f6b3000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b36a00640 1 -- 172.21.15.31:0/1449197562 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30017070 con 0x7f6b400c2330 2024-09-11T00:57:43.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b36a00640 1 -- 172.21.15.31:0/1449197562 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f6b30009ce0 con 0x7f6b400c2330 2024-09-11T00:57:43.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b36a00640 1 -- 172.21.15.31:0/1449197562 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30005280 con 0x7f6b400c2330 2024-09-11T00:57:43.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.494+0000 7f6b45000640 1 -- 172.21.15.31:0/1449197562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b401df6d0 con 0x7f6b400c2330 2024-09-11T00:57:43.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.495+0000 7f6b45000640 1 -- 172.21.15.31:0/1449197562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b40156420 con 0x7f6b400c2330 2024-09-11T00:57:43.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.496+0000 7f6b45000640 1 -- 172.21.15.31:0/1449197562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b401548b0 con 0x7f6b400c2330 2024-09-11T00:57:43.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.497+0000 7f6b36a00640 1 -- 172.21.15.31:0/1449197562 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f6b30005420 con 0x7f6b400c2330 2024-09-11T00:57:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.498+0000 7f6b36a00640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6b1c077f70 0x7f6b1c07a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.498+0000 7f6b36a00640 1 -- 172.21.15.31:0/1449197562 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 9073+0+0 (secure 0 0 0) 0x7f6b3009abf0 con 0x7f6b400c2330 2024-09-11T00:57:43.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.499+0000 7f6b35400640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6b1c077f70 0x7f6b1c07a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:43.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.500+0000 7f6b35400640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6b1c077f70 0x7f6b1c07a430 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6b401ed340 tx=0x7f6b2800b2e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:43.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.503+0000 7f6b36a00640 1 -- 172.21.15.31:0/1449197562 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f6b300622c0 con 0x7f6b400c2330 2024-09-11T00:57:43.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:43.740+0000 7f6b45000640 1 -- 172.21.15.31:0/1449197562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "b", "yes_i_really_mean_it": true} v 0) -- 0x7f6b400c6a80 con 0x7f6b400c2330 2024-09-11T00:57:44.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.306+0000 7f6b36a00640 1 -- 172.21.15.31:0/1449197562 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "b", "yes_i_really_mean_it": true}]=0 v194) ==== 99+0+0 (secure 0 0 0) 0x7f6b30066cf0 con 0x7f6b400c2330 2024-09-11T00:57:44.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 -- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6b1c077f70 msgr2=0x7f6b1c07a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:44.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6b1c077f70 0x7f6b1c07a430 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6b401ed340 tx=0x7f6b2800b2e0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 -- 172.21.15.31:0/1449197562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b400c2330 msgr2=0x7f6b4014b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:44.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b400c2330 0x7f6b4014b1d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000d770 tx=0x7f6b3000dc30 comp rx=0 tx=0).stop 2024-09-11T00:57:44.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 -- 172.21.15.31:0/1449197562 shutdown_connections 2024-09-11T00:57:44.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f6b1c077f70 0x7f6b1c07a430 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f6b40153e90 0x7f6b401ec210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b4014de40 0x7f6b400c1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 --2- 172.21.15.31:0/1449197562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b400c2330 0x7f6b4014b1d0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 -- 172.21.15.31:0/1449197562 >> 172.21.15.31:0/1449197562 conn(0x7f6b4006d560 msgr2=0x7f6b400c43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 -- 172.21.15.31:0/1449197562 shutdown_connections 2024-09-11T00:57:44.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.310+0000 7f6b45000640 1 -- 172.21.15.31:0/1449197562 wait complete. 2024-09-11T00:57:44.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm b_metadata b_metadata --yes-i-really-really-mean-it 2024-09-11T00:57:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.495+0000 7f793e600640 1 -- 172.21.15.31:0/3226479634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7938072480 msgr2=0x7f79381511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:44.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.495+0000 7f793e600640 1 --2- 172.21.15.31:0/3226479634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7938072480 0x7f79381511a0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f7928005fd0 tx=0x7f792802a670 comp rx=0 tx=0).stop 2024-09-11T00:57:44.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.495+0000 7f793e600640 1 -- 172.21.15.31:0/3226479634 shutdown_connections 2024-09-11T00:57:44.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.495+0000 7f793e600640 1 --2- 172.21.15.31:0/3226479634 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f79380c23a0 0x7f79381516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.495+0000 7f793e600640 1 --2- 172.21.15.31:0/3226479634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7938072480 0x7f79381511a0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.495+0000 7f793e600640 1 -- 172.21.15.31:0/3226479634 >> 172.21.15.31:0/3226479634 conn(0x7f79380a9750 msgr2=0x7f79380a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:44.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.495+0000 7f793e600640 1 -- 172.21.15.31:0/3226479634 shutdown_connections 2024-09-11T00:57:44.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.495+0000 7f793e600640 1 -- 172.21.15.31:0/3226479634 wait complete. 2024-09-11T00:57:44.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.497+0000 7f793e600640 1 Processor -- start 2024-09-11T00:57:44.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.497+0000 7f793e600640 1 -- start start 2024-09-11T00:57:44.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f793e600640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7938072480 0x7f79380c7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:44.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f793e600640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79380c23a0 0x7f79381ec180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:44.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f793e600640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7938153f20 0x7f79381ec6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:44.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f793e600640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7938074200 con 0x7f79380c23a0 2024-09-11T00:57:44.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f793e600640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7938074080 con 0x7f7938072480 2024-09-11T00:57:44.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f793e600640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7f7938072860 con 0x7f7938153f20 2024-09-11T00:57:44.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f792ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79380c23a0 0x7f79381ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:44.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f792ea00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79380c23a0 0x7f79381ec180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55528/0 (socket says 172.21.15.31:55528) 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f792ea00640 1 -- 172.21.15.31:0/247058606 learned_addr learned my addr 172.21.15.31:0/247058606 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f7937400640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7938072480 0x7f79380c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f7937e00640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7938153f20 0x7f79381ec6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f792ea00640 1 -- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7938153f20 msgr2=0x7f79381ec6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f792ea00640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7938153f20 0x7f79381ec6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f792ea00640 1 -- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7938072480 msgr2=0x7f79380c7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f792ea00640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7938072480 0x7f79380c7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f792ea00640 1 -- 172.21.15.31:0/247058606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79380c1980 con 0x7f79380c23a0 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.498+0000 7f7937e00640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7938153f20 0x7f79381ec6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.499+0000 7f792ea00640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79380c23a0 0x7f79381ec180 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f792000d770 tx=0x7f792000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:44.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.499+0000 7f792fe00640 1 -- 172.21.15.31:0/247058606 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7920017070 con 0x7f79380c23a0 2024-09-11T00:57:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.499+0000 7f792fe00640 1 -- 172.21.15.31:0/247058606 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f7920004330 con 0x7f79380c23a0 2024-09-11T00:57:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.499+0000 7f792fe00640 1 -- 172.21.15.31:0/247058606 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7920009c30 con 0x7f79380c23a0 2024-09-11T00:57:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.499+0000 7f793e600640 1 -- 172.21.15.31:0/247058606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79381440e0 con 0x7f79380c23a0 2024-09-11T00:57:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.499+0000 7f793e600640 1 -- 172.21.15.31:0/247058606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7938152160 con 0x7f79380c23a0 2024-09-11T00:57:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.500+0000 7f793e600640 1 -- 172.21.15.31:0/247058606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f793814ad00 con 0x7f79380c23a0 2024-09-11T00:57:44.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.501+0000 7f792fe00640 1 -- 172.21.15.31:0/247058606 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7f79200035f0 con 0x7f79380c23a0 2024-09-11T00:57:44.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.502+0000 7f792fe00640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7914077fa0 0x7f791407a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:44.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.502+0000 7f792fe00640 1 -- 172.21.15.31:0/247058606 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 9073+0+0 (secure 0 0 0) 0x7f792009aa70 con 0x7f79380c23a0 2024-09-11T00:57:44.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.503+0000 7f7937400640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7914077fa0 0x7f791407a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:44.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.503+0000 7f7937400640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7914077fa0 0x7f791407a460 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7928033000 tx=0x7f79280046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:44.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.506+0000 7f792fe00640 1 -- 172.21.15.31:0/247058606 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7f79200621d0 con 0x7f79380c23a0 2024-09-11T00:57:44.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.705+0000 7f793e600640 1 -- 172.21.15.31:0/247058606 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "b_metadata", "pool2": "b_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f793814aa20 con 0x7f79380c23a0 2024-09-11T00:57:44.758 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-11T00:57:44.759 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-11T00:57:44.759 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-11T00:57:44.763 INFO:tasks.ceph.mds.d:Started 2024-09-11T00:57:44.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:44.898 INFO:tasks.ceph.mds.d.smithi204.stdout:starting mds.d at 2024-09-11T00:57:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.923+0000 7fed82c00640 1 -- 172.21.15.31:0/811003343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed7c1518b0 msgr2=0x7fed7c0c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.923+0000 7fed82c00640 1 --2- 172.21.15.31:0/811003343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed7c1518b0 0x7fed7c0c3600 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fed74009be0 tx=0x7fed7402a900 comp rx=0 tx=0).stop 2024-09-11T00:57:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.924+0000 7fed82c00640 1 -- 172.21.15.31:0/811003343 shutdown_connections 2024-09-11T00:57:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.924+0000 7fed82c00640 1 --2- 172.21.15.31:0/811003343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed7c1518b0 0x7fed7c0c3600 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.924+0000 7fed82c00640 1 --2- 172.21.15.31:0/811003343 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fed7c0c23a0 0x7fed7c0c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.924+0000 7fed82c00640 1 -- 172.21.15.31:0/811003343 >> 172.21.15.31:0/811003343 conn(0x7fed7c0a9750 msgr2=0x7fed7c0a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.924+0000 7fed82c00640 1 -- 172.21.15.31:0/811003343 shutdown_connections 2024-09-11T00:57:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.924+0000 7fed82c00640 1 -- 172.21.15.31:0/811003343 wait complete. 2024-09-11T00:57:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.925+0000 7fed82c00640 1 Processor -- start 2024-09-11T00:57:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.925+0000 7fed82c00640 1 -- start start 2024-09-11T00:57:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.926+0000 7fed82c00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed7c072480 0x7fed7c1eb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.926+0000 7fed82c00640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fed7c0c23a0 0x7fed7c14f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.926+0000 7fed82c00640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed7c1518b0 0x7fed7c1e2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.926+0000 7fed82c00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c072860 con 0x7fed7c072480 2024-09-11T00:57:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.926+0000 7fed82c00640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c074290 con 0x7fed7c1518b0 2024-09-11T00:57:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.926+0000 7fed82c00640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fed7c0729e0 con 0x7fed7c0c23a0 2024-09-11T00:57:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.926+0000 7fed7be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed7c072480 0x7fed7c1eb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.926+0000 7fed7be00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed7c072480 0x7fed7c1eb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55542/0 (socket says 172.21.15.31:55542) 2024-09-11T00:57:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.926+0000 7fed7be00640 1 -- 172.21.15.31:0/3196134588 learned_addr learned my addr 172.21.15.31:0/3196134588 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.927+0000 7fed73400640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fed7c0c23a0 0x7fed7c14f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.927+0000 7fed80c00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed7c1518b0 0x7fed7c1e2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.927+0000 7fed7be00640 1 -- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fed7c0c23a0 msgr2=0x7fed7c14f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.927+0000 7fed7be00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fed7c0c23a0 0x7fed7c14f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.927+0000 7fed7be00640 1 -- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed7c1518b0 msgr2=0x7fed7c1e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.927+0000 7fed7be00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed7c1518b0 0x7fed7c1e2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.927+0000 7fed7be00640 1 -- 172.21.15.31:0/3196134588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7c1dd790 con 0x7fed7c072480 2024-09-11T00:57:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.927+0000 7fed7be00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed7c072480 0x7fed7c1eb820 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00c700 tx=0x7fed6c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.927+0000 7fed78c00640 1 -- 172.21.15.31:0/3196134588 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c016020 con 0x7fed7c072480 2024-09-11T00:57:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.928+0000 7fed78c00640 1 -- 172.21.15.31:0/3196134588 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fed6c00a840 con 0x7fed7c072480 2024-09-11T00:57:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.928+0000 7fed78c00640 1 -- 172.21.15.31:0/3196134588 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c00ac90 con 0x7fed7c072480 2024-09-11T00:57:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.928+0000 7fed82c00640 1 -- 172.21.15.31:0/3196134588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed7c06fdf0 con 0x7fed7c072480 2024-09-11T00:57:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.928+0000 7fed82c00640 1 -- 172.21.15.31:0/3196134588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed7c0700d0 con 0x7fed7c072480 2024-09-11T00:57:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.929+0000 7fed82c00640 1 -- 172.21.15.31:0/3196134588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed7c153f20 con 0x7fed7c072480 2024-09-11T00:57:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.930+0000 7fed78c00640 1 -- 172.21.15.31:0/3196134588 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fed6c01b400 con 0x7fed7c072480 2024-09-11T00:57:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.931+0000 7fed78c00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed58077f70 0x7fed5807a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.931+0000 7fed78c00640 1 -- 172.21.15.31:0/3196134588 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 9073+0+0 (secure 0 0 0) 0x7fed6c09af10 con 0x7fed7c072480 2024-09-11T00:57:44.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.931+0000 7fed73400640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed58077f70 0x7fed5807a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:44.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.932+0000 7fed73400640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed58077f70 0x7fed5807a430 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0c1ac0 tx=0x7fed64005f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:44.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:44.935+0000 7fed78c00640 1 -- 172.21.15.31:0/3196134588 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fed6c020020 con 0x7fed7c072480 2024-09-11T00:57:45.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.169+0000 7fed82c00640 1 -- 172.21.15.31:0/3196134588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed7c14f830 con 0x7fed7c072480 2024-09-11T00:57:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.169+0000 7fed78c00640 1 -- 172.21.15.31:0/3196134588 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) ==== 95+0+3087 (secure 0 0 0) 0x7fed6c062670 con 0x7fed7c072480 2024-09-11T00:57:45.171 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:45.171 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":194,"btime":"2024-09-11T00:57:44:305023+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":6962,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1368508756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1368508756},{"type":"v1","addr":"172.21.15.204:6835","nonce":1368508756}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":194},{"gid":8103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/22472818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":22472818},{"type":"v1","addr":"172.21.15.31:6833","nonce":22472818}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[]} 2024-09-11T00:57:45.172 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 194 2024-09-11T00:57:45.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 -- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed58077f70 msgr2=0x7fed5807a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:45.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed58077f70 0x7fed5807a430 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0c1ac0 tx=0x7fed64005f90 comp rx=0 tx=0).stop 2024-09-11T00:57:45.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 -- 172.21.15.31:0/3196134588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed7c072480 msgr2=0x7fed7c1eb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:45.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed7c072480 0x7fed7c1eb820 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00c700 tx=0x7fed6c00cbc0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 -- 172.21.15.31:0/3196134588 shutdown_connections 2024-09-11T00:57:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fed58077f70 0x7fed5807a430 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed7c1518b0 0x7fed7c1e2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fed7c0c23a0 0x7fed7c14f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 --2- 172.21.15.31:0/3196134588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed7c072480 0x7fed7c1eb820 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 -- 172.21.15.31:0/3196134588 >> 172.21.15.31:0/3196134588 conn(0x7fed7c0a9750 msgr2=0x7fed7c14cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 -- 172.21.15.31:0/3196134588 shutdown_connections 2024-09-11T00:57:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.172+0000 7fed82c00640 1 -- 172.21.15.31:0/3196134588 wait complete. 2024-09-11T00:57:45.189 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-11T00:57:45.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.555+0000 7f792fe00640 1 -- 172.21.15.31:0/247058606 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "b_metadata", "pool2": "b_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'b_metadata' removed v68) ==== 166+0+0 (secure 0 0 0) 0x7f7920069120 con 0x7f79380c23a0 2024-09-11T00:57:45.557 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'b_metadata' removed 2024-09-11T00:57:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 -- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7914077fa0 msgr2=0x7f791407a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7914077fa0 0x7f791407a460 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7928033000 tx=0x7f79280046c0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 -- 172.21.15.31:0/247058606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79380c23a0 msgr2=0x7f79381ec180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79380c23a0 0x7f79381ec180 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f792000d770 tx=0x7f792000dc30 comp rx=0 tx=0).stop 2024-09-11T00:57:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 -- 172.21.15.31:0/247058606 shutdown_connections 2024-09-11T00:57:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7f7914077fa0 0x7f791407a460 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7f7938153f20 0x7f79381ec6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79380c23a0 0x7f79381ec180 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 --2- 172.21.15.31:0/247058606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7938072480 0x7f79380c7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 -- 172.21.15.31:0/247058606 >> 172.21.15.31:0/247058606 conn(0x7f79380a9750 msgr2=0x7f7938151180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:45.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 -- 172.21.15.31:0/247058606 shutdown_connections 2024-09-11T00:57:45.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.559+0000 7f793e600640 1 -- 172.21.15.31:0/247058606 wait complete. 2024-09-11T00:57:45.579 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm b_data b_data --yes-i-really-really-mean-it 2024-09-11T00:57:45.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.744+0000 7fe26c800640 1 -- 172.21.15.31:0/2456501185 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe264072480 msgr2=0x7fe264074a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:45.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.744+0000 7fe26c800640 1 --2- 172.21.15.31:0/2456501185 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe264072480 0x7fe264074a90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe258009830 tx=0x7fe25802a480 comp rx=0 tx=0).stop 2024-09-11T00:57:45.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.744+0000 7fe26c800640 1 -- 172.21.15.31:0/2456501185 shutdown_connections 2024-09-11T00:57:45.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.744+0000 7fe26c800640 1 --2- 172.21.15.31:0/2456501185 >> v2:172.21.15.204:3300/0 conn(0x7fe264154b60 0x7fe2640c2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.744+0000 7fe26c800640 1 --2- 172.21.15.31:0/2456501185 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe264072480 0x7fe264074a90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.744+0000 7fe26c800640 1 -- 172.21.15.31:0/2456501185 >> 172.21.15.31:0/2456501185 conn(0x7fe2640a9750 msgr2=0x7fe2640a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:45.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.744+0000 7fe26c800640 1 -- 172.21.15.31:0/2456501185 shutdown_connections 2024-09-11T00:57:45.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.744+0000 7fe26c800640 1 -- 172.21.15.31:0/2456501185 wait complete. 2024-09-11T00:57:45.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.745+0000 7fe26c800640 1 Processor -- start 2024-09-11T00:57:45.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.746+0000 7fe26c800640 1 -- start start 2024-09-11T00:57:45.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.746+0000 7fe26c800640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe264072480 0x7fe2641500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.746+0000 7fe26c800640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2640c34e0 0x7fe26414dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.746+0000 7fe26c800640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe264154b60 0x7fe264141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.746+0000 7fe26c800640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe2640a7140 con 0x7fe2640c34e0 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.746+0000 7fe26c800640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe2640c23a0 con 0x7fe264072480 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.746+0000 7fe26c800640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7fe264072860 con 0x7fe264154b60 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe269400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2640c34e0 0x7fe26414dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe269400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2640c34e0 0x7fe26414dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55548/0 (socket says 172.21.15.31:55548) 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe269400640 1 -- 172.21.15.31:0/3883407870 learned_addr learned my addr 172.21.15.31:0/3883407870 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe269e00640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe264072480 0x7fe2641500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe26a800640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe264154b60 0x7fe264141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe269400640 1 -- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe264154b60 msgr2=0x7fe264141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe269400640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe264154b60 0x7fe264141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe269400640 1 -- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe264072480 msgr2=0x7fe2641500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:45.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe269400640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe264072480 0x7fe2641500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:45.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe269400640 1 -- 172.21.15.31:0/3883407870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26414e9e0 con 0x7fe2640c34e0 2024-09-11T00:57:45.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.747+0000 7fe269400640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2640c34e0 0x7fe26414dc30 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fe254009fd0 tx=0x7fe25400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.748+0000 7fe25e000640 1 -- 172.21.15.31:0/3883407870 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25400cc60 con 0x7fe2640c34e0 2024-09-11T00:57:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.748+0000 7fe25e000640 1 -- 172.21.15.31:0/3883407870 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fe25400ee70 con 0x7fe2640c34e0 2024-09-11T00:57:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.748+0000 7fe25e000640 1 -- 172.21.15.31:0/3883407870 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254004260 con 0x7fe2640c34e0 2024-09-11T00:57:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.748+0000 7fe26c800640 1 -- 172.21.15.31:0/3883407870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe264147d30 con 0x7fe2640c34e0 2024-09-11T00:57:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.748+0000 7fe26c800640 1 -- 172.21.15.31:0/3883407870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe26414ed80 con 0x7fe2640c34e0 2024-09-11T00:57:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.749+0000 7fe26c800640 1 -- 172.21.15.31:0/3883407870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2640c5120 con 0x7fe2640c34e0 2024-09-11T00:57:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.750+0000 7fe25e000640 1 -- 172.21.15.31:0/3883407870 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7fe2540035f0 con 0x7fe2640c34e0 2024-09-11T00:57:45.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.751+0000 7fe25e000640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe248077fa0 0x7fe24807a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:45.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.751+0000 7fe25e000640 1 -- 172.21.15.31:0/3883407870 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 8645+0+0 (secure 0 0 0) 0x7fe25409b130 con 0x7fe2640c34e0 2024-09-11T00:57:45.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.751+0000 7fe269e00640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe248077fa0 0x7fe24807a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:45.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.752+0000 7fe269e00640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe248077fa0 0x7fe24807a460 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe258034000 tx=0x7fe2580046c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:45.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.755+0000 7fe25e000640 1 -- 172.21.15.31:0/3883407870 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7fe2640c5120 con 0x7fe2640c34e0 2024-09-11T00:57:45.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:45.952+0000 7fe26c800640 1 -- 172.21.15.31:0/3883407870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "b_data", "pool2": "b_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe2640a6f20 con 0x7fe2640c34e0 2024-09-11T00:57:46.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.564+0000 7fe25e000640 1 -- 172.21.15.31:0/3883407870 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "b_data", "pool2": "b_data", "yes_i_really_really_mean_it": true}]=0 pool 'b_data' removed v69) ==== 154+0+0 (secure 0 0 0) 0x7fe254062a40 con 0x7fe2640c34e0 2024-09-11T00:57:46.566 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'b_data' removed 2024-09-11T00:57:46.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.567+0000 7fe26c800640 1 -- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe248077fa0 msgr2=0x7fe24807a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:46.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.567+0000 7fe26c800640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe248077fa0 0x7fe24807a460 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe258034000 tx=0x7fe2580046c0 comp rx=0 tx=0).stop 2024-09-11T00:57:46.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.567+0000 7fe26c800640 1 -- 172.21.15.31:0/3883407870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2640c34e0 msgr2=0x7fe26414dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:46.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.567+0000 7fe26c800640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2640c34e0 0x7fe26414dc30 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fe254009fd0 tx=0x7fe25400ec20 comp rx=0 tx=0).stop 2024-09-11T00:57:46.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.567+0000 7fe26c800640 1 -- 172.21.15.31:0/3883407870 shutdown_connections 2024-09-11T00:57:46.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.567+0000 7fe26c800640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7fe248077fa0 0x7fe24807a460 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:46.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.567+0000 7fe26c800640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7fe264154b60 0x7fe264141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:46.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.567+0000 7fe26c800640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2640c34e0 0x7fe26414dc30 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:46.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.567+0000 7fe26c800640 1 --2- 172.21.15.31:0/3883407870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe264072480 0x7fe2641500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:46.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.567+0000 7fe26c800640 1 -- 172.21.15.31:0/3883407870 >> 172.21.15.31:0/3883407870 conn(0x7fe2640a9750 msgr2=0x7fe26414be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:46.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.568+0000 7fe26c800640 1 -- 172.21.15.31:0/3883407870 shutdown_connections 2024-09-11T00:57:46.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:46.568+0000 7fe26c800640 1 -- 172.21.15.31:0/3883407870 wait complete. 2024-09-11T00:57:46.587 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T00:57:46.588 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-11T00:57:46.588 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:57:46.603 INFO:tasks.ceph.mds.a:Stopped 2024-09-11T00:57:46.604 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-11T00:57:46.604 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:57:46.660 INFO:tasks.ceph.mds.c:Stopped 2024-09-11T00:57:46.660 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-11T00:57:46.660 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:57:46.688 INFO:tasks.ceph.mds.b:Stopped 2024-09-11T00:57:46.688 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-11T00:57:46.688 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:57:46.742 INFO:tasks.ceph.mds.d:Stopped 2024-09-11T00:57:46.743 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T00:57:46.743 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T00:57:46.743 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:57:47.190 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:57:47.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.348+0000 7ff519400640 1 -- 172.21.15.31:0/2748278244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5141518b0 msgr2=0x7ff5140c37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:47.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.348+0000 7ff519400640 1 --2- 172.21.15.31:0/2748278244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5141518b0 0x7ff5140c37d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff50c005fd0 tx=0x7ff50c02a9d0 comp rx=0 tx=0).stop 2024-09-11T00:57:47.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.349+0000 7ff519400640 1 -- 172.21.15.31:0/2748278244 shutdown_connections 2024-09-11T00:57:47.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.349+0000 7ff519400640 1 --2- 172.21.15.31:0/2748278244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5141518b0 0x7ff5140c37d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:47.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.349+0000 7ff519400640 1 -- 172.21.15.31:0/2748278244 >> 172.21.15.31:0/2748278244 conn(0x7ff5140a9750 msgr2=0x7ff5140a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:47.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.349+0000 7ff519400640 1 -- 172.21.15.31:0/2748278244 shutdown_connections 2024-09-11T00:57:47.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.349+0000 7ff519400640 1 -- 172.21.15.31:0/2748278244 wait complete. 2024-09-11T00:57:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.350+0000 7ff519400640 1 Processor -- start 2024-09-11T00:57:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.350+0000 7ff519400640 1 -- start start 2024-09-11T00:57:47.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.351+0000 7ff519400640 1 --2- >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff514072480 0x7ff5141436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.351+0000 7ff519400640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5140c23a0 0x7ff5140c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.351+0000 7ff519400640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5141518b0 0x7ff5141ded80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.351+0000 7ff519400640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff514074080 con 0x7ff5141518b0 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.351+0000 7ff519400640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff514074200 con 0x7ff5140c23a0 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.351+0000 7ff519400640 1 -- --> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] -- mon_getmap magic: 0 -- 0x7ff514072860 con 0x7ff514072480 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.351+0000 7ff513200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5141518b0 0x7ff5141ded80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff513200640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5141518b0 0x7ff5141ded80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55554/0 (socket says 172.21.15.31:55554) 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff513200640 1 -- 172.21.15.31:0/3233226071 learned_addr learned my addr 172.21.15.31:0/3233226071 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff509e00640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5140c23a0 0x7ff5140c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff512800640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff514072480 0x7ff5141436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:47.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff513200640 1 -- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff514072480 msgr2=0x7ff5141436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff513200640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff514072480 0x7ff5141436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff513200640 1 -- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5140c23a0 msgr2=0x7ff5140c0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff513200640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5140c23a0 0x7ff5140c0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff513200640 1 -- 172.21.15.31:0/3233226071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5141de310 con 0x7ff5141518b0 2024-09-11T00:57:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff513200640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5141518b0 0x7ff5141ded80 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009cd0 tx=0x7ff50c031e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.352+0000 7ff50b400640 1 -- 172.21.15.31:0/3233226071 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c045070 con 0x7ff5141518b0 2024-09-11T00:57:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.353+0000 7ff50b400640 1 -- 172.21.15.31:0/3233226071 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7ff50c003970 con 0x7ff5141518b0 2024-09-11T00:57:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.353+0000 7ff50b400640 1 -- 172.21.15.31:0/3233226071 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c02fe60 con 0x7ff5141518b0 2024-09-11T00:57:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.353+0000 7ff519400640 1 -- 172.21.15.31:0/3233226071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5141440c0 con 0x7ff5141518b0 2024-09-11T00:57:47.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.353+0000 7ff519400640 1 -- 172.21.15.31:0/3233226071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5141de860 con 0x7ff5141518b0 2024-09-11T00:57:47.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.354+0000 7ff519400640 1 -- 172.21.15.31:0/3233226071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff514154050 con 0x7ff5141518b0 2024-09-11T00:57:47.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.355+0000 7ff50b400640 1 -- 172.21.15.31:0/3233226071 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 100601+0+0 (secure 0 0 0) 0x7ff50c004040 con 0x7ff5141518b0 2024-09-11T00:57:47.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.356+0000 7ff50b400640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff4f0077f70 0x7ff4f007a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T00:57:47.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.356+0000 7ff50b400640 1 -- 172.21.15.31:0/3233226071 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 8125+0+0 (secure 0 0 0) 0x7ff50c0bb230 con 0x7ff5141518b0 2024-09-11T00:57:47.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.356+0000 7ff512800640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff4f0077f70 0x7ff4f007a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T00:57:47.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.357+0000 7ff512800640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff4f0077f70 0x7ff4f007a430 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff504000f60 tx=0x7ff504009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T00:57:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.360+0000 7ff50b400640 1 -- 172.21.15.31:0/3233226071 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+196006 (secure 0 0 0) 0x7ff50c033020 con 0x7ff5141518b0 2024-09-11T00:57:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.595+0000 7ff519400640 1 -- 172.21.15.31:0/3233226071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff514151590 con 0x7ff5141518b0 2024-09-11T00:57:47.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.596+0000 7ff50b400640 1 -- 172.21.15.31:0/3233226071 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) ==== 95+0+3910 (secure 0 0 0) 0x7ff50c082d60 con 0x7ff5141518b0 2024-09-11T00:57:47.598 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-09-11T00:57:47.598 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":195,"btime":"2024-09-11T00:57:45:548127+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6503,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2337200751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2337200751},{"type":"v1","addr":"172.21.15.31:6835","nonce":2337200751}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":6962,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/1368508756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":1368508756},{"type":"v1","addr":"172.21.15.204:6835","nonce":1368508756}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":194},{"gid":6977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/87280044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":87280044},{"type":"v1","addr":"172.21.15.204:6837","nonce":87280044}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":195},{"gid":8103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6833/22472818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":22472818},{"type":"v1","addr":"172.21.15.31:6833","nonce":22472818}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[]} 2024-09-11T00:57:47.598 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 195 2024-09-11T00:57:47.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.598+0000 7ff519400640 1 -- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff4f0077f70 msgr2=0x7ff4f007a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.598+0000 7ff519400640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff4f0077f70 0x7ff4f007a430 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff504000f60 tx=0x7ff504009040 comp rx=0 tx=0).stop 2024-09-11T00:57:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.598+0000 7ff519400640 1 -- 172.21.15.31:0/3233226071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5141518b0 msgr2=0x7ff5141ded80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T00:57:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.598+0000 7ff519400640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5141518b0 0x7ff5141ded80 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009cd0 tx=0x7ff50c031e10 comp rx=0 tx=0).stop 2024-09-11T00:57:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.598+0000 7ff519400640 1 -- 172.21.15.31:0/3233226071 shutdown_connections 2024-09-11T00:57:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.598+0000 7ff519400640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:6800/2097467593,v1:172.21.15.204:6801/2097467593] conn(0x7ff4f0077f70 0x7ff4f007a430 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.598+0000 7ff519400640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5141518b0 0x7ff5141ded80 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.598+0000 7ff519400640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5140c23a0 0x7ff5140c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:47.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.598+0000 7ff519400640 1 --2- 172.21.15.31:0/3233226071 >> [v2:172.21.15.204:3301/0,v1:172.21.15.204:6790/0] conn(0x7ff514072480 0x7ff5141436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T00:57:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.598+0000 7ff519400640 1 -- 172.21.15.31:0/3233226071 >> 172.21.15.31:0/3233226071 conn(0x7ff5140a9750 msgr2=0x7ff51414cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T00:57:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.599+0000 7ff519400640 1 -- 172.21.15.31:0/3233226071 shutdown_connections 2024-09-11T00:57:47.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-09-11T00:57:47.599+0000 7ff519400640 1 -- 172.21.15.31:0/3233226071 wait complete. 2024-09-11T00:57:47.616 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-11T00:57:47.617 INFO:tasks.mds_thrash.fs.[a]:waiting for 117 secs before thrashing 2024-09-11T00:57:52.845 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T00:57:52.846 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T00:57:52.846 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:57:52.937 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T00:57:52.937 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T00:57:52.937 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:57:53.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-11T00:57:59.039 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T00:57:59.040 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-11T00:57:59.040 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:57:59.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-11T00:57:59.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-11T00:58:01.939 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:58:01.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:58:05.142 INFO:tasks.ceph.osd.3:Stopped 2024-09-11T00:58:05.142 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-11T00:58:05.142 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:58:05.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-11T00:58:05.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-11T00:58:11.245 INFO:tasks.ceph.osd.4:Stopped 2024-09-11T00:58:11.245 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-11T00:58:11.246 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:58:11.341 INFO:tasks.ceph.osd.5:Stopped 2024-09-11T00:58:11.341 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-11T00:58:11.341 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:58:11.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-11T00:58:11.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-11T00:58:11.438 INFO:tasks.ceph.osd.6:Stopped 2024-09-11T00:58:11.438 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-11T00:58:11.438 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:58:16.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-09-11T00:58:16.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-11T00:58:17.540 INFO:tasks.ceph.osd.7:Stopped 2024-09-11T00:58:17.541 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T00:58:17.542 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-11T00:58:17.542 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:58:17.617 INFO:tasks.ceph.mgr.y:Stopped 2024-09-11T00:58:17.617 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T00:58:17.617 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:58:17.696 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T00:58:17.697 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T00:58:17.697 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T00:58:17.697 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:58:17.721 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T00:58:17.721 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T00:58:17.721 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:58:17.741 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T00:58:17.742 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T00:58:17.742 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T00:58:17.802 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T00:58:17.803 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T00:58:17.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v 'Replacing daemon mds' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-11T00:58:17.981 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:58:17.981 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T00:58:18.237 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:58:18.237 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T00:58:18.419 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:58:18.420 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T00:58:18.626 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:58:18.626 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-11T00:58:18.829 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:58:18.830 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-11T00:58:19.194 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:58:19.195 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-11T00:58:19.376 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:58:19.376 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-11T00:58:19.590 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:58:19.591 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-11T00:58:19.753 INFO:tasks.ceph:Archiving mon data... 2024-09-11T00:58:19.755 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843/data/mon.a.tgz 2024-09-11T00:58:19.755 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2024-09-11T00:58:19.822 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.xZsmnHs4Fe 2024-09-11T00:58:19.822 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.xZsmnHs4Fe 2024-09-11T00:58:19.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.xZsmnHs4Fe 2024-09-11T00:58:20.091 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.xZsmnHs4Fe is 467KB 2024-09-11T00:58:20.181 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.xZsmnHs4Fe 2024-09-11T00:58:20.216 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843/data/mon.b.tgz 2024-09-11T00:58:20.216 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-09-11T00:58:20.285 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.yyzptdef1O 2024-09-11T00:58:20.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.yyzptdef1O 2024-09-11T00:58:20.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.yyzptdef1O 2024-09-11T00:58:20.556 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.yyzptdef1O is 491KB 2024-09-11T00:58:20.646 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.yyzptdef1O 2024-09-11T00:58:20.681 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843/data/mon.c.tgz 2024-09-11T00:58:20.681 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-09-11T00:58:20.755 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.i72fUUU1zu 2024-09-11T00:58:20.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.i72fUUU1zu 2024-09-11T00:58:20.945 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.i72fUUU1zu 2024-09-11T00:58:21.039 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.i72fUUU1zu is 491KB 2024-09-11T00:58:21.129 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.i72fUUU1zu 2024-09-11T00:58:21.164 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T00:58:21.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T00:58:21.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T00:58:21.310 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T00:58:21.345 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T00:58:21.345 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T00:58:21.347 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843/remote/smithi031/crash 2024-09-11T00:58:21.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T00:58:21.404 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843/remote/smithi204/crash 2024-09-11T00:58:21.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T00:58:21.458 INFO:tasks.ceph:Shutting down logrotate 2024-09-11T00:58:21.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:58:21.464 DEBUG:teuthology.orchestra.run.smithi204:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-11T00:58:21.560 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-11T00:58:21.616 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-11T00:58:21.671 INFO:tasks.ceph:Compressing logs... 2024-09-11T00:58:21.671 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T00:58:21.675 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T00:58:21.736 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T00:58:21.737 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112874.log 2024-09-11T00:58:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66350.log 2024-09-11T00:58:21.738 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.112874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60086.log 2024-09-11T00:58:21.738 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112874.log.gz 2024-09-11T00:58:21.739 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63275.log 2024-09-11T00:58:21.739 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66350.log.gz 2024-09-11T00:58:21.740 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35669.log 2024-09-11T00:58:21.740 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60086.log.gz 2024-09-11T00:58:21.741 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63275.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63275.log.gz 2024-09-11T00:58:21.741 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59554.log 2024-09-11T00:58:21.742 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44728.log 2024-09-11T00:58:21.742 INFO:teuthology.orchestra.run.smithi031.stderr: 76.2% -- replaced with /var/log/ceph/ceph-client.admin.35669.log.gz 2024-09-11T00:58:21.743 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57114.log 2024-09-11T00:58:21.743 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59554.log.gz 2024-09-11T00:58:21.743 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35754.log 2024-09-11T00:58:21.744 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85411.log 2024-09-11T00:58:21.744 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.57114.log.gz 2024-09-11T00:58:21.745 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85327.log 2024-09-11T00:58:21.745 INFO:teuthology.orchestra.run.smithi031.stderr: 77.9% -- replaced with /var/log/ceph/ceph-client.admin.35754.log.gz 2024-09-11T00:58:21.746 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60002.log 2024-09-11T00:58:21.746 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85411.log.gz 2024-09-11T00:58:21.747 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56880.log 2024-09-11T00:58:21.747 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85327.log.gz 2024-09-11T00:58:21.748 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60002.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60002.log.gz 2024-09-11T00:58:21.748 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55887.log 2024-09-11T00:58:21.748 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51318.log 2024-09-11T00:58:21.749 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56880.log.gz 2024-09-11T00:58:21.749 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55887.log: gzip 87.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.55887.log.gz --verbose 2024-09-11T00:58:21.749 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.59501.log 2024-09-11T00:58:21.750 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110760.log 2024-09-11T00:58:21.751 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59501.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59501.log.gz 2024-09-11T00:58:21.751 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36181.log 2024-09-11T00:58:21.752 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55719.log 2024-09-11T00:58:21.753 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.110760.log.gz 2024-09-11T00:58:21.753 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62409.log 2024-09-11T00:58:21.753 INFO:teuthology.orchestra.run.smithi031.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36181.log.gz 2024-09-11T00:58:21.754 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60889.log 2024-09-11T00:58:21.754 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55719.log.gz 2024-09-11T00:58:21.755 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51840.log 2024-09-11T00:58:21.755 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62409.log.gz 2024-09-11T00:58:21.755 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66802.log 2024-09-11T00:58:21.756 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60889.log.gz 2024-09-11T00:58:21.756 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64522.log 2024-09-11T00:58:21.757 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51840.log.gz 2024-09-11T00:58:21.757 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61542.log 2024-09-11T00:58:21.757 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66802.log.gz 2024-09-11T00:58:21.758 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57072.log 2024-09-11T00:58:21.758 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64522.log.gz 2024-09-11T00:58:21.759 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57564.log 2024-09-11T00:58:21.759 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61542.log.gz 2024-09-11T00:58:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57072.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44728.log.gz 2024-09-11T00:58:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51756.log 2024-09-11T00:58:21.760 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57072.log.gz 2024-09-11T00:58:21.760 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44248.log 2024-09-11T00:58:21.761 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57564.log.gz 2024-09-11T00:58:21.761 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57293.log 2024-09-11T00:58:21.762 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55047.log 2024-09-11T00:58:21.763 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45848.log 2024-09-11T00:58:21.763 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57293.log.gz 2024-09-11T00:58:21.764 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111054.log 2024-09-11T00:58:21.764 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55047.log.gz 2024-09-11T00:58:21.764 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46542.log 2024-09-11T00:58:21.765 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51318.log.gz 2024-09-11T00:58:21.765 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111054.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51756.log.gz 2024-09-11T00:58:21.765 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110718.log 2024-09-11T00:58:21.766 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111054.log.gz 2024-09-11T00:58:21.766 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35754.log 2024-09-11T00:58:21.767 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58460.log 2024-09-11T00:58:21.767 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110718.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44248.log.gz 2024-09-11T00:58:21.767 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52237.log 2024-09-11T00:58:21.767 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.46542.log.gz 2024-09-11T00:58:21.767 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.110718.log.gz 2024-09-11T00:58:21.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43918.log 2024-09-11T00:58:21.770 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58460.log.gz 2024-09-11T00:58:21.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64818.log 2024-09-11T00:58:21.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43253.log 2024-09-11T00:58:21.771 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35674.log 2024-09-11T00:58:21.771 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35794.log 2024-09-11T00:58:21.772 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41119.log 2024-09-11T00:58:21.772 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35794.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-11T00:58:21.772 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.1.41119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35594.log 2024-09-11T00:58:21.772 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-09-11T00:58:21.772 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35594.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-11T00:58:21.774 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T00:58:21.774 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35634.log 2024-09-11T00:58:21.774 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-11T00:58:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111366.log 2024-09-11T00:58:21.776 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64818.log.gz 2024-09-11T00:58:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36525.log 2024-09-11T00:58:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58376.log 2024-09-11T00:58:21.776 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.111366.log.gz 2024-09-11T00:58:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46199.log 2024-09-11T00:58:21.776 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45848.log.gz 2024-09-11T00:58:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58376.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58376.log.gz 2024-09-11T00:58:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101675.log 2024-09-11T00:58:21.776 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51528.log 2024-09-11T00:58:21.777 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.101675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51900.log 2024-09-11T00:58:21.777 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101675.log.gz 2024-09-11T00:58:21.777 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51612.log 2024-09-11T00:58:21.782 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.36525.log.gz 2024-09-11T00:58:21.782 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61017.log 2024-09-11T00:58:21.782 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51612.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.51900.log.gz 2024-09-11T00:58:21.782 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84178.log 2024-09-11T00:58:21.782 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51612.log.gz 2024-09-11T00:58:21.782 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64026.log 2024-09-11T00:58:21.782 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61017.log.gz 2024-09-11T00:58:21.782 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.84178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55677.log 2024-09-11T00:58:21.782 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84178.log.gz 2024-09-11T00:58:21.783 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110886.log 2024-09-11T00:58:21.783 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64026.log.gz 2024-09-11T00:58:21.783 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64264.log 2024-09-11T00:58:21.783 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55677.log.gz 2024-09-11T00:58:21.783 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54879.log 2024-09-11T00:58:21.783 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46199.log.gz -- replaced with /var/log/ceph/ceph-client.admin.110886.log.gz 2024-09-11T00:58:21.783 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T00:58:21.784 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35874.log 2024-09-11T00:58:21.784 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-11T00:58:21.784 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35874.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-11T00:58:21.784 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T00:58:21.784 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35714.log 2024-09-11T00:58:21.785 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-09-11T00:58:21.785 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35714.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T00:58:21.785 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-11T00:58:21.785 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35834.log 2024-09-11T00:58:21.787 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85538.log 2024-09-11T00:58:21.787 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64264.log.gz 2024-09-11T00:58:21.787 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36235.log 2024-09-11T00:58:21.787 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54879.log.gz 2024-09-11T00:58:21.787 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85538.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85538.log.gz 2024-09-11T00:58:21.787 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57335.log 2024-09-11T00:58:21.787 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43822.log 2024-09-11T00:58:21.787 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43030.log 2024-09-11T00:58:21.787 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57335.log.gz 2024-09-11T00:58:21.788 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58646.log 2024-09-11T00:58:21.789 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51528.log.gz 2024-09-11T00:58:21.789 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60438.log 2024-09-11T00:58:21.789 INFO:teuthology.orchestra.run.smithi031.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.36235.log.gz 2024-09-11T00:58:21.789 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61229.log 2024-09-11T00:58:21.789 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58646.log.gz 2024-09-11T00:58:21.790 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113395.log 2024-09-11T00:58:21.790 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60438.log.gz 2024-09-11T00:58:21.791 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61229.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61229.log.gz 2024-09-11T00:58:21.791 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56509.log 2024-09-11T00:58:21.791 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113395.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43918.log.gz 2024-09-11T00:58:21.791 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T00:58:21.792 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.113395.log.gz 2024-09-11T00:58:21.792 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51444.log 2024-09-11T00:58:21.793 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56509.log.gz 2024-09-11T00:58:21.793 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45156.log 2024-09-11T00:58:21.794 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100099.log 2024-09-11T00:58:21.795 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.0.45156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52152.log 2024-09-11T00:58:21.795 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.100099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63106.log 2024-09-11T00:58:21.796 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.100099.log.gz 2024-09-11T00:58:21.796 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58020.log 2024-09-11T00:58:21.797 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51048.log 2024-09-11T00:58:21.797 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63106.log.gz 2024-09-11T00:58:21.798 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50492.log 2024-09-11T00:58:21.798 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58020.log.gz 2024-09-11T00:58:21.798 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43030.log.gz 2024-09-11T00:58:21.799 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59708.log 2024-09-11T00:58:21.799 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43253.log.gz 2024-09-11T00:58:21.800 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58250.log 2024-09-11T00:58:21.800 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43822.log.gz 2024-09-11T00:58:21.800 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51048.log.gz 2024-09-11T00:58:21.800 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56593.log 2024-09-11T00:58:21.800 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59708.log.gz 2024-09-11T00:58:21.801 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.35834.log: 94.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T00:58:21.802 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61356.log 2024-09-11T00:58:21.802 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58250.log.gz 2024-09-11T00:58:21.802 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56593.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56593.log.gz 2024-09-11T00:58:21.802 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56720.log 2024-09-11T00:58:21.803 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63679.log 2024-09-11T00:58:21.803 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61356.log.gz 2024-09-11T00:58:21.804 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108547.log 2024-09-11T00:58:21.804 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56720.log.gz 2024-09-11T00:58:21.805 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62978.log 2024-09-11T00:58:21.805 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63679.log.gz 2024-09-11T00:58:21.805 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59291.log 2024-09-11T00:58:21.805 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108547.log.gz 2024-09-11T00:58:21.806 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112496.log 2024-09-11T00:58:21.806 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62978.log.gz 2024-09-11T00:58:21.807 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85496.log 2024-09-11T00:58:21.807 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.59291.log.gz 2024-09-11T00:58:21.808 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46728.log 2024-09-11T00:58:21.808 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112496.log.gz 2024-09-11T00:58:21.809 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62457.log 2024-09-11T00:58:21.809 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.85496.log.gz 2024-09-11T00:58:21.810 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62668.log 2024-09-11T00:58:21.810 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62457.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46728.log.gz 2024-09-11T00:58:21.811 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54837.log 2024-09-11T00:58:21.811 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62457.log.gz 2024-09-11T00:58:21.812 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62584.log 2024-09-11T00:58:21.812 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62668.log.gz 2024-09-11T00:58:21.813 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46812.log 2024-09-11T00:58:21.813 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54837.log.gz 2024-09-11T00:58:21.813 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50492.log.gz 2024-09-11T00:58:21.814 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46626.log 2024-09-11T00:58:21.814 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62584.log.gz 2024-09-11T00:58:21.814 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67783.log 2024-09-11T00:58:21.815 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46812.log.gz 2024-09-11T00:58:21.815 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111909.log 2024-09-11T00:58:21.816 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67783.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46626.log.gz 2024-09-11T00:58:21.816 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85722.log 2024-09-11T00:58:21.816 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67783.log.gz 2024-09-11T00:58:21.816 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113240.log 2024-09-11T00:58:21.817 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111909.log.gz 2024-09-11T00:58:21.817 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44416.log 2024-09-11T00:58:21.818 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.85722.log.gz 2024-09-11T00:58:21.818 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84708.log 2024-09-11T00:58:21.818 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113240.log.gz 2024-09-11T00:58:21.819 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43921.log 2024-09-11T00:58:21.820 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.84708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46500.log 2024-09-11T00:58:21.820 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.84708.log.gz 2024-09-11T00:58:21.820 INFO:teuthology.orchestra.run.smithi031.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44416.log.gz 2024-09-11T00:58:21.820 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43254.log 2024-09-11T00:58:21.821 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43921.log.gz 2024-09-11T00:58:21.821 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41888.log 2024-09-11T00:58:21.822 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46500.log.gz 2024-09-11T00:58:21.822 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60606.log 2024-09-11T00:58:21.823 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58900.log 2024-09-11T00:58:21.824 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113030.log 2024-09-11T00:58:21.824 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60606.log.gz 2024-09-11T00:58:21.824 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59333.log 2024-09-11T00:58:21.825 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58900.log.gz 2024-09-11T00:58:21.825 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113030.log: gzip -5 87.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43254.log.gz /var/log/ceph/ceph-client.admin.42946.log 2024-09-11T00:58:21.825 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T00:58:21.826 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113030.log.gz 2024-09-11T00:58:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59333.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59333.log.gz 2024-09-11T00:58:21.826 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46770.log 2024-09-11T00:58:21.827 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110844.log 2024-09-11T00:58:21.827 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42946.log.gz 2024-09-11T00:58:21.828 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51672.log 2024-09-11T00:58:21.828 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46770.log.gz 2024-09-11T00:58:21.829 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54711.log 2024-09-11T00:58:21.829 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.110844.log.gz 2024-09-11T00:58:21.829 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59918.log 2024-09-11T00:58:21.830 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59876.log 2024-09-11T00:58:21.831 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54711.log.gz 2024-09-11T00:58:21.831 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51672.log.gz 2024-09-11T00:58:21.832 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46073.log 2024-09-11T00:58:21.832 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59918.log.gz 2024-09-11T00:58:21.832 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59876.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.59876.log.gz 2024-09-11T00:58:21.832 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85262.log 2024-09-11T00:58:21.833 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83170.log 2024-09-11T00:58:21.834 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85262.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46073.log.gz 2024-09-11T00:58:21.834 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111971.log 2024-09-11T00:58:21.834 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85262.log.gz 2024-09-11T00:58:21.835 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.83170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56155.log 2024-09-11T00:58:21.835 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83170.log.gz 2024-09-11T00:58:21.836 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61185.log 2024-09-11T00:58:21.836 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111971.log.gz 2024-09-11T00:58:21.837 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59666.log 2024-09-11T00:58:21.837 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56155.log.gz 2024-09-11T00:58:21.837 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61185.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61185.log.gz 2024-09-11T00:58:21.838 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55418.log 2024-09-11T00:58:21.838 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50084.log 2024-09-11T00:58:21.839 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59666.log.gz 2024-09-11T00:58:21.839 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55244.log 2024-09-11T00:58:21.839 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55418.log.gz 2024-09-11T00:58:21.840 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61440.log 2024-09-11T00:58:21.841 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42482.log 2024-09-11T00:58:21.841 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55244.log.gz 2024-09-11T00:58:21.842 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43589.log 2024-09-11T00:58:21.842 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61440.log.gz 2024-09-11T00:58:21.843 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64775.log 2024-09-11T00:58:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111096.log 2024-09-11T00:58:21.844 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64775.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50084.log.gz 2024-09-11T00:58:21.845 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43589.log.gz 2024-09-11T00:58:21.845 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64775.log.gz 2024-09-11T00:58:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113794.log 2024-09-11T00:58:21.845 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101125.log 2024-09-11T00:58:21.846 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111096.log.gz 2024-09-11T00:58:21.846 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45554.log 2024-09-11T00:58:21.847 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.113794.log.gz 2024-09-11T00:58:21.847 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.101125.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.101125.log.gz 2024-09-11T00:58:21.847 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.111534.log 2024-09-11T00:58:21.848 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87545.log 2024-09-11T00:58:21.849 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58562.log 2024-09-11T00:58:21.849 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111534.log.gz 2024-09-11T00:58:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.87545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56211.log 2024-09-11T00:58:21.850 INFO:teuthology.orchestra.run.smithi031.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.87545.log.gz 2024-09-11T00:58:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65300.log 2024-09-11T00:58:21.851 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58562.log.gz 2024-09-11T00:58:21.851 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46668.log 2024-09-11T00:58:21.852 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56211.log.gz 2024-09-11T00:58:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46115.log 2024-09-11T00:58:21.853 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65300.log.gz 2024-09-11T00:58:21.853 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.45554.log.gz 2024-09-11T00:58:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112664.log 2024-09-11T00:58:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46115.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46668.log.gz 2024-09-11T00:58:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42862.log 2024-09-11T00:58:21.855 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46115.log.gz 2024-09-11T00:58:21.855 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55005.log 2024-09-11T00:58:21.856 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112664.log.gz 2024-09-11T00:58:21.856 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1%/var/log/ceph/ceph-client.admin.42862.log: -- replaced with /var/log/ceph/ceph-client.admin.42482.log.gz 2024-09-11T00:58:21.856 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43587.log 2024-09-11T00:58:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47021.log 2024-09-11T00:58:21.857 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55005.log.gz 2024-09-11T00:58:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81313.log 2024-09-11T00:58:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59126.log 2024-09-11T00:58:21.859 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41614.log 2024-09-11T00:58:21.859 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.81313.log.gz 2024-09-11T00:58:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110802.log 2024-09-11T00:58:21.860 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59126.log.gz 2024-09-11T00:58:21.861 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67263.log 2024-09-11T00:58:21.861 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54187.log 2024-09-11T00:58:21.862 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.110802.log.gz 2024-09-11T00:58:21.862 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42862.log.gz 2024-09-11T00:58:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.67263.log: gzip 87.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.67263.log.gz --verbose 2024-09-11T00:58:21.863 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/ceph-client.admin.42820.log 2024-09-11T00:58:21.863 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5%/var/log/ceph/ceph-client.admin.54187.log: -- replaced with /var/log/ceph/ceph-client.admin.43587.log.gz 2024-09-11T00:58:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63805.log 2024-09-11T00:58:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42820.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47021.log.gz 2024-09-11T00:58:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60396.log 2024-09-11T00:58:21.864 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42820.log.gz 2024-09-11T00:58:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41804.log 2024-09-11T00:58:21.865 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63805.log.gz 2024-09-11T00:58:21.866 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60743.log 2024-09-11T00:58:21.866 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60396.log.gz 2024-09-11T00:58:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62041.log 2024-09-11T00:58:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64480.log 2024-09-11T00:58:21.867 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60743.log.gz 2024-09-11T00:58:21.868 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51444.log.gz 2024-09-11T00:58:21.868 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111686.log 2024-09-11T00:58:21.869 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62041.log.gz 2024-09-11T00:58:21.869 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113114.log 2024-09-11T00:58:21.869 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64480.log.gz 2024-09-11T00:58:21.870 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36233.log 2024-09-11T00:58:21.870 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.111686.log.gz 2024-09-11T00:58:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52829.log 2024-09-11T00:58:21.871 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113114.log.gz 2024-09-11T00:58:21.871 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52506.log 2024-09-11T00:58:21.872 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66173.log 2024-09-11T00:58:21.873 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113567.log 2024-09-11T00:58:21.874 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62210.log 2024-09-11T00:58:21.875 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.66173.log.gz 2024-09-11T00:58:21.875 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.36233.log.gz 2024-09-11T00:58:21.875 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113567.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113567.log.gz 2024-09-11T00:58:21.875 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62938.log 2024-09-11T00:58:21.876 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62210.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62210.log.gz 2024-09-11T00:58:21.876 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61710.log 2024-09-11T00:58:21.877 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61794.log 2024-09-11T00:58:21.877 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62938.log.gz 2024-09-11T00:58:21.878 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57030.log 2024-09-11T00:58:21.878 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61710.log.gz 2024-09-11T00:58:21.878 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43246.log 2024-09-11T00:58:21.879 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61794.log.gz 2024-09-11T00:58:21.879 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57030.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57030.log.gz 2024-09-11T00:58:21.880 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52988.log 2024-09-11T00:58:21.880 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108740.log 2024-09-11T00:58:21.881 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87636.log 2024-09-11T00:58:21.882 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62324.log 2024-09-11T00:58:21.882 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108740.log.gz 2024-09-11T00:58:21.883 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.87636.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.43246.log.gz 2024-09-11T00:58:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50180.log 2024-09-11T00:58:21.884 INFO:teuthology.orchestra.run.smithi031.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.87636.log.gz 2024-09-11T00:58:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62324.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52988.log.gz 2024-09-11T00:58:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42102.log 2024-09-11T00:58:21.884 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62324.log.gz 2024-09-11T00:58:21.885 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35716.log 2024-09-11T00:58:21.885 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50180.log.gz 2024-09-11T00:58:21.885 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64438.log 2024-09-11T00:58:21.886 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42102.log.gz 2024-09-11T00:58:21.886 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35716.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T00:58:21.887 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57240.log 2024-09-11T00:58:21.887 INFO:teuthology.orchestra.run.smithi031.stderr: 75.9% -- replaced with /var/log/ceph/ceph-client.admin.35716.log.gz 2024-09-11T00:58:21.887 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64438.log.gz 2024-09-11T00:58:21.888 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43240.log 2024-09-11T00:58:21.889 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57240.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57240.log.gz 2024-09-11T00:58:21.889 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52359.log 2024-09-11T00:58:21.889 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46157.log 2024-09-11T00:58:21.890 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.43240.log.gz 2024-09-11T00:58:21.890 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45806.log 2024-09-11T00:58:21.891 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46157.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54187.log.gz 2024-09-11T00:58:21.891 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52359.log.gz 2024-09-11T00:58:21.891 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51402.log 2024-09-11T00:58:21.892 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60975.log 2024-09-11T00:58:21.892 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46157.log.gz 2024-09-11T00:58:21.893 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59023.log 2024-09-11T00:58:21.894 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41614.log.gz 2024-09-11T00:58:21.894 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57156.log 2024-09-11T00:58:21.894 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41804.log.gz 2024-09-11T00:58:21.894 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60975.log.gz 2024-09-11T00:58:21.894 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41888.log.gz 2024-09-11T00:58:21.895 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59023.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45806.log.gz 2024-09-11T00:58:21.895 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111848.log 2024-09-11T00:58:21.895 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59023.log.gz 2024-09-11T00:58:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55334.log 2024-09-11T00:58:21.896 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57156.log.gz 2024-09-11T00:58:21.896 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54543.log 2024-09-11T00:58:21.897 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111848.log.gz 2024-09-11T00:58:21.897 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45680.log 2024-09-11T00:58:21.897 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55334.log.gz 2024-09-11T00:58:21.898 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82714.log 2024-09-11T00:58:21.898 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54543.log.gz 2024-09-11T00:58:21.899 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52829.log.gz 2024-09-11T00:58:21.899 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35585.log 2024-09-11T00:58:21.900 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82714.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.45680.log.gz 2024-09-11T00:58:21.900 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62812.log 2024-09-11T00:58:21.900 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82714.log.gz 2024-09-11T00:58:21.901 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35585.log.gz 2024-09-11T00:58:21.901 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56361.log 2024-09-11T00:58:21.902 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53733.log 2024-09-11T00:58:21.902 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62812.log.gz 2024-09-11T00:58:21.902 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113072.log 2024-09-11T00:58:21.902 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56361.log.gz 2024-09-11T00:58:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112118.log 2024-09-11T00:58:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56635.log 2024-09-11T00:58:21.904 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113072.log.gz 2024-09-11T00:58:21.905 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83976.log 2024-09-11T00:58:21.905 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112118.log.gz 2024-09-11T00:58:21.906 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44332.log 2024-09-11T00:58:21.906 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56635.log.gz 2024-09-11T00:58:21.906 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.83976.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.83976.log.gz 2024-09-11T00:58:21.906 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103102.log 2024-09-11T00:58:21.907 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62252.log 2024-09-11T00:58:21.908 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44332.log.gz 2024-09-11T00:58:21.908 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47475.log 2024-09-11T00:58:21.908 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.103102.log.gz 2024-09-11T00:58:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112790.log 2024-09-11T00:58:21.909 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62252.log.gz 2024-09-11T00:58:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57460.log 2024-09-11T00:58:21.910 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47475.log.gz 2024-09-11T00:58:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43581.log 2024-09-11T00:58:21.910 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112790.log.gz 2024-09-11T00:58:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45764.log 2024-09-11T00:58:21.911 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57460.log.gz 2024-09-11T00:58:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54375.log 2024-09-11T00:58:21.912 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.43581.log.gz 2024-09-11T00:58:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61482.log 2024-09-11T00:58:21.913 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45764.log.gz 2024-09-11T00:58:21.914 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111806.log 2024-09-11T00:58:21.914 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54375.log.gz 2024-09-11T00:58:21.914 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112286.log 2024-09-11T00:58:21.915 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61482.log.gz 2024-09-11T00:58:21.915 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52734.log 2024-09-11T00:58:21.915 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111806.log.gz 2024-09-11T00:58:21.916 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52026.log 2024-09-11T00:58:21.916 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112286.log.gz 2024-09-11T00:58:21.917 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62626.log 2024-09-11T00:58:21.918 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60146.log 2024-09-11T00:58:21.918 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51402.log.gz 2024-09-11T00:58:21.918 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52734.log.gz 2024-09-11T00:58:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44626.log 2024-09-11T00:58:21.919 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62626.log.gz 2024-09-11T00:58:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52917.log 2024-09-11T00:58:21.920 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60146.log.gz 2024-09-11T00:58:21.920 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59792.log 2024-09-11T00:58:21.921 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52917.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.62728.log -- replaced with /var/log/ceph/ceph-client.admin.53733.log.gz 2024-09-11T00:58:21.921 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T00:58:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59792.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44626.log.gz 2024-09-11T00:58:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113751.log 2024-09-11T00:58:21.922 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59792.log.gz 2024-09-11T00:58:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64072.log 2024-09-11T00:58:21.923 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62728.log.gz 2024-09-11T00:58:21.924 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52195.log 2024-09-11T00:58:21.924 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113751.log.gz 2024-09-11T00:58:21.925 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57381.log 2024-09-11T00:58:21.925 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64072.log.gz 2024-09-11T00:58:21.925 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53189.log 2024-09-11T00:58:21.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-09-11T00:58:21.927 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46896.log 2024-09-11T00:58:21.927 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57381.log.gz 2024-09-11T00:58:21.927 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52506.log.gz 2024-09-11T00:58:21.927 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52917.log.gz 2024-09-11T00:58:21.927 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56425.log 2024-09-11T00:58:21.928 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35805.log 2024-09-11T00:58:21.928 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50922.log 2024-09-11T00:58:21.929 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56425.log.gz 2024-09-11T00:58:21.929 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51570.log 2024-09-11T00:58:21.930 INFO:teuthology.orchestra.run.smithi031.stderr: 81.1% -- replaced with /var/log/ceph/ceph-client.admin.35805.log.gz 2024-09-11T00:58:21.930 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53525.log 2024-09-11T00:58:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51570.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50922.log.gz 2024-09-11T00:58:21.931 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86893.log 2024-09-11T00:58:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53525.log: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T00:58:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36239.log 2024-09-11T00:58:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111222.log 2024-09-11T00:58:21.933 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86893.log.gz 2024-09-11T00:58:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43924.log 2024-09-11T00:58:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99871.log 2024-09-11T00:58:21.935 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111222.log.gz 2024-09-11T00:58:21.935 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42438.log 2024-09-11T00:58:21.936 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52152.log.gz 2024-09-11T00:58:21.936 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52195.log.gz 2024-09-11T00:58:21.936 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52237.log.gz 2024-09-11T00:58:21.937 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46241.log 2024-09-11T00:58:21.937 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.99871.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51570.log.gz 2024-09-11T00:58:21.938 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.99871.log.gz 2024-09-11T00:58:21.938 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42438.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53525.log.gz 2024-09-11T00:58:21.938 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose 86.5% -- -- replaced with /var/log/ceph/ceph-client.admin.36239.log.gz /var/log/ceph/ceph-client.admin.48356.log 2024-09-11T00:58:21.938 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T00:58:21.938 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43924.log.gz 2024-09-11T00:58:21.938 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42438.log.gz 2024-09-11T00:58:21.939 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63984.log 2024-09-11T00:58:21.939 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53189.log.gz 2024-09-11T00:58:21.939 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54417.log 2024-09-11T00:58:21.940 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62083.log 2024-09-11T00:58:21.940 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46241.log.gz 2024-09-11T00:58:21.940 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63984.log.gz 2024-09-11T00:58:21.941 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54417.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46896.log.gz 2024-09-11T00:58:21.941 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48356.log.gz 2024-09-11T00:58:21.941 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55521.log 2024-09-11T00:58:21.941 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54417.log.gz 2024-09-11T00:58:21.942 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62083.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62083.log.gz 2024-09-11T00:58:21.942 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113962.log 2024-09-11T00:58:21.942 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56467.log 2024-09-11T00:58:21.942 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55521.log.gz 2024-09-11T00:58:21.943 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113962.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113962.log.gz 2024-09-11T00:58:21.943 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54795.log 2024-09-11T00:58:21.944 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57978.log 2024-09-11T00:58:21.944 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56467.log.gz 2024-09-11T00:58:21.945 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56823.log 2024-09-11T00:58:21.945 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54795.log.gz 2024-09-11T00:58:21.946 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43567.log 2024-09-11T00:58:21.946 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57978.log.gz 2024-09-11T00:58:21.946 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42018.log 2024-09-11T00:58:21.947 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56823.log.gz 2024-09-11T00:58:21.947 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59834.log 2024-09-11T00:58:21.948 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59750.log 2024-09-11T00:58:21.949 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64902.log 2024-09-11T00:58:21.949 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59834.log.gz 2024-09-11T00:58:21.950 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60520.log 2024-09-11T00:58:21.950 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59750.log.gz 2024-09-11T00:58:21.951 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61269.log 2024-09-11T00:58:21.951 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64902.log.gz 2024-09-11T00:58:21.951 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43567.log.gz 2024-09-11T00:58:21.952 INFO:teuthology.orchestra.run.smithi204.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35594.log.gz 2024-09-11T00:58:21.952 INFO:teuthology.orchestra.run.smithi204.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35634.log.gz 2024-09-11T00:58:21.953 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47257.log 2024-09-11T00:58:21.953 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42018.log.gz 2024-09-11T00:58:21.953 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60520.log.gz 2024-09-11T00:58:21.953 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61269.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61269.log.gz 2024-09-11T00:58:21.953 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56763.log 2024-09-11T00:58:21.954 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58166.log 2024-09-11T00:58:21.954 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35674.log.gz 2024-09-11T00:58:21.955 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102299.log 2024-09-11T00:58:21.955 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56763.log.gz 2024-09-11T00:58:21.955 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51798.log 2024-09-11T00:58:21.955 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58166.log.gz 2024-09-11T00:58:21.957 INFO:teuthology.orchestra.run.smithi204.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35714.log.gz 2024-09-11T00:58:21.957 INFO:teuthology.orchestra.run.smithi204.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35754.log.gz 2024-09-11T00:58:21.957 INFO:teuthology.orchestra.run.smithi204.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35794.log.gz 2024-09-11T00:58:21.957 INFO:teuthology.orchestra.run.smithi204.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35834.log.gz 2024-09-11T00:58:21.958 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.102299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58418.log 2024-09-11T00:58:21.958 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102299.log.gz 2024-09-11T00:58:21.958 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55947.log 2024-09-11T00:58:21.958 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64222.log 2024-09-11T00:58:21.958 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58418.log.gz 2024-09-11T00:58:21.959 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35874.log.gz 2024-09-11T00:58:21.960 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47257.log.gz 2024-09-11T00:58:21.960 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48987.log 2024-09-11T00:58:21.960 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55947.log.gz 2024-09-11T00:58:21.960 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41976.log 2024-09-11T00:58:21.960 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64222.log.gz 2024-09-11T00:58:21.961 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48987.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51798.log.gz 2024-09-11T00:58:21.961 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48987.log.gz 2024-09-11T00:58:21.961 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43922.log 2024-09-11T00:58:21.962 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63847.log 2024-09-11T00:58:21.962 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41976.log.gz 2024-09-11T00:58:21.963 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109869.log 2024-09-11T00:58:21.963 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54139.log 2024-09-11T00:58:21.963 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63847.log.gz 2024-09-11T00:58:21.964 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109869.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-09-11T00:58:21.965 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.109869.log.gz 2024-09-11T00:58:21.965 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46031.log 2024-09-11T00:58:21.966 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52026.log.gz 2024-09-11T00:58:21.966 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43922.log.gz 2024-09-11T00:58:21.966 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mds.a.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54139.log.gz 2024-09-11T00:58:21.966 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62168.log 2024-09-11T00:58:21.967 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61668.log 2024-09-11T00:58:21.968 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42988.log 2024-09-11T00:58:21.968 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46031.log.gz 2024-09-11T00:58:21.968 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62168.log.gz 2024-09-11T00:58:21.969 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61668.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61668.log.gz 2024-09-11T00:58:21.969 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56988.log 2024-09-11T00:58:21.970 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55376.log 2024-09-11T00:58:21.970 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64690.log 2024-09-11T00:58:21.970 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56988.log.gz 2024-09-11T00:58:21.971 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51006.log 2024-09-11T00:58:21.972 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55376.log.gz 2024-09-11T00:58:21.972 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35952.log 2024-09-11T00:58:21.972 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64690.log.gz 2024-09-11T00:58:21.973 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42988.log.gz 2024-09-11T00:58:21.973 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52464.log 2024-09-11T00:58:21.974 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61584.log 2024-09-11T00:58:21.975 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53921.log 2024-09-11T00:58:21.976 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52871.log 2024-09-11T00:58:21.976 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61584.log.gz 2024-09-11T00:58:21.976 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51006.log.gz 2024-09-11T00:58:21.977 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112370.log 2024-09-11T00:58:21.977 INFO:teuthology.orchestra.run.smithi031.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.35952.log.gz 2024-09-11T00:58:21.977 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113691.log 2024-09-11T00:58:21.978 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48594.log 2024-09-11T00:58:21.978 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112370.log.gz 2024-09-11T00:58:21.979 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82330.log 2024-09-11T00:58:21.979 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.113691.log.gz 2024-09-11T00:58:21.980 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46355.log 2024-09-11T00:58:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52548.log 2024-09-11T00:58:21.981 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82330.log.gz 2024-09-11T00:58:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61626.log 2024-09-11T00:58:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52548.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53921.log.gz 2024-09-11T00:58:21.982 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113437.log 2024-09-11T00:58:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61626.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61626.log.gz 2024-09-11T00:58:21.983 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52871.log.gz 2024-09-11T00:58:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55161.log 2024-09-11T00:58:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60312.log 2024-09-11T00:58:21.984 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.113437.log.gz 2024-09-11T00:58:21.985 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55161.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55161.log.gz 2024-09-11T00:58:21.985 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56115.log 2024-09-11T00:58:21.985 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48594.log.gz 2024-09-11T00:58:21.986 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46355.log.gz 2024-09-11T00:58:21.986 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52548.log.gz 2024-09-11T00:58:21.986 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42270.log 2024-09-11T00:58:21.986 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60312.log.gz 2024-09-11T00:58:21.987 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51984.log 2024-09-11T00:58:21.987 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56115.log.gz 2024-09-11T00:58:21.988 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54627.log 2024-09-11T00:58:21.989 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112244.log 2024-09-11T00:58:21.989 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64564.log 2024-09-11T00:58:21.990 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54627.log.gz 2024-09-11T00:58:21.990 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53315.log 2024-09-11T00:58:21.990 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112244.log.gz 2024-09-11T00:58:21.991 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64564.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42270.log.gz 2024-09-11T00:58:21.991 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113649.log 2024-09-11T00:58:21.992 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64564.log.gz 2024-09-11T00:58:21.992 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53315.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51984.log.gz 2024-09-11T00:58:21.992 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64353.log 2024-09-11T00:58:21.993 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56551.log 2024-09-11T00:58:21.993 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113649.log.gz 2024-09-11T00:58:21.994 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61398.log 2024-09-11T00:58:21.994 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64353.log.gz 2024-09-11T00:58:21.995 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59207.log 2024-09-11T00:58:21.995 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.56551.log.gz 2024-09-11T00:58:21.996 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65790.log 2024-09-11T00:58:21.996 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61398.log: /var/log/ceph/ceph-client.admin.59207.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61398.log.gz 2024-09-11T00:58:21.996 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108601.log 2024-09-11T00:58:21.997 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59207.log.gz 2024-09-11T00:58:21.997 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59459.log 2024-09-11T00:58:21.997 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65790.log.gz 2024-09-11T00:58:21.998 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36153.log 2024-09-11T00:58:21.998 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108601.log.gz 2024-09-11T00:58:21.999 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59459.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59459.log.gz 2024-09-11T00:58:21.999 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43072.log 2024-09-11T00:58:22.000 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113519.log 2024-09-11T00:58:22.000 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61314.log 2024-09-11T00:58:22.001 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36153.log.gz 2024-09-11T00:58:22.001 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44500.log 2024-09-11T00:58:22.001 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113519.log.gz 2024-09-11T00:58:22.002 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44164.log 2024-09-11T00:58:22.002 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61314.log.gz 2024-09-11T00:58:22.003 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36275.log 2024-09-11T00:58:22.004 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59249.log 2024-09-11T00:58:22.005 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45722.log 2024-09-11T00:58:22.005 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59249.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43072.log.gz -5 2024-09-11T00:58:22.006 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62854.log 2024-09-11T00:58:22.006 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59249.log.gz 2024-09-11T00:58:22.006 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54227.log 2024-09-11T00:58:22.007 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66626.log 2024-09-11T00:58:22.008 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62854.log.gz 2024-09-11T00:58:22.008 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45932.log 2024-09-11T00:58:22.009 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44500.log.gz 2024-09-11T00:58:22.009 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114004.log 2024-09-11T00:58:22.010 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.66626.log.gz 2024-09-11T00:58:22.010 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.44164.log.gz 2024-09-11T00:58:22.010 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45932.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.36275.log.gz 2024-09-11T00:58:22.011 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45722.log.gz 2024-09-11T00:58:22.011 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54227.log.gz 2024-09-11T00:58:22.011 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45596.log 2024-09-11T00:58:22.011 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45932.log.gz 2024-09-11T00:58:22.012 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108189.log 2024-09-11T00:58:22.012 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.114004.log.gz 2024-09-11T00:58:22.012 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114214.log 2024-09-11T00:58:22.012 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45596.log.gz 2024-09-11T00:58:22.013 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54753.log 2024-09-11T00:58:22.013 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.108189.log.gz 2024-09-11T00:58:22.013 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114214.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52464.log.gz 2024-09-11T00:58:22.014 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.114214.log.gz 2024-09-11T00:58:22.014 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46584.log 2024-09-11T00:58:22.014 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112412.log 2024-09-11T00:58:22.014 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54753.log.gz 2024-09-11T00:58:22.015 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57606.log 2024-09-11T00:58:22.015 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46584.log.gz 2024-09-11T00:58:22.016 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52608.log 2024-09-11T00:58:22.016 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112412.log.gz 2024-09-11T00:58:22.017 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107684.log 2024-09-11T00:58:22.017 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57606.log.gz 2024-09-11T00:58:22.018 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63763.log 2024-09-11T00:58:22.019 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62542.log 2024-09-11T00:58:22.019 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53315.log.gz 2024-09-11T00:58:22.019 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52608.log.gz 2024-09-11T00:58:22.019 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107684.log.gz 2024-09-11T00:58:22.020 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87029.log 2024-09-11T00:58:22.020 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63763.log.gz 2024-09-11T00:58:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58332.log 2024-09-11T00:58:22.021 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62542.log.gz 2024-09-11T00:58:22.021 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.87029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44290.log 2024-09-11T00:58:22.022 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87029.log.gz 2024-09-11T00:58:22.022 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51090.log 2024-09-11T00:58:22.023 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58332.log.gz 2024-09-11T00:58:22.023 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44290.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T00:58:22.023 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46313.log 2024-09-11T00:58:22.024 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35874.log 2024-09-11T00:58:22.025 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102137.log 2024-09-11T00:58:22.025 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44290.log.gz 2024-09-11T00:58:22.026 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60044.log 2024-09-11T00:58:22.027 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51090.log.gz 2024-09-11T00:58:22.027 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.102137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60849.log 2024-09-11T00:58:22.028 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.102137.log.gz 2024-09-11T00:58:22.028 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60044.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60044.log.gz 2024-09-11T00:58:22.028 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85581.log 2024-09-11T00:58:22.028 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45470.log 2024-09-11T00:58:22.029 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60849.log.gz 2024-09-11T00:58:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35837.log 2024-09-11T00:58:22.030 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.85581.log.gz 2024-09-11T00:58:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60564.log 2024-09-11T00:58:22.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43252.log 2024-09-11T00:58:22.031 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52068.log 2024-09-11T00:58:22.032 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60564.log.gz 2024-09-11T00:58:22.032 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4%/var/log/ceph/ceph-client.admin.43252.log: -- replaced with /var/log/ceph/ceph-client.admin.46313.log.gz 2024-09-11T00:58:22.032 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54501.log 2024-09-11T00:58:22.033 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64156.log 2024-09-11T00:58:22.034 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55107.log 2024-09-11T00:58:22.034 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54501.log.gz 2024-09-11T00:58:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58772.log 2024-09-11T00:58:22.035 INFO:teuthology.orchestra.run.smithi031.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35874.log.gz 2024-09-11T00:58:22.035 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64156.log.gz 2024-09-11T00:58:22.036 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59065.log 2024-09-11T00:58:22.036 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45470.log.gz 2024-09-11T00:58:22.037 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55107.log.gz 2024-09-11T00:58:22.037 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113156.log 2024-09-11T00:58:22.037 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58772.log.gz 2024-09-11T00:58:22.038 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55989.log 2024-09-11T00:58:22.038 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59065.log.gz 2024-09-11T00:58:22.038 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86408.log 2024-09-11T00:58:22.038 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113156.log.gz 2024-09-11T00:58:22.039 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85875.log 2024-09-11T00:58:22.039 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55989.log.gz 2024-09-11T00:58:22.040 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41711.log 2024-09-11T00:58:22.040 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86408.log.gz 2024-09-11T00:58:22.041 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85875.log: gzip -5 86.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.35837.log.gz -- 2024-09-11T00:58:22.041 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.57847.log 2024-09-11T00:58:22.041 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.43252.log.gz 2024-09-11T00:58:22.041 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.85875.log.gz 2024-09-11T00:58:22.042 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43250.log 2024-09-11T00:58:22.042 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.52068.log.gz 2024-09-11T00:58:22.043 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60701.log 2024-09-11T00:58:22.043 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57847.log.gz 2024-09-11T00:58:22.043 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53481.log 2024-09-11T00:58:22.044 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51132.log 2024-09-11T00:58:22.045 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60701.log.gz 2024-09-11T00:58:22.045 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41711.log.gz 2024-09-11T00:58:22.045 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53481.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.43250.log.gz 2024-09-11T00:58:22.045 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42568.log 2024-09-11T00:58:22.046 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62366.log 2024-09-11T00:58:22.047 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53398.log 2024-09-11T00:58:22.048 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53819.log 2024-09-11T00:58:22.048 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62366.log.gz 2024-09-11T00:58:22.048 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112706.log 2024-09-11T00:58:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48537.log 2024-09-11T00:58:22.050 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113342.log 2024-09-11T00:58:22.050 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112706.log.gz 2024-09-11T00:58:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51714.log 2024-09-11T00:58:22.051 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53481.log.gz 2024-09-11T00:58:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113878.log 2024-09-11T00:58:22.052 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51132.log.gz 2024-09-11T00:58:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113342.log: /var/log/ceph/ceph-client.admin.51714.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113342.log.gz 2024-09-11T00:58:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87108.log 2024-09-11T00:58:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43243.log 2024-09-11T00:58:22.053 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.113878.log.gz 2024-09-11T00:58:22.054 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.87108.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T00:58:22.054 INFO:teuthology.orchestra.run.smithi031.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.87108.log.gz 2024-09-11T00:58:22.055 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42568.log.gz 2024-09-11T00:58:22.055 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63190.log 2024-09-11T00:58:22.055 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53398.log.gz 2024-09-11T00:58:22.056 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42228.log 2024-09-11T00:58:22.057 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62770.log 2024-09-11T00:58:22.057 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63190.log.gz 2024-09-11T00:58:22.058 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62499.log 2024-09-11T00:58:22.058 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107033.log 2024-09-11T00:58:22.059 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62770.log.gz 2024-09-11T00:58:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61059.log 2024-09-11T00:58:22.059 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62499.log.gz 2024-09-11T00:58:22.061 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107033.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53819.log.gz 2024-09-11T00:58:22.061 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56071.log 2024-09-11T00:58:22.061 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.107033.log.gz 2024-09-11T00:58:22.061 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61059.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51714.log.gz 2024-09-11T00:58:22.061 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112580.log 2024-09-11T00:58:22.061 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61059.log.gz 2024-09-11T00:58:22.062 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60269.log 2024-09-11T00:58:22.062 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.56071.log.gz 2024-09-11T00:58:22.063 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59596.log 2024-09-11T00:58:22.064 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112580.log: /var/log/ceph/ceph-client.admin.60269.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112580.log.gz 2024-09-11T00:58:22.064 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60269.log.gz 2024-09-11T00:58:22.064 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84686.log 2024-09-11T00:58:22.064 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42526.log 2024-09-11T00:58:22.065 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59596.log.gz 2024-09-11T00:58:22.065 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.84686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58732.log 2024-09-11T00:58:22.066 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84686.log.gz 2024-09-11T00:58:22.066 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61752.log 2024-09-11T00:58:22.067 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111306.log 2024-09-11T00:58:22.067 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58732.log.gz 2024-09-11T00:58:22.068 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61752.log: gzip -5 87.3% --verbose -- replaced with /var/log/ceph/ceph-client.admin.61752.log.gz -- 2024-09-11T00:58:22.068 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.44122.log 2024-09-11T00:58:22.068 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.43243.log.gz 2024-09-11T00:58:22.069 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111306.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42228.log.gz 2024-09-11T00:58:22.069 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111306.log.gz 2024-09-11T00:58:22.069 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42526.log.gz 2024-09-11T00:58:22.069 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57198.log 2024-09-11T00:58:22.070 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53147.log 2024-09-11T00:58:22.071 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52110.log 2024-09-11T00:58:22.071 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57198.log.gz 2024-09-11T00:58:22.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42652.log 2024-09-11T00:58:22.072 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60783.log 2024-09-11T00:58:22.073 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61957.log 2024-09-11T00:58:22.074 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48230.log 2024-09-11T00:58:22.076 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60783.log.gz 2024-09-11T00:58:22.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35999.log 2024-09-11T00:58:22.077 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61957.log.gz 2024-09-11T00:58:22.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111492.log 2024-09-11T00:58:22.077 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108936.log 2024-09-11T00:58:22.077 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44122.log.gz 2024-09-11T00:58:22.078 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108698.log 2024-09-11T00:58:22.078 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111492.log.gz 2024-09-11T00:58:22.078 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64114.log 2024-09-11T00:58:22.079 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108936.log.gz 2024-09-11T00:58:22.079 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108698.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.108698.log.gz 2024-09-11T00:58:22.079 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-11T00:58:22.080 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89554.log 2024-09-11T00:58:22.080 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64114.log.gz 2024-09-11T00:58:22.081 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29631.log 2024-09-11T00:58:22.082 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53147.log.gz 2024-09-11T00:58:22.082 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.89554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85146.log 2024-09-11T00:58:22.082 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.89554.log.gz 2024-09-11T00:58:22.083 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.29631.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52110.log.gz 2024-09-11T00:58:22.083 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29631.log.gz 2024-09-11T00:58:22.083 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44206.log 2024-09-11T00:58:22.084 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56029.log 2024-09-11T00:58:22.084 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85146.log.gz 2024-09-11T00:58:22.085 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55803.log 2024-09-11T00:58:22.086 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57522.log 2024-09-11T00:58:22.086 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42652.log.gz 2024-09-11T00:58:22.086 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56029.log.gz 2024-09-11T00:58:22.086 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48230.log.gz 2024-09-11T00:58:22.086 INFO:teuthology.orchestra.run.smithi031.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.35999.log.gz 2024-09-11T00:58:22.087 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43577.log 2024-09-11T00:58:22.087 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55803.log.gz 2024-09-11T00:58:22.087 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57522.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57522.log.gz 2024-09-11T00:58:22.088 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59960.log 2024-09-11T00:58:22.089 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42312.log 2024-09-11T00:58:22.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59960.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59960.log.gz 2024-09-11T00:58:22.090 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43590.log 2024-09-11T00:58:22.090 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108643.log 2024-09-11T00:58:22.091 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43590.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44206.log.gz 2024-09-11T00:58:22.091 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84143.log 2024-09-11T00:58:22.092 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108643.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108643.log.gz 2024-09-11T00:58:22.093 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T00:58:22.093 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.84143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57889.log 2024-09-11T00:58:22.094 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.84143.log.gz 2024-09-11T00:58:22.094 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43577.log.gz 2024-09-11T00:58:22.094 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42694.log 2024-09-11T00:58:22.095 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42312.log.gz 2024-09-11T00:58:22.095 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43590.log.gz 2024-09-11T00:58:22.096 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85263.log 2024-09-11T00:58:22.096 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57889.log.gz 2024-09-11T00:58:22.096 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48537.log.gz 2024-09-11T00:58:22.097 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64396.log 2024-09-11T00:58:22.098 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85263.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42694.log.gz 2024-09-11T00:58:22.098 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.85263.log.gz 2024-09-11T00:58:22.098 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46408.log 2024-09-11T00:58:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64396.log: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T00:58:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:gzip 87.4% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.64396.log.gz -- 2024-09-11T00:58:22.099 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.45428.log 2024-09-11T00:58:22.099 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51216.log 2024-09-11T00:58:22.100 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85453.log 2024-09-11T00:58:22.101 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46408.log.gz 2024-09-11T00:58:22.101 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54921.log 2024-09-11T00:58:22.102 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63567.log 2024-09-11T00:58:22.102 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85453.log.gz 2024-09-11T00:58:22.103 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64606.log 2024-09-11T00:58:22.103 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54921.log.gz 2024-09-11T00:58:22.104 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113477.log 2024-09-11T00:58:22.104 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63567.log.gz 2024-09-11T00:58:22.105 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55460.log 2024-09-11T00:58:22.105 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64606.log.gz 2024-09-11T00:58:22.106 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45428.log.gz 2024-09-11T00:58:22.106 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113581.log 2024-09-11T00:58:22.106 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51216.log.gz 2024-09-11T00:58:22.106 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113477.log.gz 2024-09-11T00:58:22.106 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55460.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55460.log.gz 2024-09-11T00:58:22.107 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56341.log 2024-09-11T00:58:22.107 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36071.log 2024-09-11T00:58:22.108 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113581.log.gz 2024-09-11T00:58:22.108 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46939.log 2024-09-11T00:58:22.108 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56341.log.gz 2024-09-11T00:58:22.109 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111012.log 2024-09-11T00:58:22.110 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42904.log 2024-09-11T00:58:22.111 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114046.log 2024-09-11T00:58:22.111 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111012.log.gz 2024-09-11T00:58:22.112 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36129.log 2024-09-11T00:58:22.113 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53273.log 2024-09-11T00:58:22.113 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.114046.log.gz 2024-09-11T00:58:22.114 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35605.log 2024-09-11T00:58:22.114 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36071.log.gz 2024-09-11T00:58:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60354.log 2024-09-11T00:58:22.115 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42904.log.gz 2024-09-11T00:58:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35605.log.gz 2024-09-11T00:58:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58856.log 2024-09-11T00:58:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54963.log 2024-09-11T00:58:22.117 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60354.log.gz 2024-09-11T00:58:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42144.log 2024-09-11T00:58:22.117 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58856.log.gz 2024-09-11T00:58:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43916.log 2024-09-11T00:58:22.119 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54963.log.gz 2024-09-11T00:58:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42144.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36129.log.gz 2024-09-11T00:58:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60480.log 2024-09-11T00:58:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43916.log: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-11T00:58:22.120 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53273.log.gz 2024-09-11T00:58:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51258.log 2024-09-11T00:58:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42778.log 2024-09-11T00:58:22.121 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60480.log.gz 2024-09-11T00:58:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112538.log 2024-09-11T00:58:22.123 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52650.log 2024-09-11T00:58:22.123 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42144.log.gz 2024-09-11T00:58:22.124 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83956.log 2024-09-11T00:58:22.124 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112538.log.gz 2024-09-11T00:58:22.124 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.43916.log.gz 2024-09-11T00:58:22.125 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58690.log 2024-09-11T00:58:22.125 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.83956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47063.log 2024-09-11T00:58:22.126 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.83956.log.gz 2024-09-11T00:58:22.126 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59375.log 2024-09-11T00:58:22.126 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58690.log.gz 2024-09-11T00:58:22.127 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42186.log 2024-09-11T00:58:22.128 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51258.log.gz 2024-09-11T00:58:22.128 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53357.log 2024-09-11T00:58:22.129 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59375.log.gz 2024-09-11T00:58:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42186.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42778.log.gz 2024-09-11T00:58:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110970.log 2024-09-11T00:58:22.130 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58520.log 2024-09-11T00:58:22.130 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48495.log 2024-09-11T00:58:22.131 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110970.log.gz 2024-09-11T00:58:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64648.log 2024-09-11T00:58:22.132 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58520.log: /var/log/ceph/ceph-client.admin.48495.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58520.log.gz 2024-09-11T00:58:22.132 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41846.log 2024-09-11T00:58:22.132 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52650.log.gz 2024-09-11T00:58:22.133 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36060.log 2024-09-11T00:58:22.134 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64648.log.gz 2024-09-11T00:58:22.134 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42186.log.gz 2024-09-11T00:58:22.134 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53357.log.gz 2024-09-11T00:58:22.134 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48495.log.gz 2024-09-11T00:58:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112328.log 2024-09-11T00:58:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48440.log 2024-09-11T00:58:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35902.log 2024-09-11T00:58:22.136 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41846.log.gz 2024-09-11T00:58:22.136 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112328.log.gz 2024-09-11T00:58:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48440.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.36060.log.gz 2024-09-11T00:58:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45512.log 2024-09-11T00:58:22.137 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48440.log.gz 2024-09-11T00:58:22.137 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.35902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45638.log 2024-09-11T00:58:22.138 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53649.log 2024-09-11T00:58:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56678.log 2024-09-11T00:58:22.140 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54273.log 2024-09-11T00:58:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56678.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.35902.log.gz 2024-09-11T00:58:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114129.log 87.5% 2024-09-11T00:58:22.141 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56678.log.gz 2024-09-11T00:58:22.141 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45512.log.gz 2024-09-11T00:58:22.141 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45638.log.gz 2024-09-11T00:58:22.142 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.54273.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45310.log 2024-09-11T00:58:22.143 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55204.log 2024-09-11T00:58:22.143 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.114129.log.gz 2024-09-11T00:58:22.143 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63889.log 2024-09-11T00:58:22.144 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63064.log 2024-09-11T00:58:22.145 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55204.log.gz 2024-09-11T00:58:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111138.log 2024-09-11T00:58:22.146 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54273.log.gz 2024-09-11T00:58:22.146 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63889.log.gz 2024-09-11T00:58:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63064.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63064.log.gz 2024-09-11T00:58:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44374.log 2024-09-11T00:58:22.147 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45310.log.gz 2024-09-11T00:58:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51174.log 2024-09-11T00:58:22.147 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111138.log.gz 2024-09-11T00:58:22.148 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53693.log 2024-09-11T00:58:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60648.log 2024-09-11T00:58:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44542.log 2024-09-11T00:58:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50829.log 2024-09-11T00:58:22.151 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60648.log.gz 2024-09-11T00:58:22.151 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44374.log.gz 2024-09-11T00:58:22.151 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36819.log 2024-09-11T00:58:22.151 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51174.log.gz 2024-09-11T00:58:22.152 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61999.log 2024-09-11T00:58:22.153 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36728.log 2024-09-11T00:58:22.154 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65096.log 2024-09-11T00:58:22.154 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61999.log.gz 2024-09-11T00:58:22.154 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44542.log.gz 2024-09-11T00:58:22.155 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57690.log 2024-09-11T00:58:22.155 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50829.log.gz 2024-09-11T00:58:22.156 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57793.log 2024-09-11T00:58:22.156 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.65096.log.gz 2024-09-11T00:58:22.157 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57690.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57690.log.gz 2024-09-11T00:58:22.157 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43925.log 2024-09-11T00:58:22.157 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63317.log 2024-09-11T00:58:22.158 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57793.log.gz 2024-09-11T00:58:22.158 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102713.log 2024-09-11T00:58:22.159 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57648.log 2024-09-11T00:58:22.159 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63317.log.gz 2024-09-11T00:58:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.102713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112454.log 2024-09-11T00:58:22.160 INFO:teuthology.orchestra.run.smithi031.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.102713.log.gz 2024-09-11T00:58:22.160 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43925.log.gz 2024-09-11T00:58:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42610.log 2024-09-11T00:58:22.161 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57648.log.gz 2024-09-11T00:58:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112454.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112454.log.gz 2024-09-11T00:58:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54585.log 2024-09-11T00:58:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111610.log 2024-09-11T00:58:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63488.log 2024-09-11T00:58:22.163 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54585.log.gz 2024-09-11T00:58:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111610.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42610.log.gz 2024-09-11T00:58:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55845.log 2024-09-11T00:58:22.164 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111610.log.gz 2024-09-11T00:58:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106813.log 2024-09-11T00:58:22.165 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63488.log.gz 2024-09-11T00:58:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110676.log 2024-09-11T00:58:22.166 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55845.log.gz 2024-09-11T00:58:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48398.log 2024-09-11T00:58:22.167 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.106813.log.gz 2024-09-11T00:58:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110676.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.110676.log.gz 2024-09-11T00:58:22.168 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63233.log 2024-09-11T00:58:22.169 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55288.log 2024-09-11T00:58:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63233.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63233.log.gz 2024-09-11T00:58:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42060.log 2024-09-11T00:58:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56251.log 2024-09-11T00:58:22.171 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55288.log.gz 2024-09-11T00:58:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63445.log 2024-09-11T00:58:22.172 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48398.log.gz 2024-09-11T00:58:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113198.log 2024-09-11T00:58:22.173 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56251.log.gz 2024-09-11T00:58:22.173 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42060.log.gz 2024-09-11T00:58:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81862.log 2024-09-11T00:58:22.173 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63445.log.gz 2024-09-11T00:58:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114088.log 2024-09-11T00:58:22.174 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113198.log.gz 2024-09-11T00:58:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43241.log 2024-09-11T00:58:22.175 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81862.log.gz 2024-09-11T00:58:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113920.log 2024-09-11T00:58:22.176 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.114088.log.gz 2024-09-11T00:58:22.176 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43241.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.46939.log.gz 2024-09-11T00:58:22.176 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46854.log 2024-09-11T00:58:22.177 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43241.log.gz 2024-09-11T00:58:22.177 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114172.log 2024-09-11T00:58:22.177 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.113920.log.gz 2024-09-11T00:58:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112068.log 2024-09-11T00:58:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42396.log 2024-09-11T00:58:22.179 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.114172.log.gz 2024-09-11T00:58:22.179 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53649.log.gz 2024-09-11T00:58:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58292.log 87.3% 2024-09-11T00:58:22.180 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53693.log.gz 2024-09-11T00:58:22.180 INFO:teuthology.orchestra.run.smithi031.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.112068.log.gz 2024-09-11T00:58:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60933.log 2024-09-11T00:58:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53609.log 2024-09-11T00:58:22.181 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46854.log.gz 2024-09-11T00:58:22.181 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58292.log.gz 2024-09-11T00:58:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60933.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42396.log.gz 2024-09-11T00:58:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111388.log 2024-09-11T00:58:22.182 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60933.log.gz 2024-09-11T00:58:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63402.log 2024-09-11T00:58:22.184 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.36819.log.gz 2024-09-11T00:58:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113836.log 2024-09-11T00:58:22.184 INFO:teuthology.orchestra.run.smithi031.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.111388.log.gz 2024-09-11T00:58:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112160.log 2024-09-11T00:58:22.185 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63402.log.gz 2024-09-11T00:58:22.185 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53609.log.gz 2024-09-11T00:58:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113300.log 2024-09-11T00:58:22.185 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.113836.log.gz 2024-09-11T00:58:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112988.log 2024-09-11T00:58:22.186 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.112160.log.gz 2024-09-11T00:58:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63931.log 2024-09-11T00:58:22.187 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113300.log.gz 2024-09-11T00:58:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52776.log 2024-09-11T00:58:22.188 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.112988.log.gz 2024-09-11T00:58:22.189 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53030.log 2024-09-11T00:58:22.189 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63931.log.gz 2024-09-11T00:58:22.189 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112832.log 2024-09-11T00:58:22.190 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52776.log.gz 2024-09-11T00:58:22.190 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53030.log: 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.53030.log.gz 2024-09-11T00:58:22.190 INFO:teuthology.orchestra.run.smithi031.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58124.log 2024-09-11T00:58:22.191 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112832.log: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.112832.log.gz -5 2024-09-11T00:58:22.191 INFO:teuthology.orchestra.run.smithi031.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50964.log 2024-09-11T00:58:22.192 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48314.log 2024-09-11T00:58:22.192 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58124.log.gz 2024-09-11T00:58:22.193 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.50964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41437.log 2024-09-11T00:58:22.194 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53231.log 2024-09-11T00:58:22.195 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45386.log 2024-09-11T00:58:22.196 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65734.log 2024-09-11T00:58:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45386.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50964.log.gz 2024-09-11T00:58:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100571.log 2024-09-11T00:58:22.198 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48314.log.gz 2024-09-11T00:58:22.198 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53231.log.gz 2024-09-11T00:58:22.198 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.65734.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45386.log.gz 2024-09-11T00:58:22.198 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61855.log 2024-09-11T00:58:22.198 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65734.log.gz 2024-09-11T00:58:22.199 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.100571.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100571.log.gz 2024-09-11T00:58:22.199 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47193.log 2024-09-11T00:58:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54315.log 2024-09-11T00:58:22.200 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61855.log.gz 2024-09-11T00:58:22.201 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.47193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112748.log 2024-09-11T00:58:22.202 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112202.log 2024-09-11T00:58:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51486.log 2024-09-11T00:58:22.203 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112748.log.gz 2024-09-11T00:58:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112202.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112202.log.gz 2024-09-11T00:58:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54057.log 2024-09-11T00:58:22.204 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47193.log.gz 2024-09-11T00:58:22.205 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53879.log 2024-09-11T00:58:22.205 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53777.log 2024-09-11T00:58:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53879.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54315.log.gz 2024-09-11T00:58:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66238.log 2024-09-11T00:58:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111180.log 2024-09-11T00:58:22.207 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51486.log.gz 2024-09-11T00:58:22.208 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.66238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52298.log 2024-09-11T00:58:22.208 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66238.log.gz 2024-09-11T00:58:22.209 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111747.log 2024-09-11T00:58:22.209 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111180.log.gz 2024-09-11T00:58:22.209 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58604.log 2024-09-11T00:58:22.210 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48108.log 2024-09-11T00:58:22.211 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111747.log.gz 2024-09-11T00:58:22.211 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63359.log 2024-09-11T00:58:22.211 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58604.log.gz 2024-09-11T00:58:22.212 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51360.log 2024-09-11T00:58:22.212 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63359.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54057.log.gz 2024-09-11T00:58:22.213 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63359.log.gz 2024-09-11T00:58:22.213 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41932.log 2024-09-11T00:58:22.214 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53879.log.gz 2024-09-11T00:58:22.214 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49274.log 2024-09-11T00:58:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.41932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61103.log 2024-09-11T00:58:22.215 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.49274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51942.log 2024-09-11T00:58:22.216 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61103.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61103.log.gz 2024-09-11T00:58:22.217 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58816.log 2024-09-11T00:58:22.217 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53777.log.gz 2024-09-11T00:58:22.217 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.51942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59417.log 2024-09-11T00:58:22.218 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52298.log.gz 2024-09-11T00:58:22.218 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58816.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58816.log.gz 2024-09-11T00:58:22.219 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56944.log 2024-09-11T00:58:22.219 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.59417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62896.log 2024-09-11T00:58:22.219 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.59417.log.gz 2024-09-11T00:58:22.220 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36226.log 2024-09-11T00:58:22.220 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56944.log.gz 2024-09-11T00:58:22.221 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4%/var/log/ceph/ceph-client.admin.62896.log: -- replaced with /var/log/ceph/ceph-client.admin.48108.log.gz 2024-09-11T00:58:22.221 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45890.log 2024-09-11T00:58:22.221 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62896.log.gz 2024-09-11T00:58:22.221 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51360.log.gz 2024-09-11T00:58:22.222 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36226.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41932.log.gz 2024-09-11T00:58:22.222 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63148.log 2024-09-11T00:58:22.223 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.45890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64732.log 2024-09-11T00:58:22.224 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111452.log 2024-09-11T00:58:22.224 INFO:teuthology.orchestra.run.smithi031.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63148.log.gz 2024-09-11T00:58:22.225 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64732.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49274.log.gz 2024-09-11T00:58:22.225 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64732.log.gz 2024-09-11T00:58:22.225 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53982.log 2024-09-11T00:58:22.225 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51942.log.gz 2024-09-11T00:58:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52692.log 2024-09-11T00:58:22.226 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.111452.log.gz 2024-09-11T00:58:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64860.log 2024-09-11T00:58:22.227 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43580.log 2024-09-11T00:58:22.228 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.64860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53441.log 2024-09-11T00:58:22.228 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64860.log.gz 2024-09-11T00:58:22.229 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58208.log 2024-09-11T00:58:22.229 INFO:teuthology.orchestra.run.smithi031.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.36226.log.gz 2024-09-11T00:58:22.230 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57936.log 2024-09-11T00:58:22.231 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58208.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58208.log.gz 2024-09-11T00:58:22.231 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.45890.log.gz 2024-09-11T00:58:22.231 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54099.log 2024-09-11T00:58:22.232 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43926.log 2024-09-11T00:58:22.232 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57936.log.gz 2024-09-11T00:58:22.232 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81150.log 2024-09-11T00:58:22.233 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53982.log.gz 2024-09-11T00:58:22.233 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63022.log 2024-09-11T00:58:22.234 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52692.log.gz 2024-09-11T00:58:22.234 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43580.log.gz 2024-09-11T00:58:22.234 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81150.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53441.log.gz 2024-09-11T00:58:22.234 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54099.log.gz 2024-09-11T00:58:22.234 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61145.log 2024-09-11T00:58:22.235 INFO:teuthology.orchestra.run.smithi031.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81150.log.gz 2024-09-11T00:58:22.235 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43926.log.gz 2024-09-11T00:58:22.235 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85369.log 2024-09-11T00:58:22.235 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63022.log.gz 2024-09-11T00:58:22.236 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.61145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111264.log 2024-09-11T00:58:22.236 INFO:teuthology.orchestra.run.smithi031.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61145.log.gz 2024-09-11T00:58:22.237 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.85369.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-11T00:58:22.237 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.85369.log.gz 2024-09-11T00:58:22.238 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44584.log 2024-09-11T00:58:22.238 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111264.log.gz 2024-09-11T00:58:22.238 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56273.log 2024-09-11T00:58:22.239 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.44584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52422.log 2024-09-11T00:58:22.240 INFO:teuthology.orchestra.run.smithi031.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44584.log.gz 2024-09-11T00:58:22.241 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.56273.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56273.log.gz 2024-09-11T00:58:22.241 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54669.log 2024-09-11T00:58:22.241 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.52422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36133.log 2024-09-11T00:58:22.243 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54669.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-09-11T00:58:22.243 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54669.log.gz 2024-09-11T00:58:22.243 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.36133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58961.log 2024-09-11T00:58:22.243 INFO:teuthology.orchestra.run.smithi031.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.36133.log.gz 2024-09-11T00:58:22.244 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55761.log 2024-09-11T00:58:22.245 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.58961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63721.log 2024-09-11T00:58:22.245 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58961.log.gz 2024-09-11T00:58:22.246 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.55761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83579.log 2024-09-11T00:58:22.246 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55761.log.gz 2024-09-11T00:58:22.247 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.63721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112622.log 2024-09-11T00:58:22.247 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63721.log.gz 2024-09-11T00:58:22.247 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.83579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48272.log 2024-09-11T00:58:22.248 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83579.log.gz 2024-09-11T00:58:22.248 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46450.log 2024-09-11T00:58:22.249 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112622.log.gz 2024-09-11T00:58:22.249 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.48272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60207.log 2024-09-11T00:58:22.250 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.46450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89048.log 2024-09-11T00:58:22.251 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.60207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42736.log 2024-09-11T00:58:22.251 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60207.log.gz 2024-09-11T00:58:22.252 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.89048.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48272.log.gz 2024-09-11T00:58:22.252 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 86.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.89048.log.gz -- 2024-09-11T00:58:22.252 INFO:teuthology.orchestra.run.smithi031.stderr: /var/log/ceph/ceph-client.admin.57733.log 2024-09-11T00:58:22.252 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46450.log.gz 2024-09-11T00:58:22.253 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53565.log 2024-09-11T00:58:22.253 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.57733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62126.log 2024-09-11T00:58:22.254 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57733.log.gz 2024-09-11T00:58:22.254 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.53565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42354.log 2024-09-11T00:58:22.255 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.62126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54457.log 2024-09-11T00:58:22.255 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62126.log.gz 2024-09-11T00:58:22.256 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.42354.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42736.log.gz 2024-09-11T00:58:22.256 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110928.log 2024-09-11T00:58:22.257 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53565.log.gz 2024-09-11T00:58:22.257 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.54457.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54457.log.gz 2024-09-11T00:58:22.257 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43584.log 2024-09-11T00:58:22.258 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112916.log 2024-09-11T00:58:22.258 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110928.log.gz 2024-09-11T00:58:22.260 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.43584.log: /var/log/ceph/ceph-client.admin.112916.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.112916.log.gz 2024-09-11T00:58:22.261 INFO:teuthology.orchestra.run.smithi031.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42354.log.gz 2024-09-11T00:58:22.262 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43584.log.gz 2024-09-11T00:58:22.301 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T00:58:22.577 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41437.log.gz 2024-09-11T00:58:22.580 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.52422.log.gz 2024-09-11T00:58:23.372 INFO:teuthology.orchestra.run.smithi031.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47063.log.gz 2024-09-11T00:58:23.588 INFO:teuthology.orchestra.run.smithi031.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36728.log.gz 2024-09-11T00:58:24.521 INFO:teuthology.orchestra.run.smithi031.stderr: 94.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-09-11T00:58:25.072 INFO:teuthology.orchestra.run.smithi204.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-11T00:58:25.545 INFO:teuthology.orchestra.run.smithi204.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T00:58:25.930 INFO:teuthology.orchestra.run.smithi204.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-11T00:58:26.069 INFO:teuthology.orchestra.run.smithi031.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T00:58:36.465 INFO:teuthology.orchestra.run.smithi031.stderr: 94.1% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-09-11T00:58:42.961 INFO:teuthology.orchestra.run.smithi031.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T00:58:43.056 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-11T00:58:47.489 INFO:teuthology.orchestra.run.smithi204.stderr: 94.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-09-11T00:58:47.864 INFO:teuthology.orchestra.run.smithi031.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T00:58:48.716 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-11T00:58:51.167 INFO:teuthology.orchestra.run.smithi031.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T00:58:51.652 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-11T00:58:52.094 INFO:teuthology.orchestra.run.smithi204.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-09-11T00:58:53.816 INFO:teuthology.orchestra.run.smithi031.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-11T00:58:53.973 INFO:teuthology.orchestra.run.smithi204.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-11T00:59:44.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-11T00:59:44.781 INFO:teuthology.orchestra.run.smithi031.stderr:Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)') 2024-09-11T00:59:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T00:59:44.788 ERROR:tasks.mds_thrash.fs.[a]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks/mds_thrash.py", line 121, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks/mds_thrash.py", line 239, in do_thrash status = self.fs.status() File "/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks/cephfs/filesystem.py", line 413, in status return FSStatus(self.mon_manager, epoch) File "/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks/cephfs/filesystem.py", line 88, in __init__ self.map = json.loads(self.get_ceph_cmd_stdout(*cmd)) File "/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks/ceph_manager.py", line 1687, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2024-09-11T00:59:46.938 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.fs.[a] failed 2024-09-11T00:59:46.938 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-09-11T00:59:46.938 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-09-11T00:59:46.938 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-11T00:59:46.975 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 2024-09-11T00:59:46.975 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi031.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-09-11T00:59:46.975 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T00:59:46.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:59:46.975 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-11T00:59:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-11T00:59:47.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T00:59:47.053 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-11T00:59:47.054 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-11T00:59:47.089 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052609,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497944k,nr_inodes=819200,mode=755 0 0 2024-09-11T00:59:47.090 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.1 is not mounted at ubuntu@smithi204.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1 2024-09-11T00:59:47.090 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T00:59:47.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-11T00:59:47.090 DEBUG:teuthology.orchestra.run.smithi204:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-11T00:59:47.164 INFO:teuthology.orchestra.run.smithi204.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-09-11T00:59:47.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T00:59:47.165 ERROR:tasks.ceph.osd.0:No such daemon running 2024-09-11T00:59:47.165 ERROR:tasks.ceph.osd.1:No such daemon running 2024-09-11T00:59:47.165 ERROR:tasks.ceph.osd.2:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.osd.3:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.osd.4:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.osd.5:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.osd.6:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.osd.7:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.mds.a:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.mds.c:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.mds.b:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.mds.d:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.mon.a:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.mon.b:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.mon.c:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.mgr.y:No such daemon running 2024-09-11T00:59:47.166 ERROR:tasks.ceph.mgr.x:No such daemon running 2024-09-11T01:01:13.736 INFO:teuthology.orchestra.run.smithi031.stderr: 96.0% -- replaced with /var/log/ceph/ceph-client.0.45156.log.gz 2024-09-11T01:01:13.738 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-09-11T01:01:13.738 INFO:teuthology.orchestra.run.smithi031.stderr:real 2m52.035s 2024-09-11T01:01:13.739 INFO:teuthology.orchestra.run.smithi031.stderr:user 3m51.236s 2024-09-11T01:01:13.739 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m25.464s 2024-09-11T01:01:20.341 INFO:teuthology.orchestra.run.smithi204.stderr: 95.9% -- replaced with /var/log/ceph/ceph-client.1.41119.log.gz 2024-09-11T01:01:20.344 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-11T01:01:20.344 INFO:teuthology.orchestra.run.smithi204.stderr:real 2m58.636s 2024-09-11T01:01:20.344 INFO:teuthology.orchestra.run.smithi204.stderr:user 3m31.810s 2024-09-11T01:01:20.344 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m23.374s 2024-09-11T01:01:20.345 INFO:tasks.ceph:Archiving logs... 2024-09-11T01:01:20.346 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843/remote/smithi031/log 2024-09-11T01:01:20.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T01:01:49.449 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843/remote/smithi204/log 2024-09-11T01:01:49.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T01:02:16.248 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T01:02:16.263 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T01:02:16.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T01:02:16.268 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T01:02:16.403 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T01:02:16.413 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T01:02:16.413 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T01:02:16.420 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T01:02:16.449 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-09-11T01:02:16.453 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-09-11T01:02:16.476 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T01:02:16.476 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-09-11T01:02:16.476 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 -939us[ -959us] +/- 55ms 2024-09-11T01:02:16.476 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 62 +1178us[+1158us] +/- 46ms 2024-09-11T01:02:16.476 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -584us[ -603us] +/- 69ms 2024-09-11T01:02:16.476 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:02:16.493 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T01:02:16.494 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-09-11T01:02:16.494 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv01.front.sepia.ceph.com 3 6 377 60 -923us[ -884us] +/- 55ms 2024-09-11T01:02:16.494 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 2 +1158us[+1158us] +/- 47ms 2024-09-11T01:02:16.494 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 -553us[ -553us] +/- 70ms 2024-09-11T01:02:16.494 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T01:02:16.495 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T01:02:16.503 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T01:02:16.504 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T01:02:16.512 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T01:02:16.661 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T01:02:16.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T01:02:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:02:16.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T01:02:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:02:16.907 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T01:02:16.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T01:02:16.929 INFO:teuthology.task.internal:Duration was 2356.874458 seconds 2024-09-11T01:02:16.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T01:02:16.938 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T01:02:16.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T01:02:16.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T01:02:17.059 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T01:02:17.075 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T01:02:17.616 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T01:02:17.617 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-09-11T01:02:17.617 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T01:02:17.667 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-09-11T01:02:17.668 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T01:02:17.731 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T01:02:17.732 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T01:02:17.737 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T01:02:17.796 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T01:02:17.796 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T01:02:17.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T01:02:17.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T01:02:17.989 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T01:02:17.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T01:02:18.046 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T01:02:18.102 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T01:02:18.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T01:02:18.116 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T01:02:18.206 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-09-11T01:02:18.219 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-09-11T01:02:18.249 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T01:02:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:02:18.282 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T01:02:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T01:02:18.316 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T01:02:18.327 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T01:02:18.328 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843/remote/smithi031 2024-09-11T01:02:18.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T01:02:18.397 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_16:59:47-fs-wip-hemanth-testing-2024-09-10-0723-testing-default-smithi/7899843/remote/smithi204 2024-09-11T01:02:18.398 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T01:02:18.461 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T01:02:18.461 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T01:02:18.466 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T01:02:18.500 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T01:02:18.512 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T01:02:18.512 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T01:02:18.522 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T01:02:18.522 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T01:02:18.540 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T01:02:18.571 INFO:teuthology.orchestra.run.smithi031.stdout: 393233 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 01:02 /home/ubuntu/cephtest 2024-09-11T01:02:18.575 INFO:teuthology.orchestra.run.smithi204.stdout: 393233 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 01:02 /home/ubuntu/cephtest 2024-09-11T01:02:18.577 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T01:02:18.587 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T01:02:18.611 INFO:teuthology.run:Summary data: description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} duration: 2356.8744575977325 flavor: default mon.a-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a mon.b-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a owner: scheduled_hyelloji@teuthology success: true 2024-09-11T01:02:18.611 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T01:02:18.705 INFO:teuthology.run:pass